TESTING A CHATBOT
“Hey, Google, what is the current temperature right now“- Google Assistant
“Hello Siri, can you add cheese in my
shopping list”- Apple Assistant
You must have communicated at least once with
either one of the above-listed assistants and that is why I added them in my
today’s sharing session. They serve as an amazing example of a well built and
stable chatbots and every one among us have used them.
In today’s article, I will be discussing all
about, what are Chat bot’s / automated messenger’s service, how can you test
them and all other parameters associated with handling them from a QAs
perspective.
A chatbot is the most trending innovation from
2017 and progressing. I have tested some of the top companies offering these
services like Zendesk, LivePerson, Slackbot, Facebook bots, and others. If
you haven’t got the opportunity to explore this industry, allow me to share the
experiences.
What is a chatbot
A conversational interface, computer
program mimicking as a user using the artificial intelligence technology. In a
simpler way, a virtual assistant engages the user by helping him gain any
information or perform any task.
They usually reside within social media
platform, messaging platform and do much more than just communicating (details a
head.)
Types of chatbot
There are two types of chatbots:
- A Rule-based, pre-programmed, or scripted one which works on defined rules, triggers and flows, and takes actions on the basis of selected commands.
- An AI-based; which works by interpreting user verbal commands, speech, or graphical selection. These type of bots doesn’t restrict user input. It keeps getting smarter by learning previous conversations/selections.
How to test a chatbot
There can be three simple ways to test any application:
- How it performs in an expected scenario.
- How it performs under possible scenarios
- How it performs in impossible scenarios.
Your bot design / UX is the most critical aspect in making the bot successful. A smarter and engaging bot is the one who understands customer’s requests and responds promptly. They can be used in many ways depending upon the industry for which they are built for. Following are the areas which must be dealt with while testing:
Onboarding process
Starting on with the very first step is the
onboarding process or the process where the human will engage himself with the
chatbot. Two methods are used to start the onboarding process
1.
A pre-sign-up form
2.
Engage user automatically
A pre-sign-up form is a form in where
customer provide some basic information like the name, Email ID, Phone number or
a message etc., the information is taken in pre-sign up form varies, as it depends
solely on the website owner. As soon as the user submits this information, they are
directed to the chat window where customer support personnel communicate with them
or they will be queued and receive an automated message.
The above image shows a pre-sign-up form.
The other option is smarter, a user does not input
any information and he is automatically engaged with a greeting and available
services of the bot.
The above image shows an auto engage bot.
Being a tester, you should ensure the
customer/visitor is served with a friendly greeting. If the customer arrives by
filling the pre-sign-up form, all their information is saved and passed to the
correct variables. Also, in place of ‘visitor name,’ the name provided by him
should be displayed.
For the engaging method, if the visitor
provides the information after being in chat then all the information should be
updated to relevant variables and must be retained.
Above image shows the onboarding process by the slack bot
Handling alternative responses
In this step, you need to test the bot with
variety of alternate expressions and determine whether it yields different
answer or understands the alternate words like if the bot says hello, respond
by Hi, Hey, What sup, etc.
You can throw these words randomly during a
conversation and see whether it understands and maintains the flow or breaks
up.
The above image shows the different response to alternate expressions
Allow suggested sentence/templates
Providing a user with some general templates
helps in better engagement. This helps the user in understanding how the bot works and
that his query is actually going to be answered professionally / seriously and they
are not talking with just a robot.
For example, if a user puts up a question
about ordering something, your bot should be clever enough to interpret their
command and show them the available options. If the user is browsing an
eCommerce website, the options can be; what do you want to order? Clothes,
Electronics, Home decoration, Furniture, Garden, etc. On selecting any of the
options, the user should be given more personalized choices like, if the user
selected clothing, they should be asked questions like Men clothing, women's
clothing, Baby Clothing, etc. This would help users interact with your site in a friendlier
manner, thus resulting in a better understanding user experience.
Above pictures show an example of a bot engaging the user by providing options.
Concise specific conversation
One important factor you need to know is
that the chatbot is not a human and it cannot interpret long paragraphs like a human does. Make sure the sentences are short and concise. Allow users to type
within a specified number of words. Also, make sure if a large text/paragraph is
entered by the user, the bot handles it nicely.
The above picture shows the bot replying politely to user long text input
Ensure navigation is not broken
If there are any links provided by the
chatbot and the user clicks on it, make sure on pressing the back button, the user
is redirected to the same chat and not left hanging around or simply open them
in a different tab. All the links should work.
Users often change their minds and intend to
cancel the order or selection. Ensure how the bot is behaving in such a condition
and whether he is able to comprehend such instructions.
Above image shows the user requesting to cancel the placed order.
Include feedback options
After ending any chat session with the bot
or answering the request successfully the chatbot should present the customer
with the feedback option. It not only helps to improve the bot capabilities but
also in finding out user satisfaction rate. If there is a message field in the feedback,
make sure to test the feedback option with both single line and multiline
input. Once the feedback is submitted, make sure a thank you note is displayed
to the user.
Above
image shows the feedback options.
Context of conversation
The bot should be smart enough to derive
conversation on the basis of user input and should not ask obvious questions.
For example:
User:
Hey, bot book a flight to Sydney.
Bot:
Sure, What date should I select to book the flight?
User:
Lorem ipsum dolor strawberries
Bot:
<No response>
The above example shows the bot is not
intelligent enough to interpret user response. You can always make him smarter
by implementing more options like below:
Bot:
I don’t get that, do you want me to book the flight for Tomorrow, Next Week or Next
month
User:
Next Week
Bot:
Which day of the week would you prefer Sunday, Monday, and Tuesday and so on…
User:
Wednesday
Bot:
Thank you, I have booked your flight to Sydney for coming Wednesday.
The above approach uses a smarter way to
keep the user engaged and provide him meaningful information. Make sure to test
the bot by providing each set of random and complete information and verify how
it responds.
Remembering user information
If the user has filled up a pre-sign-up form,
then the information should be saved and retained. There can be multiple cases
on verifying whether the information is saved or not
a)
Fill in a pre-chat form, talk, and
leave a chat. After the chat has ended, there is a reconnect or re-chat option.
Click on that button and chat again, see if the user name and other data is
retained on reconnecting the chat or not
b)
In an automated engaging chat,
provide the user information later. When the bot asks you for it. Check whether
the information is saved correctly or not and repeat the above-listed case a.
c)
Once the user has ended the chat,
close the tab in which the chat application was opened, after few minutes
reopen the same application and chat again. Check if the visitor is treated as
a ‘returning’ user. In this case, you should also check if the bot/messenger
is smart enough to send you a ‘Welcome back’ greeting.
d)
Another cause can be closing the
tab and clearing the history of your browser. After this is done, reopen the
chatbot application and chat. In this case, the user should be treated as a new
user. However, if the user provides the same pre-sign-up information, it should
be validated against the user database to give them a more personalized
experience.
Multi-lingual support
If you are to provide multilingual support,
you need to verify multiple languages. The first step is detecting the user’s
country information automatically and sending a greeting in their respective
language. Your job being a tester is to access the application from multiple
locations, regions and see whether the correct country and the greetings are
detected or not.
If the user wishes to get answered in
English and not in the native language, then the chatbot should be intelligent
enough to analyze the response and send a reply in English and not in the same
country language. On the other hand, if the user replies in a specific language
the service should reply in the same. Verify all the communication is smooth
and not mixed up.
Also, user can use mixed languages, make
sure to verify that the bot serves them appropriately.
The above picture gives an example of a user using a mixed language and the bot replying him appropriately
Handling inappropriate bad/input
You can have users from any part of the
world, some of them can be in a bad mood and can use inappropriate language.
The chatbot should be smart enough to identify a bad word and should have the
capability to handle unexpected inputs and reply in a positive or a generic response
like ‘Sorry I am unable to understand you this time.
Handling images
If you intend to provide file sharing
option with your bot/messenger service, you need to validate all the allowed
file types and the maximum size. Being a tester, you should verify whether the
image is getting received successfully at the back end and how is it being
processed.
Also, now the bots are intelligent enough to allow chatting in graphical interfaces or widgets like WeChat, Facebook
Messenger and Kik. For example, in Facebook messenger, if someone wishes you a
happy birthday, then you automatically see thank you images while sending a
response. So, you need to verify whether your bot can interpret speech and show
you relevant emoticons or if you have used predefined images/stamps / icons
you are able to respond in that way.
The above image shows relevant graphical widgets based on user input
Ability to answer in offline mode / non-business hours
One of the key features of your chatbot is
it can be customized, so if your business works on specific days or timings you
need to set up an offline mode response. Make sure the user is shown an offline
form or an automated response if they try to engage the bot in the non-trading
hours.
The message can be customizable like “Sorry
we are unavailable right now. You can check our operating timings here”.
Make sure the link is working and upon clicking, the user is directed to the
working days and hours frame. Also, there should be a back or close button.
Beta release
Before going live with your newly developed
bot, the last step is to push them towards the wider audiences i.e. beta
release it.
You must have tested with all the possible
scenarios you can think of and the bot must be working as planned but releasing
it to a different set of people lets you explore different behavior of your
application. Everyone has their own way of communicating and using a program.
The bot will be tested with unexpected and variety of input that could have
been missed. There can be situations where the bot stops responding to specific
input, or you can get a report on how the bot treats the different set of
instructions. You can always educate the bot and make it smarter and competent
enough to cater to all sorts of input and audience.
Meanwhile, you can also use the following
platforms, tools to beta test your bot. Click here
to see more:
Areas to be considered
While we have discussed what things should
be verified in the bots, there are some areas which must be complied for a
successful chatbot.
a)
Performance testing – The response
time has to be prompt, a user does not like long waiting.
b)
Security testing – as the conversation is taking place inside a messaging platform, it has access to user-sensitive data. Ensure there is a controlled mechanism to secure data and
spoofed messages. Ensure by passing a script or vulnerable piece of code and
see how the bot reacts.
c)
Adherence to compliance – if
the bot is made for a specific industry then it should adhere to the compliance.
For example, a bot aiding disabled people should comply with the 508 compliance,
make sure all industry standards are met and none of them are compromised.
Advantages of using chatbots
Following are some of the advantages of
having a chatbot.
a)
Cost-effective way for
businesses to connect with potential customers
b)
Chatbots can be available 24/7.
c)
Chatbots are scalable, they can
have a simultaneous conversation with multiple or hundreds of users.
d)
They can outperform in terms of
speed and accuracy.
e)
Bots eliminate the need of
long waiting/holding time on the calls.
f)
One-time investment helps
organizations save money by minimizing the required staff.
g)
Always treat the customer in a polite way thus increasing customer satisfaction to 100%.
Conclusion
With continuous advancements to machine learning,
natural language processing (NLP), AI, speech recognition chatbots will provide
a new and effective form of communication between the brand and consumer and
will continue to learn and excel in all industry sectors.
A survey conducted with over 300
participants from a wide variety of industries showed that 96% of businesses
believe that chatbots are here to stay and 67% believe that chatbots will
outperform mobile apps in the next 5 years. In addition, 75% of participants
were planning to build a chatbot for their business in 2017.
Feel free to share your experience and adding any other interesting factors which I missed out in the comments section below.
Comments
Post a Comment