• Only go outside for food, health reasons or work (but only if you cannot work from home)
• If you go out, stay 2 metres (6ft) away from other people at all times
• Wash your hands as soon as you get home

Do not meet others, even friends or family.

You can spread the virus even if you don’t have symptoms.

Summary

DVLA:

• ran the project to launch the chatbot from April 2018 to June 2019
• had a team which started with 2 chatbot administrator members and increased to 4
• had a separate team that managed Webchat Salesforce
• used the Salesforce Einstein Chatbot and Live Agent tools

Objectives

As this project was part of a pilot, the team shared their objectives with Salesforce. The team wanted to:

• identify whether chatbots would be an effective use of money and time in DVLA
• reduce the average handling time and increase webchat handling efficiency
• use natural language processing (NLP) chatbots to automate the response to customer enquiries that did not require driving record access
• provide a better service for users

The department

The DVLA is an executive agency, sponsored by the Department for Transport. The DVLA contact centre is the largest single site contact centre in government with over 1,200 staff.

The organisation holds 80 million driver and vehicle records. In a typical month they answer over:

• 1.1 million telephone calls
• 100,000 webchat enquiries
• 3,000 social media enquiries

The agency has won a Customer Service Excellence accreditation and the Contact Center Associationing (CCA) award for Best Use of Technology in 2017 and Team of the Year Public Sector in 2019. The agency recently retained CCA accreditation for the 12th consecutive year.

Using a single service platform

Since 2016, the Swansea-based contact centre has used Salesforce Live Agent as their single platform for customer interactions on telephone, email, webchat and social media. This was part of a transformation project to create a unified platform for all contact services, increase IT capability to meet DVLA’s IT strategy and support customer demand. The service cloud platform has helped teams manage different contact channels more easily as they do not have to switch between systems. The platform has also helped the DVLA to administer and configure their technology faster.

Using webchat

The contact centre used webchat on GOV.UK for their main contact services and wanted to explore automating frequent customer enquiries through a chatbot. Webchat was launched on one service which was supporting customers with transactional failures, but DVLA have now rolled it out across all services.

Integrating their webchat with their knowledge base and customer relationship management (CRM) tool has helped resolve customer enquiries faster as advisers can look up chat history and information from the platform. Other ways of working that helped DVLA manage their webchat service include:

• training advisers to handle 3 chats at the same time
• providing a telephone option for more complicated enquiries
• using a recruitment process specifically for webchat advisers
• asking customers to complete feedback surveys after using the chatbot tool

In 2018, Salesforce asked if the contact centre would take part in Salesforce’s Einstein Chatbot pilot. The team built the chatbot into their existing Salesforce Live Agent (webchat) platform.

Starting the chatbot project

The team held planning sessions both internally and with the supplier. The pilot began in April 2018 and lasted a month. The teams initially identified 2 main use cases. They deployed a menu-based chatbot for 3 months that allowed users to select a predefined set of options through buttons. They then rolled out their NLP chatbot, which read and collected information from users.

The plan was to create a chatbot users would interact with to handle their enquiries or collect information. If the user needed an adviser the chatbot would transfer them to the webchat.

Due to the DVLA IT strategy, the agency was open to the operational area of the contact centre leading on a technical initiative. Two team members, who were previously webchat advisers, led the training and administration of the pilot chatbot. The operational team was best placed to configure the technology as they understood the platform and the common user enquiries. The software needed minimal development so the team could learn how to use the chatbot with webinars and reference manuals. They also had access to the Salesforce production team for any queries.

Before the team started they had to overcome some obstacles. For example, there was some concern from contact centre staff about chatbots providing poor customer service, which would have reputational impact.

To mitigate issues or security concerns, the team:

• worked with legal and security experts to mitigate risks
• showed senior stakeholders demos of the tool early on in the project
• involved the commercial team for their expertise
• got clearance from their data security leads
• planned to roll out the new service slowly to monitor and mitigate sudden increases in call volumes

Building the chatbot

This was a low risk, simple delivery. The team had 30,000 free chatbot conversations per month supplied as part of the pilot. The team used the pilot to show that the tool met user needs before requesting any resources from DVLA.

Setting up the chatbot took one month and went live in June 2018. Over the next 3 months the team gradually rolled out menu-based bots to the service. Menu-based bots were useful in funnelling customers to the right channel. The team was able to add NLP bots to the service by August 2019. The NLP capability helped the chatbot recognise and understand more varied user requests.

Throughout the project the team:

• organised user insight sessions which led to chatbot style and content changes
• organised staff testing sessions which provided feedback on caller drop off from the chatbot to the webchat
• held weekly calls with the supplier product team
• used sprint planning so they could make almost immediate changes to the chatbot
• used speech analytics

The team found the chatbot tool easy to learn and use. There were no issues with integrating the chatbot as it was from the same suite of tools from the supplier. Another benefit for the team was having direct access to the supplier’s product team and being able to influence changes to the tool to meet the service requirements.

The chatbot also had good testing environments that were quick to build. The team could use these environments to test and prepare each version of the service before making it public.

Initial research showed there were 150 different ways that users made requests. The team had to enter each request into the tool individually so the tool could respond with the right information. Testing in these environments showed that the team created responses that were between 95% and 98% accurate.

To make sure they got feedback from users, the team used the Salesforce native reports and dashboards through their management information (MI) suite to measure and monitor the impact on existing channels. The team received approximately 20,000 individual customer feedback forms each month.

Challenges and obstacles

The team discovered issues, mainly around NLP matching. Depending on how a question is phrased, the bot could route a user to 2 different sets of information. For example, if a user only types ‘vehicle tax’ for their question the chatbot has several options it could return as answers, including checking if a vehicle is taxed and renewing your vehicle tax. The team worked with the supplier’s product team to solve these issues.

In one iteration, the team made one of the menu-based bots too complex by offering users too many options. The team learned to have an immediate rollback process in place for the times when the tool was not working for the users.

The team found that a chatbot was not the best option for very complex queries, but for the majority of customer queries it worked extremely well and did reduce demand on the telephone service and the webchat by automating answers for the simpler, regular queries.

Impact of the chatbot

The team was happy with the results as they did not expect the project to deliver so much so quickly.

Due to a combination of the chatbot and webchat, the contact centre has seen a significant drop in demand for contact through email and telephone.

Customers are benefitting from the chatbot project because it:

• reduces the average handling time from 8 minutes to 2 minutes 30 seconds
• answers 25% to 30% of customer queries without going to an adviser
• reduces webchat abandonment by 15%
• has 90% customer satisfaction rate based on the 40,000 people who completed the survey each month during the project

Wider organisational benefits of the project included:

• an increased focus on AI and automation
• growth of the operational team
• changing some strategies from building a tool to buying commercial tools and using tool configuration
• more opportunities for operational staff to get involved in significant continuous improvement work streams and make their roles more diverse

Lessons learned

The main lessons the team learned included:

• the importance of telling users at the start of the chat that they’re using a bot
• keeping changes small and simple to avoid having to rollback to previous versions
• doing a phased and gradual rollout
• giving the users an option to speak to an adviser at any point in the chat
• discovering that you do not always need developers to deliver IT enhancement
• discovering that setting up chatbots takes more time than anticipated and the AI components can have limited functionality

DVLA has also hosted successful ‘Hack to The Future’ events to find new ways to use AI and chatbot technology in the public service.

Future plans

The team plans to:

• make gradual changes and improvements to encourage a slow and steady uptake by users and assess the results
• have multiple chatbots so they can transfer users from bot to bot
• explore robotic process automation (RPA) as an interim technology to automate simple tasks and move data from one system to another

After the success of the pilot, there is a greater openness to trying pilots. It has also led to the DVLA assigning more resources and seeing chatbots as a strategic tool.

Summary

MoJ:

• ran their project from June to August 2018 with an iteration extending into September
• had a team consisting of a service designer, content designer, user researcher and developer
• trialed the Landbot tool for their chatbot

Objectives

The goal was to run a proof of concept project on the Child Arrangements Information Tool (CAIT), to understand how users would interact with content delivered through the chatbot. The team wanted to know whether separated parents are more likely to engage with content delivered by a chatbot rather than flat content. They also wanted to know which method was easiest to use and provided the most flexibility. This project would be part of a larger project around the CAIT.

The department

MoJ is responsible for the courts, prisons, probation services and attendance centres. The User Centred Policy Design team ran the chatbot project, alongside other proof of concepts with video and audio formats. They work with policy and operational teams to:

• be more user focused
• test ideas early
• make greater use of digital and design methods

Experimenting to understand user needs

The team started this project because of research which found that:

• most of their users only interact with the justice system when in a moment of crisis and might behave differently under stressful situations
• users in the moment of crisis tend to struggle reading and understanding text-heavy flat content
• users are looking for help elsewhere, for example, forums or websites where the interaction is more dynamic and conversational

The team also looked at service data and how other departments provided evidence of users accessing similar data.

Testing assumptions

The team found there was a lot of content available online but it was not engaging enough for some users. So the team set out a proof of concept to test their assumptions and to better understand what users need.

The team wrote 5 statements to test about user interaction with their content:

1. The chatbot will receive initial interaction from CAIT users.
2. The chatbot will receive significant interaction from CAIT users (after initial interaction).
3. The chatbot will have higher referral rates to support services, when compared to static page content.
4. Interacting with the chatbot will increase users’ awareness of alternatives to court
5. Users will rate the chatbot as helpful or very helpful.

The team then assigned a metric for each statement. For example, by looking at retention and completion rates to measure if users have had significant interaction with the chatbot. Each metric had a target, which the team selected based on a benchmark and research. They used this target to measure the success of the results.

Setting up the chatbot

The service team ran the project in 3 phases – planning, implementation and evaluation. They worked closely with the policy team to make sure the chatbot met policy and legal requirements.

As this was a quick, proof of concept project, the team chose to use Landbot rather than developing their own tool. The team did not need to invest in training as they could learn about the tool by using it.

The chatbot they created used a closed script which included a number of user journeys depending on what path users selected. The content designer wrote the child arrangement guidance based on current website information. They worked with their provider to make the chatbot look similar to the GOV.UK Design System.

Carrying out the project

During the main sprint, the team embedded the chatbot on the homepage of the service. It was available to the general public for 12 days and during that time 1,121 users visited the website (125 unique visitors per day) and 26% of these users initiated a conversation with the bot, with 15% of this group continuing their conversation after their initial interactions.

The team also ran 5 further face-to-face usability sessions in a lab environment to get qualitative feedback on the experiment. The team considered doing A/B testing, but the service did not have enough users visiting the site at the time the experiments were carried out.

During the first iteration the team collected feedback from users with a short questionnaire and 2 simple follow-up questions on the chatbot to rate their experience. The questions were “was this conversation useful?” and “do you know more about your options now?”.

Challenges and obstacles

The first version was a closed bot and during user testing sessions, users quickly understood that they were in a closed loop of information. In the second iteration, the team added more content and allowed users greater input.

During the first iteration, some users did not get as far as the feedback questions if they were given a link to another website as part of their journey. To resolve this in the second iteration, the team embedded the feedback in the conversation flow.

The users needed tailored content, and an automated chatbot cannot provide that. The scripted chatbot felt restrictive and not intelligent enough to provide users with the content they needed. A potential solution to this was to provide a webchat or a more intelligent chatbot with machine learning.

With the second iteration of their chatbot, the team found flaws in the analysis provided by Landbot and ended up manually analysing the data.

Due to a resource issue, there was no research testing for the second iteration of the chatbot and the team could not directly compare the first and second iteration.

Results

The participating users gave positive feedback, and were pleasantly surprised to see the government try something new. The project allowed the team to sense check their theories in an agile way to better understand and help users. They found that users are willing to engage with content delivered through chatbots, although the team needs to do more testing to make sure the users find the support they are looking for.

At the end of this project the team found:

• the users of this service wanted tailored content
• users were more likely to find and engage with pages accessed through the chatbot than through the normal website
• the number of referrals improved when using the chatbot compared to referrals going through the usual process, for example the number of referrals going to the three external websites used by the service tripled (3 to 55) during the first iteration

One of the major benefits of the project is that the policy team is more open to alternative solutions and that testing solutions before using them is essential.

Future plans

The team would like to try a minimum viable product of a webchat, possibly using Landbot which has a webchat feature. The team would also like a subject matter expert to provide content and answer questions.

The team is also considering using a structured menu so that users can navigate through the content and onto the webchat. With a structured menu, the team can test a closed bot, a semi-open bot and webchat.

• Only go outside for food, health reasons or work (but only if you cannot work from home)
• If you go out, stay 2 metres (6ft) away from other people at all times
• Wash your hands as soon as you get home

Do not meet others, even friends or family.

You can spread the virus even if you don’t have symptoms.