An AI chatbot for business makes customer service faster. So why aren't customers happier?

Research sponsored by Click4Assistance, using ContactBabel's findings from businesses and UK consumers, provides some interesting answers. While the findings could be read as unflattering to AI, the more useful question is why a technology capable of delivering faster responses and accurate information is still falling short of customer expectations.

The promise of AI in customer service has been faster responses, shorter queues, lower costs and customers getting the answers they need without waiting to speak to someone. Much of that is now being delivered, AI can respond instantly, handle multiple enquiries simultaneously and provide information without requiring a human agent to search through several systems.

So why aren't customers happier?

Faster, certainly. Better, not necessarily

One shocking aspect of the research is the number of organisations that are unable to say whether an AI chatbot for business performs better or worse than a human agent. For some metrics, almost half don’t know. There is clearly a discussion to be had about the importance of benchmarking performance before deployment and establishing measurable outcomes, but that deserves an article of its own, particularly given the investment being made in AI.

Among those organisations that do have the information, the results are interesting. When it comes to average handling time, 69% believe AI performs better than a human agent, compared with just 12% who think humans do better. That is an impressive result and demonstrates why businesses are so enthusiastic about automation. However, the picture changes when we look beyond speed.

On accuracy of information, 38% believe humans perform better, compared with 28% who think AI does a better job, however 34% see no difference. Initially we see the higher percentage favouring human performance but is that really such a disappointing result? Given that AI chatbots can respond immediately, handle multiple enquiries and operate at a fraction of the cost of human agents, delivering information that is just as accurate could be considered a success. Of course, the fact that a greater number of organisations rate humans more highly is something we shouldn't ignore, but perhaps we need to question whether AI actually needs to outperform a human agent to justify its use.

The picture is less encouraging when we look at first contact resolution, where 44% believe humans perform better, 27% rate AI more highly and 29% see no difference. This is arguably a more important measure because customers aren't simply looking for information, they want their problems resolved. Nevertheless, more than half of those expressing a view believe AI is either matching or outperforming human agents, which suggests that automation is already delivering reasonable results in some areas.

Customer satisfaction is where things become particularly interesting. In this survey, not a single organisation thought AI performed better than a human agent, with 59% rating humans more highly and 41% seeing no significant difference. As a solution provider, we often talk about improving the customer experience as one of the main reasons for introducing AI chatbots, so should we be disappointed by these findings?

What concerns me is why so many organisations still believe humans deliver a better customer experience. AI appears to be reasonably competitive when it comes to accuracy, and a significant proportion of organisations see no difference in its ability to resolve enquiries. So what is missing? Are chatbots providing the right answers but failing to understand what customers actually need, or are businesses expecting them to handle enquiries that would be better dealt with by a person? Perhaps the problem is less to do with the technology itself and more with how it is being trained and deployed.

Getting the answer right isn't the same as solving the problem

This is where the difference between accuracy and resolution gets interesting.

Imagine a customer contacting an energy provider about an unexpectedly high bill. The chatbot explains the tariff correctly, provides accurate information about meter readings and points the customer towards the relevant terms and conditions. Everything it says may be factually correct. The bot has provided information without necessarily resolving anything.

Or consider someone contacting an insurer following a difficult experience. A chatbot may explain the claims process perfectly accurately, but the customer may need reassurance, judgement or help with circumstances that don't fit neatly into a predefined process.

In both cases, the quality of the information is only one part of the interaction.

Good customer service requires an understanding of what the customer is trying to achieve, the ability to take appropriate action and recognition of when the situation calls for something more than a standard answer. This is where I suspect some AI deployments are running into difficult and the consumer findings appear to reinforce that concern.

ContactBabel (link to research document) )asked 1,000 UK consumers to compare AI customer service with their experience of human agents. The 20% who had never knowingly encountered AI customer service were excluded from the comparison.

Perhaps the most concerning finding is that 69% of customers felt AI was worse at understanding their issue, while 59% thought it was less effective at resolving their enquiry. These are fairly substantial numbers and suggest that customers are struggling to get the help they need. My question would be is that because AI doesn't understand their questions, or because it isn't equipped to deal with the problems they're bringing to it? There is an important difference, and one that deserves further consideration.

The results are a little less negative when we look at other aspects of the experience, although 54% still felt the information provided by AI was less accurate, 46% were disappointed with its friendliness and tone, and 43% found it harder to use. I wonder how much of this comes down to the way chatbots are designed and trained, rather than any fundamental limitation of the technology. After all, a chatbot that repeatedly misunderstands a question or offers irrelevant answers is hardly going to leave a customer feeling particularly positive about the interaction.

Speed was the only area where AI came out ahead, but even here the results are surprising. While 42% of customers thought AI was faster, a third actually found it slower, and 24% noticed no difference. How can that be when a chatbot can respond almost instantly? I suspect the answer lies in how we measure speed. Businesses tend to focus on handling time, whereas customers are more interested in how long it takes to get their problem sorted out. A chatbot might provide an immediate response, but that doesn't help much when the customer spends the next ten minutes going round in circles before eventually being transferred to a human agent.

Is the technology the problem?

Looking at these results, there is clearly a gap between what AI is capable of delivering and what some customers are actually experiencing. But is that a reflection of the technology itself, or how businesses are choosing to use it? We've always maintained that AI has an important role to play in supporting customer service teams, rather than replacing human interaction altogether. There will always be situations where customers need to speak to a person, and the challenge is understanding where AI can genuinely improve the experience and where human involvement remains essential.

We should remember that the widespread use of generative AI in customer service is still relatively new. While chatbots have been around for years, the capabilities now available are very different, and many businesses are still finding their feet. There has also been an enormous rush to bring AI solutions to market, with new providers emerging almost overnight. Some undoubtedly have impressive technical expertise, but understanding how AI works and understanding the complexities of delivering good customer service are not necessarily the same thing.

Customer service is about considerably more than answering questions. It involves understanding what someone is trying to achieve, recognising when a situation requires a different approach and, importantly, knowing what constitutes a successful outcome. These are considerations that experienced customer service professionals deal with every day, but they may not always be at the forefront of a technology-led deployment.

I wonder how many of the disappointing experiences highlighted in the research are the result of businesses introducing AI without sufficient guidance on where it should be used, how it should be trained and what happens when it cannot resolve an enquiry.

Perhaps we need to be more selective about the enquiries we expect AI to handle, rather than assuming that anything currently dealt with by a human agent is a potential candidate for automation.

Understanding customer service matters

This brings us to the importance of how AI is introduced into a business. There is a considerable difference between having the technical ability to build a sophisticated chatbot and knowing how to use one to improve customer service.

The influx of AI providers has given businesses an enormous choice of products, but how much attention is being given to understanding the organisation's existing customer service operation before a solution is recommended?

• Which enquiries are most common?
• Why are customers making contact?
• Where do they encounter difficulties?
• What can be resolved through information alone, and what requires access to other systems, human judgement or a decision that the chatbot isn't authorised to make?

These are just some of the questions that should help determine where AI is deployed and how it is configured. They also require knowledge of customer contact operations, not simply the underlying technology.

I think this is where an experienced, consultative solution provider can make a real difference. Businesses need guidance on selecting the right applications, preparing and training their chatbot, designing effective conversations and ensuring that automation works alongside their existing customer service teams.

It also means being realistic about what success looks like. As we considered earlier, an AI chatbot for business doesn't necessarily need to outperform a human agent in every interaction. Providing an equally satisfactory experience more quickly and at a lower cost is already a worthwhile achievement. However that depends on choosing enquiries where automation is appropriate and giving the chatbot the knowledge and capabilities it needs to deal with them properly.

There are also opportunities to use AI to support human agents, helping them find relevant information and deal with enquiries more efficiently. In these situations, customers may benefit from AI without necessarily interacting with a chatbot at all. Perhaps the focus should be less on replacing human involvement wherever possible and more on identifying where AI can make the overall service better.

We've seen something similar happen with live chat over the past 20 years. The early adopters recognised its potential and were keen to introduce it, but it took time for businesses to understand how to use it effectively. Simply putting a live chat button on a website didn't guarantee a better customer experience. Organisations had to learn when to offer chat, how to resource it, how to train their agents and how to integrate it with their wider customer service operations. Over time, those lessons helped turn live chat into an established and valuable customer contact channel.

We've seen something similar happen with live chat  for customer service over the past 20 years. The early adopters recognised its potential and were keen to introduce it, but it took time for businesses to understand how to use it effectively. Simply putting a live chat button on a website didn't guarantee a better customer experience. Organisations had to learn when to offer chat, how to resource it, how to train their agents and how to integrate it with their wider customer service operations. Over time, those lessons helped turn live chat into an established and valuable customer contact channel.

I suspect we're seeing the beginnings of a similar journey with AI chatbots. The technology has developed incredibly quickly, and businesses are understandably keen to take advantage of it, but many are still learning where it works best and how to get the most from it. As the market matures, I hope the initial excitement about what AI can do will give way to more practical questions about how it should be used, how it should be trained and what outcomes it is actually delivering.