Can AI Agents Be Hacked? Separating Headlines From Reality

When we hear that an AI agent has been hacked, it is very easy to imagine an intelligent system sitting inside a business with access to everything, making decisions and taking actions without anyone knowing what it is doing. That is not how all AI works, and it certainly is not how Click4Assistance AI agents are designed.

It seems every few weeks another story appears around the latest thing that artificial intelligence can do.

Sometimes the story is exciting. AI has discovered a new way to solve a problem, written something that would have taken a human considerably longer or advanced medical research that could greatly improve quality of life. Sometimes, however, the headline is less reassuring. Researchers have found a way to manipulate an AI agent, persuade it to behave unexpectedly or exploit the connected systems.

It is understandable that these stories attract attention. Businesses are already considering how they can use AI to automate processes, give people faster access to information, or use an AI agent for customer service so the prospect of an AI system being compromised raises questions about security, privacy and control.

To understand the real risk, we need to look at what is happening behind the headline.

There is an important distinction between an AI that has been given extensive access to external tools and systems and a conversational ai chatbot that has been deliberately designed to operate within a tightly controlled environment.

That distinction is becoming increasingly important as organisations decide where AI belongs within their own businesses.

So what exactly are we giving AI access to?

One of the fascinating things around current conversations regarding AI security is how quickly we can move from talking about the intelligence itself to talking about everything that has been connected to it.

If an AI agent is given access to the internet it can search for information beyond the material you have provided. Give it access to email and it may be able to read, interpret or potentially send messages. With access to code execution and it could suggest a piece of code or may even be capable of running it. Providing access to operating systems may potentially give it control over a production environment. At that point, we are no longer talking about an AI that simply answers questions.

This is where some of the recent research into AI security becomes particularly relevant. Researchers continue to uncover techniques that can influence the behaviour of AI agents, including attacks designed to manipulate how an agent interprets information or uses the tools available to it. In these cases, it’s not that somebody has somehow "broken into the system" in the way we might traditionally think about hacking. The concern is that an attacker has found a way to influence the AI into doing something it should not do, when the AI has been given the permissions and tools to take that action.

That is a legitimate security concern, and it is one the industry needs to take seriously, and it’s why the architecture around an AI system matters so much.

The best AI Chatbot for Business is only as powerful as the access it has been given

There is a useful comparison with employing a new member of staff.

If someone joins your business on Monday morning you would give them the information and access they need to do their job, but you probably wouldn’t hand them the keys to every office, access to every customer record, the ability to alter your databases, permission to install software on every company computer and unrestricted access to your email and financial systems.

Their intelligence would not be the reason you decided against doing it. The issue would simply be that they do not need those permissions to perform their role.

Businesses have been operated on that principle for years. People are given access according to their responsibilities, systems have different levels of permission and sensitive information is protected because not everybody needs to see it. There is no reason that the same principle should not apply to AI, and in fact, it becomes even more important when an AI agent is capable of acting at speed and at scale.

An AI designed to help developers write and test software may genuinely need access to code repositories and development environments. An AI being used as part of a cybersecurity operation may need access to logs, networks and security tools. Those systems can be incredibly useful precisely because they have been given meaningful access to the environment in which they operate.

They also require serious security controls because the consequences of that access are significant.

That is very different from an AI agent for customer support whose role is to answer a customer asking, "What time do you close today?"

The customer does not need an AI with access to the company's operating system to answer that question. They do not need it to browse the internet, execute code or install software.

That may sound obvious, but it is a distinction worth making because the term "AI agent" can make very different technologies sound as though they are essentially the same thing.

Conversational chatbots work within defined boundaries

This is where the conversation around AI security needs a little more balance.

When businesses hear about an AI agent being manipulated, it is entirely reasonable to ask whether they should be putting AI in front of their customers at all. Nobody wants to introduce a new technology only to discover that they have accidentally created a security problem.

But before making that decision, it is worth understanding what the particular AI system can actually do.

A great example is ARTI, the Click4Assistance AI Agent.

The Click4Assistance developers worked very hard to prevent the ChatGPT powered agent to wander the web looking for information. It cannot browse the internet, execute code or install software. It cannot send emails, access a customer's internal systems or modify files.

ARTI answers questions using the knowledge that an organisation provides to it.

That means the organisation remains in control of the information from which the AI is expected to answer. The business decides what information is relevant, what should be included and what the AI should know about its products, services, policies or processes.

That is a very different proposition from giving an autonomous AI access to the wider internet and asking it to work things out for itself.

Human control still matters

There is sometimes an assumption that using AI means handing over control to the machine.

In customer service the opposite approach is often much more sensible. The organisation should retain control over the information the AI uses, the boundaries within which it operates and the systems it is allowed to interact with.

That is particularly important when the AI is being used to represent a business to its customers.

If a customer asks about your opening hours, you want the answer to come from information you have approved, rather than something the AI has found somewhere on the internet. If they ask about your returns policy, you want it to use the policy you have provided. And if the AI is going to interact with another system, there should be a clear reason for that connection and an explicit permission for it to happen.

This is why permission-based integrations matter.

There may be circumstances where an organisation wants an AI agent to connect to another application via an API. Perhaps it needs to retrieve specific information or support a particular customer journey. That can be useful, but it should be a deliberate decision rather than an assumption that an AI needs access to everything in order to be effective.

So, can AI agents be hacked?

The honest answer is that any technology connected to other systems needs to be secured properly, and AI is no exception.

There will continue to be new research into attacks against AI models and AI agents, and some of those discoveries will be genuinely significant. We should want researchers to find these vulnerabilities because finding them in controlled testing gives technology providers and organisations the opportunity to address them before they become real-world problems.

However we should be careful about is treating every AI system as though it has the same capabilities and the same level of risk.

An autonomous AI agent with internet access, email access, code execution, database access, operating system permissions and the ability to take actions without human intervention is a very powerful system. If something can access and manipulate that many parts of an organisation, there are obvious reasons to take security extremely seriously.

A customer service AI operating from a controlled knowledge base is a different proposition.

The technology is not the problem, the permissions are

Perhaps this is the part of the conversation about AI security that gets lost most often. AI is not inherently a threat just because it is AI.

The risk increases when we connect an intelligent system to powerful tools and give it broad permissions without putting appropriate safeguards around those connections.

We have spent decades learning that access needs to be managed carefully. We use passwords, authentication, permissions, firewalls, encryption and security policies because giving unrestricted access to systems creates unnecessary risk. AI does not somehow make those principles irrelevant. If anything, the growing capabilities of AI make them more important.

For organisations looking at customer service AI, therefore, the question should not simply be, "Can AI be hacked?"

A much more useful set of questions is:

• What can this AI access?
• What can it change?
• What can it send?
• What systems can it connect to?
• Where does its information come from?
• Who controls that information?

Those questions tell you considerably more about the risk than the words "AI agent" ever could.

AI is moving quickly, and we should absolutely take the emerging security risks seriously. We should listen to researchers, understand new attack techniques and expect technology providers to build appropriate safeguards around their systems.

At the same time, we should not allow stories about highly autonomous AI agents with extensive system access to create the impression that every ai agent for customer service represents the same security risk.

Sometimes the most effective technology is the system that has been given the right capabilities, for the right purpose, within clearly defined boundaries.