Anthropic, the company that makes the Claude AI tool, has revealed some interesting research. It found that a relatively small of correctly formed documents can result in an AI tool delivering spurious, and even potentially dangerous, outputs.
Anthropic’s research, conducted jointly with the UK AI Security Institute and the Alan Turing Institute, found that just 250 documents could produce a vulnerability in an LLM (large language model)
The LLMs that power modern AI tools have collected their data (despite concerns about copyright and intellectual property theft) from sources all over the world. ‘Poisoned’ documents don’t have to be injected into an LLM by a cyberattacker. Data scrapers used by the creators of the LLMs can access the 250 documents from any repository they access.
It’s easy to poison an LLM
The researchers found that documents containing specific words like <SUDO> could be used. SUDO is a system command that enables a set of subsequent commands with potentially elevated system privileges. The documents could do increase the likelihood of false results or exfiltrate sensitive data.
With AI businesses accelerating their efforts to access more data, through new AI-powered web browsers, a poisoned AI could, potentially leak your sensitive data.
There are always risks with new technologies. And humans are notorious for overestimating risks and impacts – there’s a rich body of research about how people often overstate risks. There were concerns about electrocution with the introduction of electric lights. And I recall concerns when the first microwave ovens came to the market with people concerned about radiation on food. And let’s not forget cell phones and radio waves.
The issue with LLMs is that the effects are harder to detect. There’s no objective test for verifying the outputs from a generative AI application are correct. And it might only take a subtle change for an LLM to start delivering spurious outputs.
This is why I remain a sceptic when it comes to the value of the outputs of generative AI tools.
Anthony is the founder of Australian Apple News. He is a long-time Apple user and former editor of Australian Macworld. He has contributed to many technology magazines and newspapers as well as appearing regularly on radio and occasionally on TV.