UK’s NCSC Warns Against Cybersecurity Attacks on AI
UK’s NCSC Warns Against Cybersecurity Attacks on AI
Your email has been sent
The National Cyber Security Centre provides details on prompt injection and data poisoning attacks so organizations using machine-learning models can mitigate the risks.
Large language models used in artificial intelligence, such as ChatGPT or Google Bard, are prone to different cybersecurity attacks, in particular prompt injection and data poisoning. The U.K.’s National Cyber Security Centre published information and advice on how businesses can protect against these two threats to AI models when developing or implementing machine-learning models.
Jump to:
AIs are trained not to provide offensive or harmful content, unethical answers or confidential information; prompt injection attacks create an output that generates those unintended behaviors.
Prompt injection attacks work the same way as SQL injection attacks, which enable an attacker to manipulate text input to execute unintended queries on a database.
Several examples of prompt injection attacks have been published on the internet. A less dangerous prompt injection attack consists of having the AI provide unethical content such as using bad or rude words, but it can also be used to bypass filters and create harmful content such as malware code.
But prompt injection attacks may also target the inner working of the AI and trigger vulnerabilities in its infrastructure itself. One example of such an attack has been reported by Rich Harang, principal security architect at NVIDIA. Harang discovered that plug-ins included in the LangChain library used by many AIs were prone to prompt injection attacks that could execute code inside the system. As a proof of concept, he produced a prompt that made the system reveal the content of its /etc/shadow file, which is critical to Linux systems and might allow an attacker to know all user names of the system and possibly access more parts of it. Harang also showed how to introduce SQL queries via the prompt. The vulnerabilities have been fixed.
Another example is a vulnerability that targeted MathGPT, which works by converting the user’s natural language into Python code that is executed. A malicious user has produced code to gain access to the application host system’s environment variables and the application’s GPT-3 API key and execute a denial of service attack.
NCSC concluded about prompt injection: “As LLMs are increasingly used to pass data to third-party applications and services, the risks from malicious prompt injection will grow. At present, there are no failsafe security measures that will remove this risk. Consider your system architecture carefully and take care before introducing an LLM into a high-risk system.”
Data poisoning attacks consist of altering data from any source that is used as a feed for machine learning. These attacks exist because large machine-learning models need so much data to be trained that the usual current process to feed them consists of scraping a huge part of the internet, which most certainly will contain offensive, inaccurate or controversial content.
Researchers from Google, NVIDIA, Robust Intelligence and ETH Zurich published research showing two data poisoning attacks. The first one, split view data poisoning, takes advantage of the fact that data changes constantly on the internet. There is no guarantee that a website’s content collected six months ago is still the same. The researchers state that domain name expiration is exceptionally common in large datasets and that “the adversary does not need to know the exact time at which clients will download the resource in the future: by owning the domain, the adversary guarantees that any future download will collect poisoned data.”
The second attack revealed by the researchers is called front-running attack. The researchers take the example of Wikipedia, which can be easily edited with malicious content that will stay online for a few minutes on average. Yet in some cases, an adversary may know exactly when such a website will be accessed for inclusion in a dataset.
If your company decides to implement an AI model, the whole system should be designed with security in mind.
Input validation and sanitization should always be implemented, and rules should be created to prevent the ML model from taking damaging actions, even when prompted to do so.
Systems that download pretrained models for their machine-learning workflow might be at risk. The U.K.’s NCSC highlighted the use of the Python Pickle library, which is used to save and load model architectures. As stated by the organization, that library was designed for efficiency and ease of use, but is inherently insecure, as deserializing files allows the running of arbitrary code. To mitigate this risk, NCSC advised using a different serialization format such as safetensors and using a Python Pickle malware scanner.
Most importantly, applying standard supply chain security practices is mandatory. Only known valid hashes and signatures should be trusted, and no content should come from untrusted sources. Many machine-learning workflows download packages from public repositories, yet attackers might publish packages with malicious content that could be triggered. Some datasets — such as CC3M, CC12M and LAION-2B-en, to name a few — now provide a SHA-256 hash of their images’ content.
Software should be upgraded and patched to avoid being compromised by common vulnerabilities.
Disclosure: I work for Trend Micro, but the views expressed in this article are mine.
Stay up to date on the latest in technology with Daily Tech Insider. We bring you news on industry-leading companies, products, and people, as well as highlighted articles, downloads, and top resources. You’ll receive primers on hot tech topics that will help you stay ahead of the game.
UK’s NCSC Warns Against Cybersecurity Attacks on AI
Your email has been sent
Your message has been sent
TechRepublic Premium content helps you solve your toughest IT issues and jump-start your career or next project.
Microsoft is also running a grant competition for ideas on using AI training in community building.
Generative AI will be a game changer in cloud security, especially in common pain points like preventing threats, reducing toil from repetitive tasks, and bridging the cybersecurity talent gap.
Does your business need a payroll provider that offers international payroll services? Use our buyer’s guide to review the best solutions, from ADP to Oyster.
Get up and running with ChatGPT with this comprehensive cheat sheet. Learn everything from how to sign up for free to enterprise use cases, and start using ChatGPT quickly and effectively.
Looking for an alternative to monday.com? Our comprehensive list covers the best monday alternatives, their key features, pricing, pros, cons and more.
Stay up to date on the latest in technology with Daily Tech Insider. We bring you news on industry-leading companies, products, and people, as well as highlighted articles, downloads, and top resources. You’ll receive primers on hot tech topics that will help you stay ahead of the game.
Employees part ways with organizations for a variety of reasons. Some of these reasons are simple: finding a job elsewhere, taking time off to spend with the family or relocating to a different state. But some separations are more complex: a layoff due to company difficulties or reorganization, position elimination during a merger or termination …
Virtual reality is an artificial environment created with computer hardware and software and presented to the user in such a way that it appears and feels like a real environment. If you’re uncertain about what VR all means, this glossary from TechRepublic Premium will help your understanding. From the glossary: HAPTICS Haptics is the science …
A request for proposal is a common method for soliciting vendor quotes and answers about potential product or service offerings. They specifically intend to gather details involving implementation, operations and maintenance. This facilitates the decision-making process, allowing organizations to choose the RFP responses which best suit their needs. This guidebook from TechRepublic Premium offers a …
Successful IT departments are defined not only by the technology they deploy and manage, but by the skills and capabilities of their people. IT workers must keep up to date with the latest technology trends and evolutions, as well as developing soft skills like project management, presentation and persuasion, and general management. This IT Training …
source
