AI Security

What is AI security?

AI security is about protecting AI systems themselves: the models, the data they learn from, the prompts and tools around them and the applications built on them. It overlaps with ordinary application security, because an AI system still runs on servers, code and APIs. But AI adds new ways to attack, because a model’s behaviour can be changed by the data it is trained on and by the text it is given.

It is distinct from AI governance, which covers fairness, accountability and legal compliance, though the two meet, for example in the EU AI Act’s requirement that high-risk AI systems are accurate, robust and secure.

What are the attacks?

NIST’s AI 100-2 E2025, Adversarial Machine Learning, published in March 2025, gives a common taxonomy. The main groups are:

• evasion: inputs crafted to make a model get things wrong, such as altered images or text that slip past a classifier;
• poisoning: corrupting training or fine-tuning data, or the model itself, so it behaves badly or carries a hidden backdoor;
• privacy attacks: extracting training data or details of the model by querying it;
• attacks on generative AI such as large language models, including prompt injection and abuse of the model to cause harm.

What are the main risks for LLM applications?

The OWASP Top 10 for LLM Applications 2025 lists the risks most often seen in products built on large language models:

1. Prompt injection
2. Sensitive information disclosure
3. Supply chain
4. Data and model poisoning
5. Improper output handling
6. Excessive agency
7. System prompt leakage
8. Vector and embedding weaknesses
9. Misinformation
10. Unbounded consumption

Prompt injection leads because any text the model reads, including web pages, emails and documents, can carry instructions. Excessive agency matters more as AI agents are given tools to send messages, change records or spend money.

What controls help?

• Treat model output as untrusted: validate and encode it before it reaches a browser, a database or a shell.
• Limit what agents can do: give tools the least privilege needed, and require a human to approve consequential actions.
• Separate instructions from data where you can, and assume injected instructions will sometimes get through.
• Protect the supply chain: know where models, datasets and libraries come from, and check their integrity.
• Guard sensitive data: keep secrets out of prompts, control what retrieval can reach, and log access.
• Test adversarially: red-team models and applications before release and after significant changes.
• Monitor and rate-limit to catch abuse and runaway costs.

Which frameworks apply?

The UK’s AI Cyber Security Code of Practice, published on 31 January 2025, sets baseline security principles for organisations that develop and deploy AI across its lifecycle, and the UK government said it would form the basis of a new ETSI standard. ISO/IEC 42001 and the NIST AI Risk Management Framework provide the management system and risk process into which AI security controls fit, and ISO/IEC 27001 covers the underlying information security. The EU AI Act requires providers of high-risk AI systems to achieve appropriate levels of accuracy, robustness and cybersecurity.