AI safety and security
AI safety and security covers two risks: an AI system causing harm on its own (safety) and attackers exploiting it (security). Red teaming tests both with deliberate attacks.
Safety asks whether the system causes harm on its own; security, whether it's under attack.
AI safety and security covers everything that keeps an AI system from causing harm or becoming a way into the company. Safety is about the system's own behavior, such as dangerous or discriminatory output, even when nobody is pushing it. Security is about attacks such as injected instructions, poisoned training data, or data leaks. Safety asks whether the system causes harm on its own; security, whether it's under attack.
Firewalls and antivirus software inspect traffic and files, not what a sentence means. With language models, a sentence in an email or a PDF can act as a command. The matching test is red teaming: a team attacks its own system to find weak spots before attackers do.
For executives, the assessment starts with access. A chatbot that only answers questions is less dangerous than an AI agent that creates purchase orders, and the company running the system decides what it may touch.
Chapter 1 of 7
Safety and security are two different risks.
The AI Risk Management Framework from the U.S. National Institute of Standards and Technology (AI RMF 1.0, January 2023) lists them as separate traits of trustworthy AI. A safe system does not, under defined conditions, lead to a state in which human life, health, property, or the environment is endangered. A secure system protects confidentiality, integrity, and availability against unauthorized access.
Most cases sort themselves out once you ask who causes the damage. If an assistant for field technicians gives wrong instructions for shutting down a machine, that's a safety problem. If an attacker uses a crafted input to make the same assistant reveal customer data, it's a security problem. A jailbreak sits on both sides: it's an attack meant to force output the model should refuse.
The split also settles ownership. Safety sits with quality assurance, security with IT.
Chapter 2 of 7
Attacks target language, data, and access rights.
The OWASP Top 10 for LLM Applications (2025) ranks the vulnerabilities of apps built on language models. Prompt injection comes first: text gets into the input and is followed as an instruction, either directly through the chat window or indirectly through a document or web page the system reads. Other entries include poisoned training data and excessive agency.
NIST's taxonomy AI 100-2 (March 2025) covers poisoning, evasion, privacy attacks, and misuse. MITRE ATLAS collects attacker techniques against AI systems in a public knowledge base, along with case studies of real incidents.
Part of the risk starts inside the company. With shadow AI, company data ends up in services nobody has vetted. Germany's cybersecurity agency, the BSI, advises assuming that anything a generative AI model can access may be extracted.
Chapter 3 of 7
Red teaming attacks your own system before someone else does.
In its Generative AI Profile (NIST AI 600-1, July 2024), NIST describes AI red-teaming as a structured testing exercise that probes an AI system for flaws and vulnerabilities, such as inaccurate, harmful, or discriminatory output, usually in a controlled setting with the developers.
A red team might try to hijack the system through prompt injection, coax out confidential documents, get around safety rules, or use it to write phishing emails.
The BSI recommends extensive testing of generative AI models and the apps built on them before rollout and, for critical uses, red teaming that simulates concrete attacks or misuse. Any red-teaming result is a snapshot. Once the model, instructions, or data sources change, it's out of date.
Chapter 4 of 7
With AI agents, permissions matter more than filters.
OWASP and the BSI agree that prompt injection can't be fully prevented today. The workable approach is to limit the damage.
Risk grows with authority. OWASP calls it excessive agency when a system has more functionality, permissions, or autonomy than its task requires. Its example: an extension that summarizes emails only needs read access, not the ability to delete or send.
A few principles follow from this. Each system gets only the permissions its task requires. A person approves consequential steps such as payments, sending, or deleting. Output is checked before another system processes it, and a log records what the system read and did. Guardrails add to these measures but don't replace them.
Chapter 5 of 7
The AI Act requires robustness and adversarial testing.
Article 15 of the EU AI Act (Regulation (EU) 2024/1689) requires high-risk AI systems to achieve an appropriate level of accuracy, robustness, and cybersecurity. Under paragraph 5, they must resist attempts by unauthorized third parties to alter their use, outputs, or performance by exploiting vulnerabilities. The article names measures against poisoned training data (data poisoning) or pre-trained components (model poisoning), inputs designed to make the model err, and confidentiality attacks. Under the Digital Omnibus on AI (Regulation (EU) 2026/1744), this applies from December 2, 2027, for Annex III systems and from August 2, 2028, for Annex I systems.
Since August 2, 2025, Article 55 has required providers of general-purpose AI models with systemic risk to run model evaluations with documented adversarial testing.
Companies that neither deploy high-risk systems nor offer such models aren't directly affected. Specific cases call for an IT lawyer.
Chapter 6 of 7
What good implementation looks like.
It starts with an inventory of every AI system, the data each one reads, and the tools it may use. A threat analysis per system spells out who could attack it and how.
Before launch, testers run attacks from the OWASP categories and MITRE ATLAS techniques; critical applications get a red team of specialists. Findings go into a report with severity, countermeasure, and retest. After every change to the model, system prompt, or data sources, a fixed set of attack cases runs again, just as evals re-measure quality.
In operation, logs show what a system read and did. It's clear who shuts a system down after an incident and who gets told. Employees know which services are approved, and the BSI recommends thorough training. Good vendors plan the retest from day one.
Chapter 7 of 7
Where the term stops working
AI safety and security doesn't make a system error-free or legally compliant. Mistakes without an attacker are measured with evals. AI governance sets rules and ownership, and AI ethics deals with fairness and values. Guardrails and red teaming are building blocks, not the whole concept.
Frequently asked questions
What is AI safety and security?
It's protection against harm caused by AI systems and against attacks on them. Safety covers the system's own dangerous behavior, security covers attacks such as prompt injection.
What is the difference between AI safety and AI security?
AI safety is about harm a system causes on its own, AI security is about attacks on the system. NIST lists them as separate characteristics of trustworthy AI.
What is red teaming in AI?
Red teaming is a structured attack test in which a team probes an AI system for vulnerabilities and harmful output. The BSI recommends it before rollout for critical applications.
How often should you red team an AI system?
At a minimum before launch and after every significant change to the model, instructions, or data sources. The BSI advises repeating tests regularly and when new attack methods become known.
What risks does AI pose to IT security?
AI creates new attack surfaces and new tools for attackers. Your own systems can be exploited through injected text or overly broad permissions, and they can be misused to write phishing emails.
How do you protect AI systems from attacks?
Mainly by limiting the possible damage, since prompt injection can't be fully prevented today. That means narrow permissions, human approval for consequential steps, vetted data sources, logging, and regular attack tests.
What does the AI Act require for AI security?
It requires accuracy, robustness, and cybersecurity for high-risk AI systems (Article 15) and adversarial testing for general-purpose AI models with systemic risk (Article 55). The high-risk rules apply from December 2027 at the earliest.
What is the difference between AI security and cybersecurity?
AI security adds risks that only come with models. With language models, ordinary text can act as an instruction, and manipulated training data can corrupt a model.
Sources
- OWASP: Top 10 for LLM Applications, 2025 edition
- NIST: Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, January 2023
- NIST: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1, July 2024
- NIST: Adversarial Machine Learning. A Taxonomy and Terminology of Attacks and Mitigations, NIST AI 100-2 E2025, March 2025
- MITRE: ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems), knowledge base, as of November 2026
- BSI (German Federal Office for Information Security): Generative KI-Modelle. Chancen und Risiken für Industrie und Behörden, January 21, 2025 (German)
- European Union: Regulation (EU) 2024/1689 (Artificial Intelligence Act), Articles 15, 55, and 113, 2024
- European Union: Regulation (EU) 2026/1744 amending the AI Act (Digital Omnibus on AI), 2026