--- title: "AI bias: definition and example – Carl-Uwe Würfel" description: "AI bias explained: how systematic distortions enter AI systems, how algorithmic bias is measured, and what the EU AI Act and anti-discrimination law require." url: https://carl-wuerfel.de/en/glossary/ki-bias author: Carl-Uwe Würfel --- 1. [Home](https://carl-wuerfel.de/en) 2. [Knowledge](https://carl-wuerfel.de/en/knowledge) 3. [Glossary](https://carl-wuerfel.de/en/glossary) 4. AI bias # AI bias AI bias refers to systematic distortions in an AI system’s results that repeatedly favor or disadvantage certain groups or characteristics. ## On this page 1. How AI bias arises 2. Bias in language models 3. Measuring it: fairness has several definitions 4. What the law requires 5. Common misconceptions 6. What good practice looks like 7. Where the term stops working 8. Frequently asked questions 9. Sources AI bias only becomes visible when many results are compared by group. _Figure: A sequence using the example of a company that has an AI tool pre-screen apprenticeship applications. The first review after a few months shows that applications with gaps in the résumé make the shortlist far less often, which disproportionately affects women returning from parental leave. Correction: the feature is removed from the scoring, and people re-review the rejected applications from that period. A loop back to the review stands for the fixed intervals for further spot checks._ **Example** A company uses an AI tool to pre-screen apprenticeship applications. The first review after a few months shows that applications with gaps in the résumé make the shortlist far less often, which disproportionately affects women returning from parental leave. The company removes the feature from the scoring, has people re-review the rejected applications from that period and sets fixed intervals for further spot checks. AI bias is a pattern across many cases. It only becomes visible when many results are compared by group. A model that rates women’s applications lower, rejects loans in certain neighborhoods more often or always assigns a profession to one gender in its texts is biased, even if each individual decision looks plausible. The US National Institute of Standards and Technology (NIST) distinguishes three categories in its 2023 AI Risk Management Framework: systemic bias in data, organizations and society; computational and statistical bias, for example from non-representative samples; and human-cognitive bias in how systems are built and used. In statistics, bias means a systematic estimation error. What matters is the distinction between bias as a technical skew and discrimination as legally prohibited unequal treatment of people. Not every bias is unlawful. Any bias that disadvantages people because of a protected characteristic can be. NIST notes that bias can arise without any intent to discriminate. For executives, AI bias touches liability and reputation. Germany’s General Act on Equal Treatment (AGG) does not ask whether a person or software caused the disadvantage. The EU [AI Act](https://carl-wuerfel.de/en/ai-act) requires an examination of possible biases for high-risk systems, for example in recruitment. Anyone buying AI for decisions about people should know how the vendor tests for bias and plan their own spot checks. ## How AI bias arises. The most common source is the data. A model learns patterns from examples. Past unequal treatment in the examples gets learned too. With selection bias, groups are under- or overrepresented in the data. With label bias, the human judgments the model was trained on were already skewed, for example earlier hiring decisions. Then there are proxy variables: a zip code, gaps in a résumé or certain hobbies can indirectly encode a protected characteristic such as ethnicity, gender or age. The second source is the model itself. A training objective that maximizes accuracy across all cases can neglect a small group, because its errors barely move the average. The third source is deployment. A model built for one purpose or region can behave in a biased way elsewhere. And people often accept suggestions without checking them, an effect NIST classifies as human-cognitive bias. A well-known case is a recruiting tool at Amazon. According to a 2018 Reuters report, it downgraded résumés containing the word “women’s” because it had learned from applications submitted over the previous ten years, most of which came from men. Amazon abandoned the project. ## Bias in language models. Large language models learn from vast amounts of text and absorb its stereotypes. This shows up less in yes-or-no decisions than in wording: the gender of the manager in a sample text, the names that appear in success stories, the tone chosen for different customer groups. Providers dampen these patterns during training but do not remove them entirely. This matters once language models draft job ads, summarize applications, triage requests or write reviews. A summary that stresses weaknesses for some applicants and strengths for others skews the later decision, even if a person makes the final call. This can be tested by swapping attributes. The same request is sent several times, with only the name, gender or origin changed. If the results differ systematically, there is bias. Because language models can answer the same input differently, such a test needs enough repetitions to rule out chance. A careful [system prompt](https://carl-wuerfel.de/en/glossary/system-prompt) can soften bias but does not replace testing. Generated text about people should be read for substance before it goes out. ## Measuring it: fairness has several definitions. AI bias is measured by comparing results across groups. Demographic parity requires that each group receives a positive outcome, such as an interview invitation, at the same rate. Equalized odds requires equal true positive and false positive rates across groups. Calibration requires that a given risk score means the same probability in every group. These metrics cannot all be met at once. In 2016, Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan showed that three intuitive fairness conditions for risk scores cannot be satisfied simultaneously except in narrow special cases, such as equal base rates across groups or perfect prediction. Which definition applies is therefore a business and ethical decision that should be documented. Free tools such as Fairlearn and AI Fairness 360 calculate these metrics. They do not replace the groundwork: which groups are compared, where the group attributes come from, and how many cases per group are needed for a difference to be meaningful. With the small case numbers typical of mid-sized companies, a difference can arise by chance, and the absence of one proves little. ## What the law requires. As of November 2026, several rules apply. Article 10 of the AI Act requires providers of high-risk systems to examine training, validation and test data for biases that could affect health, safety or fundamental rights or lead to prohibited discrimination, and to take measures to detect, prevent and mitigate them. High-risk uses include recruitment and creditworthiness assessment. Under amending Regulation (EU) 2026/1744, these obligations apply from December 2, 2027. The same amendment inserted Article 4a, which allows special categories of personal data to be processed under strict conditions where strictly necessary to detect and correct bias, and expressly creates no testing obligation for systems outside the high-risk provider duties. The AGG prohibits discrimination on grounds of race or ethnic origin, sex, religion or belief, disability, age or sexual identity. It also covers indirect discrimination through seemingly neutral criteria. In its July 2026 response to a planned AGG amendment, the federal government stated that the act is technology-neutral and that discrimination is unlawful even when it occurs through an AI system. Article 22 of the GDPR also gives people the right not to be subject to a decision based solely on automated processing that has legal or similarly significant effects, subject to statutory exceptions. ## Common misconceptions. The most common misconception is that removing gender, age or ethnicity from the data removes the bias. It rarely does, because proxy variables carry the same information. Without the protected characteristics, bias is also harder to measure. That is why the AI Act permits processing such data for exactly this purpose, subject to safeguards. Second misconception: a machine is more neutral than a person. A model absorbs the patterns in its data and applies them uniformly to every case. A biased clerk disadvantages individual cases; a biased model may disadvantage every case of a given type. Third misconception: a model that passed a check once stays fair. Data and operating conditions change, and feedback loops can amplify bias. People who are rejected never show up in later success data, and the model confirms itself. Article 10 of the AI Act names this case explicitly, where outputs influence inputs for future operations. Fourth misconception: a good metric proves fairness. Because fairness definitions conflict, a metric only shows that one particular definition is met. Choosing it remains a decision someone has to justify. ## What good practice looks like. Good practice starts at procurement. Anyone buying an AI system for decisions about people should learn from the vendor what data it was trained on, which groups were compared, which fairness definition applies and what the testing showed. A vendor who cannot provide any of this is a warning sign. In operations, a few fixed rules help. Before launch, the company defines which groups will be compared and at what level of deviation someone takes a closer look. Spot checks continue long after rollout. Decisions with significant effects on people are made by someone who can and may actually review the recommendation, not someone who merely confirms it. Affected people get a way to have a decision reviewed. Documentation ties it together: which data, which test, which results, which corrections. For high-risk systems it is mandatory, and elsewhere it helps when a decision is challenged later. The effort depends less on the measurement than on the groundwork: preparing clean comparison data and deciding what counts as fair. ## Where the term stops working AI bias describes systematic distortions across many cases. A model that is equally inaccurate for every group is poor but not biased, and a [hallucination](https://carl-wuerfel.de/en/glossary/hallucination) is a single fabricated output. Discrimination is the legal term for prohibited unequal treatment of people; not every bias leads to it, for example when a model recognizes product photos in one category less reliably. ## Frequently asked questions What is AI bias? AI bias is a systematic distortion in an AI system’s results that repeatedly favors or disadvantages certain groups or characteristics. It shows up only when many cases are compared, not in a single answer. How does bias get into AI systems? Mostly through training data, when groups are missing, past judgments were skewed, or neutral features such as zip codes stand in for protected characteristics. The model’s training objective and deployment in a setting it was not built for add to it. What is the difference between AI bias and hallucination? A hallucination is a fabricated statement in a single case, while AI bias is a recurring pattern that disadvantages certain groups. Hallucinations are found by fact-checking, bias only by comparing many results by group. How do you test an AI system for bias? You compare results by group against fairness metrics defined in advance and repeat this during operation. For language models, swapping attributes so that only the name, gender or origin changes is a useful test. Can AI bias be eliminated completely? No, it can be measured and reduced but not eliminated. Every dataset captures only part of reality, and different fairness definitions provably cannot all be satisfied at once. What does the EU AI Act say about bias? Article 10 requires providers of high-risk systems to examine their data for bias and take countermeasures; as of November 2026 these obligations apply from December 2, 2027. Article 4a, added by the July 2026 amendment, allows special categories of personal data to be processed under strict safeguards to detect and correct bias. Does German anti-discrimination law apply to AI decisions? Yes, in the federal government’s view the AGG is technology-neutral, so discrimination based on a protected characteristic is unlawful even when an AI system is involved. As of November 2026 the act contains no specific rule on algorithmic discrimination. Assessing a specific system is a job for employment and anti-discrimination lawyers. ## Further reading * [**AI glossary**](https://carl-wuerfel.de/en/glossary) * [**AI governance: permissions, access, logs**](https://carl-wuerfel.de/en/ai-governance) * [**AI Act: classifying how you use AI**](https://carl-wuerfel.de/en/ai-act) * [**Data quality**](https://carl-wuerfel.de/en/glossary/datenqualitaet) * [**Hallucination**](https://carl-wuerfel.de/en/glossary/hallucination) * [**System Prompt** Term](https://carl-wuerfel.de/en/glossary/system-prompt) * [**Data governance** Term](https://carl-wuerfel.de/en/glossary/data-governance) ## Sources 1. [EU: Regulation (EU) 2024/1689 (Artificial Intelligence Act), Art. 10, 2024](https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689) 2. [EU: Regulation (EU) 2026/1744 amending the AI Act (Digital Omnibus on AI), Art. 4a, 2026](https://eur-lex.europa.eu/eli/reg/2026/1744/oj/eng) 3. [German Federal Ministry of Justice: General Act on Equal Treatment (AGG), official translation](https://www.gesetze-im-internet.de/englisch_agg/) 4. [German Bundestag: Printed paper 21/6669, Bundesrat opinion and federal government response on amending the AGG, 2026 (German)](https://dserver.bundestag.de/btd/21/066/2106669.pdf) 5. [EU: Regulation (EU) 2016/679 (General Data Protection Regulation), Art. 22, 2016](https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32016R0679) 6. [NIST: Artificial Intelligence Risk Management Framework (AI RMF 1.0), NIST AI 100-1, 2023](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf) 7. [Kleinberg, Mullainathan, Raghavan: Inherent Trade-Offs in the Fair Determination of Risk Scores, 2016](https://arxiv.org/abs/1609.05807)