Explainable AI

Many powerful AI models cannot be read by humans. A neural network computes its output from millions or billions of learned weights, and nothing in it states which input mattered how much. That is what people mean by a black box. Explainable AI, or XAI, is the set of methods meant to close this gap. They show which features drove a result, how the result would change with different inputs, or which rules a model follows overall.

There are two routes. You can choose a model whose logic can be read directly, such as a decision tree or a points table. Or you can explain a complex model after the fact with methods such as LIME or SHAP. An explanation can cover a single decision (local) or the behavior of the whole model (global).

For executives, explainability is a matter of liability and control. If AI helps decide on loans, job applications, pricing or benefits, the company has to be able to tell affected people, regulators and its own staff how a result came about. In the EU, the GDPR requires meaningful information about the logic involved in automated decisions. The EU AI Act requires transparency and human oversight for high-risk systems and gives affected people a right to an explanation.

Black box AI: why some models cannot be read.

A linear regression or a small decision tree fits on one page. Each feature has a weight or a branch, and anyone who reads the rules understands the model. Neural networks, large ensembles of hundreds of decision trees and language models work differently. Their knowledge is spread across a huge number of parameters that only make sense together.

So black box does not mean secret. Even with every weight in hand, you cannot read off why a particular application was declined. Many models are also bought as a service, and the vendor does not disclose training data or architecture. For the company using the model, it is closed twice over.

The consequences show up in operations. Errors are hard to pin down, skewed patterns from the training data go unnoticed, and a business team can neither justify a result nor push back on it with specifics. This is where explainability comes in. It shows what a model responds to, so people can check whether that holds up on the merits and under the law.

A readable model or an after-the-fact explanation.

The first route is interpretability. The model is built so that its logic is the explanation: a shallow decision tree, a rule list, a scoring system or a linear regression with a manageable number of features. The second route is a post hoc explanation: an extra method describes how an unchanged complex model behaves.

In 2019, computer scientist Cynthia Rudin argued in Nature Machine Intelligence that high-stakes decisions should rely on interpretable models instead of black boxes explained after the fact. Her core point: a post hoc explanation approximates the model. If it were exact in every case, it would be the model. With structured data and meaningful features, such as customer records, credit information or machine readings, her analysis finds that interpretable models often reach comparable accuracy.

With images, speech and free text, complex models usually perform much better, and post hoc explanation is the only option. The bigger the consequences of a single decision for a person, the stronger the case for a model that explains itself, even if it performs slightly worse.

How LIME, SHAP and counterfactual explanations work.

LIME (Local Interpretable Model-agnostic Explanations) comes from Marco Tulio Ribeiro, Sameer Singh and Carlos Guestrin (2016). It perturbs the input of a single case many times, watches how the prediction changes, and fits a simple, readable surrogate model around that case. The output is a list of the features that influenced this one decision the most.

SHAP (SHapley Additive exPlanations), by Scott Lundberg and Su-In Lee (2017), builds on Shapley values from cooperative game theory. Each feature gets a contribution, and the contributions add up to the difference between the prediction for this case and the model’s average. Exact computation gets expensive quickly as the number of features grows, so the tools rely on approximations.

Counterfactual explanations answer a different question: what would have had to be different for the outcome to change? For affected people, this is often the easiest form to understand. All three methods produce approximations. LIME and SHAP can highlight different features for the same case, and strongly correlated features distort both.

Language models do not explain themselves.

With large language models, the classic methods reach their limits. A language model has billions of parameters, its input is free text, and the answer is generated word by word. Weights for individual words in a prompt say little about why an answer turned out the way it did.

A language model’s own reasoning is generated text, not a record of its computation. In 2023, Miles Turpin and colleagues showed that step-by-step explanations from language models can hide what actually influenced an answer. In their experiments, deliberately planted biases in the prompt changed the models’ answers, for example ordering the answer options so that the correct one was always listed first. The explanations systematically failed to mention this influence.

For language models, explainability therefore shifts to the surroundings: answers with verifiable sources, for example through RAG; logged inputs, instructions and model versions; tests against fixed question sets; and review by a person before a result has consequences. Research into the inner workings of these models is advancing but, as of November 2026, cannot explain a single answer to an affected person.

What the EU AI Act and GDPR require.

Article 22 GDPR gives people the right not to be subject to a decision based solely on automated processing that produces legal effects or similarly significantly affects them. Where an exception such as a contract or explicit consent applies, people can demand human intervention and contest the decision. Under Article 15 they can also request meaningful information about the logic involved. The EU Court of Justice clarified this on February 27, 2025 (Case C-203/22): the controller must explain the procedures and principles actually applied. Simply handing over a complex mathematical formula is not enough. It may suffice to state to what extent a variation in the data would have led to a different result.

The EU AI Act (Regulation (EU) 2024/1689) requires in Article 13 that high-risk systems be transparent enough for deployers to interpret their outputs. Under Article 14, the people overseeing a system must be able to interpret its output correctly, including with interpretation tools. Article 86 gives affected people the right to a clear and meaningful explanation of the role the AI system played and of the main elements of the decision. It is tied to high-risk systems listed in Annex III, whose rules apply from December 2, 2027 after the amendment by Regulation (EU) 2026/1744 (as of November 2026).

What good practice looks like.

Good practice starts with the question of who needs the explanation. Developers hunting errors need feature weights across many cases. A business team wants to check whether a result is plausible. Affected people need a reason in plain language that refers to their case and shows which information drove the result. Regulators need documentation.

Other signs: the choice of model is justified, and for high-stakes decisions an interpretable model was at least evaluated. Explanations are tested on cases whose cause is known and, for important decisions, cross-checked with more than one method. Inputs, outputs and model version are logged so that a single case can be reconstructed later. After every retraining or model switch, the explanation is rebuilt, because it applied to the old model.

Explainable AI has no list price; it is not a product you buy. The costs come as compute time for post hoc methods and as staff time for review, documentation and clear writing. A model that is readable from the start can therefore cost less than a black box explained later.

Where the term stops working

Explainable AI shows what a model responds to, not whether the result is correct or fair; that takes subject-matter review and testing for AI bias. Post hoc explanations are approximations and can make a model look more sensible than it is. The term is distinct from interpretability (the model itself is readable), from transparency in the sense of disclosure and labeling, and from logging, which records what ran but not why.