Understanding Right to Explanation and Automated Decision-Making in Europe’s GDPR and AI Act

Automated decision-making (ADM) systems are used to either fully replace or support human decision-making, depending on how the system is designed and what it's used for. The goal is to improve the accuracy, efficiency, consistency, and objectivity of decisions previously made by humans alone. Examples include automated recruitment systems, healthcare triage systems, online content moderation, and predictive policing.

In liberal democracies, people have become used to consequential decisions in areas such as education, welfare entitlement, employment, healthcare, and the judiciary being subject to standardized procedures and appeals processes that are open to public scrutiny. This reflects a basic understanding that human decision-makers are neither infallible nor always fair, but that it’s possible to limit the impact of human failings by setting standards against which the fairness of consequential decisions can be evaluated.

As with human decision-making, which is subject to public scrutiny, relevant provisions in Europe's General Data Protection Regulation (GDPR) and the AI Act are intended to safeguard the substantive and procedural fairness of automated decisions. In broad terms, substantive fairness involves considerations such as distributive justice, non-discrimination, proportionality, accuracy, and reliability, while procedural fairness requires (at least) transparency, due process, consistency, human oversight, and the right to an explanation.

Recent examples of AI systems that fell short of these requirements include welfare fraud detection systems in Amsterdam and the UK, families being wrongly flagged for child abuse investigations in Japan, low-income residents being denied food subsidies in the Indian state of Telangana, and racial bias in generative AI tools when used to assist with hiring.

Given legitimate concerns about the accuracy, reliability, robustness, and fairness of ADM, provisions are made for the right to provide explicit consent to be subjected to automated decisions (GDPR art 22), be informed of the use of ADM (GDPR art 13, 14 & 15 and AIA art 26), human intervention or oversight (GDPR art 22 and AIA art 86), and the right to an explanation (GDPR art 13, 14 & 15 and AIA art 86).

In the GDPR, the right to explanation applies to decisions based solely on automated processing that produce legal or similarly significant effects concerning a natural person (art 22), though the right to an explanation of any such decisions is detailed in Articles 13-15, which requires the provision of “meaningful information about the logic involved, as well as the significance and the envisaged consequences of such processing [emphasis added].”

In the AI Act (art 86), the relevant wording refers to the “right to obtain from the deployer clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision taken [emphasis added].”

There’s much debate amongst legal and policy experts over questions such as how the requirement for “meaningful information about the logic involved” in the GDPR should be interpreted, what kind of explanation is required under the AI Act, as well as the ends such explanations are intended to serve. What’s overlooked in these debates is the technical difficulty of providing explanations for the model outputs used in ADM, as well as the inappropriateness of deploying predictive ADM in domains where human agency is a factor.

With this in mind, people should be concerned not only about whether the ADM-related provisions of the GDPR and AI Act provide the same or similar protections as those for non-assisted human decisions, but also whether there is reliable evidence that explanations of AI system outputs are suitable for this purpose.

Explainable AI

The field of Explainable AI (XAI) focuses on ways to ensure that the outputs of AI systems, e.g., automated decisions, can be explained to and understood by the persons affected by an AI system's outputs/decisions. There are two broad kinds of methods to achieve this: intrinsic and post-hoc.

Intrinsic methods are possible when the AI model is simple enough that the relationship between inputs and outputs is interpretable. For example, if a decision tree model is used for credit scoring in loan applications, it’s possible to trace each step of the model’s reasoning path from inputs (e.g., income, employment history, credit history, and loan amount) to the output of whether the applicant is eligible for the loan requested. When this is known, it’s possible to then explain to the loan applicant the basis of the automated decision.

In contrast, post-hoc methods are used when the model is too complex to trace its reasoning path from inputs to outputs, which is the case with many current AI systems (e.g., large language models such as ChatGPT, DeepSeek, Claude, Mistral, Llama, and Grok). Because complex AI models are not interpretable, post-hoc methods such as Shapley Values and LIME are employed to provide insight into a model’s reasoning without having access to its internal structure, which remains opaque. Any insight gained is only an approximation of the model’s actual reasoning path, though, which means there’s no guarantee of the accuracy or consistency of post-hoc explanations. And the more complex the model, the less reliable the approximations.

Post-hoc explanations, therefore, fall short of providing the kind of protections that are possible when human decisions are contested, as they may be inaccurate and so unhelpful, or they may create a misplaced sense of trust in the explanation provided. In either case, post-hoc explanations do not provide a reliable enough basis upon which to identify or overturn substantively unfair decisions.

This means that the right to explanation in ADM is only feasible when AI models are interpretable. But given that more complex AI models also tend to be more accurate, people are left with a trade-off between explainability and performance. This is less of a concern than it might initially seem, as there are also reasons for limiting the use of ADM when decisions are rightly subject to public scrutiny.

Even if post-hoc methods were reliable enough to identify unfair decisions and provide a valid basis for appeal, ADMs should not be deployed when they’re used to predict outcomes in domains in which human agency is a factor. For example, when predicting the likelihood of a student dropping out of university, whether a patient with active tuberculosis will adhere to prescribed treatment regimens, if an applicant will default on a car loan, or how likely a criminal defendant is to reoffend when determining bail conditions. 

Why prediction isn't enough

As Narayanan and Kapoor argue in AI Snake Oil, predictions of ‘life outcomes’ are both unreliable and ethically problematic. By ‘life outcomes,' they mean phenomena such as whether a couple will file for divorce, or what an individual’s future income will be.

In all such cases, and as demonstrated by many failed attempts in the social sciences to devise ways of predicting life outcomes, the attempt to do so becomes a factor in the system we’re trying to forecast, and the only way to avoid this is to deny the individual/s subject to such predictions their agency to act otherwise.

The alternative is to focus instead on understanding the social factors that lead to divorce, future income, or any other life outcome, so that individuals and relevant agencies can leverage this knowledge to forge a preferable future. This is a more appropriate goal for liberal societies, and also a better use case for more accurate yet non-interpretable AI models.

These more powerful models could also be reconfigured to support human decision-makers. Instead of being left with a choice of either accepting or dismissing an automated prediction, automated support systems could be designed to help human decision-makers become more efficient and consistent in their deliberations. For example, such systems could prompt human decision-makers to consider additional factors or set aside others in their deliberations, provide an analysis of how similar cases have been decided in the past, both by the individual making the decision and their peers, and provide analyses of any concerning trends in the impact of such decisions on vulnerable populations.

The implications of this for the GDPR, AI Act, and other similar regulations are that provisions designed to safeguard ADM should limit fully automated decisions to interpretable models, whose output should include a clear explanation of the decision, and that decisions that bear on circumstances in which human agency is a factor should not be automated. Failure to do so will result in many more instances of people being unfairly targeted or denied access to public goods without recourse to the same kind of safeguards that liberal democracies have previously relied upon. And given the rush to deploy AI systems in public administration, failures of this kind will very likely undermine the substantive and procedural fairness of decisions that have a significant impact on their citizens.