Privacy by Architecture Makes AI Compliance Work

Artificial intelligence presents new privacy challenges, but treating AI as if it lives outside the parameters of ordinary privacy law is fallacious. The emerging lesson from privacy regulators and enforcers is increasingly clear: there is no AI exceptionalism. The same principles that have long governed responsible data use still apply. What changes is how and where those principles need to be built. In these nascent days, AI offers an opportunity to move beyond “traditional” compliance by embedding privacy, transparency, and user control into the architecture itself.

“Traditional” privacy compliance has often focused on collection, use, disclosure, and retention policies lined up with what a company says in its statements and privacy policies. The emerging reality with AI complicates this picture. AI infers, summarizes, remembers, ranks, and increasingly acts across blurred boundaries.

That shifts AI into a category closer to delegated action, where the user is not controlling nor even observing each action. And this delegated action relationship creates a different risk profile and attack surface for privacy harms. The core concern is not only that a model may process personal data. It is that the system may decide what data it needs, access more than the task requires, move information across contexts, and act before the user fully understands what happened. What data gets inferred, shared, or disclosed might go beyond what a user is willing to share.

Despite the broader AI exceptionalism arguments, regulators are not treating AI as an exotic category outside privacy law. Instead, regulators are identifying where familiar privacy principles become harder to operationalize. Across guidance and enforcement actions, the same concerns keep surfacing: AI collapses boundaries between collection and use, between one purpose and another, between user instruction and agentic initiative.

Regulators globally have used different language to describe these risks, yet key privacy principles remain consistent and expectations to integrate them into AI-related contexts are high. A few in-progress examples of articulating privacy rules in the context of emerging AI use cases include:

• The UK Information Commissioner’s Office cautioning that AI agents should have clear purposes for collecting and processing information and that selecting what databases agents can access is fundamental to data minimization requirements.
• The European Data Protection Supervisor emphasizing human oversight as a crucial accountability mechanism for output data and inferences made by an AI model.
• Hong Kong’s Office of the Privacy Commissioner for Personal Data underscoring the importance of access right controls and continuous risk assessments as cornerstones for compliance.
• Singapore’s Personal Data Protection Commission clarifying that web-scraping datasets containing personal data can be justified by the “Publicly Available Exception” in lieu of always needing to seek consent.
• Australia’s Office of the Information Commissioner highlighting that privacy obligations apply to information inputted into AI systems, just as well as the information that is outputted.

In the United States, the Federal Trade Commission’s past enforcement actions provide relevant guidance for privacy design considerations when deploying AI:

• The Rite Aid settlement order shows the pre-deployment testing the company must conduct before deploying AI for high-risk uses.
• The Drizly matter demonstrates that data minimization and retention limits should be baseline security requirements to lessen the attack service and the risk of personal data exposure from a breach.
• GM’s settlement with the FTC highlights that connected services cannot quietly collect, use, and disclose sensitive personal data for unknown purposes.
• The Amazon Alexa matter demonstrates the need to respect users’ desire to delete data and ensure it’s not used for training models.

US state regulators have also been busy releasing guidance and developing new laws. Even without new laws, state Attorneys General can enforce existing unfairness and deception laws to address AI-specific harms.

Three key commonalities emerge from the privacy principles that regulators and enforcers are emphasizing across guidance and enforcement actions.

1. Purpose limitations

The first is that purpose limitations must become architected into AI services, and not merely contractual. In ordinary privacy programs, a purpose limitation often appears as a sentence in a notice: we use data to provide and improve services. For AI services that blur lines and can agentically act on a user’s behalf, this is not enough. “Improve productivity” or “assist the user” can become an over-broad and unclear permission to do nearly anything unless translated into technical boundaries.

In agentic systems in particular, setting clear purpose limits should be architected to honor users’ privacy and to reduce the risk of security failures. This means constraining which tools each agent can call, which data sets it can retrieve, which memories it can use, whether data can move between context, when the user must confirm an action, and whether outputs can be used for training or secondary analytics. This helps operationalize privacy requirements while adhering to longstanding security principles such as giving the least privilege to each role.

2. Accountable deployment

The second thread is accountable deployment. Regulators are skeptical of the idea that autonomy dissolves responsibility. An organization that builds, buys, deploys, or integrates AI remains responsible for how personal data is used. That means pre-deployment review cannot stop at asking whether the model is accurate; it must ask whether the system behaves within the privacy boundaries the organization promised.

Testing should include privacy-specific evaluations. Does the agent access data outside the task? Does it retain information unnecessarily? Does it resist prompt injection making system boundaries unreliable? Can the organization reconstruct what data was accessed, which tools were called, and why? Does the system perform differently across groups? What happens when the agent is wrong?

3. Transparency and use control

A third thread is that transparency and use control must move from notice to interface. Privacy notices cannot do all the work. Users need controls at the point of action. For AI services that store and use memory, this means developing memory dashboards that surface what the AI system remembers and allow users to edit or delete it. It means permission receipts explaining what data an agent accessed for a task. It means pre-action confirmations before an agent sends a message or discloses information. It means mode-switching between personal and workplace contexts. It means “do not remember this” and “why did you use that?” controls embedded into the product experience. These tools not only enhance compliance, indeed they help the user to improve the system experience.

- - -

These threads stem from principles enforcers and regulators have consistently noted are a core part of privacy requirements. They are not decorative compliance features, rather they are how privacy principles should be reflected in the design and operation of AI models and systems. The challenge now is to apply these principles to newer risks created by persistent memory, expanded permissions, and increasingly agentic systems.

Unlike the internet advertising economy, where privacy and profit have often been in tension, AI presents an opportunity to align them. Privacy, transparency, and user control can become part of the product itself. Features that strengthen trust, improve the user experience, and make AI systems more commercially attractive. The question for AI developers and deployers then, is not whether existing privacy principles can survive new technology. It is whether entities will seize the opportunity to translate those principles into the architecture, and ultimately the value proposition, of their products.