Making AI Safe Requires More Than Constraining Its Instructions

Proposals in the United States to create an independent, industry-informed regulator for advanced AI models raise an ongoing question: what should count as evidence that a model is safe? One approach has focused on system prompts, the natural-language instructions that can guide model behavior. It is an understandable approach given these prompts are legible, documentable, and express developer intent. However, treating them as primary evidence that a system is ‘safe’ risks conflating written instructions with actual system performance. Therefore, safety assessments must (at minimum) evaluate system outputs, not solely inputs.

System prompts are developer-set natural-language instructions that take priority over user-written prompts. They define roles, priorities, or constraints on certain queries. A model may be instructed to act as a helpful assistant, or to “not provide information for creating harmful substances or weapons.” These natural language instructions are often more accessible and intuitive to auditors than model weights, training data, or post-training pipelines.

There is some value in the transparency of system prompts. Prompt documentation can show design choices, track changes, and record how developers handle incidents. Versioning and provenance information can help regulators and researchers identify when, why, and by whom behavioral instructions changed. In some cases, this may expose gaps between public commitments and internal practices. Prompts can therefore contribute to transparency.

But this contribution is limited. System prompts can provide insight into what developers intend a model to do; they do not establish what the model will do in practice. The relationship between prompt wording and model behavior is contingent and unstable.

As our recent research shows, system prompts provide weak guarantees of model behavior. The stochastic nature of LLMs means prompt effects vary across tasks, contexts, and models. Any given instruction may produce different outcomes based on phrasing, formatting, context length, or the presence of other safeguards. Even within the same model family, the effects of system prompts may not be uniform.

Recent incidents illustrate this instability. Last year, xAI identified and reversed an unauthorized system prompt change to Grok that caused it to make false claims of a “white genocide.” OpenAI also made system prompt changes to GPT‑4o in response to criticism of the new model's overly sycophantic behavior. Changes to system instructions can have quick and significant impacts—but in both cases, the models required further adjustments.

Another risk arises from simply evaluating the text of system prompts and not their resulting effects. Models do not process natural language in the same way as humans. When a developer or regulator interprets an instruction such as “be truthful” or “avoid ideological bias,” they draw on social and institutional understandings of those terms. A model, by contrast, responds to statistical patterns learned during training, part of a fundamental gap between human and machine interpretation.

Take instructions to be “accurate,” “truth-seeking,” or “ideologically neutral,” as required by recent US regulation. Interpretation of these terms is complex and variable across stakeholders. In practice, a regulator might need to settle difficult questions: Does truth-seeking require the model to provide sources, distinguish fact from opinion, state uncertainty, refuse false premises, or present contested evidence? Similarly, does being ideologically neutral mean that developers must mention specific terms to the model, or avoid them entirely? In extreme cases, this could veer into misinformation or censorship.

Clear wording does not guarantee consistent or intended behavior, nor does it resolve these underlying ambiguities, as evidenced by the unintended consequences of Elon Musk trying to align Grok with his personal political views.

A regulatory approach centered on prompts therefore creates specific risks. First, it may lead to the certification of intent rather than performance. A provider could present well-crafted instructions that meet formal criteria even though the model fails to adhere to them in practice. If oversight focuses on wording, this failure may go undetected. Certification of safe models then depends on developer intent, rather than evidence of outcomes.

Second, evaluating prompt text alone may create opportunities for misleading or even malicious compliance. A provider may align prompts with regulatory expectations while also relying on techniques like fine-tuning or hidden code in prompts to shape system behavior. As a result, the machine interpretation and/or action can differ from what a human reader might infer from the prompt alone. These methods can influence behavior in ways that are not transparent in the prompt and undermine regulatory aims.

As such, a prompt content-centric approach risks overestimating what can be inferred from readable instructions alone. The object of scrutiny should be system behavior in the context of its deployment, not merely prompt text. Documentation regarding system prompts can be used to assess system behavior, where it shows how prompts are developed over time in response to observed outputs. It should detail specific versions, change logs, rationale for changes, deployment settings, and the stakeholders involved in approvals.

System prompt documentation should also show evidence of how the prompt triggers specific behaviors and how reliably it does so. This can involve risk evaluations, adversarial testing, and audits across different contexts and languages. Tests should also check for unsafe responses or unnecessary refusals on safe requests, and these tests should be iteratively repeated after updates to the prompt or its deployment context.

While the design and legal authority of a potential US AI safety agency are still unclear, any overreliance on 'prompt governance' is unlikely to produce a satisfactory outcome. While system prompts can reveal developers’ intent, certifying a model’s safety requires evidence of its behavior. Over-reliance on prompt content risks certifying aspirations that may overlook unsafe AI behavior instead of providing real assurance.