AI Agent Safety Evaluation Should Be Treated Not as “Model Testing” but as “Operating a Privileged System”

If a cybersecurity testing environment cannot reliably prevent an AI agent from reaching real systems, evaluators should treat that possibility as a security risk and apply controls to constrain and monitor the agent’s access. A performance evaluation can create a pathway to harm if its containment boundary fails.

TechCrunch’s reporting raises a central concern: agents may be able to move beyond isolated cybersecurity testing environments and access real-world systems. Based only on the provided materials, it is not possible to verify the facts of a specific case, determine whether harm occurred, or assess whether the relevant institutions had legal reporting obligations. Separately, the available materials also do not support a confident conclusion that current standards and regulatory discussions have fully closed this risk.

The European Union’s AI Act framework requires evaluation, serious incident reporting, and appropriate cybersecurity for general-purpose AI models that present systemic risk. Agent autonomy and tool use are also considered in risk management. However, the European Commission states that regulatory review related to agents is still at a preliminary stage. NIST’s work on software and AI agent identity and authorization is likewise exploring a standards-based approach and reviewing feedback. Practical guidance in the style of OWASP, such as least privilege, tool-call logging, and monitoring, points in a useful direction, but it remains limited by its voluntary nature.

The practical conclusion, therefore, is not to wait until regulation is settled. Evaluation organizations and model developers need to design testing as a system under their own operational control.

Why the Sandbox Problem Is Not Only a Model Problem

Traditional model evaluation is often structured around providing inputs and observing outputs. Agentic AI evaluation is different. An agent can call tools, use accounts, and access external resources such as networks or file systems. In this context, the risk is not only that the model may produce a bad answer. The risk is that the answer may be executed as a privileged action.

If isolation is well maintained, this distinction may seem minor. But if isolation fails, if testing credentials are connected to real system privileges, or if the tool-call path is broader than expected, the evaluation itself becomes an attack surface. Safety evaluation also creates a paradox: it may try to elicit and measure more dangerous capabilities. To test whether a model can discover vulnerabilities, escalate privileges, or perform automated tasks, evaluators may need an environment and tools that match those capabilities. If the environment design is weaker than the model’s capabilities, the evaluation becomes a way to generate risk rather than a way to discover it.

This problem cannot be solved only by asking, “Is the model sufficiently safe?” The more precise questions are these.

First, under what identity does the agent act? Second, what privileges does that identity have? Third, can tool calls and external access be logged, blocked, and recovered in real time? This is why NIST treats the identification, management, and authorization of agents as a separate topic.

Decision Criterion: If It Can Reach Real Systems, Apply Operational Deployment Standards

The following criterion is useful for organizations preparing AI agent evaluations.

If an agent can use even one of the following—an external network, a real account, real data, operational infrastructure, or reusable credentials—the evaluation should be classified not as a research experiment but as an activity with operational risk. In that case, at least four measures should come first.

First, privileges should be separated and minimized by testing purpose. The label “evaluation account” is not enough. Organizations need to confirm what actions that account can perform on real systems. They also need to examine which tools can be called and whether any privilege-escalation paths exist.

Second, all tool calls and access attempts should be logged. Reviewing only the model output after the fact makes it difficult to determine what happened and why. Records should show which tools were called from which prompts. They should also record which systems were accessed under which privileges, so that incident response and reproduction are possible.

Third, stop conditions should be defined in advance. Anthropic’s Responsible Scaling Policy presents an approach in which training or deployment is halted when certain high-risk capabilities have been identified or when the necessary safeguards cannot be applied. This does not mean that every organization should adopt the same criteria. It does mean that, before evaluation begins, organizations should document which outcomes will immediately restrict access or stop testing.

Fourth, incidents and errors should be communicated to relevant stakeholders, and the process for tracking, responding to, and recovering from them should be documented. The NIST AI RMF requires incidents and errors to be communicated to relevant AI actors and affected communities, and it requires response and recovery procedures to be followed and documented. The same principle applies to agent evaluation.

Standards Are Not Yet a Shield

Based on the currently verifiable evidence, industry and regulatory frameworks are moving in a useful direction, but they are not a completed shield. The EU AI Act has a framework that imposes evaluation, incident reporting, and cybersecurity obligations on models with systemic risk. However, agent-related review is at a preliminary stage. NIST’s work on agent identity and authorization is still at the feedback review stage. Voluntary security guidance is useful, but the provided materials do not establish that there is a unified mandatory standard that comprehensively prevents sandbox escape or the misuse or abuse of real privileges.

For that reason, responsible organizations should make conservative judgments. The default should not be “this is fine because it is an evaluation.” It should be: “because the evaluation can affect real systems, apply the same controls as deployment.” Independent evaluation organizations, in particular, need strict controls. Independence does not mean accepting access privileges loosely. The credibility of independent evaluation depends on verifiable records of what privileges were received, what was executed, and what anomalies occurred.

The practical decision is straightforward. If an agent safety evaluation plan does not include a sandbox structure, separation of identity and privileges, tool-call logs, real-time blocking, incident reporting, recovery procedures, and stop criteria, the evaluation should not begin. Before measuring model capabilities more accurately, the first task is to ensure that the measurement cannot touch real-world systems.

Aionda

AI agent evaluation is an operational security problem

AI Agent Safety Evaluation Should Be Treated Not as “Model Testing” but as “Operating a Privileged System”

Why the Sandbox Problem Is Not Only a Model Problem

Decision Criterion: If It Can Reach Real Systems, Apply Operational Deployment Standards

Standards Are Not Yet a Shield

Further Reading

References

Get updates