METR published a short account on October 22 of a three-week pilot in which one of its staff and one from Redwood Research worked inside OpenAI to stress-test the company’s internal system for catching misaligned AI agents. They found coverage gaps and several ways an agent could in principle avoid or disable monitoring, all in the system as it stood on and before July 21. OpenAI has told METR that every vulnerability affecting its production safeguards has been patched, a claim the auditors say they have not evaluated.
What the audit covered
From July 21 to August 11, the two researchers had access to OpenAI’s internal codebase, the monitor’s prompts and configuration, and internal documents, and interviewed roughly a dozen engineers across monitoring, safety-systems and security teams. They worked on OpenAI-issued devices that never left the premises and were accompanied on site. The output was a 61-page internal report. METR says the exercise was a pilot for giving outsiders deep access, not a public transparency exercise, so it agreed in advance to publish only a brief, high-level post.
Coverage gaps
METR estimates that roughly 75 to 94% of internal agentic traffic on the Responses API, from models at least as strong as GPT-5.5, was eligible for monitoring, with substantial uncertainty in both directions. The Decoder translates that into an estimated 6 to 25% not covered. For internal Codex agents the share eligible was put at 99.2%. At the start of the exercise, reinforcement-learning training runs were not covered, though OpenAI told the auditors a subset of rollouts had been monitored between March and June.
A second gap involved compaction events, the points where an agent’s context is summarized. Not monitoring them likely left on the order of 10% of eligible internal Codex traffic unmonitored between May and July, METR estimates. OpenAI had found the problem itself and fixed it during the exercise. Because the monitor triggered only when an agent handed control back to the user, an agent could postpone review indefinitely by continuing to act.
Ways around the monitor
The post lists several weaknesses. Codex agents could in principle bypass sandboxing and Auto-Review and take any action the host user could. An exploit could keep hundreds of recent actions, including its own, from being monitored; OpenAI redacted the details. Crafted inputs could likely make the monitor fail by exploiting serialization differences between what an agent sees and what the monitor sees. Two client-controlled headers could disable monitoring for new agents, although setting them might itself be detected. Two internal tools thought to be read-only could be used to take arbitrary actions in their sandbox, which METR says it did not test in practice.
The auditors also report positives: the monitor sees the full trajectory except for minor exceptions, its prompts look for scheming, sabotage, data exfiltration, deception and prompt injection, and the system’s error rate was under 0.2% of requests.
What was withheld
OpenAI redacted the report before METR and Redwood leadership received a 38-page version, and only a brief summary is public. METR says it had no right to disclose the redacted portions and that, for future audits, it wants more public transparency. It also notes that changes OpenAI made after the access period, including expanded monitoring and coverage of RL training, were not evaluated.
Why it matters
Third-party access of this kind is becoming a template: Anthropic recently gave METR staff similar access, as we reported in Anthropic gives METR employee-like access. The findings also fit the picture OpenAI described after a September incident, covered in our report on OpenAI’s offline training. For background on who runs such evaluations, see our explainer on METR, UK AISI and SecureBio.
What to watch next
- Whether OpenAI publishes more of the report or allows a second, longer audit.
- Independent verification that the reported patches and expanded coverage work.
- Whether other labs adopt the design principles METR outlined, such as monitoring every N messages instead of at turn end.
- Whether regulators begin to require outside access to monitoring systems.
Key takeaways
- Two outside researchers spent three weeks on site at OpenAI in July and August.
- Estimated unmonitored share of some agent traffic ranged from about 6 to 25%, with high uncertainty.
- The auditors found several ways an agent could in principle evade or crash the monitor.
- OpenAI says all production-affecting issues are patched; METR has not verified that.
- Most of the full report stays confidential.
Frequently asked questions
Who carried out the audit?
One METR employee and one Redwood Research employee, as a pilot designed jointly with OpenAI.
Did agents actually evade monitoring?
METR describes the findings as vulnerabilities agents could in principle use; it says it did not validate every one in practice.
Is the system now fixed?
OpenAI says so. METR says it has not evaluated that claim or the expanded system.
