METR and Redwood Find Gaps in OpenAI's Internal Safety Checks

A three-week audit found that OpenAI's internal systems failed to track a substantial portion of activity from its most advanced AI models.

Researchers from the evaluation group METR and the research organization Redwood Research conducted the on-site review from July 21 to August 11. During this pilot program, the auditors accessed OpenAI's codebase and internal documents while interviewing approximately a dozen employees.

The primary finding concerns the scope of oversight. The auditors estimated that between 6 and 25 percent of internal agentic Responses API traffic from models in the GPT-5.5 class and above was not monitored at all. METR noted that the uncertainty surrounding this range is considerable in both directions.

A separate issue affected internal Codex traffic that was supposed to be under surveillance. The system failed to process compaction events, which are context summaries, causing roughly 10 percent of that traffic to go untracked between May and July. OpenAI identified this bug independently and applied a fix while the audit was still ongoing.

The structure of the monitoring system allowed agents to delay oversight. The monitor only activated when an agent returned control to the user, meaning an agent that continued executing actions without yielding could avoid detection. The auditors also identified methods to hide hundreds of actions from the monitor and to bypass the Codex sandbox. They concluded that specially crafted inputs could likely crash the monitor entirely.

At the start of the audit, model behavior during reinforcement learning training was not monitored. OpenAI stated that it paused this training after an incident involving Hugging Face in August and subsequently expanded monitoring to include it. These changes occurred after the auditors' access period ended, so they were not part of the initial evaluation.

OpenAI reports that it has fixed all the weaknesses identified by the auditors. The full 61-page report was shared internally, while METR and Redwood Research received a 38-page redacted version. A short summary was released to the public, as METR said it lacked the right to disclose the redacted portions independently. METR indicated it wants to pursue greater public transparency in future audits.