AI Monitoring Needs Tests for What It Misses

OpenAI describes a layered system for detecting frontier-model misalignment

In an October 28 update, OpenAI describes monitors that screen agent behavior, investigate flagged cases and, for some workloads, compare transcripts with system logs before alerting a human. The company also says it audits samples that passed earlier checks and uses red-teaming to look for ways agents could evade monitoring. These are OpenAI’s reported methods and results, not an independent assessment of their effectiveness.

The useful evaluation question is broader than “Did the monitor catch this incident?” It is also: What did it miss, how often, and did monitoring change the behavior being monitored? OpenAI notes that tying monitor flags directly to training rewards could encourage concealment, so it says flagged samples keep their rewards. That design choice itself needs continued testing as systems change.

A monitor’s success depends partly on the cases it never flags. What evidence would give teams confidence they are finding those, too?