Study Finds AI Auditors Catch Most Agent Misbehavior but Flood Reviewers With False Alarms

1 reports archived1 independent sourcesupdated 11/8/2026, 00:56:38

Synthesis & Latest Updates

1 Sources Cross-Validated

Andy Hall, a researcher at the Anthropic Institute, replicated and extended METR's MALT dataset study and found that frontier models can flag most problematic agent behavior in logs, but pushing for higher recall produces a flood of false positives that mark harmless actions as suspicious. The conclusion: AI auditors are better suited to estimating incident rates than to serving as zero-miss safety gates, and those granting agents approval-free permissions should bear more responsibility.

LATEST/Andy Hall, a researcher at the Anthropic Institute, replicated and extended METR's MALT dataset study and found that frontier models can flag most problematic agent behavior in logs, but pushing for higher recall produces a flood of false positives that mark harmless actions as suspicious. The conclusion: AI auditors are better suited to estimating incident rates than to serving as zero-miss safety gates, and those granting agents approval-free permissions should bear more responsibility.

Hourly heat curve

3 fully observed hours

Now

11.6

Peak

12.001:00

24h change

–

The line compares only sources observed throughout; gap hours are interpolated to keep the trend continuous.

Coverage timeline

Total 1 reports · Latest first

The DecoderT2Nov 8, 00:56·68 pts
Study Finds AI Auditors Catch Most Agent Misbehavior but Flood Reviewers With False Alarms
- Frontier models auditing agent logs achieve high recall, but false positives rise sharply as strictness increases
- AI auditing suits incident-rate estimation rather than zero-miss safety checks
- Users granting agents approval-free execution permissions should bear more responsibility for consequences
View Related Reports & Analysis Original Source