Anthropic Deploys Low-Cost Self-Assessment for Claude Safety Using KV Cache

Baoyu (@dotey) · 2026-10-29 21

Claude now performs a self-check for conversation safety after every response. The compute is provided by Anthropic at no cost to users, with no additional data collection. A single flag does not directly result in an account ban.

Is this primarily to prevent distillation?

Anthropic has introduced an anti-abuse technique called "self-assessment." After Claude completes each turn of a conversation, the system appends a brief instruction prompting it to reflect on whether the dialogue is safe. Claude then outputs a concise safety assessment, determining if the conversation poses any risk of violating usage policies.

This mechanism has been live since the launch of Claude Fable 5 in June. Every Claude model released since Fable 5 includes this feature, covering a significant portion of Claude's global traffic.

[Impact on Users]

Cost: The self-check primarily utilizes Anthropic's own server compute. By their estimates, overall inference compute increases by less than 2%. Anthropic states that this cost is not passed on to customers.

Privacy: Anthropic claims that self-assessment does not collect additional user data. It only reads the specific conversation context that Claude was already processing during its response and operates solely on traffic that Anthropic is permitted to retain under existing policies. Traffic covered by Zero Data Retention agreements (where conversations are not stored) currently does not undergo self-assessment. Neither the appended instructions nor the assessment results are displayed to users or carried over into subsequent turns. The assessment includes a rationale that may paraphrase your conversation content; Anthropic manages this data under the same rules as user conversations.

False Positives/Bans: Being flagged does not equal a violation. Anthropic describes the assessment merely as a "signal requiring further review"; a single flag is insufficient for action. For certain harm categories, flags are aggregated nightly by organization. Only when multiple high-priority flags accumulate within the same organization are they escalated to the enforcement team for manual review. Self-assessment functions exclusively as post-hoc flagging; it does not interrupt your conversation in real-time. Real-time interception is handled by other safety classifiers.

For compliant conversations, the assessment simply returns "Not Flagged." For flagged conversations, it specifies the type of harm involved, whether Claude refused the request, the urgency level for manual review (on a scale of 1 to 10), and a brief rationale.

[Why Is It So Efficient?]

The key lies in the KV Cache. When large models process a conversation, they compute and store intermediate representations for each token, allowing subsequent generation steps to reuse these values without reprocessing the entire context from scratch. Immediately after Claude finishes responding, the intermediate results for the full conversation remain in the cache. The self-check only requires reading one short instruction and generating approximately 10 additional tokens on average. Consequently, the increase in compute load for Anthropic is less than 2%.

An alternative approach is the "Prompt-based Classifier": This involves launching a separate model, feeding it the complete safety policy along with the conversation, and having it read through everything from scratch to reason about compliance. While this is the most accurate automated review method tested by Anthropic, it is orders of magnitude more expensive per conversation. Therefore, it is reserved for scenarios requiring detailed scrutiny. For reference, Anthropic's first-generation "Constitutional Classifiers" (independent models specifically trained to identify particular harms) introduced in early 2025 increased inference costs by 23.7%.

To use an analogy: Self-assessment is like asking someone who just finished reading a document to immediately answer, "Is there anything wrong with this?" A prompt-based classifier is like hiring a separate auditor who must read the document and the entire rulebook from scratch.

[Performance Metrics]

The benchmark for evaluation is that more expensive "auditor" (the prompt-based classifier). In samples of real-world traffic from Fable 5, when false positives were controlled at 1% (meaning 1 out of 100 normal conversations is incorrectly flagged), self-assessment successfully captured 78% of the suspicious conversations identified by the auditor. Among those, it captured 94% of high-priority cases (scored 7 or above).

Anthropic believes that training primarily teaches Claude how to report issues, while its judgment of what constitutes harm largely stems from its inherent knowledge. In experiments where fraud-related or cyberattack-related samples were entirely removed from the training data, detection rates for these categories in live traffic dropped only slightly—from 83.3% to 82.1% for fraud, and from 94.2% to 90.7% for cyberattacks. Larger models perform better: across four test models of varying sizes, at the same 1% false positive rate, detection rates rose from approximately 16% for the smallest model to about 96% for the largest.

Traditional classifiers can only identify pre-defined categories of harm. Approximately one-third of the flags generated by self-assessment do not map to Anthropic's existing harm classifications.

Clustering these unclassified flags revealed several issues not specifically covered by previous defenses: unauthorized distillation (extracting Claude's reasoning processes to train other models), prompt injection (hijacking AI instructions hidden in web pages or files), and instances where Claude, acting as an agent, performed operations not authorized by the user.

Real-world cases:

• In August, a single API customer triggered over 100,000 flags related to Child Sexual Exploitation Material (CSAM). After investigation, the child safety team reported the case to the National Center for Missing & Exploited Children (NCMEC) in the US.
• In September, its flags uncovered a group using Claude Code to build and run automated romance scams affecting dozens of victims. Accounts were banned following manual review. More than half of the organizations identified had not been flagged by any other detection methods in the preceding month.

[Limitations]

It is not strong enough for real-time interception. In red-team testing involving 24 cyberattack tasks (simulating attackers repeatedly probing defenses), attackers could retry with different phrasing after being blocked. Without protection, 19 attacks succeeded (79%). With self-assessment blocking, 13 still succeeded (54%). With prompt-based classifiers, only 6 succeeded (25%). Therefore, real-time interception in production is delegated to other classifiers, while self-assessment serves only for post-hoc flagging. In offline runs where attackers receive no immediate feedback, 21 out of 24 attack attempts were flagged at least once. Anthropic also acknowledges that attackers specifically targeting self-assessment could likely bypass it.

Since it learns from the expensive "auditor" and is evaluated by the same auditor, any blind spots in the auditor are inherited, making them undetectable in evaluations. Generalization experiments only tested fraud and cyberattacks—categories Claude already knows are harmful. Its effectiveness against novel harms that Claude has not yet recognized remains unknown.

Using it to supervise Claude's own behavior is also unreliable. A compromised model might conceal issues during self-checks. Furthermore, research indicates that models tend to assign lower risk scores to their own previous actions compared to identical actions taken by others.

Original Article (dated October 16): https://t.co/8jtt1nYv44

↳ Quote @AnthropicAI (Anthropic): We’ve trained Claude to flag potential misuse in conversations, a technique we call self-assessment. Right after it responds, Claude gives a preliminary safety assessment of the conversation.

Since we began using self-assessment in June, it has surfaced misuse that our other safeguards weren’t designed to catch.

Because Claude has already read the conversation, self-assessment is very efficient. And unlike classifiers, which target a discrete set of harms, it draws on Claude’s own judgment, so it gets more accurate as our models improve.

Read more on our new Safeguards blog: