Intrusion Heads

A new Nvidia-led paper finds that irrelevant text in RAG, tool-use and long-context prompts hurts models mainly by weakening their use of the question rather than directly suppressing the right answer. The authors identify a small set of 'intrusion heads' inside open models that read unrelated text and dilute the answer; masking about 10 such heads on 7B-scale models recovered 57% of the accuracy lost to distractors on average, versus a 97.6% oracle ceiling.