Today for AI
HOT RADAR
llmHEAT 8.8°

AI CURATED HOT STORY · 10/23/2026

OASIS Introduces Attention Abstention to Boost Long-Context and Low-Bit Robustness

1 reports archived1 independent sourcesupdated 10/23/2026, 09:00:00
Synthesis & Latest Updates
1 Sources Cross-Validated

Researchers from Northwestern University and others propose OASIS, addressing activation anomalies in AttnResidual's depth routing by introducing an explicit 'abstention' (null) option. This mechanism uses token-level null probabilities to dynamically adjust depth-level branch selection, significantly reducing activation kurtosis and optimizing quantization performance. Experiments show a >40% relative improvement in GSM8K accuracy under W4A4 quantization, offering a more stable cross-layer information integration scheme for enhancing Transformer expressiveness.

LATEST/Researchers from Northwestern University and others propose OASIS, addressing activation anomalies in AttnResidual's depth routing by introducing an explicit 'abstention' (null) option. This mechanism uses token-level null probabilities to dynamically adjust depth-level branch selection, significantly reducing activation kurtosis and optimizing quantization performance. Experiments show a >40% relative improvement in GSM8K accuracy under W4A4 quantization, offering a more stable cross-layer information integration scheme for enhancing Transformer expressiveness.

HEAT TRENDHourly heat curve

14 fully observed hours
Now
8.8
Peak
12.009:00
24h change
-27%

The line compares only sources observed throughout; gap hours are interpolated to keep the trend continuous.

TIMELINECoverage timeline

Total 1 reports · Latest first
  1. 新智元 (微信公众号)T2·78 pts
    • OASIS adds explicit null routing to AttnResidual, resolving activation anomaly spikes caused by excessive attention on initial tokens.
    • Utilizes a Token-to-Depth coupling mechanism to coordinate cross-layer selection using token-level abstention signals, enhancing robustness.
    • Validated on LLaMA-3.2 and Qwen3 models, reducing average activation kurtosis by ~96% and improving GSM8K Pass@1 by 42.11% relatively under W4A4 quantization.
OASIS Introduces Attention Abstention to Boost Long-Context and Low-Bit Robustness | Today for AI