Rohan Paul (@rohanpaul_ai) · 10/10/2026, 16:01:17
Tokyo Metropolitan University Study Reveals AI Text Detectors Fail Across LLM Generations
A new paper from Tokyo Metropolitan University reveals that AI text detectors like Pangram suffer catastrophic performance drops when facing newer LLM generations. While catching 93.5% of GPT-5 rewrites, the tool missed 79.8% of content rewritten by Meta's Muse-Glimmer, demonstrating extreme sensitivity to model versions. This finding warns publishers and researchers that current detection methods are obsolete against rapid AI iteration, necessitating continuous recalibration or a shift away from reliance on static detectors.
SOURCE COVERAGEFull Original Report
Rohan Paul (@rohanpaul_ai) · 2026-10-10 08
This paper finds that AI-text detectors trained on a vendor's older models caught over 99% of rewrites before a generation change and only 3.8% after it.
Detectors that screen scientific papers for AI writing can stop working when a new LLM generation arrives, so anyone relying on them should re-test them with every model release.
↳ 引用 @rohanpaul_ai (Rohan Paul): Pangram, the AI-text detector, missed 79.8% of scientific abstracts rewritten by Meta's Muse-Glimmer, while flagging just 1 of 5,000 human abstracts.
In a new paper from Tokyo Metropolitan University, reseaerchers find the share of AI-rewritten abstracts that Pangram misses depends strongly on the LLM version
Shows that it caught 93.5% of GPT-5 rewrites but missed 79.8% from another new model. Its miss rate depended mostly on which model did the rewriting.
