Meta 推出 JiTTests 用大模型生成失败测试拦截缺陷
Meta 发布 JiTTests,让大模型针对代码 diff 生成本应失败的测试,以证明改动破坏了原有行为,再由 LLM 评审团与规则过滤器 RubFake 筛选值得工程师关注的失败。2026 年上半年在 diff 阶段捕获 74 个隐私类 SEV,隐私 SEV 发生率下降 20%,589 名工程师反馈真阳性率约 60%。论文阶段意图感知测试单测弱捕获率为 6.4%,高于此前 ACH 加固方案的 0.8%,但初期部署 41 个上报仅确认 8 个,且生产数据均为自报。
报道正文
▲ Marktechpost AI · 2026-11-16 10
▲ Meet Meta's JiTTests: LLMs write tests that are supposed to fail, proving a code change broke something before it lands.
The key idea is simple. Most AI bug tools review a diff like a faster human reviewer. JiTTests instead generates executable tests that expose a behavioral difference between the diff and its parent. An LLM judge ensemble plus a rule-based filter (RubFake) decides which failures are worth an engineer's time.
The numbers: 74 privacy SEVs caught at the diff stage in H1 2026, privacy SEV occurrence down 20%, and a 60% engineer-reported true-positive rate across 589 engineers. In the paper, intent-aware tests produced weak catches at 6.4% per test vs 0.8% for Meta's earlier ACH hardening approach. It is a trade-off though: the initial paper deployment confirmed only 8 of 41 reported catches (19.5%), and all production figures are self-reported.
Full analysis:
