Meta Unveils JiTTests Where LLMs Write Failing Tests to Catch Bugs

By ▲ Marktechpost AI (@Marktechpost)Original title: ▲ Meet Meta's JiTTests: LLMs write tests that are supposed to fail, proving a code change broke some...
72AI Score

Meta introduced JiTTests, which has LLMs generate tests expected to fail against a code diff, proving the change altered behavior before it lands; an LLM judge ensemble plus a rule-based filter called RubFake then decides which failures merit engineer attention. In H1 2026 it caught 74 privacy SEVs at the diff stage, cut privacy SEV occurrence by 20%, and drew a roughly 60% engineer-reported true-positive rate across 589 engineers. Intent-aware tests showed a 6.4% weak-catch rate per test versus 0.8% for Meta's earlier ACH hardening, though the initial paper deployment confirmed only 8 of 41 reported catches and all production figures are self-reported.

Full Report

▲ Marktechpost AI · 2026-11-16 10

▲ Meet Meta's JiTTests: LLMs write tests that are supposed to fail, proving a code change broke something before it lands.

The key idea is simple. Most AI bug tools review a diff like a faster human reviewer. JiTTests instead generates executable tests that expose a behavioral difference between the diff and its parent. An LLM judge ensemble plus a rule-based filter (RubFake) decides which failures are worth an engineer's time.

The numbers: 74 privacy SEVs caught at the diff stage in H1 2026, privacy SEV occurrence down 20%, and a 60% engineer-reported true-positive rate across 589 engineers. In the paper, intent-aware tests produced weak catches at 6.4% per test vs 0.8% for Meta's earlier ACH hardening approach. It is a trade-off though: the initial paper deployment confirmed only 8 of 41 reported catches (19.5%), and all production figures are self-reported.

Full analysis:

推文配图