Nanjing University Team Unveils TestPrism Benchmark Exposing Overstated Test Generation Quality

A team led by Jiaheng Liu at Nanjing University introduces TestPrism, a test-generation benchmark arguing that the prevailing single-reference evaluation systematically overstates Coding Agent test quality. TestPrism spans 300 test tasks from 17 sources and 3,000 candidate implementations, each task paired with a validated panel of correct and buggy implementations, and scores with a Joint Success Function (JSF) that requires tests to pass every correct implementation while rejecting every faulty one. Under this stricter criterion, Coding Agents fail collectively far more often than prior evaluations suggested, with reported pass rates inflated roughly twofold.