CodeScene Demos AI Agents Refactoring 300k Lines of C in Three Weeks Using Code Health MCP and Replay Testing

CodeScene demonstrated coding agents refactoring 300,000 lines of C code in three weeks, raising the Code Health score from 5.6 to 10.0 with a total token cost of approximately $4,000.

Quality signals (via the CodeHealth MCP Server) and a replay trajectory testing framework jointly ensured refactoring effectiveness and behavioral correctness; the agents autonomously evolved 22 refactoring patterns, including specialized modes for decompiled code; failed attempts were systematically recorded, forming an auditable evolution log.

This is suitable reading for senior software engineers, AI programming tool developers, and leaders responsible for large-scale legacy system refactoring.

CodeScene published a case study: coding agents refactored a codebase containing 300,000 lines of C code in three weeks, with a token cost of approximately $4,000. This work resulted in 2,903 commits across 726 files, modified 252,055 lines of code, and improved the codebase's Code Health score from 5.6 to 10.0. The codebase originated from an open-source decompilation project of Street Fighter III: Third Strike.

Adam Tornhill, founder of CodeScene and author of Your Code as a Crime Scene, wrote that after 30 years involved in large-scale system development, this was the first time he had witnessed what he calls "massive superhuman AI performance."

Two mechanisms supported this effort. The first is quality signals: the CodeHealth MCP Server provided the agents with a deterministic score to optimize against, used to determine whether a transformation produced positive effects. The second is correctness assurance: a replay trajectory testing framework compared rollback state hashes frame-by-frame, thereby checking system behavior after every change.

A more novel outcome was what the agents built during the process. Rather than applying a fixed catalog of refactorings, they gradually accumulated a refactoring playbook, ultimately forming 22 patterns and 82 auxiliary notes. These included common transformations like Extract Function and Guard Clauses, as well as specific solutions tailored to this codebase. Shared Index Range handled duplicate loops differing only in start/end ranges; Action Parameter handled duplicate control structures where the primary difference was the function called; Uniform Step Table converted heterogeneous calls into table-driven dispatch. Failed attempts were also recorded, including transformations that lowered the Code Health score.

Model selection was also critical. The team ultimately decided to use Claude Opus for most of the work and reported that Claude Code paired with Opus significantly outperformed Codex paired with Sol in identifying and documenting emerging patterns. When using smaller models for tasks, file scores often stagnated, appearing to hit a local optimum that could not be further breached.

Practitioner reactions on LinkedIn showed clear divergence; the controversy was not about whether this happened, but about what the result actually proves.

Paolo Perrone explained why this achievement should be taken seriously:

Most refactoring results I've read are based on test suites passing entirely, but that only indicates tests didn't fail. Trajectory replay for a fighting game sets a much higher standard.

Mats Iremark, CTO of Omda Response, stated he had done similar things himself and described the combination of CodeScene MCP and current agents as almost akin to cheating.

Skeptics primarily focused on scope. Tech lead and consultant Konrad Otrębski asked whether these changes had been merged, if they were completed via one massive merge or multiple merges, and whether this was merely an experiment on open-source code rather than work on production code generating revenue. Daniel Webb, CTO of NeoSee and one of two engineers involved in this work, responded that the changes were merged into a forked version of the main branch via 54 pull requests.

Otrębski's subsequent response raised a higher bar:

I believe the true test of AI capability is offering this refactoring service to a prominent open-source project like Grafana. Of course, the definition of done is merging it into the master branch.

Software architect and researcher Tracy Bannon objected to the description of the results rather than the methodology employed. She pointed out that describing the outcome as perfect is quite a bold claim.

Denis Baltor questioned the patterns discovered by the agents themselves. He argued that DRY focuses on duplication of knowledge and intent, not identical lines of code; if patterns treat loops differing only in range or control structures differing only in called functions as duplicates, they may conflate these two concepts.

Asko Nõmm raised methodological issues. He argued that since Claude Code and Codex are tuned specifically for their respective models, it is difficult to judge to what extent the results measure the model versus the testing framework, and the associated costs.

Some unanswered questions came from the authors themselves. Marc Bouvier asked whether non-functional behavior had improved, given that frame rate, memory usage, and input latency are critical for games; Webb stated that they were bringing in a performance specialist. When asked if the testing framework captured subtle frame timing regressions, he replied that there might not be a definitive answer, as the framework runs as a pre-commit hook, meaning some failures are fixed before they are observed. He was equally direct regarding diff size: if you are unfamiliar with the code and no longer review it line by line, how large can a single diff be? He offered no conclusion, only posing the question.

If these reactions are representative, the divergence lies not in the numbers themselves, but in the conditions under which those numbers were generated. The replay trajectory testing framework works because decompiled games provide deterministic frame-by-frame replays. Most legacy systems lack an equivalent decision mechanism, which is precisely why refactoring them carries risk. Tornhill wrote that automated tests and equivalence checks are absolutely necessary safeguards, placing the burden squarely on what unhealthy codebases often lack most.

The team’s own selection process illustrates this point. Webb noted that they considered using a Gov.UK maritime licensing codebase he had worked on, but it was too healthy to serve as a subject for subsequent research; the team chose this game partly because they played it themselves, making them users of it as well.

This study represents the next step. The improvements resulted in two functionally equivalent versions of the same system, one with a Code Health score of 5.6 and the other at 10.0. The team will collaborate with Lund University on a study where students use state-of-the-art models to implement features in both versions, comparing cost and quality.

Two other striking figures belong to future research rather than this case study. CodeScene estimates that the improvements could reduce AI-introduced defects by approximately 70% and token waste by about 45%, but both figures are extrapolated from their earlier studies and were not actually measured in this instance. The $4,000 cost and three-week duration are the actual measured results.

Original article: https://www.infoq.com/news/2026/09/agentic-refactoring-case-study/