Claude Writes Its Own Inference Engine, Cutting First-Token Latency to 0.48 Seconds
Researchers from CMU, NVIDIA and other institutions posed a task in the Singularity-Bench preprint: give Claude Opus 5.5 base server code, four GB300 GPUs and 12 hours, with no internet access, to build an inference engine using Claude Code. Replaying 16 concurrent real coding sessions, the agent-written engine reached roughly 0.48s P90 time-to-first-token versus about 12s for the mature engine SGLang v0.5.20, moving the question of whether agents can build agents from speculation to reproducible engineering evidence.
Related Entities
Full Report

New AI Era Report

You ask AI to help fix a login error.
You modify the code according to its suggestions and rerun the login test. The terminal spits out another line of red text, reporting the same error.
You paste the new log into the chat box, press Enter again, and wait for AI to tell you where to look next.
Fixing a single bug often requires several rounds of this back-and-forth.
After you submit a question each time, how long do you have to wait before you see the first character of AI's reply?
Recently, someone ran a test using real coding sessions, and Claude directly cut this waiting time from about 12 seconds to half a second.
On November 12, researchers from Carnegie Mellon University, NVIDIA, and other institutions submitted the preprint "Singularity-Bench: Can Agents Build Agents?"

In one of the tasks, researchers gave Claude Opus 5.5 a basic server codebase, 4 GB300 GPUs, and 12 hours.
During the task, the development environment restricted internet access for looking up information. Claude had to use Claude Code to write an inference engine responsible for receiving user requests, scheduling GPUs to run models, and sending back the generated responses.
After the engine was written, the researchers didn't just look at the code—they directly tested its performance using real coding sessions.
They had the two engines each run the same model on 4 GB300 GPUs, replaying the same batch of 16 concurrent real coding sessions.
The metric measured is called P90 time-to-first-token:
Sort all requests by the waiting time from when the request is sent to when the model starts replying, and the time at the 90th percentile is P90. In other words, 90% of requests can start replying within this time.
The result: the mature engine SGLang (v0.5.20) took about 12 seconds, while the engine written by Claude took only 0.48 seconds.
Under the same hardware and model conditions, Claude's self-developed inference engine achieved a P90 time-to-first-token of 0.478 seconds, compared to 12.029 seconds for SGLang.
Waiting 12 seconds, you might have already switched back to the editor and scrolled through a few lines of code following the error. Half a second? You just hit Enter, and the first character of AI's reply appears in the chat box.
Once generation starts, the two engines' generation speeds are almost identical: by median, Claude's engine generates 124.2 tokens per second, while SGLang generates 125.0.
So how exactly did Claude save those 11.5 seconds before AI starts speaking?
Changing Just One Sentence
How Much the Server Still Has to Read
Suppose you're using AI to modify a webpage.
You've already pasted the source code and error messages, and tried several versions of the solution. This round, you only add one sentence: "Move the login button a bit to the right."
In the coding sessions replayed in the paper, the chat history from previous rounds is also sent to the server. With 16 sessions running simultaneously, the median single input is 89,000 tokens.
The model must first process the input before it can start replying: this step is called "prefill."
If the beginning of a new round of requests is the same as the previous round, the engine can retrieve previously computed results and continue processing the newly added content—this is "prefix caching."
The "previously computed results" here are the computation records left when the model read that preceding text. The model used in this test leaves two copies.
One retains the information from the preceding text, making it easy for the model to look back; the other is like a balance in an account, updated each time the model reads a bit more.
Claude's engine stores both records corresponding to the same preceding text together. Next time it encounters the same beginning, the engine can continue reading from the end of that preceding text and only compute the newly added content afterward.
SGLang also uses prefix caching, and both engines have cache hit rates around 96%.
When 16 requests arrive simultaneously, the content that doesn't hit the cache still has to be processed by the GPU.
How much computation remains for each request, and which one the GPU processes first, also affects how quickly you see the first character of AI's reply.

Performance comparison between Claude's self-developed inference engine and SGLang. The two have similar cache hit rates and generation speeds, but Claude's engine has significantly lower P90 time-to-first-token.
The Less Work Remains
The Earlier It Gets Processed
Imagine a supermarket checkout scenario:
The customer ahead of you is pushing a fully loaded cart of New Year goods, with tens of thousands of brand-new tokens waiting to be scanned from scratch.
You're standing behind with just a bottle of mineral water in hand—that is, you've asked a follow-up question in an old session, and 96% of the preceding text has ready-made cache.
Stubbornly enforcing "first come, first served" in concurrent inference is like making the person buying one bottle of water stand in line for the full cart of New Year goods.
So Claude established a queuing rule for the inference engine:
Whoever has less remaining computation gets processed first.
When encountering a massive new order of tens of thousands of tokens, the engine also slices it into small chunks and processes them in batches, never letting one big task block the entire lane.
This way, the follow-up question you added in an old conversation has a chance to bypass the wait of long requests and get prioritized.
In the generation phase, the engine also layers on "speculative decoding":
The model's built-in lightweight prediction layer guesses a token first, and the main model then verifies it. If the guess is correct, the predicted result is adopted directly; if wrong, the prediction is discarded and generation continues based on the main model's result.
At the same time, the engine also bundles and merges originally scattered GPU operations, reducing unnecessary scheduling and data movement.
Even though SGLang also enabled prefix caching and speculative decoding, the engine written by Claude still held an advantage in input processing and request scheduling: on average per GPU per second, it could process 19% more uncached tokens.
The horizontal axis is P90 first-token latency, and the vertical axis is the number of uncached tokens processed per GPU-second. While reducing first-token wait time, Claude's self-developed engine also improved the processing efficiency of uncached input.
Within the same replay time limit, SGLang completed 705 requests, while the engine written by Claude completed 983, processing nearly 40% more.
Then, using the preset input and output unit prices, the completed requests were converted into "simulated revenue." Claude's engine raised output per GPU-hour from SGLang's $8.18 to $11.52, a 40.8% increase.
Developers can see AI responses earlier, and the same GPUs can also finish more requests in the same period of time.
Of course, real work scenarios have always cared only about delivered results: running extremely fast while talking nonsense is called a high-speed crash.
When submitting their answers, the researchers set two hard thresholds: first, under concurrent load, the specified response length must be generated in full; second, accuracy must pass acceptance via hidden tests.
Among the 11 AI self-developed engines in the test, in the end only 5 passed both thresholds at the same time; the other 6 either swallowed words and broke sentences midway, or their accuracy collapsed, and they were immediately given zero according to the rules.
11 AI teams build infrastructure
Claude wins five and draws one
Hand-building an inference engine is only one of the six Singularity-Bench challenges.
The research team split the core hard work of building AI into six independent closed-book exams: three on training (pretraining from scratch, mid-training data selection, and post-training reinforcement learning for coding), and three on infrastructure (inference engine, agent runtime framework, and GPU cluster scheduling).

The research team divided AI R&D into three stages—training, deployment, and application—and set six tasks to test whether agents can independently optimize the AI R&D process.
Eleven AI teams, formed by combining 7 mainstream models with two runtime frameworks, Claude Code and Codex, received starting code with the network fully disconnected and submitted their answers against a countdown, taking on mature solutions polished by humans over many years on completely identical hardware.
In addition to the major inference win, Claude Opus 5.5 paired with Claude Code also beat the human baseline on four other challenges.
When it received the pretraining exam paper (4 GB300s, 12 hours), Claude did not rush to start training. Instead, it first spent 44 minutes testing GPU performance, then determined the model size based on the test results. It did not start training until the 44th minute.
In the end, Claude used only about half the training compute to achieve a perplexity of 12.77 on the validation set, better than the human baseline's 14.22 (OLMo 2 1B with Megatron-LM).
For mid-training (4 GB300s, 12 hours), the challenge shifted to selecting the next batch of "textbooks" for an existing model, then testing the training effect with the same set of reinforcement learning tasks, examining the model's performance in math, code, and science.
Claude mainly selected data with complete solution processes and verified answers. Ultimately, its approach raised the downstream average accuracy from the human PRISM recipe's 17.5% to 31.9%, nearly doubling it.
The agent runtime framework challenge (1 GB300, 6 hours) was the most dramatic: this task tested when the model calls tools and how it checks logs after errors, and the human control group was none other than the original Claude Code.
As a result, Claude performed surgery on itself, raising the downstream average score from 0.438 to 0.468. In the same batch of tests, the new framework fixed one more bug than the original Claude Code.
Cluster scheduling (16 GB300s, 4 hours) required using 16 GPUs to serve requests for 28 models simultaneously.
Claude's actual tests found that some engines competed for resources and slowed each other down when sharing GPUs, so it adjusted the scheduling strategy to let them take turns using the GPUs.
In the end, it achieved a combined score of 0.510 on the two metrics of latency compliance rate and request completion rate, surpassing NVIDIA Dynamo with KAI Scheduler's 0.460.
It can even modify its own framework and outperform NVIDIA's solution in the cluster scheduling test. This time, Claude showed off its engineering capabilities.
The only challenge that did not produce a winner was post-training (4 GB300s, 12 hours).
Claude improved the given model's coding ability through reinforcement learning, ultimately reaching a solve rate of 53.3%, while the human control solution Miles reached 53.9%. Comparing problem by problem, each side solved 18 problems that only it solved, and the paper therefore judged it a tie.
Averaging the six results with equal weight and using the human baseline of 1 point as the reference, Opus 5.5 with Claude Code scored 1.252 with "five wins and one draw," becoming the only combination to exceed the human baseline. Second place, GPT-6.1 Sol with Codex, scored 0.969, while the remaining 10 teams averaged only 0.58.
The overall leaderboard also revealed another point: the model brain determines the upper limit of capability, and the runtime framework determines how much of that ability can be brought out.
For the same open-source model, pairing it with Codex scored 0.228, while switching to Claude Code raised it to 0.657, nearly 3 times the original. Another model was exactly the opposite: pairing it with Codex scored 0.698, while switching to Claude Code instead dropped it to 0.520.
This shows that a strong model is not enough; if the framework is not chosen correctly, it may still underperform.
Took first place overall—how much did Claude actually spend on API calls?
For API calls during the development phase alone, Opus 5.5 spent $425.41, higher than the previous generation Opus 5's $328.19, and about 5 times that of the second-place GPT-6.1 Sol ($84.52). GPU and downstream evaluation costs are counted separately.
Next question
Letting AI improve the entire R&D pipeline itself
Training models, selecting data, writing engines, managing GPUs—AI has already surpassed human baselines on multiple tasks.
Does this mean "AI iterating the next generation of AI by itself" has already been achieved? Not quite yet.
These six questions are essentially still scoped by humans, with AI answering each separately.
Post-training optimizes a given model, while the inference engine serves another ready-made model. What code each question starts from, which metric to optimize, and how much time and compute to allocate are all set in advance by humans.
The paper's authors also acknowledge that truly testing recursive self-improvement (RSI) requires more time and compute.
Building a single component and improving the entire production line are two completely different things.
The real test is whether AI can identify the biggest bottleneck in the R&D process on its own—for example, is training too slow, is the data selection wrong, or is reinforcement learning always waiting on inference results?
After finding the bottleneck, it must also decide where to start, then put the improvement back into the entire R&D pipeline for validation.
Ultimately, it's not just about how much compute and time were saved, but also whether the next-generation model trained is actually stronger.
Going further, the next-generation model must also be able to take over the previous generation's work, continue to find problems, improve processes, and deliver new results that stand up to re-testing.
When AI can find bottlenecks on its own, change processes, train a stronger next-generation model, and keep this process going continuously, recursive self-improvement will have truly begun.
Today, this research tests whether AI can save 11 seconds on a single response; what will truly be worth watching in the future is whether AI can independently complete a cross-generational upgrade, and how long that takes.
References:
https://x.com/rohanpaul_ai/status/2121330917801431326
https://arxiv.org/abs/2611.16591
https://infini-ai-lab.github.io/singularity-bench/
Chasing ASI in Seconds
⭐Like, repost, and watch—one-click triple combo⭐
Turn on the star, lock in New Intelligence Era's instant push!


