StartLux Open-Sources Prism Inference Engine with Up to 5.3x Speedup

DeepSeek and StartLux have open-sourced DeepInfer and StartLux-Prism, respectively—two inference frameworks tailored for their proprietary models. This marks a significant evolution in the domestic large model technology stack, shifting focus from model training to specialized inference systems. On a single NVIDIA H200 GPU, StartLux-Prism achieves up to 5.3x faster inference than vLLM, while supporting hybrid TP/DP parallelism and deterministic outputs.

Key highlights include:

• Support for combined Tensor Parallelism (TP) and Data Parallelism (DP) deployments, balancing low latency with high throughput;
• Single-GPU inference speeds ranging from 1.2x to 5.3x that of vLLM, with approximately 2.8x acceleration in short-query scenarios;
• Strong determinism in output results, with accuracy nearly matching BF16 native precision (92.94% vs. 92.97%).

This content is suitable for AI system engineers, inference engine developers, and large model deployment architects.

Recently, two Chinese AI companies successively open-sourced inference systems optimized for their own models. On October 20, DeepSeek released DeepInfer, an inference framework designed for its proprietary models. Shortly after, StartLux announced the open-source release of StartLux-Prism, a dedicated inference engine designed for StartLux-Decision. Although their technical approaches differ, both companies are extending their optimization focus from the models themselves to the underlying inference systems.

According to test data published by StartLux, using the StartLux-Decision v1.1 27B model on a single NVIDIA H200 GPU, Prism achieved inference speeds between 1.2x and 5.3x those of the original vLLM solution across various test scenarios. Specifically, when answering three questions based on the same document, processing time dropped from 92.9 milliseconds to 17.4 milliseconds—a 5.3x speedup. In short-query scenarios, Prism delivered approximately 2.8x acceleration.

Beyond performance optimization, Prism supports combined Tensor Parallelism (TP) and Data Parallelism (DP) deployments. TP primarily reduces processing time for individual requests, while DP increases overall throughput. These strategies can be combined based on specific deployment requirements.

Prism also places significant emphasis on optimizing the determinism of inference results. StartLux states that under supported runtime configurations, identical requests yield bit-for-bit consistent output probabilities, regardless of changes in concurrency, batching methods, or the number of GPUs. In tests covering seven evaluation datasets with a total of 12,351 decisions, Prism achieved an average accuracy of 92.94%, which is virtually on par with the 92.97% accuracy observed at BF16 native precision.

Special Requirements of Decision Models: Only Probabilities for a Few Options Are Needed

General-purpose inference engines like vLLM and SGLang are primarily optimized for generative inference needs of Large Language Models (LLMs). However, the core task of StartLux-Decision differs: it needs to read input materials and calculate probabilities for a limited set of candidate options, rather than necessarily generating long text token-by-token. This distinction creates opportunities for specialized inference optimizations.

Consider the scenario of answering multiple questions based on the same document. If each question is processed as an independent request, the system may redundantly perform steps such as tokenization, scheduling, and common context computation. For decision tasks requiring only a few candidate probabilities, computing over the full vocabulary can introduce unnecessary overhead.

In usage examples provided by StartLux, the model must answer three consecutive questions based on the same document. For each question, the decision model requires only a single forward pass to obtain the probabilities corresponding to candidate options (e.g., A, B, C) at the end of the prompt, without needing to generate a complete natural language response.

However, in previous inference solutions, these questions were still handled as independent requests. Even though multiple questions shared the same background material, this approach could lead to redundant computation and scheduling overheads. StartLux-Prism redesigned the computational flow to address these characteristics, focusing on four main areas.

Innovation 1: Computing Output Scores Only for Candidate Answers

Decision models typically need to make judgments among a limited set of options. However, the output layer of general language models calculates scores for the entire vocabulary.

Take the StartLux-Decision 27B model as an example: its vocabulary size is approximately 248,000 tokens. In standard inference flows, the model uses the LM Head to compute logits for candidate tokens in the vocabulary, followed by processing to derive output probabilities. Yet, for a decision question with only a few candidates like A, B, and C, the actual comparison may involve just a handful of tokens.

Prism redesigns the final output computation stage around this characteristic. The model still fully processes the input material, but when calculating final option probabilities, it computes output scores only for valid candidates, avoiding the need to calculate logits for every token in the full vocabulary.

If the decision task requires a normalized probability distribution among candidate options, this method yields mathematically identical results to scoring the full vocabulary and then renormalizing for the candidate set, while significantly reducing the computational load on the output layer.

According to StartLux, Prism not only narrows the scope of LM Head computations but also applies targeted optimizations to the final layer, executing full calculations only at positions where output results are required.

In a short-query test involving approximately 256 tokens, Prism reduced processing time from 53.1 milliseconds in the original inference solution to 19 milliseconds, achieving roughly 2.8x the speed of the previous method.

Innovation 2: Reducing Redundant Computation When Answering Multiple Questions Based on the Same Material

Another optimization in Prism targets scenarios where multiple questions share the same input material.

For instance, a customer support ticket might require simultaneous determination of the responsible team, response deadline, and issue severity. If these three questions are processed as separate requests, the system may need to read the same material multiple times and repeat parts of the forward computation. Prism organizes batches of tasks sharing common context into a structure resembling a "forest": common instructions form the root, shared materials constitute the trunk, and different questions serve as branches.

Through this organizational method, the model can reuse computation results for the common context before separately handling different questions, thereby reducing redundant calculations. Combined with Tree Attention, this mechanism enables multiple questions to complete inference based on a shared prefix.

In StartLux's published single-H200 benchmark, answering three questions based on the same document required 92.9 milliseconds using the original vLLM-based FP8 inference solution, whereas Prism reduced the processing time to 17.4 milliseconds—a 5.3x speedup. This represents one of the most significant performance improvements in the released tests.

However, this acceleration depends on the proportion of context shared across multiple questions. When there is little reusable content between questions, or when input materials and task structures change, the actual benefits may vary.

For business scenarios requiring multiple judgments on the same document—such as ticket classification, risk assessment, and document review—reusing common context reduces redundant computation and offers room for lowering the inference cost per task.

Innovation 3: Deterministic Computation to Reduce Numerical Variance Under Different Concurrency Conditions

For decision models relying on output probabilities, the stability of inference results is equally important. Under dynamic batching and varying concurrency configurations, the underlying order of computation may change. Since floating-point arithmetic does not strictly satisfy associativity, even with identical model weights and inputs, final probabilities can exhibit minor discrepancies. If a judgment result falls near a decision threshold, such differences could impact the final choice. To address this, Prism introduces a deterministic computation design.

According to StartLux, Prism executes relevant Projection calculations on INT8 Tensor Cores and uses INT32 accumulation to minimize numerical variations caused by changes in floating-point computation order.

StartLux states that under its supported runtime configurations, the output probabilities remain bit-for-bit consistent for the same request, regardless of which requests are processed together, how system concurrency varies, whether cache hits occur, or whether deployment uses single or multiple GPUs.

This design has practical significance for business scenarios requiring reproducible decision outcomes. For example, in financial risk control, automated auditing, or other systems that execute subsequent operations based on probability thresholds, the reproducibility of inference results aids testing, auditing, and troubleshooting.

Meanwhile, Prism's use of INT8 computation requires considering the impact of quantization on the model's judgment capability. In tests covering 7 evaluation sets with a total of 12,351 decisions, Prism achieved an average accuracy of 92.94%, compared to 92.97% for the same model running at native BF16 precision—a difference of only 0.03 percentage points.

Based on these test results, Prism largely maintains the original model's average judgment accuracy while reducing computational overhead. However, this outcome still depends on specific evaluation sets and model configurations.

Innovation 4: Sharing GPU Resources Between Thinking Tasks and Direct Answers to Prevent Long Reasoning from Blocking Short Requests

StartLux-Decision v1.1 provides an AutoThink mode: for questions the model can judge directly, it outputs results quickly; for uncertain questions, it generates a thinking process, up to 4,096 tokens. This requires the inference system to handle two types of tasks with distinct operational characteristics simultaneously. Direct-output requests typically require shorter computation times, while thinking-required requests may continuously generate large numbers of tokens. If both types of requests use simple serial scheduling, long-running thinking tasks could increase wait times for short requests.

Prism does not deploy a separate generation service for thinking tasks. Instead, it allows thinking requests and direct-output requests to share the same Scheduler and GPU memory.

Specifically, prompts for thinking tasks can complete Prefill alongside the next batch of direct-output requests. Subsequently, Decode steps during the thinking process are executed via CUDA Graphs, supporting up to 64 concurrent thinking tasks. Thinking tasks and direct-output tasks alternate GPU usage. According to StartLux's design, newly arrived short requests typically need to wait only one or two Decode Steps, rather than queuing behind a complete long thinking process.

To ensure deterministic sampling results, Prism also employs Counter-based RNG, making random numbers determined by the request itself, thereby reducing the impact of changes in batch composition. StartLux reports that in tests with concurrency levels of 16 and 64, both thinking content and final probabilities remained bit-for-bit consistent with single-request execution.

Regarding performance, on a single H200, Prism INT8 achieves a generation speed of approximately 95 tokens per second for a single thinking task. When 64 thinking tasks run concurrently, the total generation throughput reaches 3,832 tokens per second.

Beyond mixed scheduling, Prism supports Prefix Cache to reduce redundant computation when users ask follow-up questions about the same material. For instance, if the model has already read a lengthy document and the user subsequently asks new questions, the system can reuse previous common context calculation results, processing only the new additions.

In StartLux's published long-document follow-up test, this mechanism reduced the processing time for a single follow-up question from 48 milliseconds to 5.4 milliseconds.

Single H200 Benchmark: Up to 5.3x Speedup, 17% Throughput Increase at High Concurrency

To evaluate Prism's performance, StartLux used a single NVIDIA H200 GPU, the same StartLux-Decision v1.1 27B AutoThink model, identical requests, and test programs to compare Prism INT8 against the original vLLM-based FP8 inference solution. The tests covered scenarios including short questions, multi-question requests sharing materials, questions requiring thinking, long contexts, and high-concurrency throughput.

From the published data, Prism's benefits vary significantly across different scenarios. Short questions and multi-question requests sharing context benefit the most. Processing time for short questions of approximately 256 tokens decreased from 53.1 milliseconds to 19 milliseconds; processing time for answering three questions based on the same document dropped from 92.9 milliseconds to 17.4 milliseconds. For longer input materials, since the model still needs to perform substantial input computation, the proportion of performance gains attributable to specialized output optimization decreases, resulting in a relatively narrower overall speedup.

In high-concurrency scenarios, Prism also improved system throughput. According to JevBench concurrency-64 test data published by StartLux, the requests-per-second (RPS) rate increased from 26.3 to 30.7, representing an improvement of approximately 17%. This indicates that the up to 5.3x speedup primarily stems from latency optimizations in specific request scenarios and cannot be directly equated with throughput gains across all workloads.

Beyond inference pipeline optimizations, the StartLux team developed dedicated INT8 GEMM kernels for common computational shapes. In specified matrix computation tests on the H200, these kernels achieved performance ranging from 1,240 to 1,420 trillion INT8 operations per second, which is 1.3 to 2.0 times faster than NVIDIA’s cuBLASLt implementation. These results highlight the potential of kernel optimization tailored to specific computational shapes but do not imply that custom kernels will deliver similar performance advantages across all matrix sizes and hardware configurations.

From Models to Inference Engines: Specialized Optimization Is Emerging as a New Competitive Frontier

Recent developments by DeepSeek and StartLux reflect a growing emphasis among model providers on co-optimizing inference systems and model architectures.

DeepSeek’s DeepInfer targets optimizations specifically for its own models, while StartLux-Prism focuses on the operational characteristics of decision-making models by reducing unnecessary output layer computations, reusing shared context, and optimizing scheduling strategies for different types of requests. Although their implementations differ, both approaches aim to leverage known features of model structures and workloads to further enhance inference efficiency.

Compared to general-purpose inference frameworks that must support a wide variety of model architectures and task types, specialized inference engines can perform deeper optimizations tailored to specific models and business scenarios. However, this also means their applicability is relatively limited, and performance gains often depend on factors such as model structure, input length, concurrency scale, and specific hardware conditions.

Based on the data released by StartLux, Prism achieved notable latency improvements in scenarios involving short queries and multi-query tasks with shared context, whereas throughput gains in high-concurrency environments were more modest. This suggests that the benefits of specialized inference optimization are not uniformly distributed and must still be evaluated against actual workloads.

As more models are designed for specific tasks, inference systems may become increasingly fragmented. For model providers, open-sourcing model weights is only part of the delivery equation. Providing inference solutions that align with model characteristics, ensure stability, and maximize deployment efficiency is becoming another critical engineering challenge requiring sustained investment.