StartLux Launches 27B Decision Model, Tops Leaderboard with Dynamic Reasoning

StartLux Launches 27B Decision Model, Tops Hugging Face Decision Index 0.3.2; Leads in Both Public and Private Benchmarks; AutoThink Mechanism Enables Dynamic Reasoning Depth Control

Decision models are specifically designed for Agent action selection, outputting structured probabilities rather than free-form text. The dual-path Instant/AutoThink architecture balances latency and accuracy, achieving a 27% relative improvement over Jev on the Decision Index 0.3.

This article is suitable for AI system architects, Agent developers, and decision model training engineers.

The recent rise of Jev has brought decision models—specialized systems that provide judgment results for AI applications—into the developer spotlight. Unlike large language models (LLMs) that typically generate textual responses to questions, decision models take the current state and candidate options as input, directly outputting probabilities for each option to guide Agents or other programs in determining their next action.

Now, this niche has welcomed a Chinese challenger.

In the Hugging Face Decision Index 0.3.2 leaderboard updated on October 16, StartLux’s 27B Instant model from China ranked first among 174 entries with a score of 69.12. It secured the top position across all three evaluation components: public benchmarks, private tests of similar capabilities, and private tests in new domains.

Furthermore, in the team’s previously published Decision Index 0.3 public evaluation, StartLux-Decision-27B-AutoThink achieved a score of 73.37, representing an approximate 27% relative lead over Jev’s hosted version, which scored 57.96.

Another significant change introduced in this release is teaching the model to determine when it needs to "think longer."

StartLux offers two pathways: Instant and AutoThink. The former rapidly outputs probabilities via a single computation pass, while the latter decides whether to initiate additional reasoning based on the confidence level of its initial judgment.

To address when to start reasoning and how to terminate it promptly, StartLux proposed the RLCD² reinforcement learning method. Its goal is to improve the quality of decision probabilities while constraining unnecessary reasoning overhead.

This update also raises a more critical question for discussion: As AI systems increasingly rely on models for task routing, permission checks, and tool selection, how reliable are the probabilities provided by these models? To what extent does adding reasoning improve final decision-making?

I. Topping the Decision Model Leaderboard: How Does StartLux Outperform Jev?

From the latest leaderboard perspective, StartLux’s lead has been further validated by public rankings. In the Decision Index 0.3.2 leaderboard dated October 16, the 27B Instant model achieved a comprehensive score of 69.12, leading the remaining 173 entries. Specifically, it scored 72.88 on public benchmark tests, 64.82 on private tests of similar capabilities, and 61.41 on private tests in new domains, ranking first in all three sections.

In the team’s published evaluations, StartLux-Decision-27B-AutoThink surpassed Jev’s hosted version with a score of 73.37 versus 57.96, marking a relative advantage of approximately 27%. However, a deeper analysis reveals that the gap between StartLux and Jev is not solely attributable to the added reasoning capability.

The Instant version, also at the 27B scale, achieved 72.83 points on the DI 0.3 public subset without generating explicit reasoning processes, trailing the AutoThink version by only 0.54 points. In this comprehensive test suite, StartLux’s direct judgment pathway already outperformed Jev.

This advantage extends to smaller-scale models as well. The 2B Instant model scored 59.14, slightly higher than Jev’s hosted version at 57.96. The 4B and 9B Instant models scored 67.88 and 69.96, respectively. Across five scales ranging from 0.8B to 27B, StartLux Instant consistently outperformed the strongest third-party decision models of comparable size, with score advantages ranging from 7.4 to 16.9 points.

Given that Instant already achieves high scores, what specific benefits does the reasoning capability added by AutoThink bring? Another benchmark proposed by the team, DecisionPivot, offers a more concrete perspective: Can the model identify changes when key conditions in a problem shift, and adjust its answer accordingly?

On such tasks, the gap between Instant and AutoThink widens significantly. The pairwise accuracy for 27B Instant was 75.4%, whereas AutoThink reached 86.9%, an improvement of 11.5 percentage points. For comparison, Jev’s hosted version scored 31.0%, Decision 3.0 d3 scored 71.3%, Perplexity Decider v1.1 scored 70.1%, and GLM-5.3 with low reasoning intensity scored 82.2%. These results indicate that additional reasoning yields more pronounced benefits in tasks requiring the identification and handling of critical conditions.

II. "Think When Unsure": How Does AutoThink Decide Whether to Initiate Reasoning?

StartLux now provides two distinct decision pathways: Instant and AutoThink. Developers input the current state, the question, and candidate options into the model, which returns probabilities for each option. For scenarios such as task routing, tool invocation, and permission checks, programs can select subsequent actions based on these probabilities. The primary difference between the two pathways lies in whether the model generates additional reasoning processes.

Instant covers five model sizes—0.8B, 2B, 4B, 9B, and 27B—and outputs option probabilities via a single forward pass without first generating reasoning text. To fully leverage the complete input, Instant employs bidirectional attention in its full-attention layers, allowing earlier content to be computed in conjunction with subsequent questions and options, thereby improving single-pass decision accuracy.

This design also makes it suitable for high-frequency decision-making scenarios. According to inference tests published by the team, on a single NVIDIA H200 GPU using vLLM with a single concurrent request, the median latency per request for Instant models ranging from 0.8B to 27B is approximately 25 to 52 milliseconds. These results are specific to the hardware and test configuration; actual latency will vary depending on input length, deployment environment, and concurrent load.

AutoThink, covering 4B, 9B, and 27B models, retains causal attention necessary for generating reasoning. It first computes initial probabilities for each option: if the highest probability meets a threshold, the model returns directly; otherwise, it generates a reasoning process and recalculates probabilities. The system finally fuses the two results, assigning default weights of 75% to the post-reasoning result and 25% to the initial result.

The key to this mechanism is that developers can adjust the threshold for triggering reasoning. The current service defaults to a threshold of 0.6, while the main AutoThink scores reported in the paper use 0.9. A higher threshold makes it harder for the initial judgment to pass directly, increasing the likelihood that the model enters the reasoning workflow. In a set of controlled experiments in the repository, the team lowered the threshold for the 27B AutoThink model from 0.9 to 0.6. The DI 0.3 score dropped from 73.22 to 72.85, but the DecisionPivot pairwise accuracy fell significantly from 88.1% to 68.4%.

These results reveal an important trade-off in on-demand reasoning: reducing reasoning may have a minor impact on overall benchmark scores, but it can lead to more pronounced performance degradation in decision tasks requiring the identification of complex conditions. Therefore, developers cannot select thresholds based solely on aggregate leaderboard scores; they must also consider the complexity of actual tasks and the potential consequences of missing a necessary reasoning step.

The team also analyzed the actual decision processes of the models on core DecisionPivot tasks. Out of 7,800 samples, the 27B AutoThink model answered 39% of questions directly, achieving 96% accuracy on these. The remaining 61% entered the reasoning workflow, where accuracy improved from 57% (initial judgment) to 93% (after fusion). Among the questions that triggered reasoning, the system corrected 1,783 initial errors but also introduced 94 new errors.

It is important to note that "thinking" as described above specifically refers to generating additional explicit reasoning processes; it does not imply that Instant performs no computation or judgment. More critically, the trigger condition depends on the model's own initial confidence. If the model has high confidence in an incorrect answer initially, it may still skip reasoning. Thus, on-demand reasoning does not mechanistically eliminate the problem of overconfidence.

To demonstrate practical usage patterns for decision models, StartLux released several demos. In a chess demo, the original StartLux-Decision-v1-27B played White against Jev 1.13, delivering checkmate on move 19, with neither side using search algorithms.

The team also applied the model to strategic decision-making for Terran robots in StarCraft II, showcasing victories against the built-in Hard AI.

Compared to gaming tasks, continuous visual decision-making demos better illustrate how such models operate in dynamic environments. In a rocket launch demo, the model identified changes in exhaust flames, smoke, launch towers, and flight phases from continuous video frames, answering nine decision questions for each frame.

A flood rescue demo required the model to continuously track the positions and states of rescuers, survivors, slings, and rescue baskets, updating judgments on rescue progress and task priorities.

III. Reasoning Cannot Be Infinite: How Does RLCD² Reduce Computational Overhead?

Once a model determines that a question requires additional reasoning, it faces another challenge: how much reasoning content should be generated, and when should it stop? If the model continually extends its reasoning process, even if the final answer becomes more accurate, it may lose the benefit of rapid decision-making.

To address this, the StartLux team proposed the RLCD² reinforcement learning method, standing for Reinforcement Learning for Calibrated Deliberative Decisions. Instead of simply rewarding correct answers, it focuses on whether the generated reasoning improves the quality of decision probabilities and penalizes reasoning that exceeds a soft length budget.

RLCD² is positioned at the final stage of the training pipeline. Prior to this, the team established the model's foundational judgment capabilities through supervised fine-tuning and decision sample training. Next, reasoning supervision samples taught AutoThink to generate and terminate reasoning processes. Finally, reinforcement learning was used to further optimize reasoning quality and generation length. The paper discloses that base training involved million-scale samples across different model sizes, while the AutoThink reasoning supervision stage used approximately 4,000 to 21,000 samples.

During specific training, the team retained a frozen copy of the model as a scorer. This scorer compared option probabilities before and after reasoning and used probability scoring rules to measure the improvement brought by reasoning. The model responsible for generating reasoning continuously updated its parameters, learning which reasoning processes genuinely improved judgment while reducing unnecessary generation length. Since the scorer remained fixed, the training process could measure reasoning benefits against relatively stable standards.

On 2,986 held-out samples, the team tested the models before and after reinforcement learning, enabling reasoning for every question. For the 9B model, the proportion of normal reasoning termination within generation limits increased from 45.9% to 93.9%, and the average reasoning length decreased from 2,494 tokens to 868 tokens—a reduction of approximately 65%. Meanwhile, the final answer accuracy changed minimally, from 80.38% to 80.48%.

This indicates that the primary benefit of RLCD² in this set of experiments stems from inference efficiency rather than a substantial improvement in accuracy. The effects across different model scales are not entirely consistent; for instance, on DecisionPivot, the pairwise accuracy of the 27B model slightly decreased from 87.2% to 86.9% after reinforcement learning, while the 9B model improved from 53.3% to 59.9%. Therefore, a more accurate conclusion is: this method improves reasoning length and termination behavior while maintaining overall decision-making performance, but it does not guarantee further accuracy improvements for all models.

If the comparison scope is expanded to include "fully disabling reasoning" versus "on-demand reasoning," the value of additional computation becomes even more apparent. The team conducted controlled comparisons while keeping the AutoThink model unchanged: on DecisionPivot, the 9B model's pairwise accuracy increased from 22.3% to 59.9%, adding an average of 1,365 reasoning tokens per question; the 27B model’s accuracy rose from 48.0% to 86.9%, generating an average of 304 reasoning tokens per question. These averages include questions where reasoning was not triggered, thus reflecting the average generation overhead across the entire test set.

An interesting phenomenon is that although the 27B model has a larger parameter scale, it achieved higher pairwise accuracy with fewer average reasoning tokens in this experiment. This demonstrates that the number of reasoning tokens alone cannot represent a model's ability to complete decision-making tasks. However, fewer tokens do not directly equate to lower operational costs: the computational overhead per token is higher for the 27B model, so actual costs still depend on hardware, inference throughput, and deployment methods.

IV. Beyond Correct Answers: Can Models Capture Key Details and Provide Reliable Probabilities?

Beyond benchmark scores, the StartLux team focuses on two critical questions: When conditions change, can the model adjust its judgment accordingly? When programs use model-provided probabilities to filter answers or assign tasks, are these probabilities still reliable?

The DecisionPivot benchmark released by StartLux primarily aims to test models' sensitivity to key conditions. Its core component includes 13 categories of program-generated tasks, totaling 7,800 samples, covering scenarios such as permission checks, scheduling conflicts, tool call validity, and target state verification. Unlike benchmarks that only check whether a single question is answered correctly, DecisionPivot modifies key conditions within questions to observe whether the model adjusts its judgment accordingly.

Specifically, DecisionPivot uses pairwise accuracy: a group is considered correct only if the model answers all cases within a control pair correctly. For example, two questions differ by only one key condition, with correct answers being "Yes" and "No." If the model answers "Yes" to both, it gets one right but fails the entire pair. This design helps identify whether the model merely tends toward a fixed answer without adjusting its judgment based on changing conditions.

To enhance the benchmark's reproducibility, the team also published the question generator, independent solver, scoring program, and data hashes. Answers for core tasks are computed by the generation program and then verified by the independent solver.

Beyond correctness, another issue closer to practical application is: Do the confidence levels provided by the model truly correspond to the probability of answering correctly? For instance, when an Agent system filters candidate answers based on model confidence, can the system continue to trust that "90% confidence" roughly implies a 90% accuracy rate?

One experiment in the paper showed that when using the 9B model to verify answers in an independent response scenario, the actual accuracy for responses with confidence between 80% and 90% reached 92%. However, when the system selected the highest-scoring answer from 32 candidates, the accuracy for the same confidence interval dropped to 78%. This means that even if the underlying model remains unchanged, the relationship between confidence and actual accuracy may shift after filtering.

This does not imply that filtering answers necessarily reduces the system's overall accuracy. The real issue is that developers who continue to use confidence thresholds set during standalone responses may incorrectly estimate the reliability of filtered results. For example, a rule originally set as "automatically execute if confidence exceeds 90%" may no longer correspond to the expected error risk after candidate filtering or multi-model routing.

To address this, StartLux proposed policy-aware calibration. Researchers first pass labeled calibration samples through the actual filtering and routing pipelines used in production, then fit new probability calibration mappings based on the returned confidence scores and related metrics. For candidate answer filtering, this process also considers factors such as the number of candidates and average scores, without requiring changes to the underlying model parameters.

Across 184 configurations covering four types of workflows—including candidate filtering, reasoning gating, and cascaded calls—the team reported that the signed deviation between calibrated confidence and actual accuracy fell within −2.7 to +2.1 percentage points. However, when applying the calibration mapping to unseen datasets, overconfidence of up to 6.1 percentage points still occurred. This indicates that while policy-aware calibration improves probability reliability in evaluated workflows, its effectiveness is not guaranteed to remain consistent under new data distributions.

StartLux has currently made model weights and inference code publicly available, supporting deployment via vLLM, llama.cpp, and MLX, along with quantized versions of various sizes. However, developers must note license restrictions: the inference code is licensed under Apache 2.0, while the model weights use CC BY-NC 4.0, permitting non-commercial use only; commercial deployment requires separate authorization.

As Agent systems increasingly handle tasks such as tool invocation, task assignment, and permission checking, models need to provide not just an answer, but also the level of confidence in that answer and whether it is worth investing additional computation to further verify it. StartLux's research on fast decision-making, on-demand reasoning, and probability calibration aims precisely to address these challenges.

However, current leading results still stem primarily from specific evaluation environments. While DecisionPivot provides verifiable complex decision-making tasks, a gap remains between programmatically generated test cases and real-world production systems. Metrics such as latency, cost, and error rates under varying business data, call loads, and inference thresholds require further validation through actual deployments. For developers building Agent systems, these models offer a new technical option: delegating high-frequency decisions to specialized models and adding inference compute only when necessary. Whether this truly improves system reliability will ultimately be proven by performance in specific workflows.

Code, Model, and Paper Links:

• GitHub (Inference Engine & Evaluation)

https://github.com/StartLuxLabs/StartLux-Decision

https://github.com/StartLuxLabs/DecisionPivot

• Hugging Face Model Collection

https://huggingface.co/collections/startlux-models/startlux-decision-v11-autothink

• Paper