HarnessSQL: 8B Model Outperforms 32B Baselines in Long-Horizon SQL Interaction

In real-world enterprise scenarios involving complex data analysis, the notion that Large Language Models (LLMs) can "instantly comprehend a database and generate perfect SQL" is often an unrealistic fantasy.

Traditional Text-to-SQL approaches typically simplify the task into single-turn text generation: given a question and a static schema, the model generates the final SQL in one step. However, real-world databases often contain hundreds of tables, thousands of fields, and millions of records. Senior analysts never rely on "single-turn blind guessing"; instead, they execute commands like SHOW TABLES in CLI or IDE environments, inspect table structures (DDL), run probe queries, and iteratively refine logic based on error messages before submitting code.

On long-horizon interactive benchmarks such as Spider 2.0, BIRD-Interact, and LiveSQLBench, traditional statically fine-tuned models expose critical weaknesses. Trained only to "directly output the final answer," these models lack the ability to explore, track state, and recover from errors when deployed in interactive environments (Harnesses). Consequently, they easily fall into infinite loops or crash due to execution errors—a phenomenon known as Train–Deploy Mismatch.

To address this bottleneck, teams from HKUST and Microsoft have introduced HarnessSQL, the first framework for native long-horizon environment interaction training. It employs Supervised Fine-Tuning (SFT) on long-horizon trajectories and Reinforcement Learning (RL) with execution rewards within closed sandboxes and dedicated SQL Harnesses.

On Spider 2.0-SQLite, HarnessSQL achieves a 45.2% execution accuracy with just 8B parameters—nearly triple the baseline—and significantly outperforms specialized 32B models (OmniSQL at 14.8%, Arctic-Text2SQL at 16.3%) and multi-agent systems (MARS-SQL at 12.6%). The 14B model reaches an impressive 54.8%!

Paper Link: https://arxiv.org/abs/2610.12274

Open Source Code: https://github.com/YangHaolin0526/HarnessSQL

Figure 1: Evolution from single-turn Text-to-SQL to interactive database agents equipped with sandboxes and dedicated Harnesses.

Problem Formalization and the Mystery of "Train-Deploy Mismatch"

Interactive Text-to-SQL can be formalized as a Partially Observable Markov Decision Process (POMDP) mediated by an Execution Harness contract. A task instance is defined as a tuple $x = (q, D, h, v)$, corresponding to the natural language instruction $q$, the underlying database $D$, the execution harness $h$, and the hidden ground-truth validator (ExecutionOracle) $v$.

At each interaction turn $t \in [1, T]$, the agent generates reasoning and tool-calling actions $a_t \in A$ via policy $\pi_\theta$ based on environmental observations $o_t \in O$. The complete trajectory is expressed as:

where $y$ is the final SQL submitted at the termination step. Task success is evaluated by a joint reward function:

Where:

R_result(y, v) ∈ {0, 1}: Compares the extensional equivalence of the result sets produced by candidate output $y$ and ground truth $v$ on the actual database $D$, overcoming the limitations of superficial string matching.

$R_{\text{protocol}}(\tau, h) \in \{0, 1\}$: Validates whether the trajectory adheres to the Harness contract (read-only permissions, syntactic compliance, no infinite loops, and explicit submission within $T$ steps).

However, most existing post-training frameworks still rely on traditional static supervised losses:

This static training paradigm leads to three core disconnects:

1. Erasure of Intermediate Exploration States: By targeting only the final SQL output, the interactive process of probing table structures and validating hypotheses is ignored.
2. Blindness to Execution Feedback: The model never encounters runtime errors or empty result sets during training, making it prone to collapse when facing such issues during inference.
3. Lack of Action Boundary Constraints: The model fails to understand the permission boundaries and termination protocols of the database CLI, leading to tool-format hallucinations.

Forging Agents in a True Execution Sandbox

To bridge these gaps, HarnessSQL establishes three foundational pillars: a high-fidelity closed execution sandbox, a minimal SQL Harness contract, and a two-stage long-horizon post-training pipeline.

1. Closed SQL Execution Sandbox and Blueprint Synthesis

The team constructed 79 complex database sandboxes based on Spider 2.0-Lite (30 SQLite databases) and Spider 2.0-DBT (49 DuckDB databases, migrated losslessly to SQLite). They employed a SQL-First Blueprint Synthesis Method: first extracting schemas, sample values, and foreign key paths from active engines; then sampling executable SQL queries with controllable difficulty; and finally reverse-generating natural language questions and validation Oracles.

The sandbox features read-only bindings, single-statement constraints, and a 60-second timeout protection. Data quality is ensured through syntax checks, non-empty result validation, and semantic mutation tests.

2. Minimal Harness Contract Tailored for SQL Tasks (DSH-SQL)

The cumbersome Bash tools used by general-purpose programming agents often distract the model's attention. HarnessSQL adapted DeepSeek-Harness (DSH) to distill a four-action minimal contract:

• sql_list_tables: Enumerates all data tables in the database along with their row counts.
• sql_schema: Inspects the complete DDL definition, field types, and real sampled data for a specified table.
• sql_exec: Tentatively executes a read-only SQL query to retrieve the top rows of actual output or runtime error stacks.
• sql_submit: Submits the verified correct SQL artifact as the final answer and exits the long-horizon session.

The environment middleware incorporates three guardrails: Deterministic Row Truncation prevents large outputs from overflowing the context window; Loop Call Warning actively corrects repetitive invalid queries; and Pressure-Triggered Context Compression ensures that core facts are retained during long-horizon exploration.

3. Native Harness Two-Stage Post-Training: SFT Foundation + RL Peak Performance

Stage 1: Full-Trajectory Long-Horizon SFT (Cold Start). Teacher models autonomously explore within the sandbox. High-quality multi-turn trajectories that pass Oracle validation (averaging 11.8K tokens, 16 interaction rounds, and 14 real executions) are selected for full-sequence supervision:

Key Mechanism: During loss calculation, all environment observation tokens ($o_t$) are masked out. Gradients are computed only on the model's own generated reasoning, tool calls, and final submissions, forcing the model to learn how to deduce the next action based on environmental feedback.

Stage 2: Execution-Reward Reinforcement Learning (Autonomous Evolution). To equip the model with trial-and-error capabilities for unknown databases, HarnessSQL employs Dynamic Asymmetric Policy Optimization (DAPO) for continuous alignment within the sandbox, calculating group-normalized advantages for trajectories:

Reward signals rely entirely on real execution comparisons against hidden Oracles: A score of 1 is awarded for equivalent and compliant results, while errors, timeouts, or mismatched results receive a score of 0. No artificial intermediate rewards (Reward Shaping) are introduced, compelling the model to discover robust self-correction strategies.

Figure 2: Overall architecture pipeline of HarnessSQL and examples of sandbox interaction trajectories.

Experimental Evaluation

The team conducted evaluations on three real-world database benchmarks: the in-domain long-horizon interaction benchmark Spider 2.0-SQLite, the out-of-domain dynamic interaction benchmark with user simulators BIRD-Interact Mini, and the cross-domain complex reasoning benchmark LiveSQLBench Base-Lite SQLite.

Baselines included general-purpose coding models (Qwen2.5-Coder-32B), state-of-the-art SQL-specific models (OmniSQL 7B/32B, Arctic-Text2SQL 7B/32B), and multi-agent systems (MARS-SQL 3×7B). All were evaluated using Execution Accuracy.

Table 1: Comparison of Execution Accuracy across the three benchmarks. Bold indicates the best performance; underline indicates the second-best.

The experiments reveal three core findings:

1. Small 8B Model Surpasses 32B Models Across the Board: On Spider 2.0-SQLite, HarnessSQL-8B achieved 45.2%, nearly tripling its base model baseline (15.5%) and significantly outperforming OmniSQL-32B (14.8%), Arctic-Text2SQL-32B (16.3%), and Qwen2.5-Coder-32B (16.3%).

2. 14B Model Approaches the 55% Milestone: Scaling up to Qwen3-14B further increased execution accuracy to 54.8%, validating the excellent scalability of native Harness training.

3. Strong Out-of-Domain Generalization: HarnessSQL also maintained a substantial lead on unseen benchmarks such as BIRD-Interact Mini and LiveSQLBench. Notably, on LiveSQLBench, the 8B (24.8%) and 14B (28.5%) models achieved 3–4 times the performance of all 32B baselines (7.0%–8.9%), demonstrating that the models learned general exploration strategies rather than merely memorizing schemas.

Advanced Analysis

When facing interactive environment tasks, should one proceed directly with Reinforcement Learning (Direct RL) or first perform trajectory fine-tuning (SFT)? Controlled experiments provide a clear comparison:

Limited Gains from Direct RL Cold Start: Running RL directly on the 8B base model only improved accuracy slightly from 15.5% to 20.0% (+4.5%). Without prior interaction knowledge, sparse binary rewards struggle to effectively assign credit over long trajectories.

Full-Trajectory SFT Establishes Interaction Foundations: After SFT on verified trajectories, the 8B and 14B models jumped to 30.4% (+14.9%) and 37.8% (+15.6%) respectively, rapidly mastering tool invocation and error-correction protocols.

SFT + RL Achieves Qualitative Performance Leap: Introducing DAPO reinforcement learning on top of SFT yielded an additional 14.8% gain for the 8B model (reaching 45.2%) and a 17.0% gain for the 14B model (reaching 54.8%)—more than three times the improvement of direct RL alone. This establishes the golden paradigm for long-horizon agents: SFT provides the interaction framework and exploratory common sense, while RL unlocks adaptive optimization potential in unknown environments.

Four Key Ablation Studies

Regarding critical design choices in long-horizon agent training, ablation studies distilled four core insights:

Insight 1: Strong Alignment Between Training and Inference Interfaces (Training-Inference Alignment)

If inference uses the DSH-SQL interactive environment but training relies on non-interactive (Non-harness) SFT data, Spider 2.0 accuracy drops to just 9.6%, even lower than the un-fine-tuned base model (15.5%). In contrast, native Harness SFT achieves 30.4%. This proves that static training disrupts collaboration between the model and dynamic environments; training and deployment environments must be strictly isomorphic.

Cross-ablation comparison of non-interactive vs. native Harness training/inference on in-domain and out-of-domain benchmarks.

Insight 2: Full-Sequence Trajectory Organization Significantly Outperforms Single-Step Splitting (Full-Trajectory > Turn-Split)

When constructing SFT data, full-sequence supervision preserving complete trajectory context (Full-trajectory) consistently outperforms single-step splitting (Turn-split) across all three benchmarks. Long-horizon tasks heavily rely on causal dependencies across multiple turns; fragmenting samples disrupts global decision coherence.

Figure 3: Full-trajectory supervision significantly outperforms turn-split supervision.

Insight 3: Ultra-Long Context Capacity is Vital for Long-Horizon Exploration (8K -> 16K -> 32K)

Real-world database exploration involves dense schema information and intermediate result feedback, often resulting in trajectories exceeding tens of thousands of tokens. When expanding training and inference context from 8K to 16K and 32K, Spider 2.0 accuracy surged from 27.4% to 45.2%. Truncating early history effectively deprives the model of its ability to review and trace back previous steps.

Figure 4: Impact of context window length on execution accuracy, showing significant improvements with 32K context.

Insight 4: The DAPO Algorithm Is Better Suited for Long-Horizon Policy Optimization Under Sparse Rewards

Under identical budget and sandbox constraints, a comparison of GRPO (40.7%), GSPO (43.7%), and DAPO (45.2%) reveals that DAPO delivers the best performance. Its mechanisms—dynamic filtering of zero-variance trajectory groups, decoupled clipping ranges, and full-batch token normalization—effectively mitigate policy oscillation under sparse rewards.

Notable Boundaries and Future Evolution

The team also objectively outlined the current system's evolution path:

1. Cross-Database Dialect Expansion: Extending beyond the current SQLite ecosystem to enterprise-grade cloud-native dialects such as Snowflake, BigQuery, and PostgreSQL, while building more robust permission isolation and ecosystem compatibility.

2. Multimodal and Diversified Analytical Tools: Moving beyond command-line queries to integrate diverse analytical environments, including visualization charts, document reports, and interactive BI dashboards.

Summary and Outlook

Large models are accelerating their transition from "conversational assistants" to "autonomous execution agents." HarnessSQL demonstrates that for complex tasks requiring tool invocation and trial-and-error feedback, merely injecting static "correct answers" is insufficient. The interaction environment (Harness) must be integrated into the core training loop, enabling models to organically acquire long-horizon exploration and fault-tolerance capabilities within dynamic environments.

Currently, HarnessSQL’s code, sandbox environment, and synthetic trajectory assets have been fully open-sourced on GitHub, providing a directly reusable benchmark reference for Text-to-SQL, database agents, and general-purpose Agent development.

References:

https://arxiv.org/abs/2610.12274

Chasing ASI in Seconds

⭐ Like, Share, and Watch — One-Click Triple ⭐

Star us to lock in instant updates from New Intelligence Era!

Open in WeChat