IBM ALTK-Evolve: Learned Memory Tops Tau2-Bench Without Frontier Models

Back to Articles

ALTK-Evolve: How learned memory tops the τ²-bench leaderboard on customer service tasks without a frontier model

Enterprise Article Published October 20, 2026 Upvote 3

Ashwath Vaithinathan Aravindan ashwath-vaithina ibm-research Vatche Isahagian Vatche ibm-research Vinod Muthusamy vinodmut ibm-research Jayaram Radhakrishnan jayaramkr ibm-research Evelyn Duesterwald evduester ibm-research Gaodan Fang gaodan-fang ibm-research Punleuk Oum illeatmyhat ibm-research G Thomas gsthomasx ibm-research Michał Ulewicz Michal ibm-research Merve Unuvar mrvnvr ibm-research Ayhan Sebin ayhansebin ibm-research

ALTK-Evolve learns and applies a playbook using GPT-4.1, outperforming the retail entries on τ²-bench, a customer service agent benchmark—without updating model weights.

Learned memory takes a basic ReAct agent with GPT-4.1 to the top

Using GPT-4.1, an older, non-reasoning model, a basic customer-service agent reaches 67.5% Pass^4 on τ²-bench retail — meaning it solves the task on all four independent attempts for 67.5% of tasks. It scores above every retail entry in our October 12, 2026 leaderboard snapshot. It runs on the benchmark's stock ReAct scaffold, with learned guidelines appended to its system prompt.

ALTK-Evolve builds that playbook using GPT-4.1. It analyzes the agent's training-task attempts and uses the model to distill successes and failures into short behavioral rules: how to authenticate a customer, confirm a transaction, or handle a return. Those guidelines are then added to the agent's prompt for subsequent conversations. The same model powers both guideline extraction and the agent; no weight updates or human annotation are needed.

Our previous work demonstrated this approach on coding agents. Here, we test it on customer service, where success requires following domain policies, understanding a customer's request, and executing the right tool actions.

τ²-bench tests policy-bound customer service

τ²-bench (from Sierra) measures how well an agent handles customer-service conversations under real constraints. Each task pairs the agent with an LLM-simulated customer who has a goal in mind, a set of domain tools (look up an order, issue a refund, rebook a flight), and a written policy document the agent must follow. The agent has to draw out the right details, stay within policy, and take the correct — often irreversible — actions; a task counts as solved only when the final database state matches the expected outcome, a strict binary reward. We use its two customer-service domains, retail and airline.

We run τ²-bench's own implementation of a basic ReAct agent with a two-phase THINK → ACT loop, using gpt-4.1 as the agent model and gpt-5.5 as the customer-simulator model; 4 trials per task. The only change we make is a block of learned guidelines appended to the system prompt — everything else (policy, tools, scaffold) is stock τ²-bench. Concretely, the learned guidelines are tacked on after the domain policy as a plain markdown list:

You are a customer service agent that helps users according to the policy below.
Always follow the policy strictly. Use the provided tools when needed.

<policy>
# Retail agent policy

As a retail agent, you can help users:
- cancel or modify pending orders
- return or exchange delivered orders
- modify their default user address
…
</policy>

## Guidelines learned from past task attempts (follow these carefully):
- Authenticate the user before taking any order-related action. If email is unavailable, verify identity via full name and zip code using the find_user_id_by_name_zip API.
- Always confirm all intended changes (modify, cancel, return, exchange) with a clear, structured summary, and require explicit user approval before executing any database-updating action.
- When processing a return or exchange, communicate the refund with an itemized breakdown and clarify refund timing — don't wait for the user to ask.
- Take one action per step: never combine a tool call with a user-facing message in the same turn, and wait for the tool's result before replying.
- Before looking a user up, parse and clean their name — e.g. split a combined handle like `daiki_johnson_9523` into first/last name — and after a couple of failed lookups, state the exact format you need.
…

We extract guidelines from the agent's trajectories on τ²-bench train tasks using gpt-4.1, the same model that powers the agent, then apply them unchanged on the held-out test set.

Guidelines improve capability and reliability

In a prior post, we talked about how different models / tasks require different dosage of guidance. Inline with the idea, we also evaluate different amounts of guidelines provided to the model.

Each guideline carries a support count — the number of distinct training tasks it was extracted from, so higher support means a rule that generalized across more tasks rather than a one-off. We compare three configurations: baseline (no guidelines), sup3 (keep only guidelines with support ≥ 3 — a small, high-confidence core), and lossless (keep every guideline retained after consolidation, i.e. support ≥ 1).

Table 1. Guideline configurations and retained guideline counts by domain.

The metrics we track. Every task is attempted 4 times, and we read those trials two ways:

• Mean Accuracy — the average success rate across the 4 trials. This is the capability axis: how often does the agent get this right?
• Pass^4 — the percentage of tasks that passed across all 4 trials. This measures along the reliability axis and the metric the τ²-bench leaderboard reports. It matters here because a support agent that resolves a refund 3 times out of 4 still fails a quarter of real customers — Pass^4 asks whether you can trust the agent every time, not just on average.

Figure 1. Mean Accuracy and Pass^4 on held-out retail and airline tasks for agents using gpt-4.1 across four trials per task.

Agent model: gpt-4.1 (user simulator: gpt-5.5).

Table 2. Held-out agent performance using gpt-4.1 across four trials per task. Parentheses show percentage-point changes from baseline.

Mean Accuracy climbs in both domains — retail +4.4 pts (78.1 → 82.5) and airline +7.5 pts (57.5 → 65.0). Reliability moves further: the strict Pass^4 rises +10 pts in retail (57.5 → 67.5) and +10 pts in airline (35.0 → 45.0). And the gains come from better-informed calls, not longer conversations — we look at conversation length directly in Better performance costs more tokens, not more turns below.

The best amount of guidance depends on the domain

The two domains disagree about how much guidance pays off. In retail, more is better on every metric — the small sup3 core barely beats baseline on accuracy, and you need the full lossless set for the real jump. In airline the picture flips at the reliability end: lossless leads on accuracy (65% vs 60%), but the compact sup3 core (14 guidelines) is the more reliable one, taking Pass^4 to 45% against lossless's 40%. Echoing the earlier finding that delivery is a dial, not a constant, support count is a knob worth tuning per domain, not a value to hard-code.

Better performance costs more tokens, not more turns

Guidelines aren't free, but the cost lands where you'd expect — in prompt input tokens, not in a longer, more meandering conversation:

Figure 2. Total LLM cost per conversation by domain and guideline configuration on the full base split.

Table 3. Total LLM cost per conversation (agent and user simulator) on the full base split across four trials. Lower is better.

Does the agent get there by talking longer? To answer that cleanly we compare like for like: the task runs solved under both baseline and the guideline variant — identical scenario and seed, a true "same task, done right" set — and count assistant turns on each side.

Table 4. Assistant turns per conversation for matched runs solved under both settings on the test split, pooling four trials.

The agent isn't talking longer — on the shared set, guidelines make conversations slightly shorter: lossless trims ~0.3–0.4 turns in both domains (retail 10.59 → 10.16, airline 9.15 → 8.82), and no variant lengthens them. That matters beyond cost: each turn is a sequential LLM round-trip, so fewer turns means lower end-to-end latency — a faster time-to-completion for the customer, not just a cheaper one. The only real price is on the input side — the larger system prompt riding on every LLM call: retail's 292-guideline prompt pushes cost from ~$0.084 to ~$0.139 per conversation, while airline's smaller playbook is cheaper. And in production even that is largely reclaimable with prompt caching, since the guideline block is identical across every call.

So the trade is a favorable one: for a slightly higher input cost (much of it cached away in practice) the agent lands higher accuracy, higher reliability, and lower time-to-completion at once — not the usual quality-for-latency tradeoff, but all three moving the right way together.

Learned guidelines also improve agents using frontier models

So far, we have used an older, non-reasoning model — gpt-4.1. A fair objection is that learned guidelines might have less to add to an agent using a frontier model: maybe they only patch the gaps of a weaker model. So we tested it directly — we reran airline using gpt-5.5 for both the agent and guideline extraction, on the same τ²-bench ReAct scaffold, with guidelines re-mined from the agent's own runs. One detail changes with the model: none of the guidelines extracted in this setup reached support ≥ 3 (they top out at support 2), so the high-support core here is sup2 rather than sup3.

Table 5. Held-out airline performance for the gpt-5.5 agent across four trials per task.

The learned guidelines improve this setup too. On the same held-out test tasks, appending them to the prompt lifts accuracy +10 pts (72.5 → 82.5) and the strict Pass^4 by +15 pts (55.0 → 70.0) — a larger reliability gain than we observed with gpt-4.1 on these very tasks (+10 pts), and from a much higher floor. Learning guidelines around the model can improve an agent even when it starts from a stronger base. And where the baseline's reliability swung between reruns, the lossless configuration reproduced exactly — the gain it buys is dependable, not a lucky draw.

ALTK-Evolve tops the leaderboard comparison

Submissions are scored on the full base split (train ∪ test), as τ²-bench requires, and reported at Pass^4. Because that covers the full task set rather than the held-out test split, these figures differ from the test-task numbers reported above. This is where the practical upshot shows up — prompt-level memory carries agents using each base model into, or past, current frontier company:

• Retail: #1. On the base split the agent with learned guidelines reaches Pass^4 = 67.5%, ahead of every frontier model and ahead of RAFT-30B-A3B (61.4%), a model fine-tuned on synthetic τ²-bench retail data. Using stock gpt-4.1 and a prompt-appended playbook, it outperforms that fine-tuned system by +6.1 points.
• Airline: #1 using gpt-5.5, #8 using gpt-4.1 Against the published board, the compact sup3 core gives the agent using gpt-4.1 Pass^4 = 44% — 8th, beating GPT-5.2 (no reasoning) and nearly matching Claude Sonnet 4.5. Swap in the stronger base and the same approach tops the field outright: the agent using gpt-5.5 reaches 82.0%, clearing every frontier entry (66–72%).

Figure 3. Pass^4 comparison between ALTK-Evolve runs and the τ²-bench leaderboard snapshot from October 12, 2026.

Same message in both domains: learning guidelines around the model works across different base models — bringing an agent using an older, non-reasoning model into frontier company on retail, and taking an agent using a frontier model to the top on airline.

Agents can improve without weight updates

Fine-tuning a model on a benchmark is expensive, brittle, and doesn't transfer to the next domain. Here, ALTK-Evolve improves a stock agent on a stock model in both domains — and matches or beats a fine-tuned system on retail — using only prompt-level guidelines extracted from the agent's own runs, automatically, without human annotation, and without any test-set leakage.

Links:

• Prior posts