Unparalleled reliability for workflow execution
at a fraction of the cost.

What makes this submission custom

The agent under test is not one LLM behind the default τ-bench scaffold. It is the Apollo-1 runtime running an airline agent program, connected through τ²-bench’s agent interface. Under the leaderboard’s rules that makes the submission custom, and it is filed as such.

Everything on the benchmark side is unchanged:

• the tasks, the airline policy and the tool set;
• the tools’ results and the database;
• the user-simulator prompt and orchestration;
• the evaluator.

The user simulator is openai/gpt-6-luna at temperature 1.0 (the model does not accept temperature 0), with its API-default reasoning effort.

Architecture

Apollo-1 is a neuro-symbolic system. Neural models carry the language and a symbolic core carries the decisions, inside one computation per turn. Every fact the agent uses declares where it comes from:

• the customer’s own words, read by a neural model into the program’s vocabulary;
• a tool or system of record, read symbolically;
• a calculation over other facts.

Each turn runs the same six stages.

Neural elements, and why they are interchangeable

The neural side is asked two kinds of question: what did the customer say, in the program’s terms, and how should a decided outcome be worded? It is never asked whether a rule holds or what to do next. Because its job is that narrow, Apollo-1 is agnostic to the model behind it. Swapping the model changes how replies are phrased, not what the policy allows.

Inside the reasoning, Apollo-1 also uses Type-1 models: small, fast models that each answer one narrow, typed question and return a typed value the symbolic core then checks. Examples are which of the options shown the customer chose, or which event a stated date refers to. They never decide an action on their own.

This run used two neural models:

• openai/gpt-6-luna at high reasoning effort, for reading and phrasing;
• typesafe/jev as a Type-1 reader.

Where the time goes

Across 1,005 agent turns, 89.3% of reply time was spent in the general neural model and 6.4% in Type-1 calls. The symbolic core and tool I/O took the remaining 4.3% (Figure 6). The logic that decides is cheap; the cost and latency of the system are dominated by language.

From policy to program

An LLM agent receives the policy as a system prompt and is trusted to follow it. Apollo-1 instead receives a program: the policy translated into Apollo’s abstraction layer, a folder of typed YAML declarations. The public documentation at docs.aui.io describes them.

• program: identity, voice and the clock that defines “today”.
• vocabulary: the facts and records the agent can know and hold.
• sources: where facts come from and what changes the world. Here these are the benchmark’s own tools.
• derivations: facts computed from other facts, such as fees, allowances, refund amounts and eligibility.
• capabilities: the jobs the agent can do. There are 14 for airline, including book, cancel, change flights, change cabin, add bags, edit passengers, compensation and transfer.
• policies: the rules that must hold before an action proceeds, each with its refusal wording.

The program is built only from the benchmark’s airline policy and tool schemas. Every rule cites the policy section it encodes (encodes), and the policy text itself is carried as the program’s reference corpus. Below are four policy sentences and the declarations they became, excerpted from the program used in this run. Only internal development comments are removed.

Section numbers in encodes refer to the airline policy’s sections in the order they appear. Facts such as reservation.cabin or reservation.booked_within_24h are declared across the program’s files. The vocabulary is one example; derivations, sources and capabilities declare others. Each declares its origin: a tool’s result, a calculation, or the customer’s words.

The variables these declarations use (reservation.cabin, cancel.reason, pay.gift_cards and the rest) are declared by this agent’s own program. They are not part of Apollo-1’s language. Apollo-1 relies on each variable’s role, as its declaration states it: its type, where its value comes from, and which rules and actions read or change it. It never relies on the variable’s name. The same runtime runs agents in other domains, with entirely different vocabularies.

The method in one line: codify the policy, translate each customer request into the same symbolic language, compute the answer with logic on the CPU, and use neural models only to translate in and out of natural language. The model’s job is language, and the program’s job is the rules.

Consistency

pass^k measures whether an agent succeeds on every one of k independent tries at the same task. Systems that decide by sampling a model lose ground as k grows: a task solved three times in four still counts as a failure at pass^4. When the decision is computed, the same facts lead to the same decision, and variance is left mostly to the conversation itself: the simulated customer’s wording, choices and mistakes.

• Apollo-1’s pass^4 is higher than every other system’s pass^1. It succeeded on all four tries of a task (88.0%) more often than any general-purpose model here succeeded on one.
• Far fewer failures at k=4. Apollo-1 failed 12% of tasks at pass^4, against 22% for Claude Opus 5.5 (xhigh) and 26% for GPT-6.1-sol (max) on the same simulator, and 28% for GPT-5.2 (high) and 30% for Claude Opus 4.5 on the public board.
• A fraction of the cost. At $0.0130 per conversation, Apollo-1 cost 5.4× less than GPT-6.1-sol (max), 8.8× less than GPT-5.2 (high), 30× less than Claude Opus 5.5 (xhigh) and 31× less than Claude Opus 4.5. Per successful conversation it was $0.014, against $0.085, $0.137, $0.451 and $0.475. On OpenRouter’s own airline runs, Claude Fable 5 costs $0.92 per task, about 71× more (Figure 4).
• More reasoning does not buy consistency. At their highest reasoning settings, Claude Opus 5.5 and GPT-6.1-sol still lost 8.5 and 9.5 points from pass^1 to pass^4, against Apollo-1’s 2.5. See The thinking paradox of larger frontier models.
• A low-cost model inside. The neural model in this run, gpt-6-luna, is priced at $0.10 / $0.50 per million input / output tokens. The reliability comes from the architecture around it, not from a larger model.

* Run by AUI with τ²-bench v1.0.1’s official agent, prompts, tools, tasks and evaluator: 50 tasks × 4 trials, seed 300, no conversation rerun, with gpt-6-luna as the user simulator at its API defaults, as in Apollo-1’s run. Claude Opus 5.5 ran at effort xhigh and GPT-6.1-sol at reasoning max; only the model client library was updated. Cost is every agent-side call at list prices ($2 / $10 per million input / output tokens for GPT-6.1-sol, $4 / $20 for Claude Opus 5.5). Other rows are the leaderboard’s published 1.x airline submissions (read 2026-10-20), which use GPT-5.2 at low reasoning as the simulator; costs as published (“n/p”: none published). Cost per success is cost per conversation divided by pass^1.

Results in detail

Every loss

All 19 failed conversations were read. They come from six tasks, all of them tasks whose correct outcome changes the booking. For each task, the table gives what happened and the cause, using the report’s labels.

Grouped by the labels above, the 19 losses are 6 from agent behaviour or error (tasks 29, 24 and 33), 8 from policy interpretation (tasks 39 and 44), and 5 from sequencing or the simulated customer (tasks 32 and 24).

The four tasks lost in every trial are all among the 27 airline tasks revised in the τ³ task fixes, which are included in τ²-bench v1.0.1. Our program states both policy sequences those fixes enforce: cancel and rebook for a route change, and upgrade before changing a basic-economy flight. The remaining gaps are listed above as they are.

The thinking paradox of larger frontier models

Newer, larger models do not reliably score higher here: OpenRouter’s airline results show newer releases scoring below older ones¹. And on the same simulator as Apollo-1, the two newest frontier models at their highest reasoning settings, Claude Opus 5.5 (xhigh) and GPT-6.1-sol (max), still lost 8.5 and 9.5 points from pass^1 to pass^4.

Our hypothesis: on a strict, policy-bound workflow, extra thinking becomes a liability. Published work documents three effects:

• More thinking, more variance. Across agent benchmarks including τ-bench airline, more reasoning effort gave equal or lower accuracy in 21 of 36 cases². Longer thinking also makes outputs more variable³, and pass^k penalizes variance directly⁴.
• Persistence wears models down. Under repeated pressure, models give up correct positions^5,6. On τ-bench airline, two models with similar scores differed about fourfold in how often a persistent customer talked them into a policy violation⁷.
• Actions drift. Agents take actions that do not match their instructions or what their tools returned⁸. On airline, a single wrong step in a write cuts the odds of success by up to 92%⁹.

Apollo-1 is built to avoid all three. Its rules are evaluated by symbolic logic rather than reasoned about, a customer’s insistence does not change the facts a rule checks, and a write runs only after a confirmed yes to the exact change. The result is the flat line in Figure 1.

The simulated customer itself makes mistakes in a large share of airline conversations¹⁰, which is why consistency over four trials is the measure we lead with.

How the agent was built

• Program: the airline program was written from the benchmark’s airline policy and tool schemas. It contains no task identifiers, no expected actions and no rules for individual tasks. Every rule cites the policy sentence it encodes.
• Development: the public airline tasks were used during development to find and diagnose failures. A program change was accepted only if it traced to a policy sentence. The agent runs on the same general Apollo-1 model and runtime as every other Apollo-1 agent; only the program is specific to the airline.
• Calibration: some decision thresholds of the Type-1 readers were calibrated on hand-labelled answers. Those answers came from development conversations in several domains, including conversations on the public airline tasks.
• Training: no model was trained or fine-tuned by AUI on τ-bench tasks, rewards or evaluation data for this submission.
• Runtime information: the agent sees only what a standard agent sees: the conversation, the tools’ results and the airline policy, as encoded in the program.

Configuration

References

1. OpenRouter (2026). τ²-Bench Airline benchmark results, read 2026-10-20. openrouter.ai/benchmarks/tau2-bench-airline
2. Kapoor, S. et al. (2025). Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation. arXiv:2510.11977. arxiv.org/abs/2510.11977
3. Ghosal, S. S. et al. (2025). Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Models. NeurIPS 2025. arXiv:2506.04210. arxiv.org/abs/2506.04210
4. Yao, S. et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045. arxiv.org/abs/2406.12045
5. Hong, J. et al. (2025). Measuring Sycophancy of Language Models in Multi-turn Dialogues. Findings of EMNLP 2025. arXiv:2505.23840. arxiv.org/abs/2505.23840
6. Tang, L. et al. (2026). SPINE: Measuring LLM Sycophancy under Sustained Multi-Turn Pressure. arXiv:2609.09090. arxiv.org/abs/2609.09090
7. Kirgis, P. et al. (2026). Log analysis is necessary for credible evaluation of AI agents. arXiv:2605.08545. arxiv.org/abs/2605.08545
8. Zhang, W. et al. (2025). MIRAGE-Bench: LLM Agent is Hallucinating and Where to Find Them. arXiv:2507.21017. arxiv.org/abs/2507.21017
9. Cuadron, A. et al. (2025). SABER: Small Actions, Big Errors — Safeguarding Mutating Steps in LLM Agents. arXiv:2512.07850. arxiv.org/abs/2512.07850
10. Barres, V. et al. (2025). τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment. arXiv:2506.07982. arxiv.org/abs/2506.07982

Architecture overview: Beyond generative AI. Program reference: docs.aui.io.