Controllable AI in Action: Apollo-1 Aces Amazon Scenarios; Rufus Falls Short

Introduction

Generative AI excels when acting on behalf of a user, but it breaks down when an agent must act on behalf of a brand or organisation. That’s where controllable AI comes in. To prove the point, we ran a set of real-world evaluations that pit Apollo-1 against today’s flagship LLM agents. This evaluation involved 120 multi-turn shopping conversations against Amazon’s live product catalog, covering everything from product discovery and cart operations to policy guardrails, with no hand-curated prompts and no retries.

Apollo-1 is built to operate on behalf of an entity, not just as an assistant to a user. Its hybrid core fuses the fluency of generative AI with the reliability of deterministic rule engines, giving companies and organizations the control and auditability that purely stochastic LLM co-pilots cannot guarantee.

When pitted against Amazon’s own agent, Rufus, on the exact same live catalog and inventory, that design difference translated into a decisive win:

• Apollo-1: 109 / 120 full-conversation passes (90.8%)
• Rufus: 20 / 120 full-conversation passes (16.7%)

This result was achieved using a strict, one-shot scoring methodology. Unlike most leaderboards that cherry-pick the best response from batches of generations, we score the first and only response. It’s a simple, reproducible, and brutally honest snapshot of how an agent performs when a real customer is waiting.

Evaluation Method – Real-World, One-Take Scoring

Our evaluation is guided by the same principles as the tough τ-Bench benchmark, and is designed to be unforgiving. Each agent gets only one attempt to complete a scenario, and success is binary—every step must be perfect, or the entire scenario is a failure. The final score is therefore a reflection of an agent’s real-world performance.

Results at a Glance

All scores are one-shot and reflect the agent’s first responses only, no retries.

Why Apollo-1 Pulls Ahead

Built to work on behalf of entities, not users. From day one we optimised Apollo-1 for grounded action on behalf of a brand, business or organization—not for idle conversation. The architecture layers control systems that show up in every result above.

• Neuro-symbolic foundation model — a Symbolic Reasoner replaces the transformer as the model’s decision-making core, enabling controllable, context-aware interactions.
• Deterministic control on demand — lets operators inject rule-based logic whenever needed; symbolic policy guardrails guarantee the agent never departs from instructions, while still leveraging generative fluency for natural conversation.
• Real-time Control Panel — enables operators to inspect reasoning, adjust context schemas, inject rules, and replay live trajectories. In addition, the Control Panel allows for continuous fine-tuning at the sub-interaction level instead of retraining the whole stack.

The net effect: an agent that can confidently operate on behalf of a retailer, an insurance company, or a government office—executing the task, honouring every policy, and still maintain conversational fluency. 

Recent research from Salesforce Research echoes the need for this shift: on the CRMArena-Pro benchmark, leading LLM agents top out at ~58% single-turn success and collapse to ~35% in multi-turn chats, leading the authors to highlight a “significant gap between current LLM capabilities and real-world enterprise demands”.¹

Example scenario

Scoring criteria — Scenario 1 “55-inch 4K TV Under $400”

Scoring: binary (1 = pass, 0 = fail) per step. Scenario score per run = R-1 + R-2 + R-3 (max 3).

Scenario 1 Reward Logs

Apollo-1 delivers exact product details, specs, and availability in a single pass; Rufus surfaces a model but misses the follow-up spec and variant check.

Benchmark Overview & Group Analysis

The 120 live shopping runs fall into five buckets. Each bucket probes a single competency; a flawless agent would post 100% in every row.

Apollo-1 Strengths Across the Retail Buckets

• Context-aware dialogue (Specs & Comparisons) –
  In Specs & Comparisons scenarios, Apollo-1 provides quick, accurate product details and asks clarifying follow-up questions (budget, exact specs, discounts) only when necessary, consistently maintaining clarity and context throughout.
• Real-time variant availability (Variants & Stock) –
  Apollo-1 consistently returns precise, real-time information on product availability across sizes, colors, and variants, explicitly stating “not available” when appropriate—achieving a 97% accuracy.
• Deterministic cart control (Cart & Inventory) –
  In Cart & Inventory scenarios, Apollo-1 performs actual cart modifications—accurately adding/removing products—and instantly reflecting these actions in live cart contents, avoiding the “pretend add-to-cart” failures seen in Rufus.
• Advanced multi-condition filtering and arithmetic logic (Advanced Filtering) –
  Apollo-1 handles complex filtering scenarios (price, warranty, specifications), recalculates totals dynamically after each user adjustment, and maintains precise arithmetic accuracy, achieving an 83% success rate.
• Compliance and targeted recommendations (Gifts & Compliance) –
  Apollo-1 effectively recommends products based on specific use-cases or suitability criteria (e.g., occasion, pets/children) while strictly adhering to brand policies and gracefully handling constraints, reaching perfect compliance.

Areas We’re Still Tuning

• Dynamic discount detection – Sale-price vs list-price maths (Scenarios 115-120) remains brittle across both models.
• Long-tail attributes – Edge specs such as exotic ink-tank yields or obscure warranty tiers occasionally slip past the retrieval layer.

Rufus’ Persistent Gaps

• Skips variant checks (colour, size) or answers generically instead of “not available.”
• Describes—but rarely executes—cart operations.
• Miscomputes or omits arithmetic steps (price deltas, CADR values).
• Zero full passes in Multi-Constraint tasks; struggles whenever more than one hard filter is active.

Bottom Line

Across 120 one-shot retail scenarios Apollo-1 delivers a full-conversation pass on 109 runs (90.8%); Amazon Rufus manages 20 runs (16.7%). The margin widens as tasks demand live arithmetic, variant precision, or multi-constraint reasoning—hallmarks of production-grade conversational agents where deterministic, auditable behaviour is non-negotiable and controllable, neuro-symbolic agents prevail.

Appendix A: Reward logs

View Full Scenario List here 

Appendix B :

View full Trajectories Here.

References

  ¹ Huang, K-H.; Prabhakar, A.; Thorat, O.; Agarwal, D.; Choubey, P.K.; Mao, Y.; Savarese, S.; Xiong, C.; Wu, C-S. CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions. Salesforce AI Research (2025). arXiv:2505.18878.