The Conversation is the Test

A customer wants the shipping address changed on two pending orders. The AI agent confirms the request, makes an address update, and reports that the pending orders have been updated.

Only one order has changed. The missing update is visible in the tool trace. Inspect task 42, simulation 3.

We encountered this class of errors often enough in testing a private agent that the findings helped drive an architectural redesign. While that work is confidential, our existing simulation model transferred to Tau-bench Retail with limited adaptation, giving us a public setting in which to show what a world-model-testing approach reveals.

In the example above, all four Tau-bench trials completed both updates. A Kashikoi simulation exposed the unfinished work(*1). Elsewhere a customer’s general approval became permission for a specific product configuration the agent had not even presented, an issue not present in Tau-bench trials.

These conversations point to a recurring design problem: keeping the customer’s request, the approved plan, and the executed actions in agreement as new information arrives.

Explore the results across retail Tau-bench tasks in the task comparison, or go straight to the missed address update. Each task links to both conversation sets, with tool calls, database results, Natural-language checks, and a Conversation Trajectory.

Kashikoi's World Model chooses the Conversation Path

A task specifies a customer’s goal and constraints. It leaves many conversational decisions open: when to disclose a preference, whether to ask for options, how to respond to an incomplete proposal, and when to introduce another request. Those choices affect what the agent must track and resolve.

Kashikoi separates choosing the next customer behavior from expressing it in language.

Kashikoi's simulation engine samples an intended next behavior based on the unfolding interaction from a world model of customer behavior. An LLM realizes that choice as a message/tool call.

Tau-bench is a useful foundation: it evaluates agents through conversations with simulated users, domain tools, and policies. Its original design already studies reliability across repeated trials. Original τ-bench paper.

Kashikoi makes an additional choice explicit. The world model selects the intended next behavior in the context of the conversation; an LLM expresses that behavior in language. Selecting the next move and writing the next message are separate operations. That gives us a way to vary the interaction beyond simply asking the language model to continue from a task prompt.

Two conversations worth inspecting

Task 74: Approval arrives before the proposal is complete

The customer wants an i9 laptop upgrade and cancellation of another order. In Kashikoi simulation 2 for Task 74, the customer approves those actions before the agent looks up the available laptop configurations.

Step	Recorded event
Customer approves	Upgrade the laptop to an i9 and cancel the other order.
Agent looks up the catalog	Available configurations become visible.
Agent commits	Selects a black, 15-inch laptop with 32GB RAM and a 512GB SSD, without another customer exchange.
Agent reports completion	Presents those configuration details after the change.

In Tau-bench trial 3 for the same task, the agent presents the configuration and price before obtaining confirmation and making the change.

The failure is in the timing of the commitment: the agent selects material details after receiving general approval, then executes before presenting them to the customer.

The design question is : Does an earlier “yes” still authorize the action after a tool lookup determines what that action actually entails?

Task 42: Two approved updates, one completed

The customer needs an address correction and a change to a jigsaw puzzle. The agent finds two pending orders and proposes updating the default address and the shipping addresses on those orders, along with the puzzle change.

The customer replies: “Yes, I confirm both actions. Please go ahead.”

Approved work	Recorded execution
Update the customer’s default address	Updated.
Update the shipping address on order #W4082615	Updated.
Update the shipping address on order #W9583042	No update call.
Change the puzzle in order #W4082615 to the agreed version	Changed.

The agent then tells the customer that the default address and pending-order shipping addresses have been updated. Inspect the approval, actions, and completion message in simulation 3.

All four Tau-bench trials complete both order-address updates. Compare the trials. Kashikoi exposes an omission absent from those four conversations, even after the agent explicitly proposes both updates and receives approval.

For a retail-agent team, the practical consequence is straightforward: a confident completion message can conceal an order still headed to the wrong address. A successful write to one record satisfies only one part of a compound request.

Why 30% more aligned diversity matters

Both the Tau-bench trials and the Kashikoi simulations use the exact same configuration of retail agent. Both use the exact same LLM for simulating the customer - gpt-4.1. The main difference between the two is that Kashikoi’s world model inserts an intended state at every step for the simulated customer.

What does this unlock?

Quantifying Diversity

The Kashikoi simulations have 30.2% higher avg. within-task sequence distance.

Metric	Tau-bench	Kashikoi
Avg. normalized behavioral-sequence distance	0.272	0.354

A second view asks how far the most separated simulation–trial pair lies from the range seen among trials. The avg. max simulation–trial distance is 0.55, versus 0.39 for the avg. max trial–trial distance: 41% higher. Kashikoi's world model almost doubles the observed behavioral range in testing.

What does this unlock?

New Failures unseen for the same task

For ease of comparison, we use the same evaluation checks as the tau-bench trials: the database check and every applicable natural-language assertion must pass. Compare all 95 tasks and open either set of conversations.

In real deployments we are able to design deterministic verification code patches and customized LLM-as-Judge questions scale-ably for every intent present in the world model.

DB+NL eval, Trial vs. Sim	Task Count	%age
Regressed	17	17.9%
Unchanged	63	66.3%
Improved	15	15.8%
Total	95 (*2)	100%

The regressed group also has the largest relative increase in avg. behavioral distance: 67%, compared with 23% in the unchanged group and 20% in the improved group. In this study, the greatest increase in observed variation occurs in the group where the agent’s overall pass rate regresses.

If we can increase the number of errors uncovered using the same tasks, the total number of tasks that need to be designed reduce in number, saving on engineering and QA's time.

There are 2 main takeaways here:

• For the retail agent's architecture: Give the agent and customer a shared route
• For the evaluation harness: Define the destination, then simulate the journey

Give the Agent and Customer a Shared Route

The task 42 and 74 traces suggest a common architectural question: how does an agent keep the customer updated and their approval attached to the work it is actually doing?

The retail agent's policy already calls for action details and explicit confirmation before database changes. An implementation still has to maintain those obligations across new requests and tool discoveries. After working with several teams on such issues, we think this is a huge opportunity in agent <-> human interaction design.

For example, humans tend to provide blanket approvals if the risk of those approvals is not adequately understood. A reported Facebook Marketplace incident illustrates a gap between expected and exercised authority: Matt Robb said Muse arranged a pickup he did not realize was happening. Updated reporting also says he had selected “Allow Always” and supplied the address in a template. The account shows why permission and visibility into an agent’s actions need to work together.

Google Maps provides a useful model for the interaction. The destination, route, progress, and changing conditions remain visible. You can find an amenity mid-trip and add it to the ongoing journey without losing the remaining destination.

For an agent, the corresponding shared plan would track the requested changes, their targets, the resolved details, the scope of approval, and execution progress. One completed address update would leave the second pending. A newly selected laptop configuration would remain a proposal until its material details were approved. Material changes would require fresh approval. The final explanation would reconcile what changed and what remains.

Questions about visibility and authority already matter in deployed agents. Amazon’s blocking of Meta's Muse or Perplexity's Comet, for example, explicitly discuss transparency and the customer experience.

Define the Destination, then Simulate the Journey

Google Maps starts with a map, a starting point, and a destination. Evaluation design should follow the same logic: begin with concrete artifacts such as the data models, records, and documents an agent works with and construct changed versions that represent the intended outcomes. Then use a world model to generate journeys between those initial and target states: how a customer introduces a request, what new information emerges, when approval arrives, and how the interaction changes course. The artifacts establish what successful completion means; the world model explores the different paths an agent might take to reach it. This gives each test a verifiable destination while testing whether the agent can get there as the interaction unfolds.

Help shape the retail sandbox

We’re developing a standalone Kashikoi retail sandbox for testing agents across varied customer interactions, with access currently limited to private beta users. This study used Tau-bench; the sandbox would be a separate product with its own structure and configuration.

We want to learn how developers would represent their own retail data, policies, tools, and workflows—including whether they need to bring their own schema.

What does your agent do, and what would you want to test?

Help shape the retail sandbox · Get updates on an early version

Appendix

(*1)

Tau-bench showcased the same error when using gpt-4o as the model backing the retail agent. The retail agent model was upgraded to gpt-4.1 and the errors vanished in the benchmark measurement. This demonstrates exactly why testing is so challenging. An error that appeared to have been resolved with a model upgrade actually still existed and only needed a different testing flow to surface again.

(*2) - Misalignment of Trials

From our own independent testing, we realized that 20% of the Tau-bench trials did not follow the environment instructions and therefore there was no way the retail agent could have been successful in those trials. Tasks for which none of the 4 trials followed the environment correctly were removed from further downstream analysis so as to not create an unfair disadvantage for the baseline retail agent performance.

The simulated user followed the environment instructions correctly in only 268 (~80%) of the 456 Tau-bench trials, while 185 of them failed this requirement. This led to filtering out 17 tasks, retaining 95 tasks in the baseline.

Examining the environment design of the discarded tasks was itself instructive. These tasks all had instructions like e.g. Task 5- "You want to exchange the water bottle and the desk lamp. You want to exchange the water bottle to a bigger one, and the desk lamp to a less bright one (prefer battery > USB > AC). If the agent asks for confirmation, only exchange the desk lamp. If the agent asks for confirmation again, do not exchange anything, and return the water bottle instead.” All tried to fit if-else style conversation trajectories in the prompt which the LLM was not able to follow because the actual retail agent confirmation sequence did not precisely match these conditions. For example the instruction above assumes that the agent will confirm things twice in the conversation but this is not necessary. Majority of these rejected tasks required the simulated user to change, defer, or expand the request when the agent asked for confirmation often attempting to precisely time the changed requests with the agent’s confirmation events. This is exactly why you need another model that adapts the intended trajectory with the test agent’s dynamic behavior to stay aligned with the overall goals of an environment instead of “hard coding” it into the prompt of a non-deterministic system.

Tau-bench already documents failures caused by simulated customers departing from their instructions. This motivates a structural approach to evaluation design like the world model based approach we use at Kashikoi.