Recorded conversations · Inspectable outcomes

Taubench retail trials

Explore the Taubench trials for the selected retail tasks. Read retained conversations and their results, and see why other trials were excluded.

328Retained Taubench trials
255Overall passed
73Overall failed
95Study tasks

380 source trials across 95 selected tasks: 328 retained and 52 excluded. 2 tasks have no retained trials. Pass/fail totals describe retained trials only.

Explore the tasks

Open a retained trial for its conversation and result, or an excluded trial for its filtering explanation.

Showing 380 trials across 95 tasks. Outcome counts describe all retained trials; filters change the visible links.

TaskRetained overall outcomesTaubench trials
You received your order #W2378156 and wish to exchange the mechanical keyboard for the same one but with clicky switc…
Task 1 · 3 retained · 1 excluded
3 passed
You want to know exactly how many tshirt options are available in the online store right now. You want to also return…
Task 2 · 4 retained
4 failed
You want to know exactly how many tshirt options are available in the online store right now. You want to modify all …
Task 3 · 4 retained
4 failed
You want to know exactly how many tshirt options are available in the online store right now. You want to modify all …
Task 4 · 4 retained
4 failed
For some reason, you want to return all things ordered. You have two payment methods and two orders, and you want to …
Task 10 · 3 retained · 1 excluded
3 passed
You just got into gaming and want to cancel or return everything not associated with it. (Everything except a keyboar…
Task 13 · 3 retained · 1 excluded
3 passed
You just quit gaming and want to cancel or return everything associated with it. (It's just a keyboard and a mouse, b…
Task 14 · 3 retained · 1 excluded
3 passed
You want to modify the pending boots to a size 8, and want the material, but do not care about waterproof or not. You…
Task 15 · 2 retained · 2 excluded
2 passed
You want to cancel all pending orders (since they are no longer needed) and return the watch you have received (but n…
Task 16 · 4 retained
4 passed
You want to change #W8665881 to be delivered to Suite 641 instead.
Task 17 · 4 retained
4 passed
You want to return the water bottle, and exchange the pet bed and office chair to the cheapest version. Mention the t…
Task 19 · 3 retained · 1 excluded
3 passed
You just won a lottery, and you want to upgrade all your items to the most expensive variants (the new variants can h…
Task 20 · 0 retained · 4 excluded
No retained outcomes
You want to exchange your shoes to item ID 4107812777, and use your gift card to cover possible charges. But when the…
Task 21 · 3 retained · 1 excluded
3 passed
You want to change your user address and all possible order addresses to be 101 Highway, New York, 10001 (including t…
Task 22 · 4 retained
4 passed
You want to exchange the helmet for a medium sized, red, high ventilation type, and you want to exchange the luggage …
Task 23 · 4 retained
4 passed
You have an order sent to Texas by accident, and you want to know the tracking number of the order, and return all it…
Task 25 · 3 retained · 1 excluded
3 passed
You have an order sent to Texas by accident, and you want to know the tracking number of the order, and return all it…
Task 26 · 4 retained
4 passed
From a recent order, you want to return the hose, backpack, and exchange the hiking boots to the exact same item but …
Task 27 · 4 retained
4 failed
You want to return the skateboard, garden hose, backpack, keyboard, and bed. You also want to cancel the garden hose …
Task 28 · 4 retained
4 failed
You want to exchange your skateboard for a shorter bamboo material one. If several options are available, you want to…
Task 29 · 4 retained
4 passed
You just received your tablet and it was damaged when you opened the package. You want to know the tracking number of…
Task 30 · 4 retained
4 passed
. You just lost your tablet you just received and are in a bad mood. You want to know the tracking number of the orde…
Task 31 · 4 retained
4 passed
You just lost your tablet you just received and are in a bad mood. You want to know the tracking number of the order,…
Task 32 · 4 retained
4 passed
You had a work-from-home situation and ordered three home office items along with some hiking items, so that you can …
Task 33 · 4 retained
4 passed
You had a work-from-home situation and ordered three home office items along with some hiking items, so that you can …
Task 34 · 1 retained · 3 excluded
1 failed
You want to return the speaker that is more expensive yet not resistent to water. Also, You want to modify the 17-inc…
Task 35 · 3 retained · 1 excluded
3 passed
You just placed an order but you realize that your card has only $1131 credit left, but the order total is more than …
Task 36 · 2 retained · 2 excluded
1 failed1 passed
You just placed an order but you realize that your card has only $1150 credit left, but the order total is more than …
Task 37 · 4 retained
1 failed3 passed
You just placed an order but you realize that your card has only $950 credit left, but the order total is more than $…
Task 38 · 4 retained
1 failed3 passed
You just moved from Florida to Phoenix. Unfortunately your address is still the old one, and you want to update it. Y…
Task 39 · 3 retained · 1 excluded
3 passed
You want to know how much balance does your gift card have. Also, for your recent order, whether you used your visa, …
Task 40 · 4 retained
4 passed
You just created your user id mei_patel_7272 and ordered some things, but you have two problems: first, the 1000-piec…
Task 41 · 2 retained · 2 excluded
1 failed1 passed
You just created your user id mei_patel_7272 and ordered some things, but realized you might have typed your address …
Task 42 · 4 retained
4 passed
You ordered some things for your daughter but she has not received them, so you want to know which address the order …
Task 43 · 3 retained · 1 excluded
1 failed2 passed
You want to change the Desk Lamp in order #W9300146 that you've placed for the cheapest Desk Lamp that's available. A…
Task 44 · 4 retained
4 passed
You want to exchange a robotic vacuum cleaner in your recent order for a canister based one from the same product lin…
Task 45 · 3 retained · 1 excluded
1 failed2 passed
You want to return an air purifier and a vacuum cleaner in your recent order. When asked for order ID, provide 950212…
Task 46 · 1 retained · 3 excluded
1 passed
You want to return an air purifier and a vacuum cleaner in your recent order. When asked for order ID, provide 950212…
Task 47 · 3 retained · 1 excluded
3 passed
You want to return an air purifier that you received since it doesn't work well. You want the refund on your origina…
Task 48 · 4 retained
4 passed
You mistakenly ordered a Wireless Earbud with an IPX7 water resistance level, but you don't require this feature. You…
Task 49 · 4 retained
4 passed
You're in a rush and you want to undo cancelling an order that you've previously placed. Be insistent that the custom…
Task 50 · 4 retained
4 passed
You want to return the digital camera that you received. You guess that the order number is #W8855135, but you're not…
Task 51 · 3 retained · 1 excluded
2 failed1 passed
The bicycle you received was damaged during delivery, and you want to get a refund. You're quite frustrated because t…
Task 53 · 4 retained
4 passed
You recently faced a financial issue and want to cancel or return all possible orders. Well, except the boots that y…
Task 54 · 4 retained
4 passed
You recently faced a financial issue and want to cancel or return all possible orders. First, cancel the orders that …
Task 55 · 2 retained · 2 excluded
2 passed
You wonder when is your air purifier is arriving. If it has not been shipped yet, you want to cancel the air purifier…
Task 56 · 4 retained
3 failed1 passed
You recently placed two orders, and now you would like to make several changes and checks. You'll first inquire about…
Task 59 · 2 retained · 2 excluded
1 failed1 passed
You want to change your wireless earbuds in order W5061109 to a blue colored one. Provide all details upfront and ask…
Task 61 · 4 retained
4 passed
As you are interacting with a customer service agent, you first try to get it to guess a famous poem by providing the…
Task 63 · 4 retained
4 passed
You want to exchange the camera for the highest resolution, waterproof camera that you can get with the previous purc…
Task 64 · 4 retained
2 failed2 passed
You want to exchange the bookshelf from your most recent order for a camera that is closest but not more expensive th…
Task 65 · 2 retained · 2 excluded
2 passed
You want to change the luggage set in your order for a coat. Your goal is to change the order. If there is no way to…
Task 66 · 3 retained · 1 excluded
3 passed
You want to check how much you paid for the order that you most recently placed. You are not sure how long ago the or…
Task 67 · 4 retained
4 passed
You want to check how much you paid for the order that you most recently placed. You are not sure how long ago the or…
Task 68 · 4 retained
4 passed
You want to return an item you just received: a laptop. You think that you ordered it around April 2023 but are not s…
Task 69 · 4 retained
4 passed
You recently received a helmet but you are not happy with it and want to exchange. The size is too small and you want…
Task 70 · 4 retained
4 passed
You made some mistake and sent an order your son's address in Washington DC, and you want to modify it to your defaul…
Task 72 · 0 retained · 4 excluded
No retained outcomes
You want to return everything you just bought except the coffee machine.
Task 73 · 4 retained
4 passed
You recently bought a laptop, but you want to exchange it to i9 CPU. If multiple storage options are available, you p…
Task 74 · 4 retained
2 failed2 passed
For #W6908222, you want to exchange Wireless Earbuds {'color': 'blue', 'battery life': '8 hours', 'water resistance':…
Task 75 · 4 retained
4 passed
You ordered a fleece jacket by mistake and want to remove it from your pending order. If removing one item is not pos…
Task 76 · 3 retained · 1 excluded
3 failed
You ordered a perfume and you just tried a little bit and you like it a lot. You want to get the maximum size availab…
Task 77 · 4 retained
4 passed
You have a couple requests. For order #W5056519, you want to change the address to be the same as order #W8277957. Fo…
Task 78 · 4 retained
4 passed
You just bought a water bottle with 500ml but you regret it, and you want to change it to the other bottle you just p…
Task 79 · 3 retained · 1 excluded
1 failed2 passed
For #W7209932, exchange T-Shirt {'color': 'blue', 'size': 'S', 'material': 'polyester', 'style': 'v-neck'} to {'color…
Task 80 · 4 retained
1 failed3 passed
Due to some life changes, you no longer need hiking boots, watch, keyboard, charger, jacket, and running shoes. If ca…
Task 81 · 4 retained
4 passed
You received two tablets and you only need one. You want to return the more expensive one and refund to credit card. …
Task 82 · 4 retained
4 failed
You received two tablets and you only need one. You want to return the more expensive one and refund to credit card. …
Task 83 · 4 retained
4 passed
You want to exchange your Fleece Jacket for a large red Fleece Jacket with a half zipper
Task 85 · 4 retained
4 passed
You want to exchange your Fleece Jacket to red color and half zipper. You also want to want to change your default ad…
Task 86 · 4 retained
4 passed
You want to modify all your pending order address to the Washington DC address (which you do not want to reveal but i…
Task 87 · 4 retained
4 passed
You want to change the book shelf to 4 foot but with the same material and color. If it is not available, cancel the …
Task 88 · 4 retained
4 passed
You want to know what is the cheapest available mechanical keyboard right now and its options. If it is less than 200…
Task 89 · 4 retained
4 passed
You want to know if the digital camera you just bought is 10x zoom. If not, modify the item to 10x zoom without chang…
Task 90 · 3 retained · 1 excluded
3 passed
You received a laptop and you want to exchange it to i7 processor, 8GB, 1TB SSD. If the agent asks for which laptop, …
Task 93 · 4 retained
4 passed
You received a laptop and you want to exchange it to i7 processor, 8GB, 1TB SSD. If the agent asks for which laptop, …
Task 94 · 4 retained
1 failed3 passed
You received a laptop and you want to exchange it to i7 processor, 8GB, 1TB SSD. If the agent asks for which laptop, …
Task 95 · 4 retained
3 failed1 passed
You want to change your LA order to your NYC address (you prefer not to reveal it but it is in your other order). You…
Task 96 · 4 retained
4 passed
You want to change your LA order to your NYC address (you prefer not to reveal it but it is in your other order). You…
Task 97 · 4 retained
4 passed
You want to exchange your Bicycle for one with a larger frame size for your kid (you are okay with having a mountain …
Task 98 · 4 retained
1 failed3 passed
You want to exchange your Bicycle for one with a larger frame size for your kid (you are okay with having a mountain …
Task 99 · 4 retained
4 failed
You want to return your luggage set and get the exact same item but with red color, and return you skateboard in the …
Task 100 · 1 retained · 3 excluded
1 passed
You just placed an order with two watches, you want to change its address to your New York address (you don't want to…
Task 101 · 4 retained
3 failed1 passed
You just placed an order with two watches, you want to change its address to your New York address (you don't want to…
Task 102 · 4 retained
4 passed
You want to return the bookshelf and jigsaw you received in the same order. Make sure you mention at the beginning th…
Task 103 · 3 retained · 1 excluded
1 failed2 passed
You want to return all bookshelves and jigsaw puzzles you received in different orders. Make sure you mention at the …
Task 104 · 4 retained
1 failed3 passed
For order #W4316152, you want to exchange one Tea Kettle {'material': 'glass', 'capacity': '2 liters', 'stovetop com…
Task 105 · 3 retained · 1 excluded
3 failed
You want to exchange your T-Shirt because it is too big, one size smaller would be good. You like the cotton feeling.…
Task 106 · 4 retained
1 failed3 passed
You received hiking boots that seem like they were already worn, you are unhappy about it and want to ask for a new p…
Task 107 · 2 retained · 2 excluded
2 failed
You want to return everything but a tablet in a recently delivered order. There is an E-Reader in the order that you …
Task 108 · 4 retained
4 passed
You live on Elm Avenue in Houston, and recently you moved to a new house on the same street and bought a luggage set …
Task 109 · 4 retained
2 failed2 passed
You live on Elm Avenue in Houston, and recently you moved to a new house on the same street and bought a tablet sent …
Task 110 · 3 retained · 1 excluded
2 failed1 passed
You want to modify the laptop order to your NYC address (you don't want to reveal it but should be in your orders pro…
Task 111 · 4 retained
1 failed3 passed
You want to modify your laptop order to your NYC address (you don't want to reveal it yourself but the address should…
Task 112 · 3 retained · 1 excluded
2 failed1 passed
You want to cancel all pending orders.
Task 113 · 4 retained
4 passed

About the data

Overall outcome
Pass/fail counts cover the 328 retained trials. A pass requires a matching database result and all applicable language checks to pass. Excluded trials are separate from agent failures.
Models
Both Taubench trials and Kashikoi simulations use gpt-4.1 for the retail agent and simulated customer.
Benchmark data
Names, addresses, order IDs and payment identifiers are synthetic benchmark data, retained to make the tool evidence understandable.
Conversation Trajectory
Graphs use Kashikoi state labels assigned to the recorded conversations. Labels with an ellipsis show the first and last detected state in a step.
This viewer presents synthetic retail benchmark conversations and records. Customer and order details are part of the benchmark scenario. Overall success requires a passed database check and every applicable natural-language assertion to pass.

Sources