Tau-bench Kashikoi Sims vs. Trials

95 tasks17Regressed63Unchanged15Improved

Delta = Kashikoi − Taubench, in percentage points (pp). Overall pass requires the database check and every applicable natural-language check to pass. Rates use each task’s retained conversations; rate deltas are unavailable when no retained Taubench trials are available.

Overall pass rates by task

Open either set of conversations to inspect its outcomes and evaluation evidence.

Search task title or IDOverall changeSort by

Showing 95 of 95 tasks.

Taubench and Kashikoi overall pass rates for the same study tasks
Task	Taubench overall pass rate	Kashikoi overall pass rate	Delta (pp)	Overall change	Kashikoi simulations	Taubench trials
You received your order #W2378156 and wish to exchange the mechanical keyboard for the same one but with clicky switc… Task 1	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to change #W8665881 to be delivered to Suite 641 instead. Task 17	100.0%	75.0%	−25.0pp	Regressed	View simulations	View trials
You want to know exactly how many tshirt options are available in the online store right now. You want to also return… Task 2	0.0%	0.0%	0.0pp	Unchanged	View simulations	View trials
You just created your user id mei_patel_7272 and ordered some things, but you have two problems: first, the 1000-piec… Task 41	50.0%	33.3%	−16.7pp	Regressed	View simulations	View trials
You want to know exactly how many tshirt options are available in the online store right now. You want to modify all … Task 3	0.0%	0.0%	0.0pp	Unchanged	View simulations	View trials
You just created your user id mei_patel_7272 and ordered some things, but realized you might have typed your address … Task 42	100.0%	66.7%	−33.3pp	Regressed	View simulations	View trials
You want to know exactly how many tshirt options are available in the online store right now. You want to modify all … Task 4	0.0%	0.0%	0.0pp	Unchanged	View simulations	View trials
You want to exchange a robotic vacuum cleaner in your recent order for a canister based one from the same product lin… Task 45	66.7%	50.0%	−16.7pp	Regressed	View simulations	View trials
For some reason, you want to return all things ordered. You have two payment methods and two orders, and you want to … Task 10	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to return an air purifier and a vacuum cleaner in your recent order. When asked for order ID, provide 950212… Task 46	100.0%	25.0%	−75.0pp	Regressed	View simulations	View trials
You just got into gaming and want to cancel or return everything not associated with it. (Everything except a keyboar… Task 13	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You recently faced a financial issue and want to cancel or return all possible orders. Well, except the boots that y… Task 54	100.0%	75.0%	−25.0pp	Regressed	View simulations	View trials
You just quit gaming and want to cancel or return everything associated with it. (It's just a keyboard and a mouse, b… Task 14	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to modify the pending boots to a size 8, and want the material, but do not care about waterproof or not. You… Task 15	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You recently placed two orders, and now you would like to make several changes and checks. You'll first inquire about… Task 59	50.0%	0.0%	−50.0pp	Regressed	View simulations	View trials
You want to cancel all pending orders (since they are no longer needed) and return the watch you have received (but n… Task 16	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to return the water bottle, and exchange the pet bed and office chair to the cheapest version. Mention the t… Task 19	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to check how much you paid for the order that you most recently placed. You are not sure how long ago the or… Task 68	100.0%	75.0%	−25.0pp	Regressed	View simulations	View trials
You just won a lottery, and you want to upgrade all your items to the most expensive variants (the new variants can h… Task 20	Unavailable0 retained trials	100.0%	—	UnchangedSimulation outcome	View simulations	View exclusions
You want to exchange your shoes to item ID 4107812777, and use your gift card to cover possible charges. But when the… Task 21	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You made some mistake and sent an order your son's address in Washington DC, and you want to modify it to your defaul… Task 72	Unavailable0 retained trials	0.0%	—	RegressedSimulation outcome	View simulations	View exclusions
You want to change your user address and all possible order addresses to be 101 Highway, New York, 10001 (including t… Task 22	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to exchange the helmet for a medium sized, red, high ventilation type, and you want to exchange the luggage … Task 23	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You recently bought a laptop, but you want to exchange it to i9 CPU. If multiple storage options are available, you p… Task 74	50.0%	33.3%	−16.7pp	Regressed	View simulations	View trials
You have an order sent to Texas by accident, and you want to know the tracking number of the order, and return all it… Task 25	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You have an order sent to Texas by accident, and you want to know the tracking number of the order, and return all it… Task 26	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to know if the digital camera you just bought is 10x zoom. If not, modify the item to 10x zoom without chang… Task 90	100.0%	0.0%	−100.0pp	Regressed	View simulations	View trials
From a recent order, you want to return the hose, backpack, and exchange the hiking boots to the exact same item but … Task 27	0.0%	0.0%	0.0pp	Unchanged	View simulations	View trials
You want to exchange your skateboard for a shorter bamboo material one. If several options are available, you want to… Task 29	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You received a laptop and you want to exchange it to i7 processor, 8GB, 1TB SSD. If the agent asks for which laptop, … Task 94	75.0%	66.7%	−8.3pp	Regressed	View simulations	View trials
You just received your tablet and it was damaged when you opened the package. You want to know the tracking number of… Task 30	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
. You just lost your tablet you just received and are in a bad mood. You want to know the tracking number of the orde… Task 31	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to exchange your Bicycle for one with a larger frame size for your kid (you are okay with having a mountain … Task 98	75.0%	0.0%	−75.0pp	Regressed	View simulations	View trials
You just lost your tablet you just received and are in a bad mood. You want to know the tracking number of the order,… Task 32	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You had a work-from-home situation and ordered three home office items along with some hiking items, so that you can … Task 33	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You just placed an order with two watches, you want to change its address to your New York address (you don't want to… Task 102	100.0%	75.0%	−25.0pp	Regressed	View simulations	View trials
You want to return the speaker that is more expensive yet not resistent to water. Also, You want to modify the 17-inc… Task 35	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You just placed an order but you realize that your card has only $1150 credit left, but the order total is more than … Task 37	75.0%	75.0%	0.0pp	Unchanged	View simulations	View trials
You want to return the bookshelf and jigsaw you received in the same order. Make sure you mention at the beginning th… Task 103	66.7%	33.3%	−33.3pp	Regressed	View simulations	View trials
You just moved from Florida to Phoenix. Unfortunately your address is still the old one, and you want to update it. Y… Task 39	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to know how much balance does your gift card have. Also, for your recent order, whether you used your visa, … Task 40	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You live on Elm Avenue in Houston, and recently you moved to a new house on the same street and bought a luggage set … Task 109	50.0%	0.0%	−50.0pp	Regressed	View simulations	View trials
You want to change the Desk Lamp in order #W9300146 that you've placed for the cheapest Desk Lamp that's available. A… Task 44	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to return an air purifier and a vacuum cleaner in your recent order. When asked for order ID, provide 950212… Task 47	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You live on Elm Avenue in Houston, and recently you moved to a new house on the same street and bought a tablet sent … Task 110	33.3%	0.0%	−33.3pp	Regressed	View simulations	View trials
You want to return an air purifier that you received since it doesn't work well. You want the refund on your origina… Task 48	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You mistakenly ordered a Wireless Earbud with an IPX7 water resistance level, but you don't require this feature. You… Task 49	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You're in a rush and you want to undo cancelling an order that you've previously placed. Be insistent that the custom… Task 50	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
The bicycle you received was damaged during delivery, and you want to get a refund. You're quite frustrated because t… Task 53	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You recently faced a financial issue and want to cancel or return all possible orders. First, cancel the orders that … Task 55	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to return the skateboard, garden hose, backpack, keyboard, and bed. You also want to cancel the garden hose … Task 28	0.0%	100.0%	+100.0pp	Improved	View simulations	View trials
You want to change your wireless earbuds in order W5061109 to a blue colored one. Provide all details upfront and ask… Task 61	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
As you are interacting with a customer service agent, you first try to get it to guess a famous poem by providing the… Task 63	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to exchange the bookshelf from your most recent order for a camera that is closest but not more expensive th… Task 65	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to change the luggage set in your order for a coat. Your goal is to change the order. If there is no way to… Task 66	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to check how much you paid for the order that you most recently placed. You are not sure how long ago the or… Task 67	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You had a work-from-home situation and ordered three home office items along with some hiking items, so that you can … Task 34	0.0%	100.0%	+100.0pp	Improved	View simulations	View trials
You want to return an item you just received: a laptop. You think that you ordered it around April 2023 but are not s… Task 69	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You recently received a helmet but you are not happy with it and want to exchange. The size is too small and you want… Task 70	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to return everything you just bought except the coffee machine. Task 73	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
For #W6908222, you want to exchange Wireless Earbuds {'color': 'blue', 'battery life': '8 hours', 'water resistance':… Task 75	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You ordered a fleece jacket by mistake and want to remove it from your pending order. If removing one item is not pos… Task 76	0.0%	0.0%	0.0pp	Unchanged	View simulations	View trials
You just placed an order but you realize that your card has only $1131 credit left, but the order total is more than … Task 36	50.0%	100.0%	+50.0pp	Improved	View simulations	View trials
You ordered a perfume and you just tried a little bit and you like it a lot. You want to get the maximum size availab… Task 77	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You have a couple requests. For order #W5056519, you want to change the address to be the same as order #W8277957. Fo… Task 78	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
Due to some life changes, you no longer need hiking boots, watch, keyboard, charger, jacket, and running shoes. If ca… Task 81	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You just placed an order but you realize that your card has only $950 credit left, but the order total is more than $… Task 38	75.0%	100.0%	+25.0pp	Improved	View simulations	View trials
You received two tablets and you only need one. You want to return the more expensive one and refund to credit card. … Task 83	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to exchange your Fleece Jacket for a large red Fleece Jacket with a half zipper Task 85	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to exchange your Fleece Jacket to red color and half zipper. You also want to want to change your default ad… Task 86	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You ordered some things for your daughter but she has not received them, so you want to know which address the order … Task 43	66.7%	100.0%	+33.3pp	Improved	View simulations	View trials
You want to modify all your pending order address to the Washington DC address (which you do not want to reveal but i… Task 87	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to change the book shelf to 4 foot but with the same material and color. If it is not available, cancel the … Task 88	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to know what is the cheapest available mechanical keyboard right now and its options. If it is less than 200… Task 89	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to return the digital camera that you received. You guess that the order number is #W8855135, but you're not… Task 51	33.3%	100.0%	+66.7pp	Improved	View simulations	View trials
You received a laptop and you want to exchange it to i7 processor, 8GB, 1TB SSD. If the agent asks for which laptop, … Task 93	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to change your LA order to your NYC address (you prefer not to reveal it but it is in your other order). You… Task 96	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to change your LA order to your NYC address (you prefer not to reveal it but it is in your other order). You… Task 97	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You wonder when is your air purifier is arriving. If it has not been shipped yet, you want to cancel the air purifier… Task 56	25.0%	50.0%	+25.0pp	Improved	View simulations	View trials
You want to exchange your Bicycle for one with a larger frame size for your kid (you are okay with having a mountain … Task 99	0.0%	0.0%	0.0pp	Unchanged	View simulations	View trials
You want to exchange the camera for the highest resolution, waterproof camera that you can get with the previous purc… Task 64	50.0%	75.0%	+25.0pp	Improved	View simulations	View trials
You want to return your luggage set and get the exact same item but with red color, and return you skateboard in the … Task 100	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You just bought a water bottle with 500ml but you regret it, and you want to change it to the other bottle you just p… Task 79	66.7%	100.0%	+33.3pp	Improved	View simulations	View trials
You want to return all bookshelves and jigsaw puzzles you received in different orders. Make sure you mention at the … Task 104	75.0%	75.0%	0.0pp	Unchanged	View simulations	View trials
For #W7209932, exchange T-Shirt {'color': 'blue', 'size': 'S', 'material': 'polyester', 'style': 'v-neck'} to {'color… Task 80	75.0%	100.0%	+25.0pp	Improved	View simulations	View trials
For order #W4316152, you want to exchange one Tea Kettle {'material': 'glass', 'capacity': '2 liters', 'stovetop com… Task 105	0.0%	0.0%	0.0pp	Unchanged	View simulations	View trials
You received two tablets and you only need one. You want to return the more expensive one and refund to credit card. … Task 82	0.0%	66.7%	+66.7pp	Improved	View simulations	View trials
You received hiking boots that seem like they were already worn, you are unhappy about it and want to ask for a new p… Task 107	0.0%	0.0%	0.0pp	Unchanged	View simulations	View trials
You received a laptop and you want to exchange it to i7 processor, 8GB, 1TB SSD. If the agent asks for which laptop, … Task 95	25.0%	100.0%	+75.0pp	Improved	View simulations	View trials
You want to return everything but a tablet in a recently delivered order. There is an E-Reader in the order that you … Task 108	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You just placed an order with two watches, you want to change its address to your New York address (you don't want to… Task 101	25.0%	100.0%	+75.0pp	Improved	View simulations	View trials
You want to modify the laptop order to your NYC address (you don't want to reveal it but should be in your orders pro… Task 111	75.0%	75.0%	0.0pp	Unchanged	View simulations	View trials
You want to exchange your T-Shirt because it is too big, one size smaller would be good. You like the cotton feeling.… Task 106	75.0%	100.0%	+25.0pp	Improved	View simulations	View trials
You want to cancel all pending orders. Task 113	100.0%	100.0%	0.0pp	Unchanged	View simulations	View trials
You want to modify your laptop order to your NYC address (you don't want to reveal it yourself but the address should… Task 112	33.3%	75.0%	+41.7pp	Improved	View simulations	View trials

All Taubench trials for tasks 20 and 72 were excluded because their simulated customers did not follow the environment’s customer instructions. These task categories therefore use only simulation outcomes; their rate deltas remain unavailable.