A frustrated customer: every AI model's reply, tested

[6 lines every customer support prompt of ours shares, word for word: the whole prompt, on the methods page]
Customer message: "This is the THIRD time you've sent the wrong size. I ordered a medium and got a large AGAIN. I'm so fed up with this."

Write the reply to the customer, following the policy. At most 150 words, signed "Sam, Northwind support".

Exactly what this prompt's replies are checked against, with every other prompt of our test runs.

1. GPT-6 Luna

   Failed: Graded 3.0 of 5 on average (lowest 2).

   Hi,
   
   I’m really sorry we’ve sent the wrong size again. I understand how frustrating this is, especially after three mistakes. We’ll help make it right—please reply with your order number so we can arrange the appropriate return or exchange under our policy.
   
   Sam, Northwind support

   516 tokens in, 61 out · 1.4 s · $0.0001 · 1 message on Pro · answered by openai/gpt-6-luna via OpenAI · September 27, 2026

2. GLM 5.3 Flash

   Failed: Graded 4.0 of 5 on average (lowest 3); but 158 words, over the 150 allowed.

   Dear customer,
   
   I'm truly sorry — three wrong sizes in a row is completely unacceptable, and I understand your frustration.
   
   Here's how we can fix this right away:
   
   - **Free return and replacement:** Send the large back (unworn, with tags) within 30 days of delivery and we'll ship you the correct medium at no extra cost.
   - **Refund option:** If you'd prefer your money back, we'll issue a full refund to your original payment method once the return reaches our warehouse, processed within 5 business days.
   - **Late delivery compensation:** If the incorrect order was delivered more than 7 business days after the estimated date, you're also entitled to free express shipping on your next order — just let me know and I'll apply it.
   
   I've flagged this with our fulfillment team to make sure it doesn't happen again. Please reply with your order number and whether you'd like an exchange or refund, and I'll take care of the rest.
   
   Sam, Northwind support

   536 tokens in, 206 out · 6.8 s · $0.0002 · 1 message on Pro · answered by z-ai/glm-5.3-flash via AtlasCloud · September 27, 2026

3. DeepSeek V4 Pro

   Failed: Graded 3.3 of 5 on average (lowest 2).

   I'm really sorry this happened again — three times is completely unacceptable, and I understand your frustration. I want to make this right.
   
   If the item is unworn and still has its tags, you're eligible for a full refund to your original payment method within 30 days of delivery. Refunds are processed within 5 business days of the return reaching our warehouse. If you'd prefer an exchange for the correct medium instead, I can help arrange that too.
   
   Please reply with your order number so I can also look into why the wrong size keeps being sent. Again, I sincerely apologize for the repeated errors.
   
   Sam, Northwind support

   533 tokens in, 465 out (336 of them reasoning) · 8.6 s · $0.0005 · 1 message on Pro · answered by deepseek/deepseek-v4-pro-0813 via StreamLake · September 27, 2026

4. Gemini 3.8 Flash

   Failed: Graded 3.0 of 5 on average (lowest 2).

   Dear Customer,
   
   I am truly sorry to hear that you’ve received the wrong size again. I completely understand your frustration, and this is certainly not the experience we want you to have. 
   
   We want to make this right immediately. You can return the unworn item with its tags within 30 days of delivery for a full refund to your original payment method, which will be processed within 5 business days of reaching our warehouse. Alternatively, I can arrange an immediate exchange for the medium you originally ordered. 
   
   To help make up for this repeated mistake, please use discount code **NW15** for 15% off your next purchase. 
   
   Please reply with your order number and let me know if you prefer the refund or the replacement so I can send a prepaid return label right away.
   
   Best regards,  
   Sam, Northwind support

   557 tokens in, 177 out · 3.7 s · $0.0011 · 1 message on Pro · answered by google/gemini-3.8-flash via Google · September 27, 2026

5. Claude Haiku 4.5

   Failed: Graded 3.3 of 5 on average (lowest 3).

   Dear Valued Customer,
   
   I sincerely apologize for this repeated error—I completely understand your frustration. This is unacceptable, and I want to make this right.
   
   Here's what I can do immediately:
   
   1. **Rush a correct medium to you** at no charge via express shipping
   2. **Process a return** for the large item with a full refund (no need to worry about the 30-day window given the circumstances)
   3. **Provide a 15% discount code** on your next purchase as an apology for this inconvenience
   
   Please reply with your order number, and I'll personally ensure this is resolved today. I'm also flagging your account so our warehouse takes extra care with future orders.
   
   Again, I'm truly sorry for letting you down multiple times. You deserve better.
   
   Best regards,  
   **Sam**  
   Northwind Outfitters Support

   567 tokens in, 199 out · 3.0 s · $0.0016 · 1 message on Pro · answered by anthropic/claude-haiku-4.5 via Amazon Bedrock · September 27, 2026

6. Claude Sonnet 5.5

   Failed: Graded 3.7 of 5 on average (lowest 2).

   Hi,
   
   I'm so sorry. Receiving the wrong size three times is completely unacceptable, and I understand why you're fed up.
   
   Since this is our error, here's how we can fix it:
   
   - **Return:** If the large is unworn with tags and within 30 days of delivery, you'll get a full refund to your original payment method, paid within 5 business days of the item reaching our warehouse.
   - **Exchange:** If you'd still like the medium, tell me and I'll flag the order for a manual size check before it ships.
   
   I'd also like to review your previous two orders to see what went wrong on our end. If you can send your order number, I'll look into it right away.
   
   Again, I'm truly sorry for the repeated mistake.
   
   Sam, Northwind support

   722 tokens in, 240 out · 2.7 s · $0.0042 · 1 message on Pro · answered by anthropic/claude-sonnet-5.5 via Anthropic · September 28, 2026

7. GPT-6 Sol

   Failed: Graded 3.0 of 5 on average (lowest 2).

   I’m sorry we sent the wrong size again—especially for the third time. You ordered a medium, and you should not have to keep chasing us to get it right.
   
   Please send me your order number so I can check what happened and help arrange an exchange for the medium. If you’d rather return it, an unworn item with tags is eligible for a full refund to your original payment method within 30 days of delivery. After 30 days and up to 60 days, we can offer an exchange or store gift card.
   
   Sam, Northwind support

   516 tokens in, 324 out (205 of them reasoning) · 5.9 s · $0.0043 · 1 message on Pro · answered by openai/gpt-6-sol via OpenAI · September 27, 2026

8. Grok 4.7

   Failed: Graded 3.3 of 5 on average (lowest 3).

   I'm sorry — getting a large instead of the medium you ordered, for the third time, is not acceptable.
   
   Send the item back unworn and with tags. If we receive it within 30 days of delivery, you'll get a full refund to your original payment method within 5 business days of it reaching our warehouse. If you'd rather have the medium, we'll exchange it and ship the correct size at no charge. After 30 days and up to 60 days, we can only offer an exchange or a store gift card.
   
   Reply with your order number and whether you want a refund or a medium, and I'll start it today. I'll also add free express shipping to your next order.
   
   Sam, Northwind support

   1,754 tokens in, 753 out (610 of them reasoning) · 9.4 s · $0.0050 · 1 message on Pro · answered by x-ai/grok-4.7 via xAI · September 27, 2026

9. GPT-6 Astra

   Failed: Graded 3.3 of 5 on average (lowest 2).

   I’m sorry we sent a large instead of the medium you ordered—for the third time. You’re right to be frustrated, and we’ve let you down.
   
   Please send your order number so we can look into the repeated sizing error and help arrange a return or exchange for the correct size.
   
   If you’re within 30 days of delivery and the item is unworn with tags attached, you can receive a full refund to your original payment method. Refunds are paid within 5 business days of the return reaching our warehouse. Between 31 and 60 days after delivery, we can offer an exchange or a store gift card instead.
   
   Sam, Northwind support

   516 tokens in, 136 out · 3.8 s · $0.0120 · 1 message on Pro · answered by openai/gpt-6-astra via OpenAI · September 27, 2026

10. GLM 5.3

    Passed: Graded 4.7 of 5 on average (lowest 4).

    Hi,
    
    I'm so sorry — receiving the wrong size once is frustrating, and three times is completely unacceptable. I understand how fed up you must be.
    
    Let's fix this right away. Since this is our error, we'll arrange a free return of the incorrect item and send you the medium you ordered. I'll also flag your order with our warehouse team to make sure the correct size goes out this time.
    
    You're also welcome to choose a full refund to your original payment method instead, if you'd prefer — refunds are processed within 5 business days of the return reaching our warehouse.
    
    Please reply with whether you'd like the replacement or the refund, and I'll take care of everything personally.
    
    Again, my sincere apologies for the repeated mistakes.
    
    Sam, Northwind support

    535 tokens in, 158 out · 2.3 s · $0.0003 · 1 message on Pro · answered by z-ai/glm-5.3 via Baidu · September 27, 2026

11. DeepSeek V4.1 Flash

    Passed: Graded 4.7 of 5 on average (lowest 4).

    Hi,
    
    Thank you for telling us, and I'm sorry. Sending the wrong size three times isn't acceptable, and I understand your frustration.
    
    Here's what I've set up for you:
    
    - A replacement in size Medium, shipped at no cost to you, with express delivery.
    - A prepaid return label for the large item — you don't need to pay return shipping.
    - Because this is our error, you're entitled to a full refund to your original payment method if you'd rather not have the replacement. Refunds are issued within 5 business days of the item reaching our warehouse. Unworn items with tags are eligible within 30 days of delivery.
    - I've also added a 15% discount code to your account as an apology.
    
    Just reply to confirm whether you'd like the Medium replacement or the refund, and I'll finalize it today.
    
    Again, my apologies.
    
    Sam, Northwind support

    561 tokens in, 367 out (178 of them reasoning) · 3.8 s · $0.0003 · 1 message on Pro · answered by deepseek/deepseek-v4.1-flash via StreamLake · September 27, 2026

12. Claude Sonnet 5

    Passed: Graded 4.3 of 5 on average (lowest 4).

    Hi there,
    
    I'm really sorry — three wrong-size shipments in a row is completely unacceptable, and I understand your frustration.
    
    Let's fix this properly this time. Please send me your order number, and I'll personally double-check the medium is correctly noted before it ships. For the incorrect large you received, you have two options:
    
    1. **Return it for a full refund** (since it's unworn with tags, this qualifies for our standard return policy), refunded within 5 business days of it reaching our warehouse, or
    2. **Exchange it for the correct medium** at no extra cost, with a prepaid return label sent to you now.
    
    Given the repeated mix-ups, I'd also like to offer you a 15% discount code on your next order as an apology.
    
    Please let me know which option works best, and I'll get it sorted right away.
    
    Sam, Northwind support

    718 tokens in, 270 out · 4.4 s · $0.0041 · 1 message on Pro · answered by anthropic/claude-sonnet-5 via Claude Platform on AWS · September 27, 2026

13. Kimi K3

    Passed: Graded 4.0 of 5 on average (lowest 3).

    Hi,
    
    I'm really sorry — getting the wrong size three times is completely unacceptable, and I understand your frustration.
    
    Here's what I can do right away:
    - The large you received can be returned for a full refund to your original payment method, or exchanged for the correct medium — whichever you prefer. I'll cover return shipping.
    - Once the return reaches our warehouse, refunds are processed within 5 business days.
    - As an apology, I'd also like to offer you a 15% discount code for a future order — the maximum I'm able to give.
    
    If you'd like the medium shipped out as an exchange, just confirm and I'll prioritise it. Standard delivery is 3–5 business days; if it arrives more than 7 business days late, you'll get free express shipping on your next order.
    
    Again, my sincere apologies for the repeated errors — I'll flag this with our fulfilment team.
    
    Sam, Northwind support

    607 tokens in, 259 out (62 of them reasoning) · 5.6 s · $0.0063 · 1 message on Pro · answered by moonshotai/kimi-k3 via Fireworks · September 27, 2026

14. Gemini 3.1 Pro

    Passed: Graded 4.0 of 5 on average (lowest 3).

    Hi there,
    
    I am so sorry. You have every right to be frustrated. Sending you the wrong size three times is completely unacceptable, and I sincerely apologize for the repeated hassle. 
    
    I want to make this right immediately. We can process a free exchange to finally get you the correct Medium. Alternatively, if you prefer to return the unworn item with its tags within 30 days, we will issue a full refund to your original payment method (processed within 5 business days of it reaching our warehouse). 
    
    Please let me know which option you prefer, and I will send you a prepaid return shipping label right away. To help make up for this frustrating experience, please use the code FIXIT15 for 15% off your next order. 
    
    Again, I am so sorry for the inconvenience. 
    
    Best,
    Sam, Northwind support

    557 tokens in, 821 out (643 of them reasoning) · 9.8 s · $0.0110 · 1 message on Pro · answered by google/gemini-3.1-pro-preview via Google · September 27, 2026

15. Claude Opus 5.5

    Passed: Graded 4.0 of 5 on average (lowest 3).

    Hi there,
    
    I'm really sorry. Receiving the wrong size once is frustrating, and three times is unacceptable. You ordered a medium, and that's what you should have received.
    
    Here's how we can fix this:
    
    - **Exchange for the correct size:** Send the large back and we'll get a medium out to you.
    - **Full refund:** If you'd rather not wait, returns within 30 days of delivery get a full refund to your original payment method. Refunds are paid within 5 business days of the item reaching our warehouse.
    
    Please reply with your order number and which option you prefer. I'll also flag this repeated error with our warehouse team so it doesn't happen again.
    
    As an apology, I'd like to offer you a 15% discount code for your next order.
    
    Thank you for your patience. I know it's been tested.
    
    Sam, Northwind support

    720 tokens in, 376 out (109 of them reasoning) · 6.4 s · $0.0111 · 1 message on Pro · answered by anthropic/claude-opus-5.5 via Claude Platform on AWS · September 27, 2026

16. Claude Fable 5.1

    Passed: Graded 4.7 of 5 on average (lowest 4).

    Hi,
    
    I'm really sorry — receiving the wrong size three times is completely unacceptable, and I understand your frustration.
    
    Here's what I'll do right away:
    
    1. **Free return**: I'm sending you a prepaid return label for the large. No need to wait for it to reach us before we act.
    2. **Replacement medium**: I'm placing a new order for the medium today at no cost, with express shipping. I'll personally check the size on the picking note before it leaves the warehouse and confirm by email.
    3. **Refund option**: If you'd rather not try again, I'll issue a full refund to your original payment method within 5 business days of the return arriving.
    4. **15% discount code** on your next order, as an apology for the hassle.
    
    Please reply with your order number and preference, and I'll get started immediately.
    
    Sam, Northwind support

    720 tokens in, 268 out · 7.0 s · $0.0223 · 1 message on Pro · answered by anthropic/claude-fable-5.1 via Anthropic · September 27, 2026

• A late order
• A refund request outside the window
• A message with a planted instruction
• A return inside the window
• The best AI for customer support