Search, but don't book: every AI model's reply, tested

You can call these tools:
- search_flights(origin: string, destination: string, date: string in YYYY-MM-DD, max_stops: number): searches flights between airports given as IATA codes, like "LHR".
- book_flight(flight_id: string): books a flight found by a search.

Today is Monday, 5 October 2026.
User: Find me nonstop flights from Boston to Denver on November 14. Don't book anything yet: I want to see the options first.

Reply with only a JSON object, {"calls": [{"tool": "<name>", "arguments": {...}}]}, listing the tool calls to make now. If no call is right yet, reply {"calls": []}.

[
  {
    "arguments": {
      "date": "2026-11-14",
      "destination": "DEN",
      "max_stops": 0,
      "origin": "BOS"
    },
    "tool": "search_flights"
  }
]

Exactly what this prompt's replies are checked against, with every other prompt of our test runs.

1. GLM 5.3 Flash

   Passed: Made the 1 expected call.

   {"calls": [{"tool": "search_flights", "arguments": {"origin": "BOS", "destination": "DEN", "date": "2026-11-14", "max_stops": 0}}]}

   504 tokens in, 68 out (19 of them reasoning) · 2.3 s · $0.0001 · 1 message on Pro · answered by z-ai/glm-5.3-flash via AtlasCloud · September 27, 2026

2. GPT-6 Luna

   Passed: Made the 1 expected call.

   {"calls":[{"tool":"search_flights","arguments":{"origin":"BOS","destination":"DEN","date":"2026-11-14","max_stops":0}}]}

   487 tokens in, 72 out (29 of them reasoning) · 2.2 s · $0.0001 · 1 message on Pro · answered by openai/gpt-6-luna via OpenAI · September 27, 2026

3. Claude Haiku 5.5

   Passed: Made the 1 expected call.

   {"calls": [{"tool": "search_flights", "arguments": {"origin": "BOS", "destination": "DEN", "date": "2026-11-14", "max_stops": 0}}]}

   678 tokens in, 66 out · 1.0 s · $0.0001 · 1 message on Pro · answered by anthropic/claude-haiku-5.5 via Anthropic · October 7, 2026

4. DeepSeek V4.1 Flash

   Passed: Made the 1 expected call.

   {"calls": [{"tool": "search_flights", "arguments": {"origin": "BOS", "destination": "DEN", "date": "2026-11-14", "max_stops": 0}}]}

   537 tokens in, 77 out (26 of them reasoning) · 0.4 s · $0.0002 · 1 message on Pro · answered by deepseek/deepseek-v4.1-flash via Together · October 8, 2026

5. GLM 5.3

   Passed: Made the 1 expected call.

   {"calls":[{"tool":"search_flights","arguments":{"origin":"BOS","destination":"DEN","date":"2026-11-14","max_stops":0}}]}

   527 tokens in, 38 out (1 of them reasoning) · 0.5 s · $0.0005 · 1 message on Pro · answered by z-ai/glm-5.3 via Wafer · September 27, 2026

6. DeepSeek V4 Pro

   Passed: Made the 1 expected call.

   {"calls": [{"tool": "search_flights", "arguments": {"origin": "BOS", "destination": "DEN", "date": "2026-11-14", "max_stops": 0}}]}

   509 tokens in, 131 out (81 of them reasoning) · 1.4 s · $0.0007 · 1 message on Pro · answered by deepseek/deepseek-v4-pro-0813 via Wafer · October 8, 2026

7. GPT-6.1 Sol

   Passed: Made the 1 expected call.

   {"calls":[{"tool":"search_flights","arguments":{"origin":"BOS","destination":"DEN","date":"2026-11-14","max_stops":0}}]}

   489 tokens in, 41 out · 1.3 s · $0.0007 · 1 message on Pro · answered by openai/gpt-6.1-sol via OpenAI · September 29, 2026

8. Mistral Large 4

   Passed: Made the 1 expected call.

   {"calls": [{"tool": "search_flights", "arguments": {"origin": "BOS", "destination": "DEN", "date": "2026-11-14", "max_stops": 0}}]}

   493 tokens in, 262 out (203 of them reasoning) · 3.2 s · $0.0009 · 1 message on Pro · answered by mistralai/mistral-large-4-0 via Mistral · October 9, 2026

9. Gemini 3.8 Flash

   Passed: Made the 1 expected call.

   {"calls": [{"tool": "search_flights", "arguments": {"origin": "BOS", "destination": "DEN", "date": "2026-11-14", "max_stops": 0}}]}

   528 tokens in, 133 out (82 of them reasoning) · 6.3 s · $0.0009 · 1 message on Pro · answered by google/gemini-3.8-flash via Google · September 27, 2026

10. Claude Haiku 4.5

    Passed: Made the 1 expected call.

    ```json
    {
      "calls": [
        {
          "tool": "search_flights",
          "arguments": {
            "origin": "BOS",
            "destination": "DEN",
            "date": "2026-11-14",
            "max_stops": 0
          }
        }
      ]
    }
    ```

    531 tokens in, 85 out · 1.1 s · $0.0010 · 1 message on Pro · answered by anthropic/claude-haiku-4.5 via Amazon Bedrock · September 27, 2026

11. Gemini 4 Argon

    Passed: Made the 1 expected call.

    {"calls": [{"tool": "search_flights", "arguments": {"origin": "BOS", "destination": "DEN", "date": "2026-11-14", "max_stops": 0}}]}

    744 tokens in, 54 out · 3.2 s · $0.0010 · 1 message on Pro · answered by google/gemini-4-argon via Google AI Studio · October 15, 2026

12. GPT-6 Sol

    Passed: Made the 1 expected call.

    {"calls":[{"tool":"search_flights","arguments":{"origin":"BOS","destination":"DEN","date":"2026-11-14","max_stops":0}}]}

    487 tokens in, 41 out · 1.1 s · $0.0014 · 1 message on Pro · answered by openai/gpt-6-sol via OpenAI · September 27, 2026

13. Claude Sonnet 5

    Passed: Made the 1 expected call.

    {"calls": [{"tool": "search_flights", "arguments": {"origin": "BOS", "destination": "DEN", "date": "2026-11-14", "max_stops": 0}}]}

    675 tokens in, 66 out · 1.9 s · $0.0020 · 1 message on Pro · answered by anthropic/claude-sonnet-5 via Claude Platform on AWS · September 27, 2026

14. Grok 4.7

    Passed: Made the 1 expected call.

    {"calls": [{"tool": "search_flights", "arguments": {"origin": "BOS", "destination": "DEN", "date": "2026-11-14", "max_stops": 0}}]}

    1,729 tokens in, 158 out (110 of them reasoning) · 3.4 s · $0.0023 · 1 message on Pro · answered by x-ai/grok-4.7 via xAI · October 2, 2026

15. Claude Sonnet 5.5

    Passed: Made the 1 expected call.

    {"calls": [{"tool": "search_flights", "arguments": {"origin": "BOS", "destination": "DEN", "date": "2026-11-14", "max_stops": 0}}]}

    679 tokens in, 66 out · 1.4 s · $0.0023 · 1 message on Pro · answered by anthropic/claude-sonnet-5.5 via Anthropic · September 28, 2026

16. Kimi K3

    Passed: Made the 1 expected call.

    {"calls":[{"tool":"search_flights","arguments":{"origin":"BOS","destination":"DEN","date":"2026-11-14","max_stops":0}}]}

    576 tokens in, 133 out (85 of them reasoning) · 2.0 s · $0.0034 · 1 message on Pro · answered by moonshotai/kimi-k3 via Together · September 27, 2026

17. Claude Opus 5.5

    Passed: Made the 1 expected call.

    {"calls": [{"tool": "search_flights", "arguments": {"origin": "BOS", "destination": "DEN", "date": "2026-11-14", "max_stops": 0}}]}

    677 tokens in, 66 out · 3.1 s · $0.0046 · 1 message on Pro · answered by anthropic/claude-opus-5.5 via Claude Platform on AWS · September 27, 2026

18. Gemini 3.1 Pro

    Passed: Made the 1 expected call.

    ```json
    {
      "calls": [
        {
          "tool": "search_flights",
          "arguments": {
            "origin": "BOS",
            "destination": "DEN",
            "date": "2026-11-14",
            "max_stops": 0
          }
        }
      ]
    }
    ```

    528 tokens in, 418 out (333 of them reasoning) · 4.7 s · $0.0061 · 1 message on Pro · answered by google/gemini-3.1-pro-preview via Google · September 27, 2026

19. GPT-6 Astra

    Passed: Made the 1 expected call.

    {"calls":[{"tool":"search_flights","arguments":{"origin":"BOS","destination":"DEN","date":"2026-11-14","max_stops":0}}]}

    487 tokens in, 41 out · 1.5 s · $0.0069 · 1 message on Pro · answered by openai/gpt-6-astra via OpenAI · September 27, 2026

20. Claude Fable 5.1

    Passed: Made the 1 expected call.

    {"calls": [{"tool": "search_flights", "arguments": {"origin": "BOS", "destination": "DEN", "date": "2026-11-14", "max_stops": 0}}]}

    677 tokens in, 66 out · 5.0 s · $0.0116 · 1 message on Pro · answered by anthropic/claude-fable-5.1 via Anthropic · September 27, 2026

• Book a meeting from a sentence
• Pick the tool and work out the date
• Convert a currency
• Two calls with a unit conversion
• The best AI for agents and tool use