Tested prompt · RAG and answering from documents
A later amendment changes the answer: every AI model's reply, tested
We sent this hard RAG and answering from documents prompt to all 20 models in llmwise, the same way the app sends a message, and checked every reply the same way. Here's each one as it came, with whether it passed, what it cost and how long it took.
Based on 20 of our test runs on , through OpenRouter with the app's own prompt and settings. Updated .
Short answer
All 20 models passed this RAG and answering from documents prompt's check (key facts). The cheapest reply that passed was GPT-6 Luna's, at $0.000082, and the fastest too, in 1.0 s. The dearest reply, Claude Fable 5.1's, cost 201 times as much ($0.0165).
The prompt, as sent, and its check
Checked by key facts, the same way for every model.
A later amendment changes the answer (hard)
[9 lines every RAG and answering from documents prompt of ours shares, word for word: the whole prompt, on the methods page] Question: I'm an intern who started on 15 September 2026. How many days a week can I work remotely, and how big is my home office stipend?
The answer must state “2 days”, “€900”, “§7”.
Exactly what this prompt's replies are checked against, with every other prompt of our test runs.
Every model's result
All 20 models on this prompt, in catalog order.
| Model | Result | Cost | Time | Reply |
|---|---|---|---|---|
| Claude Fable 5.1Anthropic | Passed: Stated all 3 facts with the section. | $0.0165 | 7.1 s | 104 tokens |
| Claude Opus 5.5Anthropic | Passed: Stated all 3 facts with the section. | $0.0148 | 7.7 s | 188 tokens |
| Claude Sonnet 5.5Anthropic | Passed: Stated all 3 facts with the section. | $0.0031 | 1.2 s | 79 tokens |
| Claude Sonnet 5Anthropic | Passed: Stated all 3 facts with the section. | $0.0026 | 2.1 s | 79 tokens |
| Claude Haiku 5.5Anthropic | Passed: Stated all 3 facts with the section. | $0.00017 | 1.3 s | 103 tokens |
| Claude Haiku 4.5Anthropic | Passed: Stated all 3 facts with the section. | $0.0012 | 1.9 s | 100 tokens |
| GPT-6 AstraOpenAI | Passed: Stated all 3 facts with the section. | $0.0086 | 2.1 s | 40 tokens |
| GPT-6.1 SolOpenAI | Passed: Stated all 3 facts with the section. | $0.00086 | 1.3 s | 39 tokens |
| GPT-6 SolOpenAI | Passed: Stated all 3 facts with the section. | $0.0020 | 2.0 s | 31 tokens |
| GPT-6 LunaOpenAI | Passed: Stated all 3 facts with the section. | $0.000082 | 1.0 s | 33 tokens |
| Gemini 4 ArgonGoogle | Passed: Stated all 3 facts with the section. | $0.0019 | 5.7 s | 191 tokens |
| Gemini 3.1 Pro (preview)Google | Passed: Stated all 3 facts with the section. | $0.0099 | 8.6 s | 26 tokens |
| Gemini 3.8 FlashGoogle | Passed: Stated all 3 facts with the section. | $0.00088 | 4.1 s | 89 tokens |
| DeepSeek V4.1 FlashDeepSeek | Passed: Stated all 3 facts with the section. | $0.00028 | 1.0 s | 135 tokens |
| DeepSeek V4 ProDeepSeek | Passed: Stated all 3 facts with the section. | $0.00088 | 2.5 s | 32 tokens |
| Grok 4.7xAI | Passed: Stated all 3 facts with the section. | $0.0037 | 4.4 s | 21 tokens |
| Kimi K3Moonshot | Passed: Stated all 3 facts with the section. | $0.0022 | 4.8 s | 51 tokens |
| GLM 5.3Z.ai | Passed: Stated all 3 facts with the section. | $0.00059 | 1.3 s | 27 tokens |
| GLM 5.3 FlashZ.ai | Passed: Stated all 3 facts with the section. | $0.00014 | 2.2 s | 97 tokens |
| Mistral Large 4Mistral | Passed: Stated all 3 facts with the section. | $0.0034 | 38.3 s | 67 tokens |
Cost: what OpenRouter charged us for the reply. Time: from sending to the whole reply. Reply: its length in tokens, thinking not counted. In llmwise you pay per message, not per token: each of these replies counted as one message on Pro.
Every reply
Every reply passed: here they are from the cheapest up.
GPT-6 Luna
Passed: Stated all 3 facts with the section.
You can work remotely up to 2 days a week, and your home office stipend is €900 (§2, §3, §7).
659 tokens in, 33 out · 1.0 s · $0.0001 · 1 message on Pro · answered by openai/gpt-6-luna via OpenAI ·
GLM 5.3 Flash
Passed: Stated all 3 facts with the section.
You can work remotely **2 days a week**, and your home office stipend is **€900**. Since you joined on 15 September 2026 (on or after 1 September 2026), the amendments in §7 apply: the intern remote-work allowance rises from 1 to 2 days a week (§7, amending §2), and the home office stipend increases from €600 to €900 (§7, amending §3).
681 tokens in, 160 out (63 of them reasoning) · 2.2 s · $0.0001 · 1 message on Pro · answered by z-ai/glm-5.3-flash via Wafer ·
Claude Haiku 5.5
Passed: Stated all 3 facts with the section.
You can work remotely 2 days a week, and your home office stipend is €900. Both figures come from the amendment for people who joined on or after 1 September 2026 (§7), which raises the intern remote allowance from 1 day to 2 days (§2) and sets the stipend at €900 instead of €600 (§3).
920 tokens in, 103 out · 1.3 s · $0.0002 · 1 message on Pro · answered by anthropic/claude-haiku-5.5 via Anthropic ·
DeepSeek V4.1 Flash
Passed: Stated all 3 facts with the section.
As an intern who joined on or after 1 September 2026, the §7 amendments apply to you: - **Remote work: 2 days a week** — §7 raises the intern allowance in §2 from 1 day to 2 days a week for anyone joining on or after that date. - **Home office stipend: €900** — §7 sets the §3 stipend at €900 for new joiners on or after 1 September 2026. So: 2 days a week remote (§2, as amended by §7), and a one-time €900 stipend (§3, as amended by §7).
708 tokens in, 181 out (46 of them reasoning) · 1.0 s · $0.0003 · 1 message on Pro · answered by deepseek/deepseek-v4.1-flash via Together ·
GLM 5.3
Passed: Stated all 3 facts with the section.
You can work remotely 2 days a week (§7), and your home office stipend is €900 (§7).
704 tokens in, 60 out (33 of them reasoning) · 1.3 s · $0.0006 · 1 message on Pro · answered by z-ai/glm-5.3 via Wafer ·
GPT-6.1 Sol
Passed: Stated all 3 facts with the section.
You can work remotely **2 days a week** and receive a **one-time €900 home office stipend**, since you joined after 1 September 2026 (§7).
661 tokens in, 39 out · 1.3 s · $0.0009 · 1 message on Pro · answered by openai/gpt-6.1-sol via OpenAI ·
Gemini 3.8 Flash
Passed: Stated all 3 facts with the section.
* **Remote work:** You can work remotely up to 2 days a week (§7). * **Home office stipend:** Whether interns receive a home office stipend isn't in the handbook (the handbook specifies that "New staff" receive the stipend (§3), amended to €900 for those joining on or after 1 September 2026 (§7), but does not state if interns are included in "staff").
723 tokens in, 89 out · 4.1 s · $0.0009 · 1 message on Pro · answered by google/gemini-3.8-flash via Google ·
DeepSeek V4 Pro
Passed: Stated all 3 facts with the section.
Remote work: 2 days a week (§7, amending §2). Home office stipend: €900 (§7, amending §3).
680 tokens in, 178 out (146 of them reasoning) · 2.5 s · $0.0009 · 1 message on Pro · answered by deepseek/deepseek-v4-pro-0813 via Wafer ·
Claude Haiku 4.5
Passed: Stated all 3 facts with the section.
Based on the handbook: **Remote work days:** You can work remotely 2 days a week. (§2 as amended by §7) **Home office stipend:** You get a €900 one-time stipend. (§3 as amended by §7) Since you started on 15 September 2026, which is after 1 September 2026, the amended rates in §7 apply to you.
733 tokens in, 100 out · 1.9 s · $0.0012 · 1 message on Pro · answered by anthropic/claude-haiku-4.5 via Amazon Bedrock ·
Gemini 4 Argon
Passed: Stated all 3 facts with the section.
* **Remote work:** You can work remotely up to 2 days a week (§2, §7). * **Home office stipend:** The handbook does not clearly state whether interns receive the stipend, or if you receive €900: * §3 states that "New **staff**" get a one-time home office stipend (originally €600), and §2 distinguishes between "staff" and "interns." * However, §7 states that for "**everyone** who joins on or after 1 September 2026, the home office stipend in §3 is €900." * Because it isn't in the handbook whether §7 extends the §3 stipend to interns or only increases the amount for new staff, the handbook doesn't definitively say if you get the €900 stipend (§2, §3, §7).
940 tokens in, 191 out · 5.7 s · $0.0019 · 1 message on Pro · answered by google/gemini-4-argon via Google AI Studio ·
GPT-6 Sol
Passed: Stated all 3 facts with the section.
You can work remotely **2 days a week**, and your one-time home office stipend is **€900** (§7).
659 tokens in, 67 out (36 of them reasoning) · 2.0 s · $0.0020 · 1 message on Pro · answered by openai/gpt-6-sol via OpenAI ·
Kimi K3
Passed: Stated all 3 facts with the section.
You can work remotely **2 days a week** (§7, amending §2), and your one-time home office stipend is **€900** (§7, amending §3).
750 tokens in, 165 out (114 of them reasoning) · 4.8 s · $0.0022 · 1 message on Pro · answered by moonshotai/kimi-k3 via Wafer ·
Claude Sonnet 5
Passed: Stated all 3 facts with the section.
Since you started on 15 September 2026, which is after the 1 September 2026 amendment date: - **Remote work:** 2 days a week (§7, amending §2) - **Home office stipend:** €900 (§7, amending §3)
917 tokens in, 79 out · 2.1 s · $0.0026 · 1 message on Pro · answered by anthropic/claude-sonnet-5 via Claude Platform on AWS ·
Claude Sonnet 5.5
Passed: Stated all 3 facts with the section.
Since you joined on 15 September 2026, the amendments in §7 apply to you: - **Remote work:** 2 days a week (§7, amending §2). - **Home office stipend:** €900, one-time (§7, amending §3).
921 tokens in, 79 out · 1.2 s · $0.0031 · 1 message on Pro · answered by anthropic/claude-sonnet-5.5 via Anthropic ·
Mistral Large 4
Passed: Stated all 3 facts with the section.
You can work remotely **2 days a week** (§2, as amended by §7), and your home office stipend is **€900** (§3, as amended by §7).
671 tokens in, 1,574 out (1,507 of them reasoning) · 38.3 s · $0.0034 · 1 message on Pro · answered by mistralai/mistral-large-4-0 via Mistral ·
Grok 4.7
Passed: Stated all 3 facts with the section.
You can work remotely 2 days a week, and your home office stipend is €900 (§7).
1,894 tokens in, 271 out (250 of them reasoning) · 4.4 s · $0.0037 · 1 message on Pro · answered by x-ai/grok-4.7 via xAI ·
GPT-6 Astra
Passed: Stated all 3 facts with the section.
You can work remotely **2 days a week**, and you get a **one-time €900 home office stipend** because you joined after 1 September 2026 (§7).
659 tokens in, 40 out · 2.1 s · $0.0086 · 1 message on Pro · answered by openai/gpt-6-astra via OpenAI ·
Gemini 3.1 Pro
Passed: Stated all 3 facts with the section.
You can work remotely up to 2 days a week (§7). Your home office stipend is €900 (§7).
723 tokens in, 704 out (678 of them reasoning) · 8.6 s · $0.0099 · 1 message on Pro · answered by google/gemini-3.1-pro-preview via Google ·
Claude Opus 5.5
Passed: Stated all 3 facts with the section.
- **Remote work: 2 days a week.** §2 sets interns at 1 day, but the 1 September 2026 amendment raises this to 2 days (§2, §7). - **Home office stipend: €900, if interns are eligible.** For anyone joining on or after 1 September 2026, the stipend is €900 instead of €600 (§3, §7). However, §3 grants the stipend to "new staff," and §2 treats "staff" and "interns" as separate groups. The handbook doesn't say whether interns get the stipend, so it's worth confirming with the People team.
919 tokens in, 514 out (326 of them reasoning) · 7.7 s · $0.0148 · 1 message on Pro · answered by anthropic/claude-opus-5.5 via Claude Platform on AWS ·
Claude Fable 5.1
Passed: Stated all 3 facts with the section.
As an intern who joined on 15 September 2026 (on or after 1 September 2026), the amendments apply to you: - **Remote work:** 2 days a week (§7, amending §2). - **Home office stipend:** €900 one-time, since you're new staff starting after the amendment date (§7, amending §3).
919 tokens in, 104 out · 7.1 s · $0.0165 · 1 message on Pro · answered by anthropic/claude-fable-5.1 via Anthropic ·
More RAG and answering from documents prompts
The other RAG and answering from documents prompts, each with every model's reply, and the results across all five.
Questions
Which AI does best on “A later amendment changes the answer”?
All 20 models passed this RAG and answering from documents prompt's check (key facts). The cheapest reply that passed was GPT-6 Luna's, at $0.000082, and the fastest too, in 1.0 s. The dearest reply, Claude Fable 5.1's, cost 201 times as much ($0.0165).
What does a reply to “A later amendment changes the answer” cost?
Through the models' APIs, what OpenRouter charged us ran from $0.000082 (GPT-6 Luna) to $0.0165 (Claude Fable 5.1) for this prompt. In llmwise you don't pay by the token: a reply like these counts as one message on Pro, whichever model answers.
Claude, GPT, Gemini, DeepSeek, Grok, Kimi, GLM, and Mistral, in one chat.
See what a message costs before you send it. Free is 5 messages to try; sign in with an email link, no password or card.