Tested prompt · Summarization
An article in three bullets: every AI model's reply, tested
We sent this everyday summarization prompt to all 16 models in llmwise, the same way the app sends a message, and checked every reply the same way. Here's each one as it came, with whether it passed, what it cost and how long it took.
Based on 16 of our test runs on , through OpenRouter with the app's own prompt and settings. Updated .
Short answer
9 of 16 models passed this summarization prompt's check (rubric); Claude Fable 5.1, Claude Sonnet 5, GPT-6 Sol, GPT-6 Luna, Kimi K3, GLM 5.3, and GLM 5.3 Flash didn't. The cheapest reply that passed was DeepSeek V4.1 Flash's, at $0.00030; the fastest, Claude Haiku 4.5's in 1.8 s. The dearest reply, Claude Opus 5.5's, cost 57 times as much ($0.0171).
The prompt, as sent, and its check
Checked by rubric (graded), the same way for every model.
An article in three bullets (everyday)
Summarize this article in exactly 3 bullet points, at most 60 words in total.
The city of Aldmoor ended its six-month trial of protected bike lanes on Market Street last week, and the council will vote in November on whether to keep them. Bike trips on the street rose from about 900 to 2,300 a day during the trial, according to automatic counters. Car traffic fell by 14%, but travel times for drivers rose by just over a minute at rush hour. Shop owners are split: a survey of 60 businesses found 27 reported more customers, 19 reported fewer, and the rest saw no change. Two cyclist injuries were recorded on the street during the trial, down from nine in the same months last year. The lanes cost €410,000 to install; making them permanent would cost another €1.1 million for kerbs and new traffic lights.
Graded on:
- Covers the main points: Covers the result that matters: the trial, the vote, and the main numbers.
- Accurate: Every number and claim matches the article; nothing added.
- Concise: Tight, with no filler.
Exactly what this prompt's replies are checked against, with every other prompt of our test runs.
Every model's result
All 16 models on this prompt, in catalog order.
| Model | Result | Cost | Time | Reply |
|---|---|---|---|---|
| Claude Fable 5.1Anthropic | Failed: Graded 4.3 of 5 on average (lowest 4); but 62 words, over the 60 allowed. | $0.0164 | 5.1 s | 150 tokens |
| Claude Opus 5.5Anthropic | Passed: Graded 5.0 of 5 on average (lowest 5). | $0.0171 | 8.1 s | 135 tokens |
| Claude Sonnet 5.5Anthropic | Passed: Graded 4.7 of 5 on average (lowest 4). | $0.0031 | 1.9 s | 130 tokens |
| Claude Sonnet 5Anthropic | Failed: Graded 3.7 of 5 on average (lowest 3). | $0.0027 | 2.6 s | 125 tokens |
| Claude Haiku 4.5Anthropic | Passed: Graded 4.0 of 5 on average (lowest 3). | $0.0011 | 1.8 s | 102 tokens |
| GPT-6 AstraOpenAI | Passed: Graded 4.7 of 5 on average (lowest 4). | $0.0143 | 5.0 s | 101 tokens |
| GPT-6 SolOpenAI | Failed: Graded 4.0 of 5 on average (lowest 3); but 66 words, over the 60 allowed. | $0.0020 | 2.0 s | 99 tokens |
| GPT-6 LunaOpenAI | Failed: Graded 4.0 of 5 on average (lowest 3); but 61 words, over the 60 allowed. | $0.000098 | 1.5 s | 92 tokens |
| Gemini 3.1 Pro (preview)Google | Passed: Graded 4.3 of 5 on average (lowest 4). | $0.0088 | 8.0 s | 59 tokens |
| Gemini 3.8 FlashGoogle | Passed: Graded 4.3 of 5 on average (lowest 4). | $0.00072 | 3.7 s | 81 tokens |
| DeepSeek V4.1 FlashDeepSeek | Passed: Graded 4.7 of 5 on average (lowest 4). | $0.00030 | 2.5 s | 72 tokens |
| DeepSeek V4 ProDeepSeek | Passed: Graded 4.3 of 5 on average (lowest 4). | $0.0021 | 6.1 s | 84 tokens |
| Grok 4.7xAI | Passed: Graded 4.7 of 5 on average (lowest 4). | $0.0051 | 9.8 s | 90 tokens |
| Kimi K3Moonshot | Failed: Graded 4.0 of 5 on average (lowest 3); but 62 words, over the 60 allowed. | $0.0028 | 2.8 s | 99 tokens |
| GLM 5.3Z.ai | Failed: Graded 4.3 of 5 on average (lowest 3); but 68 words, over the 60 allowed. | $0.00044 | 2.8 s | 100 tokens |
| GLM 5.3 FlashZ.ai | Failed: Graded 4.7 of 5 on average (lowest 4); but 61 words, over the 60 allowed. | $0.000099 | 4.0 s | 98 tokens |
Cost: what OpenRouter charged us for the reply. Time: from sending to the whole reply. Reply: its length in tokens, thinking not counted. In llmwise you pay per message, not per token: each of these replies counted as one message on Pro.
Every reply
The 7 replies that failed first, then the ones that passed, from the cheapest up.
GPT-6 Luna
Failed: Graded 4.0 of 5 on average (lowest 3); but 61 words, over the 60 allowed.
- Aldmoor’s six-month Market Street trial boosted daily bike trips from 900 to 2,300, while car traffic fell 14%. - Rush-hour driver trips took just over a minute longer; businesses were divided on customer changes. - Cyclist injuries fell from nine to two year over year. The trial cost €410,000; permanent lanes would cost €1.1 million more, with a council vote due in November.
522 tokens in, 92 out · 1.5 s · $0.0001 · 1 message on Pro · answered by openai/gpt-6-luna via OpenAI ·
GLM 5.3 Flash
Failed: Graded 4.7 of 5 on average (lowest 4); but 61 words, over the 60 allowed.
- Aldmoor's six-month Market Street bike lane trial ended; council votes in November on making it permanent. - Cycling rose from ~900 to 2,300 daily trips, injuries dropped from nine to two, car traffic fell 14% with just over a minute added to rush-hour travel times. - Businesses are split (27 more customers, 19 fewer, 14 unchanged); lanes cost €410k, permanence needs €1.1m more.
539 tokens in, 98 out · 4.0 s · $0.0001 · 1 message on Pro · answered by z-ai/glm-5.3-flash via AtlasCloud ·
GLM 5.3
Failed: Graded 4.3 of 5 on average (lowest 3); but 68 words, over the 60 allowed.
- Aldmoor ended its six-month protected bike lane trial on Market Street; the council votes in November on making them permanent. - Daily bike trips rose from about 900 to 2,300, cyclist injuries fell from nine to two, and car traffic dropped 14%, though rush-hour drives got slightly longer. - Businesses are split (27 reported more customers, 19 fewer), and permanence would cost an extra €1.1 million beyond the €410,000 spent.
539 tokens in, 218 out (118 of them reasoning) · 2.8 s · $0.0004 · 1 message on Pro · answered by z-ai/glm-5.3 via Morph ·
GPT-6 Sol
Failed: Graded 4.0 of 5 on average (lowest 3); but 66 words, over the 60 allowed.
- Aldmoor’s six-month protected bike-lane trial on Market Street has ended; the council will vote in November on whether to keep the lanes. - Daily bike trips rose from about 900 to 2,300, while cyclist injuries fell from nine to two. Car traffic dropped 14%, and rush-hour drives took just over a minute longer. - Businesses reported mixed results. Installation cost €410,000; permanent lanes would cost another €1.1 million.
522 tokens in, 99 out · 2.0 s · $0.0020 · 1 message on Pro · answered by openai/gpt-6-sol via OpenAI ·
Claude Sonnet 5
Failed: Graded 3.7 of 5 on average (lowest 3).
- Aldmoor's 6-month trial bike lanes tripled cyclist trips (900→2,300/day), cut car traffic 14%, but added ~1 min to rush-hour drives. - Business impact mixed: 27 of 60 shops saw more customers, 19 fewer. - Injuries dropped (9→2); permanent install would cost €1.1M more.
725 tokens in, 125 out · 2.6 s · $0.0027 · 1 message on Pro · answered by anthropic/claude-sonnet-5 via Claude Platform on AWS ·
Kimi K3
Failed: Graded 4.0 of 5 on average (lowest 3); but 62 words, over the 60 allowed.
- Aldmoor's six-month protected bike lane trial on Market Street ended; the council votes in November on making them permanent. - Bike trips rose from 900 to 2,300 daily, cyclist injuries dropped from nine to two, while car traffic fell 14% with only slightly longer rush-hour drive times. - Businesses were split on customer impact; installation cost €410,000, with permanent lanes requiring €1.1 million more.
608 tokens in, 109 out (10 of them reasoning) · 2.8 s · $0.0028 · 1 message on Pro · answered by moonshotai/kimi-k3 via Phala ·
Claude Fable 5.1
Failed: Graded 4.3 of 5 on average (lowest 4); but 62 words, over the 60 allowed.
- Aldmoor's six-month protected bike lane trial on Market Street ended; council votes in November on making it permanent. - Daily bike trips rose from 900 to 2,300, car traffic fell 14%, rush-hour driver times rose about a minute, and cyclist injuries dropped from nine to two. - Businesses were split (27 gained customers, 19 lost); permanence would cost €1.1 million beyond the €410,000 installation.
727 tokens in, 150 out · 5.1 s · $0.0164 · 1 message on Pro · answered by anthropic/claude-fable-5.1 via Anthropic ·
DeepSeek V4.1 Flash
Passed: Graded 4.7 of 5 on average (lowest 4).
- Aldmoor's six-month Market Street bike lane trial ended; council votes in November. - Cyclist trips rose 900→2,300 daily, injuries fell 9→2; car traffic fell 14%, rush-hour delays +1 minute. - Businesses split; €410k installed, €1.1m more to make permanent.
562 tokens in, 310 out (238 of them reasoning) · 2.5 s · $0.0003 · 1 message on Pro · answered by deepseek/deepseek-v4.1-flash via StreamLake ·
Gemini 3.8 Flash
Passed: Graded 4.3 of 5 on average (lowest 4).
* Aldmoor's six-month trial increased daily bike trips from 900 to 2,300 and reduced cyclist injuries from nine to two. * Car traffic fell 14% with minor delays, while local shop owners reported mixed impacts on customer numbers. * The city council votes in November on making the lanes permanent, which would cost €1.1 million.
560 tokens in, 81 out · 3.7 s · $0.0007 · 1 message on Pro · answered by google/gemini-3.8-flash via Google ·
Claude Haiku 4.5
Passed: Graded 4.0 of 5 on average (lowest 3).
• **Bike usage tripled** (900 to 2,300 daily trips); cyclist injuries dropped from 9 to 2 during six-month trial. • **Mixed business impact**: 27 of 60 surveyed shops gained customers; 19 lost customers; car travel times increased ~1 minute. • **Council votes November** on permanent lanes costing €1.1 million after €410,000 installation trial.
561 tokens in, 102 out · 1.8 s · $0.0011 · 1 message on Pro · answered by anthropic/claude-haiku-4.5 via Amazon Bedrock ·
DeepSeek V4 Pro
Passed: Graded 4.3 of 5 on average (lowest 4).
- Bike trips on Market Street rose from 900 to 2,300 daily during the trial; cyclist injuries fell from nine to two. - Car traffic dropped 14%, but rush-hour driver travel times rose by just over a minute; shop owners are split. - Making the lanes permanent would cost €1.1 million on top of the €410,000 already spent; the council votes in November.
534 tokens in, 983 out (899 of them reasoning) · 6.1 s · $0.0021 · 1 message on Pro · answered by deepseek/deepseek-v4-pro-0813 via Ionstream ·
Claude Sonnet 5.5
Passed: Graded 4.7 of 5 on average (lowest 4).
- Aldmoor's six-month Market Street bike-lane trial raised daily bike trips from about 900 to 2,300, cut car traffic 14%, and added just over a minute to rush-hour drives. - Injuries fell from nine to two; businesses were split, with 27 reporting more customers, 19 fewer. - Council votes in November; permanence would cost another €1.1 million.
729 tokens in, 130 out · 1.9 s · $0.0031 · 1 message on Pro · answered by anthropic/claude-sonnet-5.5 via Anthropic ·
Grok 4.7
Passed: Graded 4.7 of 5 on average (lowest 4).
- Aldmoor’s Market Street bike-lane trial ended; the council votes in November. Daily bike trips rose from 900 to 2,300. - Car traffic fell 14% and rush-hour delays rose about one minute; cyclist injuries dropped from nine to two. - Of 60 shops, 27 saw more customers and 19 fewer. Installation cost €410,000; permanence would add €1.1 million.
1,758 tokens in, 756 out (666 of them reasoning) · 9.8 s · $0.0051 · 1 message on Pro · answered by x-ai/grok-4.7 via xAI ·
Gemini 3.1 Pro
Passed: Graded 4.3 of 5 on average (lowest 4).
* Daily bike trips jumped to 2,300 and cyclist injuries dropped sharply. * Car traffic fell 14% with minor delays, while local business impacts were mixed. * The city council votes in November on spending €1.1 million to make the lanes permanent.
560 tokens in, 636 out (577 of them reasoning) · 8.0 s · $0.0088 · 1 message on Pro · answered by google/gemini-3.1-pro-preview via Google ·
GPT-6 Astra
Passed: Graded 4.7 of 5 on average (lowest 4).
- Aldmoor’s six-month bike-lane trial ended; November’s council vote decides permanence, costing another €1.1 million after €410,000 installation. - Daily bike trips rose from 900 to 2,300; car traffic fell 14%, rush-hour driving times increased just over a minute, and cyclist injuries fell from nine to two. - Of 60 businesses, 27 gained customers, 19 lost customers, and 14 saw no change.
522 tokens in, 182 out (81 of them reasoning) · 5.0 s · $0.0143 · 1 message on Pro · answered by openai/gpt-6-astra via OpenAI ·
Claude Opus 5.5
Passed: Graded 5.0 of 5 on average (lowest 5).
- Aldmoor's six-month Market Street protected bike lane trial ended; council votes in November on keeping them. - Daily bike trips rose from 900 to 2,300, car traffic fell 14%, rush-hour drive times rose about a minute, and cyclist injuries dropped from nine to two. - Businesses were split; permanence would cost €1.1 million beyond the €410,000 installation.
727 tokens in, 674 out (539 of them reasoning) · 8.1 s · $0.0171 · 1 message on Pro · answered by anthropic/claude-opus-5.5 via Claude Platform on AWS ·
More summarization prompts
The other summarization prompts, each with every model's reply, and the results across all five.
Questions
Which AI does best on “An article in three bullets”?
9 of 16 models passed this summarization prompt's check (rubric); Claude Fable 5.1, Claude Sonnet 5, GPT-6 Sol, GPT-6 Luna, Kimi K3, GLM 5.3, and GLM 5.3 Flash didn't. The cheapest reply that passed was DeepSeek V4.1 Flash's, at $0.00030; the fastest, Claude Haiku 4.5's in 1.8 s. The dearest reply, Claude Opus 5.5's, cost 57 times as much ($0.0171).
What does a reply to “An article in three bullets” cost?
Through the models' APIs, what OpenRouter charged us ran from $0.000098 (GPT-6 Luna) to $0.0171 (Claude Opus 5.5) for this prompt. The cheapest that passed was DeepSeek V4.1 Flash's, $0.00030. In llmwise you don't pay by the token: a reply like these counts as one message on Pro, whichever model answers.
Claude, GPT, Gemini, DeepSeek, Grok, Kimi, and GLM, in one chat.
See what a message costs before you send it. Free is 5 messages to try; sign in with an email link, no password or card.