A study with a negative result: every AI model's reply, tested

Summarize the findings of this study in 2 or 3 sentences for a general reader.

In a 12-week randomized trial of 480 adults with frequent migraines, the drug Veltrazine did not reduce the number of migraine days compared with a placebo (4.1 vs 4.3 days a month; the difference was not statistically significant). In an exploratory analysis, participants who took the drug within an hour of a migraine starting reported shorter attacks, but the authors caution that this subgroup was not planned in advance and needs its own trial. Side effects, mostly mild nausea, were reported by 11% of the drug group and 6% of the placebo group.

Exactly what this prompt's replies are checked against, with every other prompt of our test runs.

1. GPT-6 Luna

   Passed: Graded 4.7 of 5 on average (lowest 4).

   In this 12-week trial, Veltrazine did not significantly reduce monthly migraine days compared with a placebo. People who took it soon after a migraine began reported shorter attacks, but that finding was exploratory and needs further study; mild side effects, mainly nausea, were more common with the drug.

   477 tokens in, 63 out · 1.3 s · $0.0001 · 1 message on Pro · answered by openai/gpt-6-luna via OpenAI · September 27, 2026

2. GLM 5.3 Flash

   Passed: Graded 4.3 of 5 on average (lowest 3).

   In a 12-week trial of 480 adults with frequent migraines, the drug Veltrazine failed to reduce monthly migraine days compared with a placebo, showing no statistically significant benefit. While an exploratory analysis hinted that taking the drug early in an attack might shorten it, the authors caution this finding wasn't planned and needs confirmation in a dedicated trial. Side effects were mild, mainly nausea, affecting slightly more people on the drug (11%) than placebo (6%).

   494 tokens in, 96 out · 3.7 s · $0.0001 · 1 message on Pro · answered by z-ai/glm-5.3-flash via AtlasCloud · September 27, 2026

3. DeepSeek V4.1 Flash

   Passed: Graded 4.3 of 5 on average (lowest 4).

   In a 12-week trial of 480 adults with frequent migraines, the drug Veltrazine worked no better than a placebo at reducing migraine days (4.1 vs. 4.3 per month). A secondary analysis suggested people who took the drug within an hour of an attack had shorter migraines, but the researchers say this needs confirmation in a new trial. Side effects were mostly mild nausea, reported by 11% on the drug versus 6% on placebo.

   518 tokens in, 107 out (6 of them reasoning) · 1.4 s · $0.0002 · 1 message on Pro · answered by deepseek/deepseek-v4.1-flash via Novita · September 27, 2026

4. Gemini 3.8 Flash

   Passed: Graded 4.3 of 5 on average (lowest 4).

   A recent study found that the drug Veltrazine was no more effective than a placebo at reducing how many days adults experienced migraines. While taking the drug immediately at the start of a migraine appeared to shorten the attack, researchers emphasize that further studies are needed to confirm this unverified benefit. The medication was generally well-tolerated, with mild nausea being the most common side effect.

   506 tokens in, 76 out · 3.6 s · $0.0007 · 1 message on Pro · answered by google/gemini-3.8-flash via Google · September 27, 2026

5. GLM 5.3

   Passed: Graded 4.7 of 5 on average (lowest 4).

   In a 12-week trial of 480 adults with frequent migraines, the drug Veltrazine failed to reduce monthly migraine days compared with a placebo (4.1 vs. 4.3 days), though an unplanned exploratory analysis suggested attacks may be shorter if the drug is taken within an hour of onset — a finding the authors say needs confirmation in its own dedicated trial. Side effects, mostly mild nausea, were somewhat more common with the drug (11% vs. 6% with placebo).

   517 tokens in, 105 out (1 of them reasoning) · 0.8 s · $0.0008 · 1 message on Pro · answered by z-ai/glm-5.3 via Wafer · September 27, 2026

6. Claude Haiku 4.5

   Passed: Graded 4.0 of 5 on average (lowest 3).

   In a 12-week study of 480 migraine sufferers, the drug Veltrazine failed to reduce the overall frequency of migraines compared to placebo. However, when taken immediately as a migraine began, it showed promise in shortening individual attacks, though researchers caution this finding needs confirmation in a dedicated study. The drug caused mild nausea in some patients, appearing only slightly more often than in the placebo group.

   523 tokens in, 97 out · 3.4 s · $0.0010 · 1 message on Pro · answered by anthropic/claude-haiku-4.5 via Amazon Bedrock · September 27, 2026

7. DeepSeek V4 Pro

   Passed: Graded 5.0 of 5 on average (lowest 5).

   In a 12-week study of 480 adults with frequent migraines, the drug Veltrazine did not reduce the number of migraine days compared with a placebo, so the main goal wasn't met. A secondary look suggested it might shorten attacks when taken within an hour of a migraine starting, but that finding wasn't planned in advance and needs its own study. Side effects were mostly mild nausea, reported by 11% of people taking the drug versus 6% on placebo.

   490 tokens in, 251 out (156 of them reasoning) · 3.5 s · $0.0016 · 1 message on Pro · answered by deepseek/deepseek-v4-pro-0813 via Parasail · September 27, 2026

8. GPT-6 Sol

   Passed: Graded 4.7 of 5 on average (lowest 4).

   In a 12-week study of 480 adults, Veltrazine did not significantly reduce the number of migraine days compared with a placebo. People who took it soon after a migraine began reported shorter attacks, but that finding needs confirmation in a separate trial. Mild nausea and other side effects were reported more often with Veltrazine (11%) than with placebo (6%).

   477 tokens in, 78 out · 1.7 s · $0.0017 · 1 message on Pro · answered by openai/gpt-6-sol via OpenAI · September 27, 2026

9. Grok 4.7

   Passed: Graded 4.7 of 5 on average (lowest 4).

   In a 12-week study of 480 adults with frequent migraines, Veltrazine did not reduce monthly migraine days compared with a placebo. People who took it within an hour of a migraine starting reported shorter attacks, but that finding was exploratory and needs a separate trial. Mild nausea was a bit more common with the drug than with placebo.

   1,712 tokens in, 207 out (137 of them reasoning) · 3.0 s · $0.0020 · 1 message on Pro · answered by x-ai/grok-4.7 via xAI · September 27, 2026

10. Kimi K3

    Passed: Graded 4.7 of 5 on average (lowest 4).

    In a 12-week trial of 480 adults with frequent migraines, the drug Veltrazine failed to reduce monthly migraine days any more than a placebo did. While a small hint suggested it might shorten attacks if taken within the first hour, that finding came from an unplanned analysis and needs to be confirmed in a separate study. The drug was generally well tolerated, though mild nausea was somewhat more common among those taking it (11% vs. 6% on placebo).

    563 tokens in, 112 out (7 of them reasoning) · 3.2 s · $0.0021 · 1 message on Pro · answered by moonshotai/kimi-k3 via Phala · September 27, 2026

11. Claude Sonnet 5

    Passed: Graded 4.7 of 5 on average (lowest 4).

    A 12-week study of 480 adults with frequent migraines found that the drug Veltrazine did not significantly reduce the number of migraine days compared to a placebo. An early, unplanned analysis suggested it might shorten attacks if taken within an hour of onset, but researchers say this needs to be confirmed in a dedicated follow-up study. Mild nausea was somewhat more common in those taking the drug (11%) than the placebo (6%).

    667 tokens in, 146 out · 2.7 s · $0.0028 · 1 message on Pro · answered by anthropic/claude-sonnet-5 via Claude Platform on AWS · September 27, 2026

12. Claude Sonnet 5.5

    Passed: Graded 4.7 of 5 on average (lowest 4).

    In a 12-week trial of 480 adults with frequent migraines, the drug Veltrazine did not reduce the number of migraine days compared with a placebo (about 4.1 vs. 4.3 days a month, a difference too small to be meaningful). An unplanned side analysis hinted that taking the drug within an hour of a migraine starting may shorten attacks, but this is only a preliminary signal that would need its own trial to confirm. Mild side effects, mostly nausea, were more common with the drug (11%) than with the placebo (6%).

    671 tokens in, 178 out · 2.4 s · $0.0034 · 1 message on Pro · answered by anthropic/claude-sonnet-5.5 via Anthropic · September 28, 2026

13. Claude Opus 5.5

    Passed: Graded 5.0 of 5 on average (lowest 5).

    In a 12-week trial of 480 adults with frequent migraines, Veltrazine did not reduce the number of monthly migraine days any more than a placebo did. People who took the drug within an hour of a migraine starting reported shorter attacks, but this wasn't part of the original study plan, so a separate trial is needed to confirm it. Side effects, mostly mild nausea, were somewhat more common with the drug (11%) than with the placebo (6%).

    669 tokens in, 168 out (20 of them reasoning) · 3.7 s · $0.0066 · 1 message on Pro · answered by anthropic/claude-opus-5.5 via Claude Platform on AWS · September 27, 2026

14. Gemini 3.1 Pro

    Passed: Graded 4.3 of 5 on average (lowest 3).

    In a recent study, the drug Veltrazine was found to be no better than a placebo at reducing the overall number of monthly migraine days for adults with frequent migraines. However, taking the drug immediately when symptoms start might shorten the length of the migraine, though researchers emphasize that more studies are needed to confirm this. Overall, the drug was well-tolerated, with a small percentage of users experiencing mild nausea.

    506 tokens in, 540 out (457 of them reasoning) · 8.2 s · $0.0075 · 1 message on Pro · answered by google/gemini-3.1-pro-preview via Google · September 27, 2026

15. GPT-6 Astra

    Passed: Graded 4.7 of 5 on average (lowest 4).

    In a 12-week study of 480 adults, Veltrazine did not significantly reduce monthly migraine days compared with a placebo. People who took it within an hour of a migraine starting reported shorter attacks, but this finding came from an unplanned analysis and needs confirmation in a separate trial. Side effects, mostly mild nausea, occurred in 11% of people taking Veltrazine versus 6% taking placebo.

    477 tokens in, 87 out · 2.3 s · $0.0091 · 1 message on Pro · answered by openai/gpt-6-astra via OpenAI · September 27, 2026

16. Claude Fable 5.1

    Passed: Graded 4.7 of 5 on average (lowest 4).

    In a 12-week study of 480 adults with frequent migraines, the drug Veltrazine did not meaningfully reduce how many migraine days people had per month compared with a placebo. A secondary look at the data hinted that taking the drug within an hour of a migraine starting might shorten attacks, but this wasn't part of the original plan and would need a dedicated study to confirm. Side effects were generally mild (mainly nausea) and somewhat more common with the drug than with placebo.

    669 tokens in, 156 out · 4.9 s · $0.0160 · 1 message on Pro · answered by anthropic/claude-fable-5.1 via Anthropic · September 27, 2026

• An article in three bullets
• An email thread in one sentence
• Decisions and action items from a meeting
• A quarterly memo for the CEO
• The best AI for summarization