A study with a negative result: every AI model's reply, tested

Summarize the findings of this study in 2 or 3 sentences for a general reader.

In a 12-week randomized trial of 480 adults with frequent migraines, the drug Veltrazine did not reduce the number of migraine days compared with a placebo (4.1 vs 4.3 days a month; the difference was not statistically significant). In an exploratory analysis, participants who took the drug within an hour of a migraine starting reported shorter attacks, but the authors caution that this subgroup was not planned in advance and needs its own trial. Side effects, mostly mild nausea, were reported by 11% of the drug group and 6% of the placebo group.

Exactly what this prompt's replies are checked against, with every other prompt of our test runs.

1. GPT-6 Luna

   Passed: Graded 4.7 of 5 on average (lowest 4).

   In this 12-week trial, Veltrazine did not significantly reduce monthly migraine days compared with a placebo. People who took it soon after a migraine began reported shorter attacks, but that finding was exploratory and needs further study; mild side effects, mainly nausea, were more common with the drug.

   477 tokens in, 63 out · 1.3 s · $0.0001 · 1 message on Pro · answered by openai/gpt-6-luna via OpenAI · September 27, 2026

2. GLM 5.3 Flash

   Passed: Graded 4.3 of 5 on average (lowest 3).

   In a 12-week trial of 480 adults with frequent migraines, the drug Veltrazine failed to reduce monthly migraine days compared with a placebo, showing no statistically significant benefit. While an exploratory analysis hinted that taking the drug early in an attack might shorten it, the authors caution this finding wasn't planned and needs confirmation in a dedicated trial. Side effects were mild, mainly nausea, affecting slightly more people on the drug (11%) than placebo (6%).

   494 tokens in, 96 out · 3.7 s · $0.0001 · 1 message on Pro · answered by z-ai/glm-5.3-flash via AtlasCloud · September 27, 2026

3. Claude Haiku 5.5

   Passed: Graded 4.7 of 5 on average (lowest 4).

   The study found that Veltrazine did not significantly reduce monthly migraine days compared with a placebo in 480 adults over 12 weeks. An unplanned analysis suggested shorter attacks when the drug was taken within an hour of onset, but the authors say this needs confirmation in a dedicated trial. Mild nausea was more common with the drug (11%) than with placebo (6%).

   670 tokens in, 127 out · 1.4 s · $0.0001 · 1 message on Pro · answered by anthropic/claude-haiku-5.5 via Anthropic · October 7, 2026

4. DeepSeek V4.1 Flash

   Passed: Graded 5.0 of 5 on average (lowest 5).

   In a 12-week trial of 480 adults with frequent migraines, the drug Veltrazine worked no better than a placebo at reducing monthly migraine days (4.1 vs. 4.3). A secondary analysis hinted that taking the drug within an hour of a migraine's start might shorten attacks, but the authors say that finding is preliminary and needs a dedicated trial. Side effects, mostly mild nausea, were somewhat more common with the drug (11%) than with placebo (6%).

   518 tokens in, 106 out (3 of them reasoning) · 0.5 s · $0.0002 · 1 message on Pro · answered by deepseek/deepseek-v4.1-flash via Together · October 8, 2026

5. DeepSeek V4 Pro

   Passed: Graded 5.0 of 5 on average (lowest 5).

   A 12-week trial of 480 adults found that the migraine drug Veltrazine did not reduce monthly migraine days compared with a placebo, so it failed its main goal. A hint that taking it early during an attack might shorten migraines was unproven and needs its own study. Side effects, mostly mild nausea, were somewhat more common with the drug than the placebo.

   490 tokens in, 100 out (21 of them reasoning) · 2.1 s · $0.0005 · 1 message on Pro · answered by deepseek/deepseek-v4-pro-0813 via Wafer · October 8, 2026

6. Gemini 3.8 Flash

   Passed: Graded 4.3 of 5 on average (lowest 4).

   A recent study found that the drug Veltrazine was no more effective than a placebo at reducing how many days adults experienced migraines. While taking the drug immediately at the start of a migraine appeared to shorten the attack, researchers emphasize that further studies are needed to confirm this unverified benefit. The medication was generally well-tolerated, with mild nausea being the most common side effect.

   506 tokens in, 76 out · 3.6 s · $0.0007 · 1 message on Pro · answered by google/gemini-3.8-flash via Google · September 27, 2026

7. GLM 5.3

   Passed: Graded 4.7 of 5 on average (lowest 4).

   In a 12-week trial of 480 adults with frequent migraines, the drug Veltrazine failed to reduce monthly migraine days compared with a placebo (4.1 vs. 4.3 days), though an unplanned exploratory analysis suggested attacks may be shorter if the drug is taken within an hour of onset — a finding the authors say needs confirmation in its own dedicated trial. Side effects, mostly mild nausea, were somewhat more common with the drug (11% vs. 6% with placebo).

   517 tokens in, 105 out (1 of them reasoning) · 0.8 s · $0.0008 · 1 message on Pro · answered by z-ai/glm-5.3 via Wafer · September 27, 2026

8. GPT-6.1 Sol

   Passed: Graded 4.7 of 5 on average (lowest 4).

   In a 12-week trial of 480 adults with frequent migraines, Veltrazine did not significantly reduce migraine days compared with a placebo. An unplanned analysis suggested that taking it within an hour of a migraine starting might shorten attacks, but this needs confirmation in a separate trial. Side effects, mostly mild nausea, occurred in 11% of people taking Veltrazine versus 6% taking the placebo.

   479 tokens in, 86 out · 1.7 s · $0.0009 · 1 message on Pro · answered by openai/gpt-6.1-sol via OpenAI · September 29, 2026

9. Claude Haiku 4.5

   Passed: Graded 4.0 of 5 on average (lowest 3).

   In a 12-week study of 480 migraine sufferers, the drug Veltrazine failed to reduce the overall frequency of migraines compared to placebo. However, when taken immediately as a migraine began, it showed promise in shortening individual attacks, though researchers caution this finding needs confirmation in a dedicated study. The drug caused mild nausea in some patients, appearing only slightly more often than in the placebo group.

   523 tokens in, 97 out · 3.4 s · $0.0010 · 1 message on Pro · answered by anthropic/claude-haiku-4.5 via Amazon Bedrock · September 27, 2026

10. Gemini 4 Argon

    Passed: Graded 4.0 of 5 on average (lowest 3).

    In a 12-week study of 480 adults, the drug Veltrazine worked no better than a placebo at reducing the number of monthly migraine days. While patients who took the drug within an hour of a migraine starting reported shorter attacks, researchers caution that this specific finding needs further testing to be confirmed. Side effects were uncommon and consisted mostly of mild nausea.

    724 tokens in, 80 out · 3.2 s · $0.0011 · 1 message on Pro · answered by google/gemini-4-argon via Google AI Studio · October 15, 2026

11. Mistral Large 4

    Passed: Graded 4.3 of 5 on average (lowest 3).

    In a 12-week trial of 480 adults, the migraine drug Veltrazine failed to reduce the number of headache days compared to a placebo. An exploratory analysis hinted that taking the drug within an hour of a migraine starting might shorten attacks, but the authors caution this wasn't a planned finding and requires its own dedicated trial. Mild nausea was the most common side effect, occurring in 11% of those taking the drug versus 6% of those on placebo.

    487 tokens in, 540 out (499 of them reasoning) · 5.8 s · $0.0015 · 1 message on Pro · answered by mistralai/mistral-large-4-0 via Mistral · October 9, 2026

12. GPT-6 Sol

    Passed: Graded 4.7 of 5 on average (lowest 4).

    In a 12-week study of 480 adults, Veltrazine did not significantly reduce the number of migraine days compared with a placebo. People who took it soon after a migraine began reported shorter attacks, but that finding needs confirmation in a separate trial. Mild nausea and other side effects were reported more often with Veltrazine (11%) than with placebo (6%).

    477 tokens in, 78 out · 1.7 s · $0.0017 · 1 message on Pro · answered by openai/gpt-6-sol via OpenAI · September 27, 2026

13. Kimi K3

    Passed: Graded 4.7 of 5 on average (lowest 4).

    In a 12-week trial of 480 adults with frequent migraines, the drug Veltrazine failed to reduce monthly migraine days any more than a placebo did. While a small hint suggested it might shorten attacks if taken within the first hour, that finding came from an unplanned analysis and needs to be confirmed in a separate study. The drug was generally well tolerated, though mild nausea was somewhat more common among those taking it (11% vs. 6% on placebo).

    563 tokens in, 112 out (7 of them reasoning) · 3.2 s · $0.0021 · 1 message on Pro · answered by moonshotai/kimi-k3 via Phala · September 27, 2026

14. Grok 4.7

    Passed: Graded 4.3 of 5 on average (lowest 4).

    In a 12-week study of 480 adults who get frequent migraines, Veltrazine did not reduce how many migraine days people had compared with a placebo. People who took it within an hour of a migraine starting sometimes had shorter attacks, but that finding was exploratory and needs to be tested in its own study. Mild nausea was a bit more common with the drug than with the placebo.

    1,713 tokens in, 113 out (33 of them reasoning) · 2.8 s · $0.0024 · 1 message on Pro · answered by x-ai/grok-4.7 via xAI · October 2, 2026

15. Claude Sonnet 5

    Passed: Graded 4.7 of 5 on average (lowest 4).

    A 12-week study of 480 adults with frequent migraines found that the drug Veltrazine did not significantly reduce the number of migraine days compared to a placebo. An early, unplanned analysis suggested it might shorten attacks if taken within an hour of onset, but researchers say this needs to be confirmed in a dedicated follow-up study. Mild nausea was somewhat more common in those taking the drug (11%) than the placebo (6%).

    667 tokens in, 146 out · 2.7 s · $0.0028 · 1 message on Pro · answered by anthropic/claude-sonnet-5 via Claude Platform on AWS · September 27, 2026

16. Claude Sonnet 5.5

    Passed: Graded 4.7 of 5 on average (lowest 4).

    In a 12-week trial of 480 adults with frequent migraines, the drug Veltrazine did not reduce the number of migraine days compared with a placebo (about 4.1 vs. 4.3 days a month, a difference too small to be meaningful). An unplanned side analysis hinted that taking the drug within an hour of a migraine starting may shorten attacks, but this is only a preliminary signal that would need its own trial to confirm. Mild side effects, mostly nausea, were more common with the drug (11%) than with the placebo (6%).

    671 tokens in, 178 out · 2.4 s · $0.0034 · 1 message on Pro · answered by anthropic/claude-sonnet-5.5 via Anthropic · September 28, 2026

17. Claude Opus 5.5

    Passed: Graded 5.0 of 5 on average (lowest 5).

    In a 12-week trial of 480 adults with frequent migraines, Veltrazine did not reduce the number of monthly migraine days any more than a placebo did. People who took the drug within an hour of a migraine starting reported shorter attacks, but this wasn't part of the original study plan, so a separate trial is needed to confirm it. Side effects, mostly mild nausea, were somewhat more common with the drug (11%) than with the placebo (6%).

    669 tokens in, 168 out (20 of them reasoning) · 3.7 s · $0.0066 · 1 message on Pro · answered by anthropic/claude-opus-5.5 via Claude Platform on AWS · September 27, 2026

18. Gemini 3.1 Pro

    Passed: Graded 4.3 of 5 on average (lowest 3).

    In a recent study, the drug Veltrazine was found to be no better than a placebo at reducing the overall number of monthly migraine days for adults with frequent migraines. However, taking the drug immediately when symptoms start might shorten the length of the migraine, though researchers emphasize that more studies are needed to confirm this. Overall, the drug was well-tolerated, with a small percentage of users experiencing mild nausea.

    506 tokens in, 540 out (457 of them reasoning) · 8.2 s · $0.0075 · 1 message on Pro · answered by google/gemini-3.1-pro-preview via Google · September 27, 2026

19. GPT-6 Astra

    Passed: Graded 4.7 of 5 on average (lowest 4).

    In a 12-week study of 480 adults, Veltrazine did not significantly reduce monthly migraine days compared with a placebo. People who took it within an hour of a migraine starting reported shorter attacks, but this finding came from an unplanned analysis and needs confirmation in a separate trial. Side effects, mostly mild nausea, occurred in 11% of people taking Veltrazine versus 6% taking placebo.

    477 tokens in, 87 out · 2.3 s · $0.0091 · 1 message on Pro · answered by openai/gpt-6-astra via OpenAI · September 27, 2026

20. Claude Fable 5.1

    Passed: Graded 4.7 of 5 on average (lowest 4).

    In a 12-week study of 480 adults with frequent migraines, the drug Veltrazine did not meaningfully reduce how many migraine days people had per month compared with a placebo. A secondary look at the data hinted that taking the drug within an hour of a migraine starting might shorten attacks, but this wasn't part of the original plan and would need a dedicated study to confirm. Side effects were generally mild (mainly nausea) and somewhat more common with the drug than with placebo.

    669 tokens in, 156 out · 4.9 s · $0.0160 · 1 message on Pro · answered by anthropic/claude-fable-5.1 via Anthropic · September 27, 2026

• An article in three bullets
• An email thread in one sentence
• Decisions and action items from a meeting
• A quarterly memo for the CEO
• The best AI for summarization