How to compare LLM models

Public benchmarks measure narrow skills on fixed test sets, and new model versions arrive often. Neither tells you whether a model writes in the tone your team uses, knows your domain, or keeps to the format you need. The only way to find out is to try it on real work.

1. Pick three real tasks: one easy, one typical, one hard. Include the files you'd really attach.

2. Decide what good means before you look at any answers, and write it down (the checklist below helps).

3. Run each task on two or three models. In llmwise, ask on one model, then switch models in the same chat and ask again; or use a new chat per model for a blind test.

4. Score the answers against what you wrote down, not against each other.

5. Keep the cheapest model that passes. Re-run the three tasks when models change.

• Accuracy

  Is it right? Check the facts and numbers you can check.

• Following instructions

  Did it do what you asked: the length, the format, the things you said to include or leave out?

• Editing needed

  How much would you have to fix before you could use it?

• Tone

  Does it sound the way you need: plain, formal, friendly?

• Honesty about gaps

  Does it say when it isn't sure, or does it guess confidently?

• Price

  Once two models are both good enough, the cheaper one wins.

In llmwise the model picker shows how many messages you have left on each model, before you send. Start on an everyday model (Claude Haiku 5.5, GPT-6 Luna, DeepSeek V4.1 Flash, and GLM 5.3 Flash) and only move up when it isn't good enough; often it is. See pricing for the allowances.