LLM failover: keeping answers coming when a provider fails

• Overload

  A provider under heavy load turns requests away with a rate limit or an “overloaded” error. Anthropic's API errors page lists one for exactly this: 529, overloaded.

• Outages

  The provider's servers fail and answer with an error, for a minute or for hours. Each big provider logs its incidents on a status page, such as OpenAI's and Anthropic's.

• Network failures and timeouts

  The connection drops, or the provider takes too long to answer at all.

• Request errors (not outages)

  The request itself is the problem: the chat is too long for the model, or the model refuses. Retrying elsewhere doesn't fix these.

• Retries with backoff

  Try the same request again after a short wait, waiting longer each time. Cheap, and enough for brief blips.

• Another route, same model

  Send the same request to the same model through another route: another host, another region, or a service in front of several providers.

• Another model

  Fall back to a different model when the first one is down. It keeps you answering, but the answer may differ in quality, style and price.

• Circuit breakers

  After several failures in a row, stop sending to a route for a while and use the fallback straight away, instead of waiting on each request.

Most apps combine them: a quick retry, then another route, with a circuit breaker so a dead route doesn't slow every request. Falling back to a different model is the most drastic step, because the person asking gets an answer from a model they didn't choose.

Our test runs of September 27, 28, 29 and October 2, 7, 8, 9, 2026 sent the same prompts to 19 models through OpenRouter, which reports the host that answered each one. 16 of the 19 came from a single host every time; the table names it. The open-weight models were another matter: 3 of them were answered by three hosts or more, and GLM 5.3 Flash by 9.

That's what a second route looks like in practice. A model many hosts serve can fail over to another host without anyone noticing. A model one company serves can only be retried, or reached through a cloud that also hosts it.

Hosts as OpenRouter named them in each reply. The runs, prompt by prompt.

• Claude, GPT, and Gemini: sent to the model's maker whenever llmwise can reach it directly. If the maker fails before the reply starts (an overload, a server error, a dropped connection), llmwise sends the same request to the same model through OpenRouter instead.

• DeepSeek, Grok, Kimi, GLM, and Mistral: served only through OpenRouter, by hosts that don't store or train on prompts. Where a model has more than one such host, OpenRouter moves the request to another when one is down.

• Once a reply has started, a failure isn't retried behind your back: the reply stops, and you can send it again or switch models.

• It never swaps your model for another. If a model keeps failing, pick a different one in the model picker: it sees the whole chat.

Nothing, when the provider is at fault: a reply cut off by a network error, an outage, an overload or a timeout isn't counted, however much of it had arrived. A request the model itself rejects is different: it's counted like any reply once something came back, and not counted if nothing did. The limits and reliability page has the details.