LLM provider failover routes an eligible model request to an alternative when the preferred route cannot serve it. It can reduce dependence on one provider, but useful recovery also depends on the task, the replacement model, and what already happened in the agent workflow.
The planning question is concrete: if this model call cannot finish, what may the system safely do next?
Fall back only to an approved alternative; otherwise route the request for review.
Distinguish retry, rerouting, and recovery
A retry repeats a request, often after a delay. Failover changes the serving route. Workflow recovery determines whether the broader task can continue safely. Switching regions within one service and switching between providers are different designs; neither automatically restores agent state.
Set a bounded policy for waiting and attempting alternatives. Repeated attempts can add latency and cost, so define when the application should stop and report an incomplete result.
Choose a fallback routing pattern
When the preferred model fails, there are two common ways to pick the next one. An ordered fallback tries approved models in a fixed order. A capability-matched fallback tries only the models that can handle this task: its inputs, its output format, its tools, and where its data may be processed. Either way, the rule for when to stop matters as much as the list of alternatives. Every extra attempt adds time and can add cost, and a model that cannot do the job is not a recovery.
Here is how one task plays out. An agent is drafting an inspection follow-up when its preferred model stops responding before producing anything useful. It retries once and the call fails again, so it switches to the approved alternative, which has already passed the same tests. The draft comes back complete. Before offering to send it, the agent checks the delivery record. An earlier attempt may already have gone out, and the new model has no way of knowing that.
The agent's next move depends on what it can see:
Scroll horizontally to see all columns.
| What the agent sees | What it does | What it needs to decide |
|---|---|---|
| A brief error before any useful output | Tries again, within a set number of attempts and a time limit. | What went wrong, how long it has been, and how many tries are left. |
| The preferred model is unavailable | Switches to an approved alternative, if the rules allow. | Whether the alternative is approved for this task and this data. |
| No suitable model, or no tries left | Stops and shows the task as incomplete. | Who to tell, and what that person should do next. |
| Only part of a draft came back | Marks the draft as partial and asks whether to start a fresh one. | What came back, and what the user has already seen. |
| An action, such as sending a message, may already have happened | Checks the connected system before repeating anything. | Confirmation from that system, not a guess from the model. |
Judge a failover service by the whole task
Ask any supplier which errors trigger a new attempt, how approved alternatives are chosen, when attempts stop, and what your operators can see afterward. Ask whether the service only reroutes model requests or also keeps track of where the task stands. Test ordinary requests and failures on every approved alternative before you rely on it.
A successful model call cannot tell you whether an earlier email went out or a record update went through. Recovering the workflow takes that separate check. Compare the quality of each alternative, and its cost per completed task, not just whether it is available.
Check the alternative before it is needed
An alternative endpoint must be approved for the work it receives. Check these questions with the system owner:
Scroll horizontally to see all columns.
| Check | Why it matters |
|---|---|
| Input compatibility | The replacement must accept the required context and input types. |
| Output and tool compatibility | Validate required fields and tool arguments rather than assuming matching behavior. |
| Quality | Run the task's evaluation cases on each approved candidate. |
| Data destination | Confirm where requests may be processed before widening a route. |
| Cost and capacity | An alternative has its own pricing and limits. |
| Operational visibility | Record which route was attempted and whether a usable result reached the application. |
A model that can answer ordinary questions may still be unsuitable for a document extraction step. Keep fallback eligibility tied to the task.
Example: recovering a draft without repeating an action
A fictional service company uses an agent to draft a follow-up after an inspection. The inspection record is read-only during drafting. If the preferred model is unavailable before producing a usable draft, the team permits an evaluated alternative to attempt that drafting step.
Sending the follow-up is a separate action requiring approval. If a later failure occurs after a send attempt, the application checks the delivery record before offering another attempt. Changing models is not evidence that the earlier send failed.
For an interrupted response, the interface should distinguish a partial draft from a finished result. Do not append a new model's output to partial text as if it were one coherent completion unless that behavior has been designed and verified.
Test failure states alongside ordinary requests
Use a controlled pilot to exercise:
- Preferred route unavailable before output begins.
- Alternative unable to satisfy the required output format.
- No approved alternative available.
- Response interrupted after partial text.
- Tool action with an uncertain completion state.
For each case, specify the visible user state, operator evidence, and permitted next action. A clear “not completed; review required” result can be the correct outcome. Track task completion separately from successful model calls, and review agent observability when deciding what evidence operators need.
In AgentShelf
AgentShelf's published architecture describes automatic failover across 7+ providers. The infrastructure page explains configured model access; availability depends on configuration, credentials, region, plan, and catalog. Retry limits, what happens to an interrupted response, whether a tool action is ever repeated, and any availability commitment depend on your setup. Test recovery for your own workflow before relying on it.
For the surrounding architecture, read what LLM infrastructure includes and what a gateway handles.