Failover decision tree (OpenRouter → OpenAI → Anthropic)

Photo by Rafael Minguet Delgado from Pexels

When your production chat feature depends on a single LLM provider, every outage becomes a customer-facing incident. A well-designed failover chain, OpenRouter as the primary aggregator, OpenAI direct as the first fallback, and Anthropic direct as the last resort, can keep time-to-first-token (TTFT) within your SLO even when one provider is degraded. This guide walks through the exact decision tree, the latency signals that trigger each transition, and the code you need to wire it all together.

TL;DR

• Use OpenRouter as your default routing layer; it already handles model availability across dozens of providers.
• Fall back to OpenAI direct when OpenRouter TTFB exceeds your P95 baseline or returns 5xx errors for more than two consecutive probes.
• Fall back to Anthropic direct when OpenAI direct also breaches thresholds or is unreachable.
• Automate the decision with health-check probes running every 60 seconds from at least three regions.
• Monitor all three legs continuously with Observinio's multi-region probes so you have the data to tune thresholds over time.

Why a three-tier failover chain matters

Most teams start with a single provider and add a second one only after a painful outage. The problem is that adding a fallback reactively, during an incident, means you haven't tested latency baselines, token-mapping differences, or prompt compatibility ahead of time. A three-tier chain gives you two independent fallback paths, which statistically reduces the probability of a total outage from a single-provider failure rate of, say, 0.5 % per month to a combined failure rate well below 0.001 % per month (assuming independent failure modes).

The three tiers serve distinct roles:

1. OpenRouter (primary): Aggregates multiple upstream providers, offers automatic model fallbacks, and often routes to the fastest available backend for a given model family. It is your default because it maximises optionality.
2. OpenAI direct (fallback 1): Bypasses the aggregation layer entirely. Useful when OpenRouter's routing adds latency or when OpenRouter itself is experiencing control-plane issues.
3. Anthropic direct (fallback 2): A completely separate provider infrastructure. If both OpenRouter and OpenAI are degraded, for example during a shared cloud-region incident, Anthropic on its own infrastructure provides genuine diversity.

The decision tree explained

The failover logic is a series of if/else checks evaluated per request (or per health-check cycle, depending on your architecture). Here is the full tree in plain language:

Step-by-step decision flow

• Send the request to OpenRouter.
• Check the response status and TTFB.
• If the response is 2xx and TTFB is below your P95 baseline (e.g., 800 ms for gpt-4o), use the response. Done.
• If the response is 429 (rate limit), apply exponential back-off up to two retries on OpenRouter. If still 429, proceed to step 3.
• If the response is 5xx, timeout, or TTFB exceeds 2× your P95 baseline, proceed to step 3.
• Send the request to OpenAI direct.
• Check the response status and TTFB.
• If 2xx and TTFB is acceptable, use the response. Done.
• If 5xx, timeout, or TTFB exceeds 2× baseline, proceed to step 5.
• If 429, retry once, then proceed to step 5.
• Send the request to Anthropic direct.
• Map the prompt to Anthropic's Messages API format (system prompt goes into the system field, not as a message).
• If 2xx, use the response. Done.
• If all three fail, return a graceful degradation response to the user (cached answer, queue for retry, or an honest error message).

Choosing your thresholds

Thresholds should come from your own baseline data, not from provider marketing pages. Here is a practical approach:

• Collect TTFB and TTFT measurements for each provider from every region you serve traffic in. Observinio's daily probes across 21 regions give you exactly this data without building custom instrumentation.
• Compute P50, P95, and P99 for each provider-region pair over a rolling 7-day window.
• Set the failover trigger at 2× P95 for latency-based decisions. This avoids flapping on normal variance while still catching genuine degradation.
• For error-based decisions, trigger after two consecutive non-2xx responses within a 120-second window.

"Documentation IndexFetch the complete documentation index at: /docs/llms.txtUse this file to discover all available pages before exploring further."

OpenRouter's own model fallback feature can handle intra-OpenRouter failover (e.g., switching from one upstream provider of claude-3.5-sonnet to another). The decision tree in this article covers the outer layer, what happens when OpenRouter itself is the problem.

Implementation: a Python failover client

Below is a production-ready skeleton in Python. It uses httpx for async HTTP and keeps provider-specific logic isolated.

import httpx
import time
import os

OPENROUTER_URL = "https://openrouter.ai/api/v1/chat/completions"
OPENAI_URL = "https://api.openai.com/v1/chat/completions"
ANTHROPIC_URL = "https://api.anthropic.com/v1/messages"

TTFB_THRESHOLD_MS = 1600  # 2x your measured P95; adjust per model

async def call_openrouter(payload: dict, client: httpx.AsyncClient) -> dict | None:
    headers = {
        "Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}",
        "Content-Type": "application/json",
    }
    start = time.monotonic()
    try:
        resp = await client.post(OPENROUTER_URL, json=payload, headers=headers, timeout=10.0)
        ttfb_ms = (time.monotonic() - start)  1000
        if resp.status_code == 200 and ttfb_ms < TTFB_THRESHOLD_MS:
            return resp.json()
    except (httpx.TimeoutException, httpx.ConnectError):
        pass
    return None


async def call_openai(payload: dict, client: httpx.AsyncClient) -> dict | None:
    headers = {
        "Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}",
        "Content-Type": "application/json",
    }
    start = time.monotonic()
    try:
        resp = await client.post(OPENAI_URL, json=payload, headers=headers, timeout=10.0)
        ttfb_ms = (time.monotonic() - start)  1000
        if resp.status_code == 200 and ttfb_ms < TTFB_THRESHOLD_MS:
            return resp.json()
    except (httpx.TimeoutException, httpx.ConnectError):
        pass
    return None

async def call_anthropic(payload: dict, client: httpx.AsyncClient) -> dict | None:
    # Map OpenAI-style payload to Anthropic Messages API
    anthropic_payload = {
        "model": "claude-sonnet-4-20250514",
        "max_tokens": payload.get("max_tokens", 1024),
        "system": next(
            (m["content"] for m in payload.get("messages", []) if m["role"] == "system"), ""
        ),
        "messages": [m for m in payload.get("messages", []) if m["role"] != "system"],
    }
    headers = {
        "x-api-key": os.environ["ANTHROPIC_API_KEY"],
        "anthropic-version": "2023-06-01",
        "Content-Type": "application/json",
    }
    try:
        resp = await client.post(ANTHROPIC_URL, json=anthropic_payload, headers=headers, timeout=12.0)
        if resp.status_code == 200:
            return resp.json()
    except (httpx.TimeoutException, httpx.ConnectError):
        pass
    return None

async def completion_with_failover(payload: dict) -> dict:
    async with httpx.AsyncClient() as client:
        result = await call_openrouter(payload, client)
        if result:
            return result

result = await call_openai(payload, client)
        if result:
            return result

result = await call_anthropic(payload, client)
        if result:
            return result

raise RuntimeError("All three providers failed. Check status at /status.")

Key implementation notes

• Timeout values differ per tier. OpenRouter gets 10 s, Anthropic gets 12 s (its TTFT tends to be slightly higher for long-context requests).
• Prompt mapping is critical. Anthropic's Messages API requires the system prompt in a separate system field. If you skip this, you will get unexpected behaviour or errors.
• Log which tier served each request. This telemetry is essential for tuning thresholds and for postmortems. Tag every response with provider: openrouter | openai | anthropic in your structured logs.
• Do not retry indefinitely. The tree above allows at most one retry on OpenRouter (for 429s) and one on OpenAI. After that, move to the next tier. Total worst-case latency for the full chain is roughly 10 + 10 + 12 = 32 s, which you can tighten by lowering individual timeouts.

Operational checklist

Use this checklist before enabling the failover chain in production:

Monitoring the chain with regional data

Failover logic is only as good as the health signals feeding it. If you rely solely on a single health-check endpoint in us-east-1, you will miss regional degradation that affects your users in Europe or Asia-Pacific.

Observinio runs synthetic probes from 21 global regions against OpenRouter and OpenAI endpoints every day. This means you get:

• Regional TTFB baselines that reflect real network paths, not just provider-side processing time.
• Degradation alerts when a specific region's latency drifts above its historical baseline, before your users notice.
• Weekly summary emails that show trend lines, so you can spot gradual performance shifts (e.g., a provider getting slower in eu-west-1 over three weeks).

Common pitfalls to avoid

1. Symmetric timeouts across tiers. If every tier gets the same 30 s timeout, your worst-case user-facing latency is 90 s. Tighten timeouts on earlier tiers and give slightly more room to the last resort.
2. Ignoring prompt compatibility. A prompt that works perfectly with GPT-4o may produce poor results with Claude if you rely on OpenAI-specific features like response_format: { type: "json_object" }. Test each tier's output quality, not just connectivity.
3. No circuit breaker. Without one, every single request during a 10-minute OpenRouter outage will wait for the full timeout before falling back. A circuit breaker that opens after three consecutive failures and stays open for 60 seconds eliminates this waste.
4. Failing over on cost, not just availability. Some teams forget that Anthropic's pricing differs from OpenAI's. A sudden failover to Anthropic under high traffic can blow through budget. Set spend alerts alongside latency alerts.
5. Not testing the failover path regularly. Run a chaos-engineering drill monthly: block traffic to OpenRouter in a staging environment and verify that the chain falls through cleanly to OpenAI and then Anthropic.

Start monitoring before you need to fail over

The best time to set up failover monitoring is before your first outage. Observinio's multi-region probes give you the TTFB baselines you need to set accurate thresholds, and degradation alerts notify you the moment a provider starts drifting, so your failover chain activates on data, not guesswork. Check the OpenRouter provider page to see how latency varies across 21 regions today.

Additional Resources

• Model Fallbacks - Automatic Failover Between Models - OpenRouter handles this fallback routing itself. The fallbacks parameter does not use Anthropic's server-side fallback feature.
• Provider Failover vs Model Fallbacks Explained - Provider failover is automatic; model fallbacks are opt-in. Learn how OpenRouter routes around outages, what triggers a fallback, and where ...
• Access OpenAI, Anthropic, Google Gemini, Mistral via ... - OpenRouter acts as a single gateway to multiple LLMs from different providers. Instead of managing separate APIs for OpenAI, Anthropic, Mistral, ...