Photo by cottonbro studio from Pexels
When your chat application slows down for users in São Paulo but runs fast in London, the culprit often isn't your code, it's the route your LLM API requests take through OpenRouter's provider network. Picking the right route requires more than intuition; it demands real latency data measured from your users' regions. This guide shows how to test OpenRouter routes systematically, interpret the results, and make routing decisions backed by evidence rather than guesswork.
TL;DR
- OpenRouter routes requests to different model providers (OpenAI, Anthropic, etc.) based on availability, cost, and fallback logic; latency varies by provider and region.
- Use synthetic probes from multiple global regions to measure Time-to-First-Byte (TTFB) and Time-to-First-Token (TTFT) for each route in your production regions.
- Regional variance often exceeds 200 ms per request; a provider fast in North America may be slow in Asia-Pacific.
- Set up daily probes for your top 3–5 routes and alert when latency degrades 15–20% above baseline.
- Combine latency data with cost per token to make routing decisions; sometimes a slower route saves money without hurting UX.
Why Provider Route Latency Matters
OpenRouter abstracts away the complexity of managing multiple LLM provider accounts, but that abstraction hides a critical truth: not all routes are equally fast from every region. OpenRouter's /api/v1/models endpoint returns a list of available routes, each pointing to a different underlying provider and sometimes different model versions or quantizations.
For Alex, the ML platform engineer shipping chat to users worldwide, this matters enormously. A 300 ms latency difference between two routes is the difference between a snappy 800 ms chat response and a sluggish 1.1 second one. Users notice. They complain. They churn.
The core problem: provider latency is not uniform. OpenAI's API might have lower latency from US east coast but higher latency from Europe. Anthropic might be the opposite. Smaller providers route through different infrastructure. Unless you measure from your actual user regions, you're flying blind.
"Most individual agent tools support one API key at a time.">, OpenRouter vs Direct Provider APIs: Routing Your Agent Traffic
This is where OpenRouter's aggregation model becomes effective: instead of picking one provider and hoping, you can load-balance or fallback across routes, conditional on latency. But first, you need baselines.
Understanding OpenRouter Routes and Latency Metrics
Before testing, clarify what you're measuring.
Routes: When you call OpenRouter's /chat/completions endpoint, you specify a model ID like anthropic/claude-3-sonnet. Behind the scenes, OpenRouter may route this to Anthropic's direct API, or to a reseller, or to a cached endpoint. Each routing path has different latency characteristics.
Time-to-First-Byte (TTFB): The elapsed time from sending a completion request to receiving the first byte of response (usually the opening { of the JSON). This includes network transit, API processing, and model tokenization. For most LLM use cases, TTFB dominates perceived latency. Target: under 400 ms from your primary region.
Time-to-First-Token (TTFT): For streaming endpoints, the time to the first token chunk. This is crucial for chat UX because users see "thinking" feedback faster. Often 50–100 ms faster than TTFB for full-response mode, depending on model and query complexity.
Regional variance: The difference in TTFB for the same route measured from different geographic regions. Real-world OpenRouter latency varies by 100–400 ms depending on whether you measure from US-East, EU-Central, or Asia-Pacific. Ignoring this is a common mistake.
Setting Up Your Latency Test Suite
Step 1: Choose Your Test Regions
Pick regions where your actual users are located, plus one or two you're considering for future expansion. A typical setup for a global app:
- North America (us-east-1 equivalent)
- Europe (eu-central-1 equivalent)
- Southeast Asia (ap-southeast-1 equivalent)
- Japan (ap-northeast-1 equivalent) [optional, if you serve Japanese market]
Step 2: Select Your Candidate Routes
List the 3–5 routes you're considering. For most teams, this includes:
- Default fallback route (e.g.,
openai/gpt-4-turbo) - Cost-optimized route (e.g.,
meta-llama/llama-2-70b-chat) - Secondary provider route (e.g.,
anthropic/claude-3-sonnet) - Experimental route (e.g., a newer quantized model)
Your progress is saved automatically in your browser.
Step 3: Craft Your Probe
Create a simple HTTP POST to OpenRouter's /chat/completions endpoint:
curl -w "TTFB: %{time_starttransfer}s\nTotal: %{time_total}s\n" \
-X POST https://openrouter.io/api/v1/chat/completions \
-H "Authorization: Bearer $OPENROUTER_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "anthropic/claude-3-sonnet",
"messages": [{"role": "user", "content": "Say hello."}],
"max_tokens": 10
}' \
--connect-timeout 5 --max-time 10
The curl flag -w "TTFB: %{time_starttransfer}s" captures time-to-first-byte. Repeat this probe for each route from each region, collecting timestamps.
Step 4: Run Probes Daily
Set up a scheduled job (cron, Lambda, or Observinio's probe scheduler) to run your test suite every 24 hours. Consistency matters; you want apples-to-apples comparison.
Record three metrics for each (route, region) pair:
- TTFB (seconds)
- Status code (200 or error)
- Timestamp (ISO 8601)
Interpreting Latency Data
Once you have 2–4 weeks of probe data, analyze it:
Calculate Baseline Percentiles
For each (route, region) pair, compute:
- p50 (median) TTFB: The middle value. If you have 28 probes, sort them and take the 14th. Typical: 200–400 ms for OpenRouter.
- p95 TTFB: The 95th percentile. This is closer to user experience for slower days. Typical: 400–700 ms.
- p99 TTFB: Tail latency. Useful for detecting rare outliers or infrastructure hiccups. Typical: 600–1200 ms.
Spot Regional Patterns
Create a table:
| Route | US-East (p50) | EU-Central (p50) | AP-Southeast (p50) | Best Region |
|---|---|---|---|---|
| OpenAI GPT-4 | 240 ms | 310 ms | 520 ms | US-East |
| Claude 3 Sonnet | 280 ms | 220 ms | 480 ms | EU-Central |
| Llama 2 70B | 350 ms | 380 ms | 420 ms | AP-Southeast |
Identify Degradation Thresholds
Set an alert threshold at p50 baseline + 15–20%. For example, if Claude 3 usually hits 220 ms in EU, alert when it exceeds 253 ms (15% overhead). This catches real degradation without false alarms from normal variance.
Making Routing Decisions
Decision Framework
Combine latency data with business metrics:
- User Experience (UX) requirement: What's the maximum acceptable end-to-end latency for your chat? If it's 1000 ms total, and your infrastructure contributes 200 ms, your LLM API budget is 800 ms. A provider at 600 ms p95 leaves a 200 ms safety margin.
- Cost per token: OpenRouter shows cost breakdowns. Compare
$/1M input tokensand$/1M output tokensacross routes. A 15% slower route might be 30% cheaper.
- Reliability (uptime): Track error rates alongside latency. A route with 250 ms latency but 2% errors is worse than a 300 ms route with 0.1% errors.
- Regional distribution: Weight regions by traffic volume. If 60% of your users are in North America, optimize for US latency even if EU suffers slightly.
Example Routing Policy
Based on your data:
IF user_region == "US" THEN
PRIMARY: openai/gpt-4-turbo (p50: 240ms, cost: $0.03/1k tokens)
FALLBACK: anthropic/claude-3-sonnet (p50: 280ms, cost: $0.015/1k tokens)
ELSE IF user_region == "EU" THEN
PRIMARY: anthropic/claude-3-sonnet (p50: 220ms)
FALLBACK: openai/gpt-4-turbo (p50: 310ms)
ELSE IF user_region == "APAC" THEN
PRIMARY: meta-llama/llama-2-70b-chat (p50: 420ms, cost: $0.0008/1k tokens)
FALLBACK: anthropic/claude-3-sonnet (p50: 480ms)
This policy prioritizes latency in each region while providing fallbacks. If the primary route fails or degrades, your system automatically tries the next.
Monitoring and Alerting
After you've chosen your routes, set up continuous monitoring:
Daily Baseline Checks
Run your probe suite every 24 hours. Log results to a time-series database (InfluxDB, CloudWatch, or Observinio's native storage). Track:
- Absolute TTFB per route per region.
- Rolling 7-day average to smooth out noise.
- Degradation alerts: page your on-call if latency exceeds threshold.
Weekly Summary Reports
Send your team a weekly email with:
- Average TTFB and p95 TTFB per route per region (vs. previous week).
- Any routes that degraded >20% or improved >10%.
- Recommended actions (e.g., "Claude 3 latency in AP-Southeast increased 18%; consider fallback to Llama").
Incident Triggers
Automate escalation:
- Yellow alert (15–20% degradation): Log it, notify Slack.
- Red alert (>20% degradation or >30% error rate): Page on-call; prepare rollback plan.
Common Pitfalls and How to Avoid Them
Pitfall 1: Testing only from your office
You'll measure latency from one ISP in one city. This doesn't reflect user experience worldwide. Always test from at least 3 geographic regions.
Pitfall 2: Averaging across all regions
"My routes average 300 ms" is useless if US is 200 ms and Asia is 500 ms. Always report by region.
Pitfall 3: One-time test
A single latency measurement tells you nothing about variance or reliability. Run probes daily for at least 4 weeks before making major routing decisions.
Pitfall 4: Ignoring cost in your decision
Optimizing for the lowest latency route can double your LLM costs. Always factor in cost per token and your user's willingness to wait slightly longer to save money.
Pitfall 5: Not alerting on degradation
If you're not probing daily, you won't notice when a provider's latency creeps up due to their infrastructure changes. Set up automated alerts at 15–20% above baseline.
FAQ
Frequently Asked Questions
Next Steps: Monitor and Iterate
Testing OpenRouter routes is not a one-time exercise. Market conditions change: OpenRouter adds new providers, model prices fluctuate, user geography shifts. The infrastructure that's optimal today may be suboptimal in six months.
Set up daily synthetic probes from your key regions, establish clear latency baselines, and configure degradation alerts. Observinio makes this seamless, no custom scripts required. You can monitor OpenRouter from 21 global regions, set per-region thresholds, and receive email alerts when latency drifts beyond acceptable bounds.
Start with your top 3 routes, collect 4 weeks of data, and make your first routing decision. Then iterate. Most teams find that systematic latency testing surfaces 1–2 quick wins per quarter: a cheaper fallback route that's only 10% slower, a regional provider that dominates in one area, or a new model that's both faster and more cost-effective.
Latency is measurable. Route selection can be data-driven. Your users, and your margins, will thank you.
Quick Reference: Latency Decision Matrix
Use this matrix during your weekly latency reviews to standardize decision-making across teams.
Additional Resources
- OpenRouter vs Direct Provider APIs: Routing Your Agent Traffic - To minimize latency when using a router, choose providers located geographically closer to you and use streaming responses.
- Provider Routing - Eden AI Documentation - Eden AI picks which provider serves it, by price, speed or latency, with health-aware failover. this one call answers both questions: which names route, and ...
- OpenRouter provider not routing requests to models #67989 - OpenClaw fails to route requests through the OpenRouter provider and returns blank responses in the chat UI.
