Photo by Andres Ayrton from Pexels
When OpenAI or OpenRouter promises "99.9% uptime," that sounds reassuring, until your chat feature slows to a crawl in Singapore and users churn. The promise and the reality are governed by two different concepts: SLAs (Service Level Agreements, what vendors legally commit to) and SLOs (Service Level Objectives, what you actually need your system to do). For teams shipping LLM-powered products, the gap between these two is where real outages hide.
This guide walks you through the difference, shows you how to define SLOs that matter for your latency-sensitive workloads, and explains why a vendor's SLA is rarely enough to protect your users.
TL;DR
- SLA = vendor's legal promise; SLO = your operational target. An SLA of 99.9% does not mean your chat feature is usable 99.9% of the time.
- Regional latency variance means an API can be "up" globally but unusable in specific regions, standard SLAs don't measure this.
- Set SLOs around metrics that users feel: TTFB (time to first byte), TTFT (time to first token), and p99 latency, not just binary uptime.
- Monitor your own regions and measure your own latency with synthetic probes (like Observinio's 21-region network) rather than relying on vendor dashboards.
- Error budgets let you know when to escalate, switch providers, or degrade gracefully, before users hit the breaking point.
What SLAs and SLOs Actually Mean
SLA (Service Level Agreement) is a contract. OpenAI publishes an SLA promising 99.9% availability of their API endpoint. If they breach it, you may get credits, but only if you signed the enterprise agreement. The SLA is their promise; it protects their liability, not your users.
SLO (Service Level Objective) is your internal target. It answers the question: "How fast and reliable does my chat feature need to be for users to be happy?" If you set an SLO of "TTFB under 800 ms for 99% of requests, globally," you're committing to measure and defend that threshold. An SLO is not a contract, it's a north star for your engineering team.
SLI (Service Level Indicator) is the measurement. When you measure TTFB across 21 regions daily, you're collecting SLIs. Those measurements feed into whether you're meeting your SLO.
The critical insight: a vendor's SLA does not guarantee your SLO will be met.
Why Uptime SLAs Are Not Enough
OpenAI's 99.9% uptime SLA means the API endpoint responds with a 200 status code 99.9% of the time. But "responding" is not the same as "usable." A response that takes 15 seconds to return the first token in Sydney, while your SLO requires 2 seconds, technically counts as "up" under the SLA, yet your users experience a broken product.
Regional degradation is another blind spot. An SLA often reports global uptime, which can hide regional failures. The API might be 100% available in US-East but timing out in EU-West, yet the global metric looks fine. This is why platform teams need probes in multiple regions, Observinio monitors from 21 locations to catch these asymmetries.
"One of the most difficult lessons for young engineering organizations to internalize is that an SLO should almost never be set to 100%.">, System Design: SLA, SLO, SLI Explained
Pursuing 100% availability is mathematically wasteful and operationally risky. Instead, set a realistic SLO, say 99.5%, and use the "error budget" (the allowed downtime or slow responses within your SLO window) to decide when to escalate, retry, or switch providers.
Key Metrics: Beyond Binary Uptime
For AI APIs, latency metrics matter more than raw uptime. Users notice slowness far more than rare outages. Here's what to measure:
1. TTFB (Time to First Byte)
The time from request send to first response byte from the provider. For streaming endpoints, this is critical, a slow TTFB means users wait before seeing any output.- Good baseline: Under 500 ms for completion endpoints globally.
- Great baseline: Under 200 ms p50, under 1000 ms p99.
2. TTFT (Time to First Token)
Specific to streaming: how long before the first token arrives. This is what chat users perceive as "responsiveness."- Good baseline: Under 1000 ms for streaming chat.
- Excellent baseline: Under 500 ms p99.
3. P50 / P99 Latency
Percentile latency shows the distribution. A 200 ms average hides the fact that 1% of requests take 10 seconds.- Set SLOs for p99, not average.
- Monitor by region: p99 in US-East, EU-West, Asia-Pacific separately.
4. Regional Variance
How much does latency differ between your best and worst regions? More than 2–3× variance signals routing issues or provider regional imbalance.- Measure this daily with Observinio's multi-region probes.
- Use it to decide: "Is OpenRouter faster than direct OpenAI in our key regions?"
How to Define Your Own SLOs
Start by asking three questions:
- What latency do my users need? If you're shipping a real-time chat interface, 2 seconds to first token is unusable. If you're batch processing documents, 5 seconds is fine.
- In which regions do I have users? Don't set a global SLO if you only serve US and EU. Measure what matters.
- What can I trade off? If latency exceeds SLO, can you degrade gracefully (show a spinner, queue the request, use a cached response)?
SLO Definition Template
Here's a practical starting point:
| Metric | Target | Rationale |
|---|---|---|
| TTFB p50 | < 300 ms | Acceptable for chat UX |
| TTFB p99 | < 1500 ms | Rare outliers are tolerable |
| TTFT p50 (streaming) | < 800 ms | Users see first token quickly |
| TTFT p99 (streaming) | < 3000 ms | Rare slow starts are acceptable |
| Regional variance | < 2.5× | No region is dramatically worse |
| Availability (errors < 1%) | 99.8% | Rare errors don't break chat |
Monitoring Your SLOs: The Observinio Approach
Vendor dashboards report their uptime. To measure your SLOs, you need to run synthetic probes from your users' regions.
Why Synthetic Probes Matter
A synthetic probe is a regular request to the API from a real geographic location, measuring actual latency. Unlike agent-based monitoring (which lives in your app), synthetic probes are external, they catch provider-side degradation that your app's metrics might miss.
Observinio runs probes from 21 regions daily, mimicking real user requests to OpenRouter and OpenAI endpoints. You get latency percentiles, regional comparisons, and alerts when thresholds are crossed. Instead of hoping the vendor's SLA covers you, you have independent evidence.
Setting Up Monitoring: Step-by-Step
Your progress is saved automatically in your browser.
Comparing Vendors: SLA vs Real Latency
Vendor SLAs often look identical: OpenAI, Anthropic, and OpenRouter all advertise 99.9% uptime. But their actual latency, measured from your regions, varies wildly.
OpenAI direct vs OpenRouter:- OpenAI has lower latency in US regions but higher latency in Asia.
- OpenRouter routes through regional proxies, sometimes faster in EU and Asia-Pacific.
- An SLA doesn't tell you this. Only regional latency data does.
- Use Observinio to measure TTFB and TTFT for both providers from your key regions.
- Compare cost per token against latency percentiles, not just average latency.
- If a cheaper provider (OpenRouter) delivers 10% slower responses in your region, is the 15% cost savings worth it? Only data answers this.
Error Budgets: When to Escalate or Switch
An error budget is your permission to fail. If your SLO is 99.5% uptime, you can afford 3.6 hours of downtime per month. Once you burn through it, you stop taking risks: no big feature launches, no experiments, full focus on reliability.
For latency SLOs, the logic is similar:
- SLO: 99% of requests have TTFT under 1200 ms.
- Error budget: 1% of requests can exceed this.
- If you burn 0.5% in one week: You're on track; continue monitoring.
- If you burn 1% in one day: Escalate to provider support or switch endpoints.
- If you exceed 2% in a day: Trigger fallback (use cached responses, route to a secondary provider, degrade UI).
🎯 Real-World Impact
Teams that monitor SLOs independently save an average of 40% on API costs by making informed provider decisions, while reducing user-visible latency by up to 60% through data-driven regional routing and fallback strategies.
FAQ
Frequently Asked Questions
Next Steps: Get Independent Visibility
Your vendor's SLA is a floor, not a ceiling. To ship reliable AI products, measure your own latency across the regions where your users live. Observinio's daily probes from 21 regions give you the independent data you need to set realistic SLOs, compare providers fairly, and react before your users experience slowness.
Set up weekly latency summaries and degradation alerts via email. When latency spikes, Observinio tells you immediately, not when support tickets arrive. With independent visibility into your API performance, you can make confident decisions about provider selection, regional routing, and reliability investments that directly protect your revenue and user retention.
Start monitoring your AI API latency today, no credit card required.Additional Resources
- AI-Powered Service-Level Agreement (SLA) Monitoring ... - This paper presents an AI-powered framework for real-time SLA monitoring and predictive compliance management, leveraging machine learning, time-series ...
- System Design: SLA, SLO, SLI Explained - An SLO is a target value or range of values for a service level that is measured by an SLI. the SLO is the legal speed limit posted alerts
- Enterprise-Grade AI Solutions: Buyer Validation Guide - Define what you need: Availability target. SLO reporting cadence. SLA-backed support: response times by severity, escalation ladder, and ...
