Monitoring OpenAI API latency in production (2026) (before production cutover)

Photo by Solen Feyissa from Pexels

You are about to cut production traffic over to a new OpenAI-powered feature. The model works, the prompts are tuned, and staging looks great. But staging is one region, one concurrency level, and one time of day. Production is none of those things. Before you flip the switch, you need a latency monitoring baseline that tells you what "normal" actually looks like, across regions, across hours, and across the specific endpoints your users will hit. This article walks through exactly how to build that baseline and what to watch once traffic starts flowing.

TL;DR

• Establish a latency baseline from multiple regions before production cutover, not after the first incident.
• Track Time to First Byte (TTFB) and Time to First Token (TTFT) separately, they reveal different failure modes.
• Regional variance on OpenAI endpoints can exceed 300 ms between the fastest and slowest probe locations.
• Synthetic probes running on a fixed schedule catch provider-side regressions that your application metrics will miss.
• Observinio's daily probes from 21 regions, degradation alerts, and weekly summaries give you this coverage without building custom infrastructure.

Why pre-cutover monitoring matters

Most teams add latency monitoring reactively. A customer in São Paulo reports that the chat feature "feels slow," an engineer checks the dashboard, sees an aggregate p50 of 420 ms, and concludes everything is fine. The problem is that the aggregate hides a p95 of 1,800 ms in South America while North American users enjoy 350 ms. By the time the support ticket arrives, the damage, churn, negative reviews, lost trust, is already done.

Pre-cutover monitoring flips the timeline. You collect latency data from every region your users occupy before a single real request is served. That data becomes your baseline. When production traffic begins, any deviation from the baseline triggers an alert, not a support ticket. The difference between "we detected a 40 % TTFT increase in ap-southeast-1 at 03:12 UTC" and "users in Singapore say the app is broken" is the difference between a five-minute remediation and a five-hour postmortem.

The metrics that matter: TTFB, TTFT, and end-to-end

Not all latency numbers are created equal. Here are the three you should track and what each one tells you:

1. Time to First Byte (TTFB), the interval between sending the HTTP request and receiving the first byte of the response. This captures DNS resolution, TLS handshake, network transit, and the provider's initial processing overhead. A spike in TTFB with stable TTFT usually points to network or load-balancer issues on the provider side.
2. Time to First Token (TTFT), the interval between sending the request and receiving the first generated token in a streaming response. TTFT includes everything in TTFB plus the model's prefill phase. A spike in TTFT with stable TTFB suggests the model itself is under load or the prompt is unusually large.
3. End-to-end (E2E) latency, the total time from request sent to last byte received. For streaming completions, this is dominated by the number of output tokens and the model's decode speed. E2E is important for user-perceived performance but is less useful for diagnosing provider issues because it conflates generation length with infrastructure speed.

"Intuition: Prompt tokens add very little latency to completion calls."

This means that if your TTFT suddenly jumps by hundreds of milliseconds, the cause is almost certainly infrastructure, not your prompt getting longer. Tracking TTFT separately from E2E lets you isolate that signal cleanly.

Building your pre-cutover baseline

A baseline is only useful if it reflects the conditions your production traffic will face. Here is a step-by-step process to build one that holds up under real load.

Step 1: Identify your user regions

Pull your analytics data and list every region where at least 5 % of your user base is located. For a typical SaaS product shipping an AI chat feature in 2026, that list often includes US East, US West, Western Europe, and at least one APAC region. Do not skip regions with smaller user counts, those users are often the first to experience degradation and the last to be noticed.

Step 2: Choose your probe endpoints

Your baseline probes should mirror your production calls as closely as possible. If your application calls POST /v1/chat/completions with gpt-4o and streaming enabled, your probe should do the same. Use a fixed prompt and a fixed max_tokens value so that variance in the results comes from infrastructure, not from generation randomness. A short system prompt plus a one-sentence user message with max_tokens: 50 is a good starting point.

Step 3: Run probes on a schedule

Run probes at least every hour from each region for a minimum of seven days. Seven days captures weekday/weekend patterns and at least one provider maintenance window. Record TTFB, TTFT, E2E, HTTP status code, and the x-request-id header from OpenAI (invaluable for support escalations). Store results in a time-series database or, more simply, let Observinio handle it, its daily probes from 21 regions already follow this pattern and store historical data for baseline comparison.

Step 4: Compute baseline thresholds

For each region, calculate p50, p90, p95, and p99 for TTFB and TTFT. Your alert thresholds should be based on the p95 values, not the p50. A common starting point:

• Warning: current p50 exceeds baseline p95 for two consecutive probe cycles.
• Critical: current p50 exceeds baseline p99, or any single probe returns an HTTP 5xx.

Step 5: Validate with a shadow traffic test

Before full cutover, route a small percentage of production traffic (5–10 %) through the new path while continuing to probe synthetically. Compare real-user latency distributions against your synthetic baseline. If they diverge significantly, investigate whether prompt length, concurrency, or authentication differences explain the gap.

Pre-cutover monitoring checklist

Use this checklist to confirm readiness before flipping the production switch:

What regional variance actually looks like

Regional variance is not a theoretical concern. Observinio probes from 21 global regions consistently show that TTFT for the same model and prompt can differ by hundreds of milliseconds depending on the source region. Regions geographically closer to OpenAI's inference clusters (primarily US-based) tend to see lower TTFB, which directly reduces TTFT. APAC and South American regions often carry an additional 150–350 ms of network transit time that no amount of prompt optimization can eliminate.

This variance has practical consequences for production cutover planning:

• Timeout configuration: A 3-second timeout that works perfectly from us-east-1 may cause intermittent failures from ap-southeast-1 where the p95 TTFT is already 2.6 seconds.
• Retry budgets: If your retry policy allows two retries with a 5-second total budget, users in high-latency regions may exhaust that budget on a single slow-but-successful request.
• User experience thresholds: Research consistently shows that users perceive delays above 1 second as "slow." If your baseline TTFT from a given region already sits at 900 ms, you have almost no headroom before users notice degradation.

Practical probe script example

Below is a minimal Python snippet that measures TTFB and TTFT for an OpenAI streaming completion. Use it as a starting point for custom probes or rely on Observinio to handle this automatically.

import time
import openai

client = openai.OpenAI()

start = time.perf_counter()
ttfb = None
ttft = None

stream = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "Reply concisely."},
        {"role": "user", "content": "What is latency monitoring?"},
    ],
    max_tokens=50,
    stream=True,
)

for chunk in stream:
    now = time.perf_counter()
    if ttfb is None:
        ttfb = now - start
    if chunk.choices and chunk.choices[0].delta.content:
        if ttft is None:
            ttft = now - start

end = time.perf_counter()
e2e = end - start

print(f"TTFB: {ttfb1000:.0f} ms | TTFT: {ttft1000:.0f} ms | E2E: {e2e*1000:.0f} ms")

Run this from each target region on a cron schedule, log the results, and you have the raw data for your baseline. Alternatively, skip the infrastructure work and let Observinio's 21-region probe network collect the same data points automatically, with degradation alerts and weekly summaries delivered to your inbox.

Start monitoring before you need to

The best time to set up latency monitoring is before your first production user hits the endpoint. Observinio probes OpenAI and OpenRouter endpoints from 21 global regions every day, compares results against historical baselines, and sends you an email alert when latency degrades, no custom infrastructure required. Visit the status page to see current latency data, or get in touch to start building your pre-cutover baseline today.

Additional Resources

• Production best practices | OpenAI API - Explore best practices for transitioning your AI projects from prototype to production, including scaling, security, and cost management.
• What AI Monitoring Actually Requires in Production - Latency in an AI system is more complex than a single response time figure. total generation time determines how much can be produced before ...
• Latency optimization | OpenAI API - This guide covers the core set of principles you can apply to improve latency across a wide variety of LLM-related use cases.