Photo by https://kaboompics.com/ from Pexels

When you call OpenAI's streaming chat completion endpoint, the response arrives as a series of server-sent events rather than a single JSON blob. That changes everything about how you measure latency. A single "average response time" number is almost meaningless for streaming, you need a baseline that captures Time to First Byte (TTFB), Time to First Token (TTFT), inter-token interval, and total stream duration. This guide walks you through building a baseline calculator that turns raw probe data into actionable thresholds you can alert on.

TL;DR

  • Streaming APIs require at least four latency metrics, TTFB, TTFT, inter-token interval (p50/p95), and total stream duration, not just one average.
  • A baseline calculator collects samples over a rolling window (typically 7 days), computes percentile bands, and flags deviations.
  • You can build a minimal calculator in Python with fewer than 80 lines of code using the OpenAI SDK's streaming iterator.
  • Regional variance matters: a baseline built from US-East probes will misfire when applied to traffic from Singapore or Frankfurt.
  • Observinio's daily probes across 21 regions give you ready-made baseline data without maintaining your own probe infrastructure.
Key takeaway: A single "average response time" is meaningless for streaming APIs. Build per-region, per-model baselines using TTFB, TTFT, inter-token interval, and total duration, then alert on p95 deviations with a consecutive-breach rule to catch real degradation without drowning in false positives.
0+
Streaming latency metrics tracked
0
Global probe regions
0-day
Default rolling baseline window

Why Streaming Latency Needs Its Own Baseline

network monitoring dashboard screen
Photo by Keysi Estrada from Pexels

Non-streaming API calls have a simple lifecycle: request out, response back, measure the gap. Streaming calls are fundamentally different. The server opens a connection, sends an initial chunk (often just the role and an empty delta), and then drips tokens one by one until the [DONE] sentinel arrives. Users perceive quality through two moments: how quickly the first word appears on screen, and how smoothly subsequent words flow. A baseline that only tracks total duration misses both.

The four metrics that matter

  1. TTFB (Time to First Byte): The interval between sending the HTTP request and receiving the first byte of the response. This captures network round-trip plus any queue time on OpenAI's side.
  2. TTFT (Time to First Token): The interval between the request and the first non-empty content delta in the SSE stream. TTFT is always ≥ TTFB, and the gap between them reveals serialization and model warm-up overhead.
  3. Inter-token interval (ITI): The time between consecutive content deltas. Compute p50 and p95 across all tokens in a single stream, then aggregate across streams. Spikes in p95 ITI indicate GPU contention or throttling.
  4. Total stream duration: Wall-clock time from request to [DONE]. Useful for capacity planning but less useful for user-experience alerting.
Without separating these four, you cannot tell whether a slowdown is caused by queuing (high TTFT, normal ITI), throttling (normal TTFT, high ITI), or a network issue (high TTFB across all models).

Collecting Raw Samples

server room data center
Photo by panumas nikhomkhai from Pexels

Before you can calculate a baseline, you need a consistent stream of measurements. The key principles are:

  • Fixed prompt and parameters. Use a deterministic prompt (e.g., "Explain Newton's second law in three sentences.") with temperature: 0 and a fixed max_tokens value. This removes output variability from your measurements.
  • Consistent region. Tag every sample with the region it was collected from. A probe running in us-east-1 and another in eu-west-1 should never be mixed into the same baseline bucket.
  • Sufficient sample size. Aim for at least 30 samples per region per day. Fewer than that and your percentile estimates will be noisy.
  • Regular cadence. Run probes at evenly spaced intervals (e.g., every 30 minutes) rather than in bursts. Bursts can trigger rate limits and skew your data.

Probe script example

Below is a minimal Python script that calls the OpenAI streaming endpoint and records the four metrics. It uses only the official openai library and the standard library's time module.

import time
import json
from openai import OpenAI


client = OpenAI()  # uses OPENAI_API_KEY env var



def measure_stream(model: str = "gpt-4o-mini", max_tokens: int = 128) -> dict:
    """Send a fixed prompt via streaming and return latency metrics."""
    prompt = "Explain Newton's second law in three sentences."
    t_start = time.perf_counter()
    first_byte_at = None
    first_token_at = None
    token_times = []



stream = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
        temperature=0,
        stream=True,
    )



for chunk in stream:
        now = time.perf_counter()
        if first_byte_at is None:
            first_byte_at = now
        delta = chunk.choices[0].delta if chunk.choices else None
        if delta and delta.content:
            if first_token_at is None:
                first_token_at = now
            token_times.append(now)



t_end = time.perf_counter()



# Compute inter-token intervals
    iti_list = [
        token_times[i] - token_times[i - 1]
        for i in range(1, len(token_times))
    ]



return {
        "ttfb_ms": round((first_byte_at - t_start)  1000, 1),
        "ttft_ms": round((first_token_at - t_start)  1000, 1),
        "iti_p50_ms": round(sorted(iti_list)[len(iti_list) // 2]  1000, 1)
            if iti_list else 0,
        "iti_p95_ms": round(sorted(iti_list)[int(len(iti_list)  0.95)]  1000, 1)
            if iti_list else 0,
        "total_ms": round((t_end - t_start)  1000, 1),
        "token_count": len(token_times),
    }



if __name__ == "__main__":
    result = measure_stream()
    print(json.dumps(result, indent=2))

Run this on a cron schedule, append results to a JSON-lines file or a database table, and you have the raw material for baseline calculation.

Computing the Baseline

Baseline calculator for OpenAI streaming APIs process
Figure 1: Baseline calculator for OpenAI streaming APIs at a glance.

A baseline is not a single number, it is a percentile band computed over a rolling window. Here is the step-by-step process:

  1. Choose a rolling window. Seven days is a good default. It smooths out weekday/weekend traffic patterns on OpenAI's infrastructure without hiding gradual regressions.
  2. Group samples by region and model. Never mix regions. A gpt-4o baseline from Tokyo is a different distribution than one from Virginia.
  3. Compute percentiles for each metric. Calculate p50 (median), p75, p90, and p95 for TTFB, TTFT, ITI-p50, ITI-p95, and total duration.
  4. Set alert thresholds. A common approach: alert when the latest probe exceeds the p95 of the rolling window by more than 20%. This catches real degradation while ignoring normal tail variance.
  5. Recalculate daily. Shift the window forward each day so the baseline adapts to legitimate changes (e.g., OpenAI deploying a faster model version).
p50 – Median latency
0%
p90 – High-confidence band
0%
p95 – Alert threshold input
0%

Threshold formula

For each metric M, the alert threshold is:

threshold(M) = baseline_p95(M) × 1.2

If three consecutive probes exceed the threshold, fire an alert. The "three consecutive" rule prevents one-off network hiccups from waking someone up at 3 AM.

Alert Threshold Calculator


Alert threshold: 384.0 ms

"Just built one today with all the updated models pricing: OpenAI API Pricing Calculator | 100% Free."
>, Looking for Pricing Calculator*

Cost calculators are useful for budgeting, but they tell you nothing about whether the API is actually fast enough for your users. A baseline calculator fills that gap by turning latency samples into quantitative expectations.

Regional Variance: The Hidden Variable

One of the most common mistakes is building a single global baseline. OpenAI's infrastructure is not uniformly distributed. Requests routed from São Paulo experience different network paths, different load balancer pools, and potentially different GPU clusters than requests from London. In practice, TTFT can vary by 200–400 ms between regions for the same model and prompt.

How to handle regional baselines

  • Maintain one baseline per (region, model) pair. If you monitor from 5 regions across 3 models, that is 15 independent baselines.
  • Compare like with like. When a user in Frankfurt reports slowness, compare against the Frankfurt baseline, not the global average.
  • Watch for regional divergence. If one region's p50 TTFT suddenly jumps while others stay flat, the issue is likely network or regional infrastructure, not a model-level regression.
Observinio's probe network covers 21 regions worldwide, which means you get per-region baseline data out of the box. Instead of deploying and maintaining your own probe fleet, you can rely on Observinio's daily measurements and receive email alerts when any region crosses its baseline threshold. Check the status page for a live view of current latency across all monitored regions.

Baseline Calculator Checklist

Use this checklist when setting up your own baseline calculator for OpenAI streaming APIs:

Your progress is saved automatically in your browser.

Common Pitfalls

Mixing streaming and non-streaming data

If your application uses both streaming and non-streaming calls, keep the baselines completely separate. Non-streaming TTFB includes the full generation time; streaming TTFB does not. Mixing them produces meaningless percentiles.

Ignoring warm-up effects

The first request after a cold period (e.g., a new deployment or a long idle gap) often has elevated TTFT due to connection setup and potential model loading. Exclude or flag the first probe of each session to avoid inflating your baseline.

Over-alerting on ITI spikes

Inter-token intervals are inherently noisy. A single slow chunk can push ITI-p95 above threshold without any user-visible impact. Use the consecutive-breach rule and consider alerting only on ITI-p95 rather than ITI-p50.

Not accounting for model updates

OpenAI periodically updates model weights and infrastructure. After a known model update, consider resetting the baseline window to avoid comparing against stale data. A 7-day window naturally flushes old data, but a manual reset gives you cleaner signal during transitions.

Key takeaway: A single "average response time" is meaningless for streaming APIs. Build per-region, per-model baselines using TTFB, TTFT, inter-token interval, and total duration, then alert on p95 deviations with a consecutive-breach rule to catch real degradation without drowning in false positives.

Frequently Asked Questions

TTFB (Time to First Byte) measures when the very first byte of the HTTP response arrives, which is typically the SSE header or an initial empty chunk. TTFT (Time to First Token) measures when the first actual content token appears in the stream. TTFT is always equal to or greater than TTFB. The gap between them reflects server-side processing before the model starts generating output.
A minimum of 30 samples per region per day gives you statistically meaningful percentile estimates. For production-critical applications, 48 samples per day (one every 30 minutes) is a practical target. Over a 7-day window, that yields 336 data points per baseline bucket, more than enough for stable p95 calculations.
No. Different models have different latency profiles. For example, gpt-4o-mini typically has lower TTFT and faster inter-token intervals than gpt-4o because it is a smaller model. Always maintain separate baselines per model. If you switch from one model to another, you need to build a new baseline from scratch.
Daily recalculation with a 7-day rolling window is the standard approach. This ensures the baseline adapts to gradual infrastructure changes while still catching sudden regressions. If you operate in a highly dynamic environment (e.g., frequently switching models or regions), you might shorten the window to 3–5 days, but be aware that shorter windows increase sensitivity to noise.
Yes. Observinio runs daily probes against OpenAI (and OpenRouter) endpoints from 21 global regions. The collected TTFB and TTFT data feeds into per-region baselines, and you receive degradation alerts via email when latency crosses established thresholds. You can view current and historical latency on the status page and receive weekly summary reports without deploying any probe infrastructure yourself.

Start Monitoring Your Baselines Today

Building a baseline calculator is a one-time investment that pays off every time OpenAI's latency shifts and you catch it before your users do. If you would rather skip the infrastructure work, Observinio already runs streaming probes from 21 regions daily, computes baselines automatically, and sends you email alerts when any region degrades. Visit the status page to see live latency data, or get in touch to set up alerts for your specific models and regions.

Additional Resources