Photo by Sanket Mishra from Pexels

Every production system that calls the OpenAI Chat Completions API needs a latency baseline, a documented set of expected response-time numbers broken down by model, region, and payload size. Without one, you are flying blind: you cannot tell whether a 2.8-second P95 is normal for gpt-4o in Frankfurt or a sign that something is degrading. This template gives you a repeatable, copy-and-adapt framework for capturing those numbers, storing them, and turning them into actionable alerts.

TL;DR

  • A latency baseline records expected TTFB, TTFT, and end-to-end times for each model-region-payload combination you use in production.
  • You need at least seven consecutive days of probe data before the baseline is statistically useful.
  • The template below covers metric selection, probe design, storage format, threshold math, and alert rules.
  • Baselines drift, schedule a monthly review or automate it with Observinio weekly summaries.
  • Start with the three regions that serve 80 % of your traffic, then expand.
Key takeaway: A latency baseline is only valuable if it is segmented by model, region, and payload size, collected over at least seven days, and refreshed monthly. Without regular re-baselining, your alert thresholds drift and your team loses the ability to distinguish real degradation from normal variance.
0+
Core metrics tracked per probe
0 days
Minimum baseline collection window
0
Global regions monitored by Observinio

Why you need a latency baseline before anything else

cloud infrastructure operations
Photo by Pixabay from Pexels

Most teams discover latency problems reactively: a customer complains, an SRE spots a spike in a general-purpose dashboard, or a Slack channel fills with "is the API slow for anyone else?" messages. The root cause is almost always the same, nobody wrote down what "normal" looks like.

A baseline solves three problems at once:

  1. Incident detection, you can set thresholds that fire only when latency genuinely deviates, not on every random fluctuation.
  2. Provider comparison, when evaluating OpenAI direct versus OpenRouter, or comparing gpt-4o against gpt-4o-mini, you need apples-to-apples numbers collected under identical conditions.
  3. Capacity planning, if your P95 is already at 3.4 seconds and your SLO promises 4 seconds, you know exactly how much headroom you have before you need to add a fallback model or a second region.
Without a baseline document, every latency conversation devolves into anecdotes. With one, you have a shared source of truth that the platform team, the product team, and the on-call rotation can all reference.

Metrics to include in your baseline

latency performance analytics
Photo by Rafael Minguet Delgado from Pexels

Not every latency number is equally useful. The template focuses on five core metrics:

Core metric definitions

  • TTFB (Time to First Byte): The interval from sending the HTTP request to receiving the first byte of the response. This captures network round-trip plus provider queue time. For streaming endpoints, TTFB is the single most important user-perceived metric because it determines how quickly the first token appears on screen.
  • TTFT (Time to First Token): Closely related to TTFB but measured at the application layer after SSE parsing. In practice, TTFT = TTFB + a small deserialization overhead (usually < 20 ms). Track both if you use streaming; for non-streaming calls, TTFB alone is sufficient.
  • End-to-end latency: The total wall-clock time from request sent to last byte received. For non-streaming calls this is the number your users feel. For streaming calls it matters less for perceived speed but is critical for throughput planning.
  • Token throughput (tokens/second): Divide completion_tokens by end-to-end latency. This normalizes across different output lengths and lets you compare models fairly.
  • Error rate: Percentage of requests returning 4xx/5xx or timing out. A baseline with a 0.3 % error rate tells you that 0.8 % is worth investigating.
For each metric, record the mean, median (P50), P90, P95, P99, min, max, and standard deviation. The community has already surfaced how dramatically these can differ across API styles:
"Responses: mean=4.268s median=2.349s min=1.421s max=21.711s stdev=4.903s
Chat : mean=1.354s median=1.298s min=0.902s max=2.385s stdev=0.330s Statistical: Store = False." >, Stateful Responses API Much Slower Than Chat Completions

That quote illustrates exactly why baselines matter: the Chat Completions endpoint shows a tight distribution (stdev 0.330 s) while the Responses API is all over the map (stdev 4.903 s). If you only tracked the mean, you would miss the massive tail latency in the Responses path.

Step-by-step: building your baseline from scratch

OpenAI chat completions latency baseline template process
Figure 1: OpenAI chat completions latency baseline template at a glance.

Follow these eight steps to go from zero to a production-ready baseline document.

Step 1 – List model-region-payload combinations
0%
  1. List your model-region-payload combinations. Create a table with every model you call in production (e.g., gpt-4o, gpt-4o-mini), every region your users connect from (e.g., us-east-1, eu-west-1, ap-southeast-1), and two or three representative prompt sizes (short: ~50 tokens, medium: ~500 tokens, long: ~2 000 tokens). Each row in this table is a "probe configuration."
Step 2 – Design a deterministic probe prompt
0%
  1. Design a deterministic probe prompt. Use a fixed system message and user message so that output length is roughly consistent across runs. A good pattern is a factual question with a constrained answer length: "Summarize the HTTP/2 protocol in exactly three sentences." Avoid creative prompts, they produce variable output lengths that add noise to your measurements.
Step 3 – Choose probe frequency
0%
  1. Choose your probe frequency. For the initial baseline period, run each probe configuration every 15 minutes. That gives you 96 data points per day per configuration. After the baseline is established, you can drop to every 30 or 60 minutes for ongoing monitoring.
Step 4 – Run probes for seven days
0%
  1. Run probes for at least seven days. Weekday and weekend traffic patterns differ on the provider side. Seven days captures both. If you can afford 14 days, even better, you will catch biweekly maintenance windows and model update rollouts.
Step 5 – Store raw results
0%
  1. Store raw results in a structured format. Each probe result should include: timestamp (UTC), region, model, prompt token count, completion token count, TTFB (ms), TTFT (ms), end-to-end latency (ms), HTTP status code, and any error message. A simple JSON Lines file or a Postgres table works fine.
Step 6 – Compute summary statistics
0%
  1. Compute summary statistics. After the collection window closes, calculate mean, median, P90, P95, P99, min, max, and standard deviation for each metric, grouped by model-region-payload. These numbers are your baseline.
Step 7 – Set alert thresholds
0%
  1. Set alert thresholds. A common starting point: alert when the rolling 15-minute P95 exceeds the baseline P95 by more than 40 %, or when the error rate exceeds twice the baseline error rate. Tune these multipliers after a few weeks of real alerts.
Step 8 – Document and version the baseline
0%
  1. Document and version the baseline. Store the baseline table in your repository alongside your infrastructure code. Include the collection date range, probe prompt text, and any provider-side notes (e.g., "collected during OpenAI API version 2025-06-01"). When you re-baseline, keep the old version for comparison.

Baseline template (copy and adapt)

Below is a ready-to-use Markdown table you can paste into your team wiki or repo. Fill in the numbers from your probe data.

ModelRegionPayloadTTFB P50TTFB P95E2E P50E2E P95Tok/s P50Error %Collection Period
gpt-4ous-east-1shortmsmsmsms%YYYY-MM-DD – MM-DD


gpt-4ous-east-1mediummsmsmsms%


gpt-4oeu-west-1shortmsmsmsms%


gpt-4o-minius-east-1shortmsmsmsms%


gpt-4o-miniap-southeast-1mediummsmsmsms%

Probe configuration checklist

Use this checklist every time you set up or modify a probe:

Your progress is saved automatically in your browser.

Keeping the baseline current

developer checking api metrics
Photo by Daniil Komov from Pexels

A baseline is not a one-time artifact. OpenAI ships model updates, adjusts rate limits, and changes infrastructure routing regularly. A baseline from three months ago may be dangerously stale.

When to re-baseline

  • After a model version change. When OpenAI updates gpt-4o to a new snapshot, run a fresh seven-day collection. Compare the new numbers to the old baseline and update your alert thresholds.
  • After you change regions. If you add ap-northeast-1 to your deployment, you need baseline data for that region before you can set meaningful alerts.
  • Monthly, as a hygiene practice. Even without obvious changes, provider-side infrastructure evolves. A monthly re-baseline catches gradual drift. Observinio's weekly summary emails make this easy, they show you the trailing seven-day P50 and P95 for every model-region pair, so you can compare against your stored baseline without running a manual collection.
  • After an incident. If you experienced a prolonged degradation event, exclude that window from your baseline data or re-collect. Contaminated baselines lead to overly generous thresholds.

Automating baseline comparison

Instead of manually comparing spreadsheets, write a simple script that pulls the latest week of probe data, computes summary statistics, and diffs them against the stored baseline. Flag any metric where the new value exceeds the baseline by more than your chosen tolerance (e.g., 20 % for P50, 40 % for P95). This script can run as a weekly cron job or be triggered by Observinio's webhook integration.

A minimal Python snippet for the comparison logic:

def check_drift(current: dict, baseline: dict, tolerance: float = 0.4) -> list:
    """Return list of metrics that drifted beyond tolerance."""
    alerts = []
    for metric in ["ttfb_p95", "e2e_p95", "error_rate"]:
        if baseline[metric] == 0:
            continue
        drift = (current[metric] - baseline[metric]) / baseline[metric]
        if drift > tolerance:
            alerts.append({
                "metric": metric,
                "baseline": baseline[metric],
                "current": current[metric],
                "drift_pct": round(drift  100, 1),
            })
    return alerts

Alert threshold calculator

Enter your baseline P95 latency and desired tolerance to compute the alert threshold.





Click Calculate to see your threshold.

Common mistakes to avoid

Teams that build baselines for the first time often fall into these traps:

  1. Using a single global average. A 1.8-second global mean hides the fact that ap-southeast-1 runs at 3.2 seconds. Always segment by region.
  2. Collecting for only one day. One day of data captures one traffic pattern. You need at least seven days to see weekday/weekend variance and provider maintenance windows.
  3. Ignoring prompt size. A 50-token prompt and a 2 000-token prompt produce very different latency profiles on the same model. Baseline both.
  4. Setting thresholds too tight. If your alert fires on every 10 % deviation, you will get paged constantly and start ignoring alerts. Start at 40 % above P95 and tighten gradually.
  5. Never re-baselining. A six-month-old baseline is worse than no baseline because it gives false confidence. Schedule re-collection or use automated drift detection.
Key takeaway: A latency baseline is only valuable if it is segmented by model, region, and payload size, collected over at least seven days, and refreshed monthly. Without regular re-baselining, your alert thresholds drift and your team loses the ability to distinguish real degradation from normal variance.

FAQ

Frequently Asked Questions

Aim for at least 672 data points per probe configuration (96 per day × 7 days at 15-minute intervals). This gives you enough samples to compute stable P95 and P99 values. If your probe frequency is lower, say, every 30 minutes, extend the collection window to 14 days to compensate.
Yes. Streaming responses have a meaningful TTFB/TTFT that non-streaming calls do not expose in the same way. The end-to-end latency profile also differs because the provider starts sending tokens before generation is complete. If your production code uses streaming, your baseline probes must use streaming too.
Absolutely. Observinio runs daily probes from 21 global regions against OpenAI and OpenRouter endpoints. You can use the status page data and weekly summary emails as your baseline source, especially for regions where you do not have your own infrastructure. The advantage is that Observinio probes are already standardized and run consistently, removing the operational burden of maintaining your own probe fleet.
Do not remove outliers from your raw data, they represent real-world conditions your users may experience. Instead, use percentile-based metrics (P50, P95, P99) rather than means for your alert thresholds. Percentiles are naturally robust to outliers. If you see a cluster of extreme values (e.g., 20+ second responses), investigate whether they correlate with a known provider incident and annotate your baseline accordingly.
It depends heavily on prompt size and region. For short prompts (under 100 tokens) from US East, a P95 TTFB under 800 ms and a P95 end-to-end under 2.5 seconds is a reasonable starting expectation based on community reports. However, you should always derive your target from your own measured baseline rather than adopting someone else's numbers. Regional variance alone can add 200–600 ms depending on proximity to OpenAI's inference clusters.

Start monitoring before your next incident

Building a latency baseline is the first step; keeping it current is the ongoing work. If you would rather skip the probe infrastructure and get straight to the data, Observinio already tracks OpenAI Chat Completions latency from 21 regions with daily probes, degradation alerts, and weekly summary emails. Set up alerts once and let the baseline come to you, so the next time latency drifts, you find out from an email, not from a customer.

Additional Resources