Monitoring OpenAI API latency in production (2026) (in monorepo setups)

Photo by Andrew Neel from Pexels

When you ship LLM features across multiple services in a monorepo, latency stops being a single number. It becomes a web of dependencies: your routing layer, regional provider variance, token throughput, and queueing delays all interact in ways that production dashboards rarely expose. By 2026, teams treating OpenAI API latency as a single metric have already lost visibility of regional slowdowns and cascading failures.

This article covers how to set up production-grade latency monitoring for OpenAI and similar endpoints in monorepo environments, with practical steps to measure what matters (TTFB, TTFT, regional variance) and alert before your users do.

TL;DR

• OpenAI latency in production varies dramatically by region and depends on your routing logic; aggregate averages hide degradation.
• Measure TTFB (time-to-first-byte) and TTFT (time-to-first-token) separately; they surface different bottlenecks in your stack.
• In monorepos, inject lightweight probes at package boundaries and route them through regional endpoints to catch provider issues early.
• Set baselines for each region and provider, then alert when latency crosses thresholds for more than 2–3 consecutive requests.
• Use weekly trend reports to spot patterns (e.g., slower regions at peak times) and inform provider or routing decisions.

Key takeaway: Regional latency monitoring in monorepo setups separates signal from noise by measuring TTFB and TTFT independently, routing probes through multiple datacenters, and alerting only when trends emerge (3+ consecutive slow requests), not on single spikes.

Why regional latency matters in monorepo setups

Monorepos consolidate chat services, embedding pipelines, and fallback logic under one codebase. When OpenAI latency spikes in Europe, a chat service in the US may not notice, until European users start timing out or switching features off. Single-region or aggregate monitoring misses this entirely.

OpenAI operates datacenters worldwide, but latency from your infrastructure to their API differs by geography. A request from Sydney adds 150–250 ms of network transit alone. When you route through a US-based gateway to save costs, you inherit both regional and proxy latency. In monorepos, different services may route differently (one service hits the API directly, another goes through a shared inference gateway), making diagnosis harder.

Moreover, provider performance degrades unpredictably. OpenAI published data showing that completion latency can shift 10–30% month-over-month due to model updates or traffic patterns. Without proactive measurement from multiple regions, you discover these shifts when support tickets arrive.

Measuring latency: TTFB vs. TTFT

Most teams conflate "latency" into one number. In reality, two metrics drive user experience:

Time-to-First-Byte (TTFB), the span from request send to the first response byte arriving. This includes network latency, OpenAI's request queueing, and their initial inference pipeline startup. TTFB is critical for perceived responsiveness; users feel slow chat responses within 200–300 ms.

Time-to-First-Token (TTFT), a streaming-specific metric: the time from send to the first token in the completion stream. For chat, TTFT feels more responsive than TTFB because tokens continue arriving incrementally. A 500 ms TTFT is often acceptable if tokens flow every 50–100 ms afterward.

"Intuition: Prompt tokens add very little latency to completion calls."

>, Production best practices

In a monorepo, your routing layer and batch processing can distort both metrics. If your chat service queues requests before sending them to OpenAI, TTFB includes queue wait time. If your inference gateway defers streaming setup, TTFT suffers even if OpenAI is fast.

Actionable measurement approach:

• Log TTFB for all completion calls (synchronous and streaming).
• For streaming, also log TTFT and time-between-tokens.
• Tag logs with provider (OpenAI direct vs. OpenRouter), region (inferred from your service's datacenter), model, and user tier.
• Export to a time-series database so you can slice by region and model.

Setting up regional probes in a monorepo

Production monitoring requires synthetic probes, lightweight requests that mirror real traffic but run independently of user requests. In a monorepo, you can instrument this at multiple layers.

Step-by-step setup:

1. Create a shared probe library in your monorepo (e.g., packages/latency-probes).

This package exports a function probeOpenAI() that sends a fixed test prompt to OpenAI, measures TTFB and TTFT, and logs results with metadata (region, provider, timestamp).

1. Route probes from multiple regions.

Deploy probe jobs (cron or event-driven) in each region your chat service supports (US-East, US-West, EU-Central, APAC, etc.). Each probe uses regional network routing to OpenAI, either direct API calls or through a regional proxy.

1. Use 21 regions for global coverage (or a subset matching your user base).

Observinio covers 21 regions; align your probes with those endpoints so degradation detection is consistent.

1. Run probes every 5–10 minutes.

Frequent enough to catch regional issues within 10–15 minutes, low overhead for provider rate limits.

1. Store baseline latencies per region and model.

Calculate rolling 30-day medians. Use these to define alert thresholds (e.g., alert if TTFB > baseline + 1.5 seconds for 3 consecutive probes).

Lightweight probe example (pseudocode):

function probeOpenAI(region, model, apiKey) {
  const prompt = "Respond with one word: fast or slow?"
  const startTime = now()
  const ttfbTime = null
  const stream = openai.createCompletion({
    model: model,
    messages: [{role: "user", content: prompt}],
    stream: true,
    timeout: 30000
  })
  
  stream.on('data', (chunk) => {
    if (!ttfbTime) {
      ttfbTime = now() - startTime
      logMetric('ttft', ttfbTime, {region, model})
    }
  })
  
  stream.on('end', () => {
    const totalTime = now() - startTime
    logMetric('ttfb', totalTime, {region, model})
  })
  
  stream.on('error', (err) => {
    logError('probe_failed', err, {region, model})
  })
}

Alerting on latency degradation

Not every latency spike requires a page. A single slow request is noise; a trend is a signal.

Alert criteria:

• Regional threshold breach: If TTFB exceeds baseline + 1.5 seconds for 3 consecutive probes from the same region, trigger a degradation alert.
• Global degradation: If all regions show +20% latency increase simultaneously, suspect a provider-wide issue or model update.
• Specific model slowdown: If gpt-4-turbo latency spikes but gpt-3.5-turbo remains normal, investigate provider-side model routing.

Route alerts to:

• Email digest (daily summary of regional trends).
• PagerDuty (critical: all regions degraded, or latency > 3 seconds for chat).
• Slack channel (informational: regional alerts, trends).

In a monorepo, wire probes to your shared alerting service so chat and inference services consume the same signal.

Integrating latency data into routing decisions

Latency monitoring loses value if your routing layer ignores it. In a monorepo, a shared inference package can read baseline data and prefer faster regions or providers.

Example routing logic:

function selectProvider(userRegion, model) {
  const latencies = {
    'openai-direct-us': 120,
    'openai-direct-eu': 180,
    'openrouter-us': 140,
    'openrouter-eu': 160
  }
  
  // Prefer provider/region combo with lowest recent TTFB
  const sorted = Object.entries(latencies)
    .filter(([key]) => key.includes(userRegion) || key.includes('us'))
    .sort(([, a], [, b]) => a - b)
  
  return sorted[0][0] // "openrouter-eu"
}

Refresh latency baselines hourly. If a region degrades and stays slow for 30+ minutes, fail over to a secondary region (e.g., EU-Central to EU-West).

Monorepo-specific instrumentation checklist

When multiple services share OpenAI logic, consistency matters. Use this checklist to avoid gaps:

Weekly summaries and trend analysis

Raw probe data is useful; trends are actionable. Create a weekly latency report:

• Regional latency table: Median TTFB and TTFT per region, week-over-week change.
• Provider comparison: OpenAI direct vs. OpenRouter; which is faster in each region?
• Model breakdown: Is gpt-4-turbo slower than gpt-3.5-turbo? By how much?
• Anomalies: Hours/days with unexpected spikes; correlation with known events (provider maintenance, model updates).
• Recommendations: "Switch EU-Central users to OpenRouter" or "Increase timeout for gpt-4-turbo to 8 seconds."

Observinio's weekly summaries help, but in a monorepo you often need custom logic (e.g., filtering probes by service tag). Export Observinio data (or your own probe logs) to a BI tool and build a dashboard.

Handling false positives and data quality

Probes can lie: network blips, stale DNS, or probe-side clock skew distort results. Mitigate:

• Discard outliers: Remove TTFB measurements > 99th percentile (likely failures, not signal).
• Require consensus: Wait for 3–5 probes to agree a region is degraded before alerting.
• Cross-check with user metrics: If probes say EU is slow but user request latency is normal, the probe is probably wrong (bad routing, stale baseline).
• Log failures separately: Distinguish "request timed out" from "slow response." Timeouts might indicate a connectivity or rate-limit issue, not provider load.

FAQ

Next steps

Start small: pick one critical region and one model, run probes for two weeks, and establish baseline latencies. Then expand to all regions and route based on data. If you need managed probes across 21 regions with built-in alerting, Observinio handles the infrastructure; focus your monorepo on consuming latency data and routing intelligently.

For support or to set up a monitoring dashboard, visit Observinio's contact page or view live provider latency on Observinio's status page.

Additional Resources

• Production best practices | OpenAI API - In this section, we will discuss some factors that influence the latency of our text generation models and provide suggestions on how to reduce it. The latency ...
• Latency optimization | OpenAI API - This guide covers the core set of principles you can apply to improve latency across a wide variety of LLM-related use cases. These techniques come from ...
• How to Create Latency Monitoring - Learn to create latency monitoring for tracking and optimizing LLM response times.