On-call runbook: AI API latency spike

Photo by Kampus Production from Pexels

When a user's chat request hangs for 5 seconds instead of the usual 800 milliseconds, your on-call phone buzzes. The incident is real: an AI API latency spike is degrading production. This runbook walks you through diagnosis, mitigation, and recovery in the first 30 minutes, and how to prevent the next one.

Key takeaway: Regional monitoring, clear latency thresholds, and a pre-tested failover strategy reduce incident response time from 30–60 minutes to 10–15 minutes. Most latency spikes are provider-side degradation, regional routing issues, or retry storms amplifying the problem. Stop retrying before you understand the root cause, and always compare latency across regions to distinguish between global outages and localized problems.

TL;DR

• Latency spikes are regional and variable. A single aggregate metric hides whether slowness is global, provider-wide, or isolated to your routing path. Compare TTFB (time to first byte) and TTFT (time to first token) across regions via monitoring tools like Observinio's daily probes.
• Retry storms make spikes worse. If an API tool starts timing out at 20% of calls, the agent retries and burns token budget. Stop retrying before you understand the root cause.
• Provider status pages lag reality by 5–15 minutes. Check your own regional probes first; then verify with the provider's dashboard.
• Failover to a backup provider or region takes 2–5 minutes. Have a pre-tested runbook and alert thresholds set before the spike happens.
• Post-incident: analyze trends, not just the spike. Weekly summaries reveal whether latency is creeping upward or if this was an outlier.

---

Minute 0–2: Confirm the spike is real

Before escalating or panicking, verify that the latency spike is not a local issue.

Step 1: Check your own application metrics.

• Open your APM tool (DataDog, New Relic, etc.) and look at API response times for the last 10 minutes.
• Isolate the latency to the external AI API call, not your entire request pipeline. Filter by service and endpoint.
• Note the percentile affected: Is 100% of requests slow, or just the p99?

Step 2: Check regional variance.

• If you call the AI API from multiple regions (US East, EU, Asia-Pacific), compare latency across each origin point.
• A global provider outage affects all regions evenly; a regional incident affects one or two.
• Use Observinio's dashboard to see TTFB and TTFT from 21 global probe regions. This gives you a data-driven view in seconds, without waiting for internal logs to backfill.

Step 3: Verify it is not your rate limit.

• Check your provider's rate-limit headers in the API responses: x-ratelimit-remaining, x-ratelimit-reset.
• If remaining quota is near zero, you are rate-limited. This is not a provider outage; you need to increase quota or shed load.

Decision point: If the spike is confirmed (latency is up 2–5x from baseline, across multiple regions or requests), move to Minute 2–5.

---

Minute 2–5: Identify the root cause

Latency spikes have three common causes: provider degradation, regional routing trouble, or your own retry storm. Narrow it down quickly.

Root Cause #1: Provider-side degradation

Symptoms:

• Latency spike is global (all regions affected equally, or only regions that route to that provider).
• Error rate is stable or low; requests are succeeding, just slowly.
• Your application code and configuration have not changed.

Diagnosis checklist:

1. Check the provider's status page. OpenAI's status page at status.openai.com, OpenRouter's at status.openrouter.io, and similar services. Look for "Investigating" or "Degraded Performance" posts in the last 15 minutes.
2. Compare latency to Observinio baseline. If your baseline TTFB was 300 ms and you are now seeing 1.2 seconds from multiple regions, the provider is slow. Check the baseline graph on /status or your own dashboard.
3. Check provider's own dashboards. If you have direct OpenAI or provider account access, log in and verify response times from their first-party metrics.
4. Query recent batch jobs or traffic spikes. Use the provider's API logs or account page to see whether a large batch job or viral use case is consuming the provider's capacity.

Action if confirmed provider degradation:

• Post an incident in your Slack #incidents channel with: "OpenAI latency degraded globally. TTFB baseline 300 ms → 1.2 s. Failover to OpenRouter in progress."
• Move to the "Failover" section (Minute 5–10).
• Do not keep retrying the slow provider; every retry burns tokens and delays recovery.

Root Cause #2: Regional routing issue

Symptoms:

• Only one or two regions are slow; others are normal.
• Error rate is stable.
• Your provider's own status page shows no issues.

Diagnosis:

1. Compare latency per region in Observinio's 21-region probes. If US East is 300 ms and US West is 2.5 seconds, the issue is routing between your data center and the provider's West Coast endpoint, not the provider itself.
2. Check your regional failover configuration. Is a DNS entry pointing to the wrong endpoint? Is a regional load balancer broken?
3. Contact your provider's support and mention the specific region. They can see whether their endpoint in that region is healthy or degraded.

Action if confirmed regional routing issue:

• Route traffic away from the slow region to an alternate endpoint or fallback provider.
• Update DNS or load-balancer rules to test the alternate path.
• Expect 2–5 minutes for DNS or LB rule changes to propagate.

Root Cause #3: Retry storm (your code is amplifying the spike)

Symptoms:

• Error rate is up (e.g., 5–10% of requests are timing out).
• Total request volume to the provider doubled or tripled in the last 5 minutes.
• Your retry logic is aggressive (e.g., retry every 100 ms, 10 times).

Diagnosis:

1. Check your API logs for patterns. Are the same user session retrying the same request three times? Are you retrying on all error types?
2. Compare incoming request volume to outgoing API calls. If you have 100 user requests but 400 API calls, you are retrying 3x per user request on average.

Action if confirmed retry storm:

• Stop retrying immediately. Set a feature flag to disable retries and flip it in production.
• Update your retry logic to: retry only on transient errors (5xx, timeout), not on client errors (4xx). Use exponential backoff (wait 500 ms, then 1 s, then 2 s) to avoid slamming the provider.
• Burn down the queue of queued retries before resuming normal traffic.

---

Minute 5–10: Failover or mitigate

Once you have identified the root cause, execute the appropriate mitigation. This is where pre-planning saves minutes.

If provider is degraded: Failover to backup provider

Pre-incident setup (do this before you are on-call):

1. Configure a secondary provider in your AI API routing layer. For example: primary = OpenAI, secondary = OpenRouter or Anthropic Claude.
2. Set a latency threshold for failover (e.g., if p99 latency exceeds 2 seconds for 2 consecutive minutes, failover).
3. Test failover in staging. Verify that swapping providers does not break your prompt formatting or token limits.
4. Document the failover in your runbook (this file).

During incident:

1. Open your feature-flag or config-management tool (LaunchDarkly, Unleash, etc.).
2. Create or enable a flag called use_backup_provider: true and roll it out to 100% of traffic.
3. Monitor the p99 latency for 30 seconds. If it drops back to normal (< 500 ms), the failover worked.
4. If the backup provider is also slow, roll the flag back and escalate to your CTO or incident commander.

"If an API tool starts timing out at 20% of calls, the agent retries and burns token budget."

>, The On

If regional issue: Reroute or shed load

For regional degradation:

1. Update your DNS or load-balancer to route affected regions to an alternate endpoint or provider.
2. If you use OpenRouter, try a different model or provider tier (e.g., switch from openai/gpt-4-turbo to openrouter/anthropic-claude-opus). Latency may differ.
3. Shed non-critical load: if you offer background jobs or batch processing, pause them for 10 minutes to reduce pressure on the provider.

Time to mitigation: 2–5 minutes for DNS changes, 30 seconds for load-balancer rules, instant for feature flags.

If retry storm: Kill and drain

1. Disable retry logic via feature flag immediately. Set max_retries: 0.
2. Monitor the queue of pending retries. As they fail and are drained from memory, outgoing API request volume will fall.
3. Once volume stabilizes (5–10 minutes), gradually re-enable retries with exponential backoff (start at 500 ms, max 5 seconds, max 3 attempts).

---

Minute 10–20: Communicate and monitor

Communication checklist:

Post incident summary in #incidents Slack channel: "Incident: OpenAI latency spike, 12:05–12:15 UTC. Root cause: Provider-side degradation. Mitigation: Failover to OpenRouter. Status: Resolved. P50 latency now 400 ms." Update your status page (e.g., Statuspage.io or a custom page at /status). Set an incident post to "Investigating" or "Resolved" with a 1-sentence summary. Notify affected customers if latency was user-facing for > 5 minutes. Email or in-app banner: "We experienced elevated latency between 12:05–12:15 UTC. We have restored normal service. Thank you for your patience." Tag the incident with metadata: severity (SEV2/SEV3), provider(s) affected, root cause, duration.

Monitoring checklist:

Watch Observinio dashboard (or your own APM tool) for the next 10 minutes. Ensure latency stays below your SLO (e.g., p99 < 800 ms). Check error rate. It should return to baseline (< 0.5%) within 5 minutes of failover. If using a backup provider, verify that user experience (response quality, latency consistency) is acceptable. Some providers have longer TTFT even if TTFB is fast. Monitor your token spend if you switched providers. Some providers charge differently or have higher usage on the same prompt.

---

Minute 20–30: Stabilize and document

Once latency has been normal for 10 minutes and error rate is stable, the incident is effectively over. But do not skip stabilization.

Stabilization steps:

1. Gradually roll back failover. If you failed over to a backup provider, slowly shift 10% of traffic back to the primary provider every 2 minutes. Watch for latency spikes. If the primary is still degraded, keep the failover in place and move to post-incident review.
2. Verify no cascading issues. Check downstream services that depend on your AI API responses. Did any timeouts or queue backlogs accumulate?
3. Update your dashboards and alerts. If you discovered a blind spot (e.g., regional latency was not monitored), set up a new alert or dashboard panel now.

Documentation (for post-incident review):

• Duration: start time, end time, total minutes of impact.
• Root cause: one sentence.
• Mitigation: what action fixed it and how long it took.
• Why we did not catch this earlier: did your alerts fail? Was the threshold too high?
• Prevention: what will you do differently next time?

---

Preventing the next spike: Setup and thresholds

Latency spikes are not random. They cluster around certain events: provider maintenance windows, traffic spikes from viral content, or gradual capacity exhaustion. Use data to predict and prevent.

Daily probes and baseline comparison

Use Observinio or a similar synthetic monitoring tool to:

• Run daily probes from 21 regions. Each probe measures TTFB, TTFT, and error rate. This gives you a global latency map, not a single aggregate number.
• Compare against rolling 7-day baseline. If today's TTFB is 50% higher than the 7-day average, an anomaly detector should fire an alert.
• Set region-specific thresholds. US East latency might be 300 ms on average, but EU latency might be 450 ms due to routing. Set alerts relative to each region's baseline, not a global threshold.

Alert configuration

Recommended alert thresholds for P1 (critical) incidents:

• P99 latency > 1.5 seconds for > 2 minutes, on any region: page on-call immediately.
• Error rate > 5% for > 2 minutes: page on-call immediately.
• Latency degradation > 50% from 7-day baseline for > 5 minutes, on any region: send email alert (not page) to track trends.

Recommended alert thresholds for P2 (warning):

• P99 latency > 800 ms for > 5 minutes, on any region: send email to #incidents channel.
• Error rate > 2% for > 5 minutes: send email alert.

Weekly trend review

Every Monday morning, review the previous week's latency from Observinio's weekly summary email or your APM tool:

• Max latency per region. Did any region hit new highs?
• Baseline drift. Is latency creeping upward week-over-week, or stable?
• Error spike incidents. Were there any brief spikes that your alerts did not catch?
• Provider comparison. If you use multiple providers, which one is fastest and most stable?

Use this data to decide: Do we need to increase failover thresholds? Switch providers? Negotiate higher quota?

---

Checklist: Before you are on-call

Print or bookmark this checklist. Complete every item before your on-call shift.

---

FAQ

---

Next steps

Latency spikes will happen. But with regional monitoring, clear thresholds, and a tested failover strategy, you can go from alert to resolution in 10–15 minutes instead of 30–60.

Start today:

1. Set up Observinio daily probes from 21 regions to compare your OpenRouter and OpenAI latency against a rolling baseline. Alerts will fire when latency drifts, before users complain.
2. Configure your backup provider and test failover in staging this week.
3. Share this runbook with your team. Run a brief training session or post-mortem walkthrough so everyone knows what to do when their phone buzzes.
4. Review your weekly latency trends every Monday. Spot trends early and avoid the emergency on Friday at 5 PM.

For more on observing AI API latency across regions, see our providers page to compare OpenRouter, OpenAI, and others side by side, or contact us to set up a custom monitoring plan for your infrastructure.

Additional Resources

• The On-Call Runbook for AI Systems That Nobody Writes - A latency spike could be caused by: a model generating longer outputs (upstream content change), KV cache pressure from concurrent long-context ...
• On-Call Runbook Template: A Framework That Works at 3AM - Here's the framework we've tested across hundreds of incidents — it works whether you're responding to a latency spike, a spike in 500s, or a ...
• The State of On-Call Runbook Automation in 2026 - In 2026, AI agents can execute these runbooks for you — querying your observability stack, analyzing the data, and surfacing a root cause in ...