Incident runbook when OpenAI degrades in one region (in monorepo setups)

Photo by Rodolfo Gaion from Pexels

A single-region OpenAI degradation is one of the trickiest incidents to handle in a monorepo architecture. Your aggregate dashboards stay green, error rates barely move, yet users in São Paulo or Frankfurt experience TTFB spikes that double or triple normal values. This runbook gives you a step-by-step playbook, from detection through resolution, designed specifically for teams that ship multiple services from one repository and route inference traffic through shared gateway code.

TL;DR

• Regional OpenAI degradations are invisible to global-average metrics; you need per-region latency baselines to catch them.
• In a monorepo, a single shared API client configuration means one degraded region can cascade into queue build-ups across services.
• This runbook covers five phases: Detect → Triage → Contain → Communicate → Review.
• Automated probes from multiple regions (like Observinio's 21-region checks) cut detection time from minutes to seconds.
• Every phase includes concrete commands, thresholds, and escalation criteria you can paste into your incident management tool.

Why regional degradations are uniquely dangerous in monorepos

In a polyrepo world each service typically owns its own OpenAI client, timeout settings, and retry policy. When latency spikes in one region, the blast radius is limited to that service's deployment. Monorepos change the equation in three important ways:

1. Shared client libraries. A single openai-client package in packages/ or libs/ is imported by every service. Timeout and retry defaults propagate everywhere.
2. Unified deployment pipelines. A CI/CD change to the gateway module redeploys all consumers simultaneously, making it harder to roll back just one service.
3. Common environment variables. API keys, base URLs, and region routing rules often live in a shared .env or secrets manager path. Switching a region endpoint means touching configuration that affects every service in the repo.

These coupling points mean a regional degradation that looks minor in isolation can cascade: retry storms from one service saturate the shared rate-limit pool, starving other services of tokens. Understanding this coupling is the first step in your runbook.

"The July 25 incident is consistent with, and adds a further data point to, the concentration-risk pattern CSA's research has modeled as structural rather than incidental."

>, The ChatGPT Outage Pattern: Concentration Risk in Practice

Phase 1: Detect, catching the signal before users do

Detection speed determines everything. The goal is to move from "a user filed a ticket" to "an automated alert fired" as your primary detection channel. Here is what you need:

Regional baseline comparison

Set up synthetic probes that call the same model endpoint (e.g., gpt-4o completions with a fixed 50-token prompt) from every region your users occupy. Compare each probe result against a rolling 7-day baseline for that specific region. A probe that returns TTFB above 2× the regional baseline for two consecutive checks is your trigger.

Alert thresholds checklist

• Warning: TTFB > 1.5× regional baseline for 2 consecutive probes.
• Critical: TTFB > 2× regional baseline for 2 consecutive probes OR any probe returns a 5xx status.
• Emergency: 3+ regions simultaneously exceed 2× baseline (this likely indicates a global incident, not regional).

Observinio's daily probes across 21 regions and degradation email alerts handle this detection layer out of the box, no custom Prometheus exporters or cron jobs required. You can check the current state at any time on the Observinio status page.

Phase 2: Triage, confirming scope and impact

Once an alert fires, the on-call engineer needs to answer three questions within the first five minutes:

1. Is it regional or global? Check latency from at least three other regions. If only one region is degraded, proceed with the regional playbook. If multiple regions spike, switch to your global outage runbook.
2. Which services are affected? In a monorepo, run a quick dependency query to find every service importing the shared OpenAI client. A command like grep -r "from.openai-client" packages//src --include="*.ts" -l gives you the list in seconds.
3. What is the user-facing impact? Map degraded services to product features. A chat completion service degrading in eu-west-1 might affect all European chat users, while an embedding service in the same region might only slow down a nightly batch job.

Severity matrix

Condition	Severity	Escalation
Single region, non-critical service	SEV-3	On-call engineer handles alone
Single region, user-facing service	SEV-2	Notify engineering lead, begin containment
Multiple regions or rate-limit pool exhaustion	SEV-1	Page incident commander, open war room

Phase 3: Contain, stopping the cascade

Containment in a monorepo requires surgical precision because blunt changes ripple across services. Follow these steps in order:

Step-by-step containment procedure

1. Reduce retry amplification. In your shared client config, lower max_retries for the degraded region from the default (typically 3) to 1. This single change can cut outbound request volume by 60 % and relieve pressure on the shared rate-limit pool.

1. Enable circuit breaker for the affected region. If your gateway supports feature flags (LaunchDarkly, Unleash, or even a simple environment variable), flip the circuit breaker for the degraded region. Requests from that region should fall back to a secondary provider or a cached response.

   # Example feature flag in monorepo config (packages/gateway/config.yaml)
   circuit_breakers:
     openai:
       eu-west-1:
         enabled: true
         fallback: "openrouter"  # route through OpenRouter as secondary
         ttl_seconds: 300
   

1. Isolate rate-limit pools. If you share a single API key across services, the degraded region's retries consume tokens that healthy regions need. Temporarily assign a separate key to the degraded region's traffic so healthy regions are unaffected. In a monorepo this usually means updating the secrets manager path referenced in packages/gateway/src/config.ts.

1. Throttle non-critical consumers. Identify batch or background services (embeddings, summarization pipelines) that can tolerate delay. Pause their queues or reduce their concurrency to free up rate-limit headroom for user-facing services.

1. Verify containment. After each step, check your regional probes. TTFB for healthy regions should return to baseline within one to two probe cycles (typically under 10 minutes). The degraded region may still show elevated latency, but the cascade should stop.

Containment verification checklist

Phase 4: Communicate, keeping stakeholders informed

During a regional degradation, internal communication is just as important as technical containment. Use this template for your first status update, posted within 15 minutes of detection:

Incident title: Elevated OpenAI latency in [REGION]
Status: Investigating / Contained
Impact: [Service names] experiencing increased response times for users in [geographic area]. Other regions are unaffected.
Current action: Circuit breaker activated; traffic rerouted to fallback provider.
Next update: In 30 minutes or when status changes.

Post updates to your internal incident channel (Slack, Teams) and, if you maintain a public status page, update it with the same information. Observinio's status page can serve as an external reference point, link to it so stakeholders can independently verify whether the degradation is on the provider side.

Phase 5: Review, turning the incident into prevention

Within 48 hours of resolution, run a blameless postmortem focused on three areas:

What to cover in the postmortem

1. Detection gap. How long between the start of degradation and your first alert? If it was more than five minutes, your probe frequency or threshold configuration needs tuning.
2. Monorepo coupling. Did the shared client library, shared rate-limit pool, or shared config amplify the blast radius? Document specific coupling points and assign follow-up tasks to decouple them.
3. Fallback effectiveness. Did the circuit breaker and fallback provider (e.g., OpenRouter) actually deliver acceptable latency? Compare TTFB from the fallback path against your SLO targets.

Recommended post-incident improvements

• Per-service rate-limit pools. Assign separate API keys or use header-based rate-limit partitioning so one service's retries cannot starve another.
• Region-aware retry budgets. Instead of a global max_retries value, configure retries per region in the shared client. Degraded regions get aggressive backoff; healthy regions keep normal settings.
• Automated circuit breaker triggers. Wire your regional probe alerts directly into your feature flag system so the circuit breaker flips without human intervention.
• Weekly latency trend reviews. Use Observinio's weekly summary emails to spot gradual regional degradation before it becomes an incident. A region whose P95 TTFB has crept up 20 % over three weeks is a leading indicator.

Monorepo-specific configuration tips

Managing OpenAI client configuration in a monorepo deserves special attention. Here are patterns that reduce incident blast radius:

Layered configuration

Structure your config so that each service can override shared defaults:

packages/
  openai-client/
    src/
      defaults.ts      # shared defaults: timeout=30s, retries=3
  chat-service/
    config/
      openai.ts        # overrides: timeout=10s (user-facing, tighter SLO)
  embedding-service/
    config/
      openai.ts        # overrides: retries=5 (batch job, can tolerate retries)

This pattern lets you tighten timeouts for latency-sensitive services without affecting batch consumers, and vice versa.

Region routing as a first-class config

Do not bury region routing in application code. Expose it as a top-level configuration value that your gateway reads at startup and that feature flags can override at runtime. This makes containment step 2 (circuit breaker activation) a config change rather than a code deploy.

---

If you are tired of discovering regional OpenAI degradations from user complaints instead of dashboards, Observinio can help. Our probes run from 21 global regions, compare every result against per-region baselines, and send you an email alert the moment latency deviates. Set up alerts in minutes at observinio.com and check real-time provider status on our status page.

Additional Resources

• The ChatGPT Outage Pattern: Concentration Risk in Practice - The downstream consequences of the outage extended well past the roughly one-hour window in which OpenAI's own services were degraded. Startups ...
• ilert's Post - OpenAI's July 2026 incident shows how cloud infrastructure maintenance reduced capacity for an internal identity service. The remaining regional ...
• The OpenAI and Hugging Face security incident: why AI ... - OpenAI models broke AI agent containment and reached Hugging Face servers during a benchmark test, showing why agentic systems require ...