OpenAI probe configuration worksheet (with custom token sets)

Photo by Jeswin Thomas from Pexels

Running synthetic probes against the OpenAI API sounds straightforward, until you realize that a 10-token "hello world" completion and a 2 000-token summarization request produce wildly different latency profiles. If your monitoring probes do not mirror the token shapes your production traffic actually sends, the numbers you collect are noise, not signal. This worksheet walks you through designing probe configurations that use custom token sets so every measurement reflects real workload behavior.

TL;DR

• Default single-token probes underestimate real-world TTFB and total latency by 30–60 % for longer completions.
• Custom token sets let you match probe payloads to your actual prompt/completion distribution (short chat, medium RAG, long summarization).
• A three-tier probe strategy (small / medium / large token set) covers most production traffic patterns.
• Pair each token set with region-specific scheduling to catch variance across OpenAI's global infrastructure.
• Observinio's daily probes from 21 regions can run each token set independently, giving you per-shape, per-region baselines.

Why default probes fall short

Most monitoring setups ship with a single "ping" probe: a minimal prompt that asks the model to return one word. That probe is useful for checking whether the endpoint is alive, but it tells you almost nothing about how the API behaves under realistic load. Here is why:

• Prompt-processing time scales with input tokens. A 50-token prompt and a 1 500-token prompt hit different code paths in the inference stack. Prefill time grows roughly linearly with input length, so a short probe hides the latency your users actually experience.
• Completion length affects streaming TTFB differently than total latency. When you request max_tokens: 800, the model's decode phase dominates wall-clock time. A probe capped at max_tokens: 5 never exercises that path.
• Token-budget throttling is invisible to tiny probes. OpenAI applies rate limits in tokens-per-minute. A probe that consumes 15 tokens per call will never trigger the throttle, even when your production traffic is being shaped.

The fix is straightforward: define multiple probe configurations, each with a token set that mirrors a real traffic class.

Anatomy of a custom token set

A "token set" in this context is a pair of values plus supporting metadata:

1. Input token count, the approximate number of tokens in the probe prompt.
2. Requested output token count, the max_tokens (or max_completion_tokens for newer API versions) value.
3. Model identifier, e.g., gpt-4o, gpt-4o-mini, gpt-3.5-turbo.
4. Temperature, keep it at 0 for deterministic, reproducible latency measurements.
5. Region tag, which Observinio probe region(s) should execute this set.

Example token-set definitions

Set name	Input tokens	Max output tokens	Model	Use case
chat-short	~60	150	gpt-4o-mini	Quick chatbot replies
rag-medium	~800	400	gpt-4o	RAG-augmented Q&A
summarize-long	~2 000	800	gpt-4o	Document summarization

Each set exercises a different part of the latency curve. When you review your Observinio weekly summary, you can compare TTFB and total response time per set and immediately see whether a regression is prompt-size-dependent or global.

Step-by-step: building your probe worksheet

Follow these seven steps to go from zero to a production-grade probe configuration.

1. Inventory your traffic classes. Pull the last 7 days of API logs and bucket requests by input-token range: 0–100, 100–500, 500–1 500, 1 500+. Note the median max_tokens for each bucket.
2. Select representative prompts. For each bucket, pick (or craft) a prompt that lands in the middle of the token range. Avoid prompts that contain PII or proprietary data, probes run on external infrastructure.
3. Tokenize and verify. Use tiktoken to confirm your prompt hits the target count:

   import tiktoken
   enc = tiktoken.encoding_for_model("gpt-4o")
   tokens = enc.encode(prompt_text)
   print(f"Token count: {len(tokens)}")
   

1. Define the token sets. Create a JSON or YAML config for each set:

   probe_sets:
name: chat-short
       model: gpt-4o-mini
       prompt_file: prompts/chat_short.txt
       max_tokens: 150
       temperature: 0
       regions:
us-east-1
eu-west-1
ap-northeast-1
name: rag-medium
       model: gpt-4o
       prompt_file: prompts/rag_medium.txt
       max_tokens: 400
       temperature: 0
       regions: all
name: summarize-long
       model: gpt-4o
       prompt_file: prompts/summarize_long.txt
       max_tokens: 800
       temperature: 0
       regions: all
   

1. Set scheduling cadence. Short probes can run every 5 minutes with minimal cost. Medium and long probes are more expensive, schedule them every 15–30 minutes, or align them with Observinio's daily probe windows to keep spend predictable.
2. Establish baselines. Run each set for at least 72 hours before setting alert thresholds. Record p50, p90, and p99 for both TTFB and total response time per region.
3. Configure alerts. In Observinio, set degradation alerts per probe set. A 20 % increase over the p90 baseline for summarize-long in eu-west-1 is a meaningful signal; the same threshold on chat-short might be too tight because absolute values are smaller.

Choosing the right regions for each token set

Not every token set needs to run from all 21 Observinio regions. A practical allocation strategy looks like this:

• chat-short, run from the 3–5 regions where your users are concentrated. Short probes are cheap, so you can afford higher frequency in fewer locations.
• rag-medium, run from all regions. RAG workloads are latency-sensitive and often serve global users. Full regional coverage reveals routing asymmetries in OpenAI's infrastructure.
• summarize-long, run from all regions but at lower frequency. Long completions are expensive; a probe every 30 minutes from each region still gives you 48 data points per day per region, more than enough for trend analysis.

Regional variance and token length

Empirically, regional latency variance increases with token count. A chat-short probe might show a 40 ms spread between the fastest and slowest region, while summarize-long can show a 600 ms+ spread. This is because longer decode phases amplify any difference in GPU queue depth or network path between regions. Monitoring only short probes masks this effect entirely.

Interpreting results: what to look for

Once your probes are running, focus on these signals in your Observinio dashboard and weekly summaries:

• TTFB divergence between token sets. If chat-short TTFB stays flat but rag-medium TTFB spikes, the provider is likely experiencing prefill congestion, the model is slower to start generating when the input context is larger.
• Total latency ratio. Divide summarize-long total latency by chat-short total latency. A stable ratio (e.g., 5.2×) means the API is scaling linearly. A growing ratio means decode throughput is degrading for longer completions.
• Region-specific regressions. A single region showing elevated p90 for all three token sets points to a network or data-center issue. A single region showing elevated p90 only for summarize-long suggests GPU capacity constraints in that region.
• Baseline drift over weeks. Use Observinio's weekly summary emails to track whether baselines are creeping up. A 5 % week-over-week increase in rag-medium p50 across all regions often precedes a larger degradation event.

"The amount it can pull at once from that kind of data submitted at once is limited and based on a similarity search."

>, Token limit for Custom GPT's

This observation from the OpenAI community underscores why token-set sizing matters: the API's internal retrieval and processing behavior changes with payload size, and your probes need to reflect that.

Probe maintenance checklist

Probes are not set-and-forget. Use this checklist monthly:

Putting it all together: a sample worksheet

Below is a filled-out worksheet for a team running a customer-support chatbot with RAG and an internal document summarizer.

This single worksheet gives the team full visibility into three distinct latency profiles across every region Observinio monitors.

FAQ

Start monitoring with real token shapes

If your current probes only send a handful of tokens, you are measuring an API that your users never see. Observinio's daily probes from 21 regions let you define distinct token-set configurations and track TTFB, total latency, and degradation per set. Set up your custom token sets today, establish baselines, and let Observinio's email alerts tell you the moment a specific workload shape starts slowing down, before your users notice.

Additional Resources

• Token limit for Custom GPT's - Token limit for Custom GPT's. The maximum size allowed for any individual file uploaded to OpenAI's API is 5 megabytes (MB).
• How to restrict the model to only consider a set of possible ... - Hi, how do I make openai.Completion.create() calls to only consider a small set of tokens as possible so that the choices and most likely tokens ...
• OpenAI - Configure OpenAI models. The providers list takes a config key that allows you to set parameters like temperature. Use the functions config to define custom ...