Prompt token count vs latency worksheet

Photo by Dash Cryptocurrency from Pexels

When you send a request to an LLM API, latency depends on many factors, but prompt token count is not the primary driver most teams assume it is. Many production incidents blamed on "too many input tokens" are actually caused by regional network variance, provider queue depth, or model routing decisions. This worksheet helps you measure the real relationship between token count and response time across your infrastructure and providers.

TL;DR

• Prompt token count has a measurable but small impact on Time to First Byte (TTFB); output token generation dominates perceived latency.
• Regional latency variance (North America vs. Europe vs. Asia) typically swamps the effect of 100–500 token differences.
• Establish a baseline for your specific provider and region using daily synthetic probes before optimizing tokens.
• Use this worksheet to run controlled tests, plot token count against TTFB, and identify whether token reduction is worth the engineering effort.
• Observinio's multi-region probe data makes this comparison systematic and defensible in postmortems.

Key takeaway: Token count optimization is a fine-tuning tactic, not a primary latency fix. Regional routing and provider selection typically deliver 15–30× greater latency improvement than token reduction. Measure regional and provider variance first before investing in token optimization.

Why token count matters less than you think

The conventional wisdom is straightforward: fewer input tokens → faster requests. In practice, the relationship is weak. As one detailed analysis found:

"Many sources online state that reducing the number of input tokens does result in lower latency, but this is usually not a significant factor."

>, How input token count impacts the latency of AI chat tools

Here's why:

Token encoding is fast. Modern tokenizers (GPT-3, Claude, etc.) process thousands of tokens per millisecond on the provider's hardware. A prompt that grows from 500 to 1000 tokens adds microseconds to encoding, not milliseconds.

TTFB latency is dominated by the network and queue. Time to First Byte, the elapsed time from request send to the first token of the response, includes network round-trip time, server routing, queue wait, and model initialization. Even at a high-traffic provider like OpenAI or OpenRouter, the first token typically arrives in 200–800 ms from a US region. Token count contributes perhaps 10–50 ms of that, depending on the model and provider load.

Output tokens are where latency scales linearly. Once the model begins generating the response, each output token takes 30–100 ms (for fast inference servers). A 500-token response will take 15–50 seconds to stream entirely. Token count in the output is the primary latency lever, not the input.

Regional and provider differences dominate. A request from Sydney to OpenAI's US-only endpoint will always be slower than one from us-east-1, regardless of prompt size. Similarly, OpenRouter's routing overhead or a provider's momentary queue surge will add 200+ ms of latency, far more than optimizing 100 tokens away.

The practical implication: token optimization is a fine-tuning tactic, not a latency fix. If your chat feels slow, measure regional variance and provider queuing first. If you've already addressed those and token count is still a concern, then run this worksheet.

Understanding latency components

Before measuring, clarify what latency you're optimizing for:

Time to First Byte (TTFB)

The elapsed time from when your client sends a request to when the first token of the response arrives. This is what users feel as "responsiveness" in a chat interface.

• Includes network latency, server routing, queue wait, and model initialization.
• Typically 200–1000 ms depending on region and provider.
• Prompt token count affects this by only 5–15% in most scenarios.

Time to Full Token (TTFT)

The total elapsed time for the complete response to arrive (all tokens streamed). This is the wall-clock time a user waits for a full answer.

• Dominated by output token count × time-per-token.
• For a 500-token response at 50 ms/token, expect ~25 seconds total.
• Prompt token count is negligible relative to output streaming time.

Streaming latency

Real-time LLM APIs (OpenAI, OpenRouter) return tokens as a stream, not batched at the end. Latency perceived in a chat UI is the inter-token time, typically 30–100 ms per token after the first arrives.

Prompt token count has almost no effect on streaming latency once the model is running. What matters is inference speed (batch size, model weights, hardware) and network congestion.

Setting up your measurement

To run a fair test, you must control variables:

Variables to fix

1. Same model and parameters. Test against a single model (e.g., gpt-4-turbo) with identical temperature, max_tokens, and system prompt structure.
2. Same provider and endpoint. Use one provider for each test (OpenAI direct, OpenRouter, etc.). Provider overhead varies widely.
3. Same time of day and region. Provider queue depth and latency shift throughout the day and by geography. Run tests at consistent times and from a fixed region (or use Observinio's 21 regions to capture variance).
4. Same prompt semantics. Vary only token count; keep the logical content the same. Use prompt compression techniques (removing redundant examples, shortening explanations) rather than truncating randomly.

Variables to measure

1. Prompt token count (input_tokens, reported by the API).
2. Time to First Byte (TTFB in milliseconds).
3. Output token count (completion_tokens).
4. Total latency (time from send to last token).
5. Region (if using multi-region probes).
6. Timestamp and provider queue estimate (if available).

Step-by-step test procedure

1. Create 5–10 prompt variants at different token counts (e.g., 100, 300, 500, 800, 1200 tokens). Keep the task and expected output identical.
2. Run 10–20 requests per variant (more is better for statistical noise reduction).
3. Record TTFB and total latency for each request. Use libraries like openai.timings or manual timestamp capture.
4. Compute mean and standard deviation for each token count level.
5. Plot token count vs. TTFB on a scatter or line chart. Fit a trend line.
6. Calculate the slope: How many milliseconds of TTFB do you lose per 100 additional input tokens?
7. Compare to your baseline variance. If token count explains <10% of TTFB variance compared to regional or time-of-day differences, it's not a priority.

The worksheet: token count vs. latency

Use this template to structure your test:

Test metadata

Field	Value
Model	(e.g., gpt-4-turbo)
Provider	(OpenAI / OpenRouter / other)
Region	(e.g., us-east-1)
Test date & time	(YYYY-MM-DD HH:MM UTC)
Temperature	0.7
Max tokens	500

Prompt variants

Variant	Token Count	Task / Description
A	~100	Minimal prompt, core task only
B	~300	Base prompt + one example
C	~500	Base prompt + two examples + detail
D	~800	Detailed prompt + three examples + system notes
E	~1200	Comprehensive with context, examples, and constraints

Latency results (10 runs per variant)

Variant	Run 1 TTFB (ms)	Run 2 TTFB (ms)	...	Mean TTFB	Std Dev	Output Tokens
A (100 tk)	245	267	...	256	18	127
B (300 tk)	268	251	...	263	22	131
C (500 tk)	287	279	...	284	25	129
D (800 tk)	301	298	...	302	19	128
E (1200 tk)	325	318	...	321	24	130

Analysis

• TTFB increase from A to E: 321 − 256 = 65 ms
• Token count increase: 1200 − 100 = 1100 tokens
• Slope: 65 ms / 1100 tokens ≈ 0.06 ms per token
• Coefficient of variation (CV): Std dev / mean across runs. If CV > 10%, regional or time-based variance dominates.

Interpretation

• Slope < 0.1 ms/token: Token count is negligible. Focus on routing and regional latency.
• Slope 0.1–0.3 ms/token: Moderate effect. Optimization worth considering for latency-critical paths (e.g., sub-100 ms SLOs).
• Slope > 0.3 ms/token: Significant effect. Token reduction is a valid optimization, but verify it's not queue/region variance.

Multi-region and cross-provider comparison

A single-region test is incomplete. Latency behavior changes by geography and provider:

Why regions matter

A Sydney user querying OpenAI's US endpoint sees ~150–200 ms network latency baseline. A US user sees ~30–50 ms. Token count effects get hidden by geography. If you're routing dynamically (Sydney → OpenRouter Australia, US → OpenAI direct), you must test each route separately.

Why providers matter

OpenRouter incurs routing overhead (~50–100 ms extra) compared to direct provider calls. This fixed cost dominates token count effects. A 100-token optimization saves maybe 6 ms; OpenRouter's overhead adds 75 ms. The token work doesn't matter.

Multi-region test strategy

1. Use Observinio's 21-region probe network (or your own multi-region synthetic tests).
2. Run the same token-count test from 3–5 regions (e.g., us-east-1, eu-west-1, ap-southeast-1).
3. Plot regional TTFB variance: Likely 100–300 ms spread based on geography alone.
4. Overlay token-count trends: Does the slope hold across regions? Or does geography dominate?
5. Calculate: Token effect as % of regional variance. If token effect is 20 ms and regional variance is 200 ms, tokens explain 10%.

Cross-provider snapshot

If evaluating providers:

Provider	Variant A (100 tk) TTFB	Variant E (1200 tk) TTFB	Slope	Region
OpenAI direct	245 ms	321 ms	0.076 ms/tk	us-east-1
OpenRouter (gpt-4)	312 ms	398 ms	0.086 ms/tk	us-east-1
OpenAI direct	398 ms	467 ms	0.069 ms/tk	eu-west-1
OpenRouter (gpt-4)	478 ms	551 ms	0.073 ms/tk	eu-west-1

Takeaway: OpenRouter adds ~67 ms of overhead independent of token count. Token optimization (0.07–0.09 ms/tk) is a second-order effect compared to provider selection.

When token count does matter

Token optimization is worth pursuing in these specific scenarios:

1. Latency SLO < 300 ms TTFB, at scale.

If you've already optimized provider and routing, and you're chasing the last 50 ms, token reduction can help. Each 100-token reduction yields ~6 ms gain (at 0.06 ms/token baseline).

1. High-concurrency batch processing.

When processing thousands of requests in parallel, model throughput becomes queue-bound. Reducing prompt size per request lets the model process more requests per second, lowering queue wait. This shows up as TTFB improvement on aggregate.

1. Cost-critical with tight SLO.

If you're paying per token and have both cost and latency constraints, token reduction optimizes the tradeoff. But this is a financial optimization, not a pure latency fix.

1. Long context (10k+ token prompts).

At extreme prompt sizes, tokenizer and context-window overhead becomes more visible. The 0.06 ms/token slope may increase to 0.1–0.2 ms/token. Worth testing with your actual data.

1. Small, fixed-size responses.

If output is a single word or short phrase (e.g., classification task), reducing prompt tokens proportionally increases token count sensitivity. TTFB becomes the only latency that matters.

Practical token-reduction techniques

If your test reveals that token optimization is worthwhile, here are evidence-backed strategies:

Prompt compression

• Remove redundant examples: Two high-quality examples often outperform five mediocre ones; saves 50–200 tokens.
• Use concise instruction format: "Output JSON. Fields: name, age, role." instead of full prose; saves 30–50 tokens.
• Inline few-shot examples: Instead of 10 separate examples, use 1–2 exemplars; saves 100–300 tokens.

System prompt optimization

• Consolidate constraints: Merge repeated do's and don'ts; saves 20–40 tokens.
• Remove explanations: "Respond in JSON" instead of "Please respond in valid JSON format for downstream processing"; saves 10–20 tokens per occurrence.

Context pruning

• Retrieve only relevant context: Use vector search or BM25 to fetch 1–2 relevant documents instead of full knowledge base; saves 500–2000 tokens.
• Summarize older messages: In chat history, replace old turns with a summary; saves 100–500 tokens per conversation.

Semantic compression

• Abstractive summaries of user input: Paraphrase the user's question more concisely before sending; saves 20–100 tokens.
• Structured data instead of free text: If possible, pass structured input (JSON, CSV) instead of prose; saves 10–30% tokens.

A/B test the optimization

After applying a compression technique, re-run your latency test. Confirm that TTFB improves and that output quality does not degrade (use an evaluation metric or manual spot check on 10 responses).

FAQ

Getting started with Observinio

If you're measuring latency across multiple regions and providers, manually running this worksheet becomes tedious. Observinio handles the legwork: daily synthetic probes to OpenAI and OpenRouter from 21 global regions, automatic baseline comparison, and degradation alerts via email.

Use Observinio to:

• Run the worksheet automatically: Schedule probes with different prompt token counts and get TTFB trends plotted weekly.
• Compare providers and regions at scale: See token impact across geography and provider in one dashboard.
• Set latency SLOs: Define TTFB thresholds by region and get alerted when token optimization (or other changes) regress your baselines.

Start by logging the regions your users query from, then set up daily probes from those regions. Within a week, you'll have statistically significant data on whether token count is worth optimizing.

Check our status page for current provider latency and regional variance, or contact our team to discuss a custom monitoring setup for your use case.

Additional Resources

• How input token count impacts the latency of AI chat tools - Many sources online state that reducing the number of input tokens does result in lower latency, but this is usually not a significant factor.
• An evaluation framework for measuring prompt wise ... - Each prompt category contributes unique diagnostic information: core metrics such as response latency and tokens per second uncover throughput characteristics,
• How to track LLM token usage (2026): Prompt, completion ... - Finding the runaway step requires token counts for each span, so teams can sort the trace by prompt tokens, completion tokens, cost, or latency.