TTFB vs TTFT: What to Measure for Chat Completions

Photo by Oleksandr Petroniuk from Pexels

When your chat completion endpoint starts feeling sluggish, the first question is always the same: what exactly should you measure? Two acronyms dominate the conversation, TTFB (Time to First Byte) and TTFT (Time to First Token), and they are not interchangeable. Picking the wrong metric can send you chasing network ghosts while the real bottleneck sits inside the inference pipeline, or vice versa. This article breaks down both metrics, explains when each one matters, and gives you a concrete measurement checklist you can apply to OpenAI, OpenRouter, or any streaming chat completion API.

TL;DR

• TTFB measures the time from the HTTP request leaving your client to the first byte of the HTTP response arriving back. It includes DNS, TLS, TCP, and server processing time.
• TTFT measures the time from request submission to the first meaningful token appearing in the streamed response. It is always equal to or greater than TTFB.
• For streaming chat completions, TTFT is the metric that correlates with perceived user latency, it is what your end-user actually waits for before text starts appearing.
• TTFB is still valuable for diagnosing network-layer and routing issues between your infrastructure and the provider.
• Monitoring both metrics across multiple regions gives you the full picture. Tools like Observinio probe from 21 global regions daily so you can separate network variance from inference variance.

Why the Distinction Matters for LLM APIs

Traditional web APIs return a complete JSON payload in one shot. In that world, TTFB is a perfectly reasonable proxy for responsiveness: once the first byte arrives, the rest of the body follows almost immediately. Chat completion endpoints break that assumption. When you call /v1/chat/completions with stream: true, the server sends back an initial HTTP response (often just headers and the first SSE frame), and then tokens trickle in one by one as the model generates them.

This means the first byte you receive might be an HTTP header, a keep-alive frame, or a data: prefix, none of which contain an actual generated token. TTFB captures when that first byte lands. TTFT captures when the first decoded token lands. The gap between the two can range from a few milliseconds to hundreds of milliseconds, depending on how the provider's streaming implementation works, whether there is a load balancer buffering the response, and how much prompt processing the model needs to do before generation begins.

If you only track TTFB, you might conclude that your provider is fast when in reality the user is still staring at a blank chat bubble for another 300 ms waiting for the first token. Conversely, if you only track TTFT, you might miss that a regional network path is adding 150 ms of pure transport latency that could be solved by switching to a closer endpoint or a different provider route.

A Quick Analogy

Think of ordering food at a restaurant. TTFB is when the waiter acknowledges your order and brings you a glass of water. TTFT is when the first actual dish arrives at your table. Both tell you something useful about the restaurant's efficiency, but only the dish arrival time tells you when you actually start eating.

Defining TTFB and TTFT Precisely

To avoid ambiguity, here are the precise definitions as they apply to streaming chat completion APIs:

TTFB, Time to First Byte

1. Start: The client sends the HTTP POST request (the last byte of the request body leaves the client's network stack).
2. End: The first byte of the HTTP response is received by the client.
3. Includes: DNS resolution, TCP handshake, TLS negotiation, request transmission, server-side routing to the inference backend, and the initial response header generation.
4. Does not include: Token generation time (unless the provider happens to send the first token in the very first response chunk).

TTFT, Time to First Token

1. Start: Same as TTFB, the moment the request is fully sent.
2. End: The first content token from the model's generation appears in the response stream. In OpenAI's SSE format, this is the first data: chunk where choices[0].delta.content is non-empty.
3. Includes: Everything in TTFB, plus prompt tokenization, KV-cache lookup or computation, any queuing in the inference pipeline, and the generation of the very first output token.
4. Does not include: Subsequent token generation (that falls under inter-token latency or total generation time).

The Gap Between Them

The difference TTFT - TTFB isolates the inference startup overhead: prompt processing, attention computation for the input context, and any provider-side queuing. This delta is especially useful when you are comparing models of different sizes or prompt lengths. A 128k-context request will have a much larger TTFT–TTFB gap than a 500-token prompt, even on the same model and region.

How to Measure Both in Practice

Below is a practical step-by-step approach you can implement in any language. The examples use Python, but the logic applies universally.

Step-by-Step Measurement Checklist

Example: Measuring with curl

curl -o /dev/null -s -w "TTFB: %{time_starttransfer}s\nTotal: %{time_total}s\n" \
  -X POST https://openrouter.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"openai/gpt-4o","stream":true,"messages":[{"role":"user","content":"Say hello"}]}'

This gives you TTFB via time_starttransfer. For TTFT, you need to parse the streamed output and timestamp the first non-empty content delta, curl alone cannot do that, so you will need a small script wrapper or a dedicated tool.

Example: Measuring with Python

import httpx
import json
import time

url = "https://openrouter.ai/api/v1/chat/completions"
headers = {"Authorization": "Bearer YOUR_KEY", "Content-Type": "application/json"}
payload = {
    "model": "openai/gpt-4o",
    "stream": True,
    "messages": [{"role": "user", "content": "Say hello"}],
}

t_start = time.perf_counter()
t_first_byte = None
t_first_token = None

with httpx.stream("POST", url, headers=headers, json=payload, timeout=30) as resp:
    for chunk in resp.iter_lines():
        if t_first_byte is None:
            t_first_byte = time.perf_counter()
        if chunk.startswith("data: ") and chunk != "data: [DONE]":
            data = json.loads(chunk[6:])
            content = data["choices"][0]["delta"].get("content", "")
            if content and t_first_token is None:
                t_first_token = time.perf_counter()
                break

print(f"TTFB:  {(t_first_byte - t_start)  1000:.0f} ms")
print(f"TTFT:  {(t_first_token - t_start)  1000:.0f} ms")
print(f"Delta: {(t_first_token - t_first_byte)  1000:.0f} ms")

This script gives you both metrics in a single run. Wrap it in a loop across regions (or let Observinio handle that for you) and you have a proper baseline dataset.

Regional Variance: Where the Metrics Diverge Most

0+

Global probe regions monitored by Observinio

0

Core latency metrics to track (TTFB, TTFT, delta)

0ms

Typical TTFB variance between US and Asia regions

TTFB is heavily influenced by geographic distance and network path quality. A request from us-east-1 to an OpenAI endpoint in the US might show a TTFB of 80 ms, while the same request from ap-southeast-1 in Singapore could show 280 ms. The TTFT–TTFB delta, however, should remain roughly constant for the same model and prompt length, because inference startup time does not depend on where the client is located.

This is exactly why measuring from a single region is misleading. If you only probe from your primary data center, you cannot tell whether a latency spike is caused by a network path degradation (TTFB increase, delta stays the same) or an inference pipeline slowdown (TTFB stays the same, delta increases). You need both data points from multiple vantage points.

What Regional Data Reveals

• TTFB variance across regions with stable TTFT delta → The provider's inference is fine; the issue is network routing or CDN configuration.
• Stable TTFB with increasing TTFT delta → The provider is experiencing inference queuing, model loading delays, or GPU saturation.
• Both TTFB and TTFT increasing in one region only → Likely a regional network issue (ISP peering, submarine cable degradation, etc.).
• Both increasing globally → Provider-wide incident. Check the Observinio status page to confirm.

"Thanks @Diet , yeah, well, time to first token is a riduculous comparison."

>,

This community sentiment highlights an important nuance: TTFT alone can be misleading if you do not account for total generation time and the specific API architecture (e.g., Assistants API adds overhead that inflates TTFT without necessarily meaning slower end-to-end performance). Always pair TTFT with inter-token latency and total completion time for a complete picture.

When to Use Which Metric

Not every situation calls for both metrics. Here is a decision guide:

• You are debugging a user complaint about "slow chat" → Start with TTFT. That is what the user perceives as the wait before text appears.
• You are evaluating a new provider or region → Measure both. TTFB tells you about network quality; TTFT tells you about inference performance.
• You are setting SLOs for your product → Define the SLO in terms of TTFT (e.g., "p95 TTFT < 800 ms from any monitored region"). Use TTFB as a diagnostic sub-metric.
• You are comparing OpenRouter vs. direct OpenAI → The TTFB difference shows routing overhead; the TTFT difference shows whether the extra hop affects inference startup. Check Observinio's OpenRouter provider page for current data.
• You are optimizing prompt length or model selection → Focus on the TTFT–TTFB delta. That isolates inference startup from network noise.

Building a Monitoring Baseline

A single measurement is an anecdote. A week of measurements is a baseline. Here is how to build one:

1. Choose your probe locations. At minimum, cover the regions where your users are concentrated. Ideally, probe from all major cloud regions.
2. Standardize the test prompt. Use a fixed prompt (e.g., "Say hello in one sentence") so that prompt token count does not introduce variance.
3. Run probes at consistent intervals. Daily probes catch trends; hourly probes catch incidents. Observinio runs daily probes from 21 regions automatically.
4. Store raw TTFB and TTFT values with full metadata (timestamp, region, model, provider).
5. Compute percentiles weekly. p50, p95, and p99 for both metrics, per region. This is your baseline.
6. Set alert thresholds at 2× your p95 baseline. When TTFT crosses that threshold in any region, you want to know immediately, not when a user files a support ticket.

Start Measuring What Matters

If you are running chat completions in production and only tracking one latency number, you are flying partially blind. Set up TTFB and TTFT measurement from at least the regions your users care about, build a weekly baseline, and alert on deviations. If you would rather not build and maintain that infrastructure yourself, Observinio already probes OpenRouter and OpenAI from 21 regions daily and sends you degradation alerts and weekly summaries, so you can focus on shipping features instead of debugging latency in the dark. Check the status page or get in touch to get started.

Additional Resources

• Performance analysis of Assistants versus Chat completion ... - Tracking time to first token (ttft), and tokens per second (tps) would probably be the most useful metrics. This is better than completion time,
• Understand LLM latency and throughput metrics - Time to first token (TTFT) refers to the time elapsed between submitting a prompt and receiving the first token of the model's response.
• How to Measure Time To First Token (TTFT) in AI Systems - Time To First Token (TTFT) measures the elapsed time between sending a prompt request and receiving the first token in the they are working on ...