Time To First Token
AI latency often gets reported as two numbers: time to first token (TTFT) and total time to completion (sometimes called end-to-end latency). TTFT measures how quickly the system starts streaming output after you send a prompt. Total time measures how long it takes until the model finishes the response or reaches a stopping condition. In practice, TTFT affects perceived responsiveness, while total time affects how long you wait for the full answer.
Consider a clinical note assistant that streams text as it generates. If TTFT is 2 seconds, you see the first words quickly, even if the full summary takes 12 seconds. If TTFT is 8 seconds, the interface feels stalled, even when the final answer arrives in 10 seconds. That difference matters for workflows where users decide whether to continue reading, ask follow-ups, or correct the prompt after the first few tokens.
Streaming also changes the meaning of “latency.” A system that streams tokens can show progress even when the model is still computing later parts. A non-streaming system returns only at the end, so TTFT and total time collapse into a single user experience. Many dashboards still report TTFT even when the product does not stream, which can mislead readers who compare numbers across tools.
Main Latency Pain Points
People often treat TTFT and total time as interchangeable, then blame the model for delays caused elsewhere. In real deployments, the prompt travels through multiple stages: client-side networking, gateway routing, authentication, prompt assembly, retrieval (if used), model inference, and post-processing. Any stage before the first generated token can inflate TTFT without changing total time much.
Another common misunderstanding involves tokenization and output length. Total time grows with the number of generated tokens, but TTFT can stay similar across short and long outputs because the model starts generating early. If you ask for a one-sentence answer versus a 1,000-word draft, total time changes dramatically while TTFT might not. That pattern shows up when the system uses the same model and similar prompt size, yet different max output settings.
Supporting technologies also shape latency. Retrieval-augmented generation adds time for embedding lookup and document reranking before generation begins. Tool use adds time for function calls, which can happen before or during generation depending on the orchestration. Rate limiting and load balancing can add queueing delays that show up as higher TTFT, and queueing can vary minute to minute.
Even the “same” request can behave differently due to caching. Some systems cache embeddings, some cache retrieval results, and some cache model outputs for repeated prompts. Caching can reduce TTFT by skipping earlier steps, while total time may still depend on the generation length. A dashboard that averages across mixed cache hit rates can hide the user-facing variability.
How To Measure Both Metrics
Separate Prompt Time From Generation
Measure TTFT as the time from the client sending the final prompt payload to the first streamed token arrival. Measure total time as the time from the same send point to the final token or stop event. If your system uses retrieval or tool calls, log timestamps for each stage: request received, retrieval start/end, model start, first token, last token. Without stage logs, you cannot tell whether TTFT is dominated by queueing, retrieval, or model warm-up.
In a test harness, record at least 30 runs per scenario and report median plus a tail metric like p95. Averages hide jitter, and health-related workflows often care about worst-case delays when users are waiting for a draft. I’ve seen teams report mean TTFT from 10 samples and then get surprised when p95 doubles after a traffic spike, which is frustrating because the dashboard looked “fine.”
Control Output Length And Settings
Hold prompt content constant and vary only one factor at a time. For total time, vary max output tokens or a target length. For TTFT, keep output length fixed and test different prompt sizes. Sampling parameters like temperature and top-p can change how long the model takes to reach a stop condition, but the dominant driver for total time remains the number of generated tokens.
When you test streaming, confirm the client actually receives tokens incrementally. Some SDKs buffer output until a minimum chunk size, which makes TTFT look worse than the server’s first-token time. In one internal evaluation I ran in early 2024 (tooling: a simple Node.js script with an HTTP streaming parser), the server logged first token at 1.1s while the client observed it at 1.6s due to buffering—small, but enough to change conclusions.
Account For Network And Queueing
TTFT includes network round-trip time and any queueing before inference begins. To isolate model-side effects, run tests from the same region as the inference endpoint and compare against a baseline request that uses a minimal prompt. If TTFT jumps while total time stays stable, queueing or pre-inference steps likely changed. If both TTFT and total time rise, the system may be under load during inference or generating more tokens than expected.
Also track concurrency. Latency under load often follows a non-linear pattern because requests compete for GPU time and memory. A single-user test can look great while a clinic’s batch of simultaneous requests produces higher TTFT. If you cannot run load tests, at least test with a small concurrency level that matches your expected usage.
Interpret Metrics For User Experience
TTFT maps to “time until the user sees something.” Total time maps to “time until the task completes.” In patient-facing drafting, TTFT helps users decide whether the system is responsive enough to keep them engaged, while total time affects whether they can finish a review cycle before moving to the next task.
For triage-style workflows where users scan the first few lines, TTFT can matter more than total time. For document summarization where the full output is required, total time dominates. A practical approach is to set acceptance thresholds per workflow, then tune prompts and system settings to meet them.
Case Examples
Clinic Drafting With Retrieval
An anonymized outpatient clinic uses a retrieval-augmented assistant to draft follow-up instructions. The system performs retrieval and reranking before generation. In tests, TTFT averages 3.5 seconds with p95 around 6 seconds, while total time averages 18 seconds for a 250-token draft. When the clinic switches to a smaller retrieval set, TTFT drops to 2.6 seconds and total time drops to 16 seconds, suggesting retrieval time contributed to both metrics.
After a traffic spike, TTFT rises to 7 seconds while total time stays near 18 seconds. Stage logs show model start delayed by queueing, while generation speed remains stable. The clinic responds by reducing concurrent requests and adding a short retry policy for non-idempotent prompts, which avoids compounding load.
Non-Streaming Summaries
An anonymized telehealth team uses a summarizer that returns a single response at the end rather than streaming tokens. Their dashboard reports “latency” as total time only, so TTFT is not observable. When they compare two prompts—one requesting 100 tokens and one requesting 400 tokens—total time scales roughly with output length, and users report the interface feels unresponsive for longer prompts because nothing appears until completion.
After switching to streaming, the team measures TTFT separately and sees first tokens arrive around 2 seconds for both prompts. Total time still increases with output length, but user feedback improves because they can stop early if the summary format is wrong. The team also updates its UI to show partial text and a “generation in progress” indicator, which reduces repeated prompt submissions.
Latency Checklist And Table
| Metric | What It Measures | Common Causes Of Slowdowns | What To Tune First |
|---|---|---|---|
| TTFT | Time until first streamed token arrives | Queueing, retrieval/tool pre-steps, client buffering, auth/gateway delays | Reduce pre-inference work, verify streaming, test from same region, lower concurrency |
| Total Time | Time until completion or stop condition | Output length, decoding settings, long context, post-processing | Cap max tokens, shorten prompts, adjust stop criteria, measure post-processing time |
| p95 Tail | Upper-bound behavior under variability | Load spikes, cache misses, network jitter | Use concurrency limits, add retries for safe steps, monitor cache hit rate |
Step-by-step checklist for a trustworthy latency comparison:
- Log timestamps for: request send, retrieval start/end, model start, first token, last token, and response assembly.
- Use the same prompt text and the same output constraints across runs.
- Run at least 30 trials per scenario and report median and p95, not only the mean.
- Test with streaming enabled and confirm the client receives incremental chunks.
- Separate cache-hit and cache-miss runs if your system caches retrieval or embeddings.
- Repeat tests at two concurrency levels to detect queueing effects.
Common Mistakes That Skew Results
One frequent mistake is comparing TTFT from a streaming endpoint to total time from a non-streaming endpoint. The numbers describe different user experiences, so the comparison rewards the wrong system. Another mistake is mixing prompts with different lengths and then attributing the difference to the model rather than the number of generated tokens.
Teams also misread dashboards that average across heterogeneous requests. If some requests include retrieval and others do not, TTFT averages become meaningless. A related issue appears when the system retries internally after a failure; the dashboard might show a single latency value that includes retry delays, which hides the original failure rate.
Client-side buffering can distort TTFT. If the SDK batches tokens before emitting them to your UI, the measured TTFT becomes a property of the client library rather than the model. I’ve seen teams upgrade a dependency (for example, a streaming parser update around mid-2023) and accidentally change observed TTFT without changing the backend.
Finally, people sometimes tune prompts for speed by removing context, then treat the resulting quality drop as an unavoidable trade-off. A more trustworthy approach is to measure both latency metrics and outcome metrics like factual consistency checks, formatting compliance, and user acceptance on a small labeled set.
FAQ
What Is Time To First Token Used For?
TTFT predicts how quickly a streaming AI response becomes visible to the user. It helps you judge responsiveness for workflows that start reviewing after the first few tokens.
Does Total Time Depend Only On The Model?
Total time depends on model generation plus pre- and post-processing steps such as retrieval, tool calls, and response assembly. If those steps change, total time changes even when the model stays the same.
Why Can TTFT Stay Flat While Total Time Increases?
TTFT can remain similar when the system starts generating at the same point, while total time grows with the number of tokens produced. Longer outputs extend the completion time.
How Do I Compare Two AI Services Fairly?
Use the same prompt, the same output limits, the same streaming mode, and the same concurrency level. Measure median and p95, and separate retrieval/tool steps from pure generation time if logs exist.
What Should I Do If p95 TTFT Is Much Worse Than Median?
Investigate queueing and cache misses, then test under realistic concurrency. Add stage logging to identify whether delays occur before model start or during token generation.
Author's Insight
TTFT and total time describe different parts of the request lifecycle, so they should be measured and interpreted separately. In systems that stream tokens, TTFT often reflects queueing and pre-inference work, while total time reflects how many tokens get generated and how long post-processing takes. Stage-level timestamps turn latency from a single number into a diagnosis. If you only track one metric, you end up tuning the wrong component and blaming the model for delays caused by retrieval, buffering, or load.
Key Takeaways
- TTFT measures “when the first output appears,” while total time measures “when the response finishes.”
- Pre-inference steps like retrieval, tool calls, and gateway delays can dominate TTFT.
- Total time scales strongly with output length and any post-processing after generation.
- Use median and p95, log stage timestamps, and test with realistic concurrency to avoid misleading comparisons.
- Streaming changes user experience; verify that your client actually receives incremental tokens.
