Time to first token (TTFT) explained: what it measures, why it has become one of AI infrastructure's defining metrics, and what drives it up or down across the stack.
Time to first token, or TTFT, measures the time between a user submitting a request and an AI model generating its first output token. The clock starts the moment a prompt arrives and stops when the first output token is generated. As inference workloads continue to grow, TTFT has become an increasingly important measure of AI responsiveness and infrastructure efficiency.
What TTFT actually measures
TTFT is measured from the moment an AI system receives a request until it generates the first output token. It includes the entire prefill stage and the generation of the first output token, but not the time taken to complete the rest of the response.
AI inference consists of two stages: prefill and decode. During prefill, the model processes the entire prompt and builds the key-value (KV) cache, which stores information from the prompt for reuse during generation. During decode, the model generates output tokens one at a time using that cached information.
.png)
Before the first token can be generated, a request must travel across the network, be scheduled onto available compute, and complete the prefill stage. That's why TTFT reflects the performance of the entire serving stack rather than the model alone. It captures the cumulative impact of networking, scheduling, model execution, and hardware performance.
Several factors influence TTFT, including prompt length, context size, network latency, request queueing, GPU availability, and model architecture. Longer prompts and larger context windows increase the amount of work during prefill, which is why retrieval-augmented generation (RAG), agentic AI, and applications with long conversation histories often have higher TTFT than simpler chatbot interactions. For example, increasing prefill throughput from 1,000 to 2,500 tokens per second reduces the time required to process a 100,000-token prompt from around 100 seconds to 35-40 seconds. The impact becomes even greater in long-running agentic workloads, such as complex coding or architecture tasks, where large contexts may be processed repeatedly across many interactions. In these cases, improvements to prefill performance can compound over the course of a task, potentially translating into hours or even days of difference in overall completion time.
TTFT versus other latency metrics
TTFT is often confused with related numbers that measure different things:
- Time per output token (TPOT), or inter-token latency: the average gap between tokens once generation has started. This governs how smooth the streaming experience feels after the first token appears.
- End-to-end latency: the total time from request to the last token, roughly approximated as TTFT plus output tokens multiplied by TPOT.
- Throughput: the total tokens or requests a system handles per second, a measure of sustained capacity rather than responsiveness.
- Cost per token: a measure of inference efficiency, not user-perceived speed.
A system can have excellent throughput and cost per token and still feel slow if TTFT is high, because the user is staring at a blank screen while the prefill phase works through a long prompt. This is why TTFT has become a defining metric alongside cost per token rather than a replacement for it: as providers increasingly monetize inference through token-based services rather than raw compute time, how fast infrastructure starts serving a request matters as much as what each token costs to produce.
What drives TTFT up or down
TTFT is rarely explained by a single component. It reflects how well compute, memory, storage, and networking work together across the entire stack.
Prompt length and context. Longer prompts and larger context windows mean more work during prefill, which is one of the most direct levers on TTFT. This is especially relevant for retrieval-augmented generation and agentic workloads, where retrieved context and conversation history can push prompt length up significantly.
Queueing and load. When a serving system is busy, requests wait before they are even processed. Under high concurrency, queue time can dominate TTFT more than the model's raw compute speed.
Memory and the KV cache. Inference places more emphasis on memory bandwidth and moving data efficiently than training does, which is dominated by raw matrix math. This shift is already reflected in newer GPU architecture designed specifically for inference workloads rather than training.
Data infrastructure. GPUs deliver value when they spend their time performing inference rather than waiting for data. That means data has to be in the right format and arrive fast enough to keep the GPUs busy. As inference workloads grow, data infrastructure has become a critical factor in TTFT. Reference architectures are increasingly bringing data services closer to the compute layer, reducing the latency of moving data between storage and GPUs.
Networking. Connections between GPUs, and between GPUs and storage, add up. Newer interconnect technology that removes physical cabling between GPUs is one example of infrastructure-level work aimed squarely at reducing that overhead.
The throughline across all of this is that optimizing GPUs alone is not enough to consistently deliver a fast first token. TTFT is a systems problem, not a single-component one.
Why TTFT is a full-stack metric
TTFT is often thought of as a measure of model serving performance. In reality, it reflects decisions made across every layer of AI infrastructure.
Software optimizations such as prompt caching, batching, and scheduling all influence how quickly a model begins responding. But those techniques depend on foundations that include GPU architecture, memory bandwidth, networking, storage, power, and data center design. Each layer affects the next, and delays introduced at any layer ultimately contribute to the latency experienced by the end user.
That's why consistently reducing TTFT requires more than optimizing model serving. It requires a full-stack approach, where improvements across infrastructure, hardware, and software combine to deliver faster, more responsive AI.
The takeaway
TTFT is not a line item any one team owns. It gets faster or slower with every decision made across an AI system, from networking and storage to compute, models, and applications. Cost per token tells you what a system costs to run. TTFT tells you how quickly that system starts to feel responsive to the person using it. Both matter, and neither one replaces the other.
.png){kind=icon}
