Time to first token (TTFT) brings infrastructure performance closer to what users actually feel.
Inference is becoming the workload that defines AI infrastructure, and inference is judged in milliseconds. The gap between a prompt and a response is where AI products win or lose users. Closing it is a full-stack job. That’s why this month’s announcement matters. Our agreement to acquire Anyscale brings together Nscale's AI infrastructure with the software developers use to build, run, and scale AI workloads.
It’s an important step in Nscale’s full-stack story, and a timely moment to look at why AI infrastructure performance is increasingly a full-stack question.
For years, infrastructure investments at scale have been described primarily in terms of capacity: more GPUs, bigger clusters, more compute. Those numbers still matter. But users don't experience compute.
They experience time.
How quickly does an AI assistant respond? How long does a coding agent take to get to work? How soon does a model turn infrastructure into something useful?
At the RAISE Summit in July, conversations with NVIDIA, VAST Data, and Nscale pointed to the same shift: performance is increasingly about how quickly infrastructure translates compute into a high-performance AI experience.
One metric brings that into focus: Time to first token (TTFT). TTFT measures the interval between a user submitting a prompt and the model beginning its response. On the surface, it's a latency metric. But look beneath it, and TTFT tells a full-stack story.
Infrastructure has a new scorecard
Before the first token reaches the user, a lot has to happen behind the scenes. Workloads are scheduled, models loaded into memory, data moved across storage and networking, requests routed, and the inference pipeline orchestrated.
TTFT puts a clock on that entire journey. A bottleneck at any point can affect when a response begins.
That makes TTFT a systems metric rather than a component metric. No single team or technology determines the result, so reducing latency becomes a shared objective across the stack. Two platforms built on the same class of GPU can therefore deliver different levels of time-to-first-token performance depending on how that stack is designed and operated.
And those differences aren't just visible in a benchmark. They're perceptible to users. Decades of human-computer interaction research show how quickly users notice latency:
- Within ~100 ms: The response feels instantaneous.
- Within ~1 second: Users remain engaged with the task.
- Beyond ~1 second: Users notice the delay and can begin to disengage.
TTFT isn't a universal benchmark, however. Model choice, workload, prompt characteristics, and concurrency can all affect the result. Like any performance metric, it is most meaningful when compared within the broader context.
The economics of milliseconds
Speed isn't the only objective. Milliseconds also have an economic dimension. GPU capacity is at a premium, and optimizing for lower latency with smaller batches can lead to more spare capacity. The challenge is to make each request fast enough for the user without unnecessarily increasing the cost of serving it.
Batch size shows the trade-off clearly. Processing more requests together can improve throughput and lower costs, but it can also make users wait longer for a response.
TTFT is therefore most useful alongside other performance metrics, not as a target to minimize in isolation. The goal isn't the lowest possible TTFT. It's the right balance of latency, throughput, and cost for the workload.
That balance is shaped by product decisions as much as by hardware. Batching, routing, scheduling, and the defaults customers inherit all influence the result.
From first token to first answer
TTFT may not be the end of the story. As reasoning models take on more complex tasks and longer reasoning chains, a fast first token won't always mean a fast answer. Users ultimately care about when the first meaningful answer reaches them, not simply when a response begins.
That is giving rise to another metric: Time to first answer. It brings reasoning effort into the performance equation, alongside model optimization and inference infrastructure, as another factor shaping real-world response times.
The metric may evolve, but the direction of travel is clear. AI infrastructure is increasingly being evaluated not just by what goes into it, but by what comes out. Delivering high-performance means optimizing every layer of the stack, from infrastructure and orchestration through to the software serving the model. TTFT moves the scorecard closer to the thing infrastructure ultimately exists to deliver. Time to first answer may move it closer still.
Watch the conversation
Hear more from NVIDIA, VAST Data and Nscale on the infrastructure behind AI performance in our RAISE Summit panel video.
This article was originally published as part of Nscale's Full Stack AI newsletter, where we share perspectives on AI infrastructure, engineering, and emerging industry trends.
Subscribe to receive future editions.
