Insights from Nscale, NVIDIA, and VAST Data on why time to first token (TTFT) is becoming one of AI infrastructure's most useful metrics.
Speaking at the RAISE Summit in Paris, Rod Evans, VP of Supercomputing and AI Cloud Infrastructure at NVIDIA, noted that a 20-megawatt cluster of 10,000 GPUs can cost just under $2 billion to stand up. At that scale of investment, every second before those systems begin producing useful work has an economic cost. That’s why time to first token (TTFT) has become a useful indicator of how effectively an AI system is designed to respond. It measures how quickly the AI stack begins delivering a response to the end user, making it a practical signal of both interactivity and full-stack system performance.
At a panel moderated by Nscale's Managing Director, Kristin Zwez, alongside Rod Evans and Jeff Denworth, co-founder of VAST Data, the discussion converged on one conclusion: TTFT is more than just the hardware layer. Delivering faster responses at scale requires coordinated optimization across every layer of the AI infrastructure stack. While TTFT is measured in milliseconds, the value of reducing it compounds across millions of inference requests. Faster responses improve the user experience, increase infrastructure utilization, and ultimately extract more value from every GPU in the cluster.
TTFT as a signal of full-stack performance
Cost per token remains a critical top-level measure of inference economics. TTFT provides a complementary view of system responsiveness: how quickly an AI system begins responding once a request arrives.
For cloud and inference providers serving token-based workloads at scale, delays before a response begins can indicate friction across scheduling, model serving, memory, networking, and the data layer. Addressing those bottlenecks can make the system more responsive, improve the use of available infrastructure, and contribute to lower overall operating costs.
Rod Evans argued that improving TTFT is fundamentally a full-stack infrastructure challenge. Performance depends on how compute, storage, networking, and the rest of the stack work together. Optimizing GPUs alone is not enough to consistently deliver the first token quickly.
Kristin Zwez reinforced the point by referencing NVIDIA CEO Jensen Huang's "five-layer cake." Every layer depends on the one beneath it, and delays anywhere in the stack ultimately surface as latency for the end user. TTFT can therefore reveal how effectively the entire AI stack works together.
Inference reshapes the data layer
One of the key technical themes of the discussion was how inference is reshaping the relationship between compute, memory, and storage. As inference workloads scale, memory requirements are evolving to support both short-term context, such as KV caching during inference, and the longer-term context that enables agentic applications. Therefore, data movement plays a crucial role in the overall system performance.
Evans argued that one of the biggest barriers to bringing AI systems into production is still data infrastructure. GPUs only create value when they're actively producing tokens, which means data must be in the right format and arrive quickly enough to keep them busy. Reducing latency therefore depends not only on faster GPUs, but on removing bottlenecks across the data layer.
Denworth expanded on that idea, arguing that reference architectures are fundamentally changing how AI infrastructure is designed. Rather than treating storage as a separate system connected to compute, modern AI platforms are integrating data services into the compute layer itself, reducing latency and helping improve time to first token.
What is changing at the hardware level
NVIDIA's latest architecture reflects this shift directly. Training focuses on processing enormous volumes of mathematical calculations as quickly as possible, so raw computing power, measured in FLOPs, is critical. Inference shifts the priority towards responsiveness, making fast access to memory and efficient data movement just as important as compute. That thinking underpins NVIDIA's Vera Rubin architecture and Rubin CPX platform, alongside networking advances such as cable-less NVLink connections between GPUs that reduce infrastructure complexity. As Evans described it, the goal is to build a fully integrated "token factory," where hardware, networking, storage, models, and applications are optimized together to deliver tokens as quickly as possible.
The takeaway for enterprises
Neither speaker argued that organizations should rebuild their infrastructure from scratch. Evans encouraged enterprises to augment existing systems with AI, such as introducing AI agents into established service desk workflows, while staying focused on the business outcomes those investments are expected to deliver. Denworth's advice was equally pragmatic: assign a single accountable owner for AI initiatives and adopt proven reference architectures rather than reinventing the stack.
The underlying point both were making is that TTFT is not a line item any one team owns. It gets faster or slower with every decision made across the AI stack, from networking and storage to compute, models, and applications. When those decisions are made together, rather than in isolation, TTFT becomes the number the end user actually feels.
