Cost per token is the all-in operational cost of generating a single unit of AI output. It is shaped by an entire infrastructure ecosystem, including compute performance, networking, cooling, power distribution, software-driven optimization, deployment strategy, and operational resilience. As generative AI scales from experimentation to production, it has become the defining KPI for AI factory economics, eclipsing raw compute or chip-level performance as the measure of success.
No single factor sets the number. But power and energy efficiency have emerged as two of the largest and most controllable levers, and they are what this article examines in detail, from the shift to 800 VDC distribution to the software that manages it.
For years, computing success was measured in processing power. Faster chips, larger servers, denser racks. But as generative AI moves from pilot to production, the conversation has shifted. The new measure for the engineers, specifiers, and infrastructure teams building these facilities is not how powerful the AI is. It is how efficiently the underlying infrastructure converts grid power into AI output.
The calculation itself is simple. The inputs are not.
Cost per token = (power + cooling + compute + networking + facilities + operations) ÷ total tokens generated
The most efficient AI facilities are not always the ones with the fastest GPUs. They are the ones that produce the most tokens for the least infrastructure overhead.
What cost per token means for AI data center design
A token is the small unit of data an AI model uses to read or generate content. Every AI interaction, whether answering a question, summarizing a document, or generating an image, consumes thousands of tokens.
The numbers are striking. NVIDIA’s most recent benchmarks show its Blackwell GB300 NVL72 platform delivers a cost of $0.12 per million tokens, compared with $4.20 on the prior-generation Hopper HGX H200. That is a 35x reduction, driven by tighter integration between compute, networking, and power systems.
But cheaper tokens have not made AI cheaper to run. Inference costs per token have dropped roughly 10x over the past two years, while consumption has risen more than 100x. It is a classic case of the Jevons paradox, where falling unit costs drive demand faster than savings can offset them.
The result: even as the per-token math improves, total AI infrastructure costs are climbing. For data center designers, specifiers and builders, that means every design decision now ties directly to operating economics. Cost per token is the metric that translates voltage choices, conversion losses, and equipment selection into the units of work AI actually produces.
Why AI is changing the design problem
Traditional data centers were engineered around CPU workloads with rack densities of 5 to 10 kW. Today’s AI training racks now routinely exceed 100 kW with a clear path toward 1 MW per rack.
The International Energy Agency projects global data center electricity demand will more than double from 415 TWh in 2024 to roughly 945 TWh by 2030, approximately the entire annual electricity consumption of Japan today. AI-focused facilities are forecast to triple their power use over the same period.
In the United States, the impact is sharper. The IEA expects data center electricity demand to grow 130% by 2030, with these facilities accounting for nearly half of all U.S. electricity demand growth between now and the end of the decade. Goldman Sachs Research forecasts global power demand from data centers will increase as much as 175% by the end of the decade compared with 2023.
AI workloads change every assumption in the powertrain:
- Distribution voltage: legacy 400 V AC and 48 VDC architectures were not built for current rack densities
- Cooling integration: thermal loads of 80 to 120 kW per square foot require liquid cooling tightly coupled to the power architecture
- Equipment density: gray-space footprints must shrink to leave more floor for compute
- Continuous load profile: GPUs run at near-peak utilization, eliminating the load fluctuations traditional designs relied on
Why cooling costs compound cost per token
Cooling is one of the largest operational cost drivers in an AI facility. The International Energy Agency puts cooling and environmental control at roughly 7% of total electricity use in efficient hyperscale data centers, and over 30% in less efficient enterprise facilities.
As rack densities climb past 100 kW toward 1 MW, power and cooling stop behaving like separate systems. Every watt lost in power conversion becomes heat that has to be removed, so an electrical inefficiency gets charged twice, once at the meter and again at the chiller. Improving electrical efficiency therefore reduces energy consumption and cooling load at the same time. That is why Schneider Electric designs AI factory infrastructure end to end, from grid to chip and chip to chiller.
Every inefficiency between the utility tie-in and the chip directly increases cost per token. Lose one percent to cooling. Lose another to power conversion. Multiply that across millions of tokens per hour and the dollars move quickly.
Why 800 VDC has become the industry’s answer
Energy efficiency is no longer a sustainability story. It is the economic foundation of AI. Every wasted watt inside a data center is added to the cost of every AI interaction. As models scale, small efficiency improvements deliver outsized returns. Hyperscalers and the broader industry are responding by rethinking how power moves through the entire stack, from the utility grid to the chip.
The most consequential shift is moving AC-to-DC conversion out of the IT rack, distributing at 800 VDC. Today’s AI training racks routinely exceed 100 kW and are headed toward 1 MW per rack, and legacy AC and 48 VDC architectures cannot scale to meet that load efficiently.
A new architecture, codified in the Open Compute Project’s Mt. Diablo specification, moves AC-to-DC conversion into a dedicated power rack, also called a sidecar rack, supporting densities up to 1 MW. . Schneider Electric, among the power-system partners advancing 800 VDC,is developing a power rack capable of supporting IT racks up to 1.2 MW. The result is more rack space for compute, smaller conductors, and lower thermal losses.
For designers and specifiers, that translates to:
- More rack space available for compute equipment
- Smaller conductors and lower I²R losses across distribution
- Power capacity scaled independently of compute capacity
- Simplified maintenance, with power equipment serviced independently of servers
- Support for rack densities up to 1 MW
“Cost per token is a design constraint, not just a finance metric. Every decision we make on power conversion, distribution voltage, and resilience translates directly into the unit economics of AI. That is why Schneider Electric is helping shape 800 VDC standards through the Open Compute Project, to take losses out of the path between the grid and the GPU.”—Marta Asack, Senior Vice President, Power Products, Schneider Electric
How software lowers cost per token
Infrastructure efficiency does not stop at electrical design. Monitoring, analytics, predictive maintenance, and power management software increasingly shape the economics of AI operations. Real-time visibility into capacity utilization, power quality, and energy consumption helps operators find inefficiencies before they affect performance or uptime.
Digital management platforms help reduce stranded capacity, improve resilience, optimize energy use, and direct power to where AI workloads create the most value. EcoStruxure™ IT gives operators that visibility across power, cooling, and IT infrastructure. ETAP digital twins let design teams model electrical performance and test decisions before equipment is installed, simulating AI factory power from grid to chip with NVIDIA Omniverse.
Advanced power architecture sets the ceiling on efficiency. Software decides how much of that ceiling is actually reached. As AI environments scale, pairing the two is one of the most practical ways to bring cost per token down.
What cost per token means for the next generation of AI data centers
Computing performance will continue to matter. But the economic conversation around AI has expanded. Efficiency, resilience, and energy management now sit alongside raw capability as the metrics that decide which AI businesses scale profitably and which do not.
For the engineers, specifiers, and infrastructure teams building these facilities, cost per token is the language that turns architectural decisions into operating outcomes.
From grid to chip, the convergence of electrification, automation, and digital intelligence is what makes scalable, resilient, energy-efficient AI infrastructure possible, and what determines the real cost of intelligence.
Read the full solutions overview: From Grid to Chip: Scalable Power Distribution for AI Data Centers: reference designs, equipment specifications, and the end-to-end view of how Schneider Electric is solving AI power from utility to chip.
Frequently Asked Questions
What is cost per token in AI?
Cost per token is the total operational cost of producing a single token of AI output, including electricity, cooling, hardware depreciation, networking, and uptime. For AI data center designers, it is the metric that ties every architectural decision back to operating economics.
Why do AI data centers need different infrastructure than traditional ones?
AI workloads rely on GPU-dense racks that draw an order of magnitude more power than conventional CPU racks, and they run continuously at near-peak utilization. As a result, rack power densities have jumped from 5 to 10 kW to over 100 kW with a path to 1 MW, making legacy power and cooling architectures untenable.
What is 800 VDC and why does it matter for AI data centers?
800 VDC is an emerging high-voltage DC distribution architecture for AI data centers, codified in the Open Compute Project’s Mt. Diablo specification. It moves AC-to-DC conversion out of the IT rack into a dedicated power rack, freeing rack space for compute, cutting conductor size and thermal losses, and supporting rack densities up to 1 MW.
How does power architecture affect AI economics?
AI workloads draw vast amounts of power continuously at near-peak utilization. Every watt lost to inefficient conversion, distribution, or cooling adds to the cost of every token. Architectural decisions on distribution voltage, where AC-to-DC conversion happens, and UPS operating mode determine how much energy actually reaches the GPU.
How can designers reduce cost per token through infrastructure decisions?
Specify higher-voltage DC distribution architectures like 800 VDC to cut conversion losses. Choose UPS systems with efficient operating modes (such as eConversion) that bypass continuous regeneration when grid power is stable. Specify modular, prefabricated power solutions to accelerate deployment. Integrate intelligent monitoring at every level of the powertrain to identify and reduce energy losses in operation.
How does lower cost per token accelerate AI ROI?
Lower cost per token improves the margin on every AI workload an operator runs. It shortens the payback period on infrastructure investment, makes more use cases viable at production scale, and frees capacity for additional workloads inside the same power envelope. The practical result is that efficiency gains in power, cooling, and software compound into faster returns on the AI investment itself.
Add a comment