DeepSeek Open-Sources Deep Infra: How It Serves 155T Tokens a Day
DeepSeek open-sourced its Deep Infra inference stack, showing how prefill/decode splitting and caching serve 155 trillion tokens a day.
What is DeepSeek’s Deep Infra?
Deep Infra is the inference engine DeepSeek built to run its own API, and the company recently open-sourced it along with a detailed technical write-up covering the parts it couldn’t release in code. According to DeepSeek, a single cluster running this stack serves 155 trillion tokens a day. The release matters because it shows, in concrete architectural terms, how a frontier lab serves a model to millions of concurrent users without every GPU sitting idle or every request stepping on every other request.
TL;DR
- DeepSeek open-sourced Deep Infra, the inference engine behind its API, along with a write-up describing the proprietary pieces it didn’t release as code.
- The system splits every request into two separate hardware jobs, prefill and decode, running on different machines tuned for different bottlenecks.
- DeepSeek reports a 96% cache hit rate on input tokens by keeping the KV cache on disk for up to 7 days for chat traffic and 3 days for agent traffic, backed by roughly 100 petabytes of SSD storage.
- A technique called SWA bonded replay reuses sliding-window attention computation instead of recomputing it, nearly halving total prefill work.
- Decode spreads a mixture-of-experts model like DeepSeek V4.1 Flash across 64 or more GPUs, rebalancing popular experts every few thousand steps to avoid bottlenecks.
- Speculative decoding with a confidence head that predicts which guessed tokens will survive adds roughly 6% throughput on top of standard speculative decoding gains.
- A batch-invariant mode exists specifically because batching can change model outputs even at temperature zero, which matters for anyone running evals or debugging agents.
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
How does serving an LLM to millions of users differ from running it locally?
Running a model on your own machine is a single-user, single-GPU job: the model reads your prompt, writes tokens back one at a time, and waits for your next message. There’s no contention, so there’s little need for sophisticated engineering.
Serving the same model to millions of people breaks that model completely. Requests arrive simultaneously, with wildly different shapes. A ten-word question might land in the queue milliseconds after an autonomous agent dumps an entire codebase for context. Every one of those requests hides two opposite hardware jobs: prefill, where the model reads the entire prompt in one large parallel pass and builds its working memory (the KV cache), and decode, where the model writes the answer one token at a time, with each token depending on the one before it.
Prefill is a burst of math done all at once. Decode is a small amount of math repeated thousands of times in sequence. Running both jobs on the same generic GPU cluster means they interfere with each other: a large incoming prompt can stall everyone else’s token generation. Deep Infra’s core design answer is to stop treating “inference” as one job and start treating it as two, run on separate hardware.
How does the prefill and decode split actually work?
A request enters as JSON, gets tokenized, and is routed through a component called DeepSeek Recipe (also open-sourced), which converts the input from OpenAI- or Anthropic-style formats into the exact tokens the model expects, and converts the model’s raw output back into a usable response.
This step solves a subtle security problem. Models use special control tokens to mark where a user’s message ends and the assistant’s reply begins. If a user types one of those markers as plain text, a naive server can treat it as a real control token, tricking the model into thinking its turn has already started. This is a known prompt injection vector. DeepSeek Recipe is built so that anything a user types stays plain text and can never be reinterpreted as a control token.
Before any new computation happens, the system checks the KV cache, the model’s saved memory of everything it has already read in that conversation. Only new tokens need to go through prefill. DeepSeek reports keeping this cache on disk for 3 days for agent traffic and 7 days for chat traffic, stored on a dedicated cluster with roughly 100 petabytes of SSD. The result: 96% of all input tokens are cache hits, meaning only about 4 of every 100 input tokens actually get read and processed fresh. This is also why cached input tokens are billed far more cheaply than fresh ones, and why keeping a stable prompt prefix in agent workloads is one of the simplest cost optimizations available.
New tokens then go to prefill, where a load balancer sorts incoming requests into length buckets so a short prompt doesn’t get stuck behind a 250,000-token one. Each GPU gets a time budget per step, and a scheduler predicts how much prompt it can process within that window.
Once prefill finishes, the request moves to decode, which runs on different hardware entirely: prefill machines are regular 8-GPU nodes built for compute throughput, while decode runs on “super nodes” with hundreds of GPUs built for fast memory access, since decode has to repeatedly read model weights and conversation memory for every single token. Running them separately lets DeepSeek tune time-to-first-token and total response time independently. In production, the system reportedly needs roughly one prefill GPU for every decode GPU.
What is SWA bonded replay and why does it matter?
DeepSeek’s model uses sliding window attention (SWA) in parts of its layers, meaning those layers only attend to the most recent 128 tokens rather than the full context. Instead of recomputing that sliding window from scratch through every layer on every request, Deep Infra replays just the short window at the end of the prompt. DeepSeek says this nearly halves total prefill compute, a meaningful saving given that prefill is the most compute-intensive half of the pipeline.
How does decode handle a mixture-of-experts model at scale?
DeepSeek V4.1 Flash is a mixture-of-experts (MoE) model, meaning each token only activates one group of specialized sub-networks out of a much larger pool. In decode, DeepSeek spreads those experts across 64 or more GPUs, so each GPU holds only a handful of experts and can serve a large batch of users simultaneously.
The problem with MoE at scale is load imbalance: some experts get hit far more often than others, and the slowest GPU sets the pace for the whole group. Deep Infra addresses this by keeping spare copies of popular experts on standby and shifting those copies around every few thousand steps based on live traffic patterns, smoothing out hotspots before they become bottlenecks.
The last layer of speedup comes from speculative decoding: a small draft model guesses the next five tokens, and the full model verifies all five in a single pass instead of generating them one by one. DeepSeek adds a confidence head on top that predicts how likely each guessed token is to be accepted, so the system skips verification compute on guesses unlikely to survive. DeepSeek reports this confidence scheduling adds about 6% additional throughput, which is a meaningful number of GPUs saved at this scale.
Why can the same prompt at temperature zero give different answers?
This is one of the more surprising findings in DeepSeek’s write-up. In theory, temperature zero should produce deterministic output. In practice, on a busy server, it often doesn’t, because GPUs sum floating-point numbers in a different order depending on how many requests are batched together at once. Those tiny rounding differences can flip which token the model picks, even though nothing about your individual prompt changed.
Without shadow IT.
Deep Infra includes a batch-invariant mode that forces the math to execute the same way regardless of what else is in the batch. It’s off by default because it’s slower, but it’s the setting to enable for reproducible evals or when debugging an agent where consistency matters more than raw speed. Notably, this behavior isn’t unique to DeepSeek’s infrastructure: it’s a property of batched floating-point computation in general, and it shows up even on small models running locally on modest hardware.
What are the limits of the open-sourced version?
Deep Infra as released is not something you can download and run as a full production API server. The open version only supports offline inference, with no built-in API server. The KV cache system and load balancers described in the write-up are tied to DeepSeek’s internal infrastructure, so the paper explains how they work, but the code for them hasn’t been released.
There’s also a hardware constraint: the kernels are written specifically for Nvidia’s data center Blackwell chips (B200-class), and they don’t run out of the box on other Blackwell variants used in workstation-class hardware. The attention library only builds for those data center chips, and the matrix multiplication library throws an unsupported architecture error on anything else. Recompiling doesn’t fix this, since it’s a targeted kernel design choice, not a build configuration issue.
Any revenue-per-GPU or revenue-per-gigawatt figures floating around based on this release should be treated with caution. They assume constant off-peak pricing, 100% utilization, and access to the proprietary caching system that hasn’t been open-sourced.
If you want to run DeepSeek V4.1 Flash yourself today, the model already works on standard inference engines like vLLM, which Deep Infra was originally built on top of. DeepSeek is open-sourcing both its models and parts of its infrastructure while continuing to build on existing open source tools, a two-way relationship worth noting for anyone evaluating how “open” this release really is.
Frequently Asked Questions
What is the difference between prefill and decode in LLM inference?
Prefill is the step where a model reads an entire input prompt in one parallel pass and builds its working memory (the KV cache). Decode is the step where the model generates the reply one token at a time, with each token depending on the previous one. Prefill is compute-heavy and bursty; decode is memory-bound and sequential, which is why DeepSeek runs them on different hardware.
How does DeepSeek achieve a 96% cache hit rate on input tokens?
By keeping the KV cache (the model’s memory of previously processed prompt tokens) on disk for extended periods: 3 days for agent traffic and 7 days for chat traffic, backed by around 100 petabytes of SSD storage. This means repeated or continued conversations rarely need to reprocess earlier context from scratch.
Can I run Deep Infra on my own hardware?
Not as a full production system. The open-sourced version only supports offline inference with no API server, and its performance kernels are built specifically for Nvidia data center Blackwell GPUs like the B200, not other Blackwell variants. You can, however, run the underlying DeepSeek models on standard engines like vLLM.
Why would the same prompt produce different answers at temperature zero?
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
Because GPU floating-point math sums numbers in different orders depending on what else is being processed in the same batch, and those tiny rounding differences can change which token the model selects. Deep Infra’s batch-invariant mode forces consistent math ordering to eliminate this, at the cost of some speed.
What is speculative decoding and how does DeepSeek improve on it?
Speculative decoding uses a small draft model to guess several upcoming tokens at once, which the full model then verifies in a single pass instead of generating tokens one by one. DeepSeek adds a confidence head that predicts which guesses are likely to be correct, skipping verification work on unlikely guesses, which reportedly adds about 6% extra throughput.
