- TurboQuant is a vector quantization algorithm from Google Research and Google DeepMind that can compress AI model memory—specifically the KV cache—by up to 6x with no measurable quality loss.
- Unlike traditional product quantization or GPTQ-style methods, TurboQuant is data-oblivious and works online, requiring zero training or codebook construction before deployment.
- The technique could reshape the economics of running large language models by dramatically reducing the memory and hardware needed for inference.
Every large language model has a dirty secret: most of the GPU memory it consumes during inference isn’t storing the model’s weights. It’s storing the KV cache—the growing record of everything the model has seen so far in a conversation. The longer the conversation, the bigger the cache. And the bigger the cache, the more expensive each response becomes to generate.
In March 2026, a team from Google Research and Google DeepMind published a paper on an algorithm called TurboQuant that attacks this problem at the mathematical root. The result: a way to compress that memory by a factor of six with zero measurable accuracy loss. The day the research blog post went live, memory stocks from Samsung to Micron dropped up to 6.2%. Three of the most important companies in the global semiconductor industry moved sharply on a blog post about math.
What TurboQuant Actually Does
At its core, TurboQuant is a vector quantization algorithm. Quantization, in AI terms, means representing numbers with fewer bits—turning precise 16-bit floating-point values into rougher 4-bit or 3-bit approximations to save memory and compute. The challenge has always been doing this without destroying the model’s output quality.
TurboQuant specifically targets high-dimensional vectors—the kind that make up an LLM’s KV cache during inference. Instead of storing hundreds of 16-bit numbers per attention head per token, TurboQuant compresses those vectors down to 3.5 bits per channel while keeping the model’s output indistinguishable from the uncompressed version. Push it to 2.5 bits and you get only marginal quality degradation. The paper demonstrates compression ratios of at least 4.5x compared to uncompressed representations.
The algorithm was developed by Amir Zandieh and Vahab Mirrokni at Google Research, alongside Majid Daliri from NYU and Majid Hadian from Google DeepMind. The paper, submitted in April 2025, is rooted in Shannon’s source coding theory—the mathematical framework for how much you can compress a signal without losing information.
How It Works: The Three-Step Trick
Here’s the core problem with quantizing AI model data: the numbers involved—weights, activations, attention cache values—are not well-behaved. Some values are tiny, clustered near zero. Others are massive outliers that spike wildly. It’s like trying to grade a class where most students score between 60 and 80, but three students somehow scored 9,000, negative 400, and 12 million. Any simple rounding scheme will either destroy those outliers (losing critical information) or waste precision on them (making everything else worse).
TurboQuant sidesteps this problem entirely with a technique so elegant it almost feels like cheating. It doesn’t fight the outliers. It doesn’t learn a lookup table. It just rearranges the numbers until they become easy to compress, then compresses them simply. Here’s how.
Step 1: Shuffle the deck (random rotation). Imagine you have a bookshelf where the books are organized chaotically—some shelves are overflowing, others are nearly empty. Now imagine you could spin the entire bookshelf at a precise mathematical angle so that every shelf ends up with roughly the same number of books. That’s what the random rotation does.
Specifically, TurboQuant multiplies each input vector by a random orthogonal matrix constructed using the Fast Walsh-Hadamard Transform (FWHT). This isn’t a slow matrix multiplication—it runs in O(d log d) time, making it nearly as fast as a simple element-wise operation. The effect is dramatic: those heavy-tailed outlier distributions that make quantization hard get spread evenly across all coordinates. The spikes vanish. What remains is a smooth, concentrated distribution—like shaking a jar of mixed nuts until they settle uniformly.
Step 2: Compress each coordinate independently (scalar quantization). Here’s where the magic really kicks in. After rotation, something remarkable happens at a statistical level. Each coordinate of the rotated vector follows a concentrated Beta distribution that, in high dimensions, converges to a Gaussian—a bell curve. And crucially, any two distinct coordinates become nearly independent—not just uncorrelated, but statistically almost independent, which is a much stronger mathematical property.
Think of it this way: imagine you have a room full of people, each holding a number. Before rotation, everyone’s number is somehow related to their neighbor’s—it’s a tangled mess of correlations. After rotation, it’s as if everyone in the room has been given a random number that has nothing to do with anyone else’s. You can now evaluate each person in isolation.
This means TurboQuant can quantize each coordinate separately using a simple, precomputed scalar quantizer—essentially a 1-dimensional lookup that maps continuous values to the nearest representative point. No need to account for interactions between dimensions. No expensive multi-dimensional clustering. The scalar quantizers are found by solving a 1-dimensional k-means problem using the Max-Lloyd algorithm, then precomputed and stored for common bit-widths. At inference time, it’s just a lookup.
Step 3: Fix the bias (inner product correction). There’s a catch. Step 2 is optimized for mean-squared error—minimizing the average distance between original and reconstructed values. But in transformer attention, what actually matters is the inner product (dot product) between key and value vectors. MSE-optimal quantizers introduce a subtle bias in inner product estimation: they systematically over- or under-estimate attention scores.
TurboQuant handles this with a second pass. After the initial quantization, it computes the residual error (the difference between the original vector and its quantized approximation), then applies a 1-bit Quantized Johnson-Lindenstrauss (QJL) transform to that residual. Think of it as a correction stamp: the first pass gives you a compressed version that’s good on average, and the second pass precisely cancels out the specific bias in inner product calculations. The result is unbiased inner product estimation—the attention scores computed from compressed vectors match the original.
The paper proves that this two-stage approach—MSE quantization plus QJL correction—achieves near-optimal distortion for both objectives simultaneously. And the whole pipeline, including the rotation, runs in linear time relative to the vector dimension. No training. No calibration. No codebook. Just rotate, quantize, correct.
How TurboQuant Differs From Existing Methods
The quantization landscape is crowded, and understanding where TurboQuant fits requires a quick tour of the existing approaches.
GPTQ (2022) was a breakthrough for weight quantization, but it’s an offline method—it needs calibration data and processes weights layer by layer using second-order optimization. It’s accurate but slow to apply, and it can’t handle dynamic data like the KV cache. AWQ improved on this by identifying salient weight channels and protecting them during quantization, but it still requires offline analysis of representative data. SmoothQuant migrates quantization difficulty from activations to weights, enabling 8-bit inference, but it operates within the same offline paradigm and struggles below 8 bits.
KIVI and PolarQuant target the KV cache specifically—the same problem TurboQuant solves—but they leave newly generated tokens unquantized, creating an inconsistency that degrades quality over long sequences. TurboQuant applies quantization even during the streaming generation process, so every token gets the same treatment.
Product Quantization (PQ), widely used in vector databases and nearest neighbor search, relies on k-means clustering to build codebooks. As the number of bits increases, the codebook grows exponentially, making high-bit-width PQ impractical. PQ also requires separate training data to construct the codebook, adding overhead. TurboQuant has no codebook at all—it uses precomputed scalar quantizers that work regardless of the input distribution.
The key differentiators are threefold. First, TurboQuant is online and data-oblivious—no calibration, no training data, no codebook construction. Second, it’s provably near-optimal: the paper formally proves its distortion is within a factor of approximately 2.7x of the theoretical lower bound, a guarantee no prior method has demonstrated. Third, it works across bit-widths from 2-bit to 4-bit and beyond, and it handles both MSE and inner product distortion objectives.
Why the KV Cache Problem Matters
To understand why TurboQuant matters, you need to understand the KV cache bottleneck. When a transformer model processes a sequence of tokens, it computes key and value vectors for each token at every attention layer. These vectors are stored in memory so the model can attend to all previous tokens when generating the next one.
For a model like Llama 3 with 128K context length, the KV cache can consume tens of gigabytes of GPU memory—often more than the model weights themselves. This is why long-context inference is so expensive, why models struggle with extended conversations, and why context windows are artificially capped despite models theoretically supporting longer sequences.
Current solutions are brute-force: buy more GPUs, add more HBM, or limit context length. Companies are spending tens of billions on AI infrastructure partly to work around this bottleneck. TurboQuant offers a software-only alternative: compress the KV cache by 4.5 to 6x while maintaining full model quality, potentially making much of that hardware investment unnecessary.
Real-World Performance
The paper’s experiments paint a convincing picture. On standard LLM benchmarks, TurboQuant achieves absolute quality neutrality at 3.5 bits per channel—meaning the compressed model’s outputs are statistically indistinguishable from the full-precision version. At 2.5 bits, the degradation is measurable but small.
On the Needle-In-A-Haystack test—a benchmark that measures a model’s ability to find specific information buried in long documents—TurboQuant maintains perfect accuracy even with aggressive compression. This is particularly significant because it means models can use long context windows without paying the full memory cost.
On LongBench end-to-end generation tasks, the compressed models match or nearly match uncompressed baselines across multiple evaluation categories. For nearest neighbor search—a different application entirely—TurboQuant outperforms existing product quantization techniques in recall while reducing indexing time to virtually zero. That’s because there’s no codebook to build. You just rotate and quantize.
The algorithm has already been adopted by other researchers. A team applied TurboQuant to protein language models (TurboESM), solving the KV cache bottleneck for biological sequence processing. Another group integrated it into 3-bit LLM weight quantization (ITQ3_S), pushing the boundary of how aggressively model weights can be compressed.
Use Cases and Implications
The implications of TurboQuant extend well beyond academic benchmarks. In practical terms, this technology could reshape how AI is deployed at every level:
Longer context on fewer GPUs. If the KV cache shrinks by 6x, a model that previously needed 8 GPUs for 128K context could potentially run on 2 or 3. This directly impacts the cost of serving enterprise AI workloads, legal document analysis, codebase understanding, and any application that requires processing long inputs.
More capable on-device AI. The push for running AI models on phones and laptops is severely constrained by available memory. Compressing the KV cache means longer conversations and more complex tasks on the same hardware, without relying on cloud inference.
Lower inference costs across the industry. Inference—running trained models—is where most AI spending goes. If TurboQuant cuts the memory footprint of inference by 6x, cloud providers can serve more customers per GPU cluster. The per-token cost of AI drops. New use cases that were previously too expensive become viable.
Scientific and specialized models. Protein language models, genomic analysis tools, and drug discovery pipelines all face severe memory constraints when processing long biological sequences. TurboQuant’s adaptation to these domains—demonstrated by the TurboESM work—shows the technique generalizes beyond standard NLP.
Vector databases and search. Beyond LLMs, TurboQuant’s near-optimal performance on nearest neighbor search tasks could improve the efficiency of vector databases, recommendation systems, and retrieval-augmented generation (RAG) pipelines. Zero indexing time is a significant advantage over product quantization in these applications.
The Bigger Picture
TurboQuant is not a product. It’s not something you download or install. It’s a mathematical result—a proof that the memory bottleneck in transformer inference is more compressible than anyone had formally demonstrated before.
But mathematical results have a way of becoming products within 12 to 18 months. If TurboQuant’s approach gets integrated into inference frameworks like vLLM or Hugging Face Transformers, the cost of running large language models could drop meaningfully without requiring any new hardware.
The AI industry has spent the past two years building bigger models and buying more GPUs. TurboQuant suggests the smarter play might be compressing what we already have. As researchers also explore fundamentally new architectures like CALM that rethink how models generate text, the combined effect of algorithmic innovation on both the generation side and the compression side could be transformative.
The paper is available on arXiv. Whether Google integrates TurboQuant into its own serving infrastructure—Gemini, Gemma, or otherwise—remains to be seen. But the math is public, and the race to implement it has already started.
Frequently Asked Questions
What is Google TurboQuant?
TurboQuant is a compression technique developed by Google Research that reduces the memory footprint of large language models by up to 6x with no measurable accuracy loss. It works by compressing the KV cache—the largest memory consumer during inference—using a three-step process of chunking, rotation, and quantization.
How does TurboQuant differ from standard quantization?
Standard quantization compresses model weights but doesn’t address the KV cache, which grows linearly with context length and dominates inference memory. TurboQuant specifically targets the cache, achieving 6x compression where traditional methods only compress weights by 2-4x. It’s complementary to weight quantization—you can use both together.
How much could TurboQuant reduce AI inference costs?
By compressing the KV cache 6x, TurboQuant could reduce GPU memory requirements proportionally, which directly lowers the hardware cost per token. Inference memory is the primary bottleneck for serving long-context models, so a 6x reduction could translate to significantly cheaper API pricing for applications that process long documents or conversations.