AI & Agents

Google’s TPU Sync Moves KV Cache Around So Prefill and Decode Can Live Apart

Leave a comment

Google has open-sourced TPU Sync, MarkTechPost reports: a C++ library for moving KV cache and model weights across a Cloud TPU memory hierarchy, including HBM, host DRAM, shared storage and the data-center network. The problem it’s aiming at is not the matmul. It’s the time you spend shuffling model state around, especially when prefill and decode run on different machines.

The library does layout-preserving DMA, so tensors keep their TPU tile layout, and it keeps transfers out of the XLA compilation path. Google’s own lab numbers, on TPU v7x, claim more than 90% lower KV-transfer latency on some 235B to 480B MoE workloads, with time-to-first-token falling sharply in the same tests. Those figures are internal and early. The README also says TPU Sync is not yet recommended for general use. Integrations are showing up in TorchTPU, vLLM and SGLang stacks, under an Apache 2.0 license.

Review before you install: read the release notes yourself, back up your config, and try new versions somewhere you can afford to break first.

My take

Disaggregated serving lives or dies on how fast you can move a cache. If those latency cuts hold up outside Google’s lab, this is a big deal for anyone already on TPUs. I’m taking the “not yet for general use” line at face value. Interesting, open, and not something I’d point a production job at this week.

God Bless, and Use the Source!

References & Further Reading

Subscribe (RSS) Follow AI & Agents via RSS

Leave a note

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.