Tencent's Sherry Quantization: How a 1.5TB Model Shrank to 214GB
Tencent's Angel Slim toolkit uses ternary Sherry quantization to compress a 770B model 7x, from 1.5TB to 214GB, with barely any quality loss.
What is Sherry quantization?
Sherry is a quantization method from Tencent that rounds a model’s weights down to just three possible values: -1, 0, or +1. It’s built into Angel Slim, Tencent’s toolkit for shrinking large language models, and it’s the technique behind a widely shared result: taking a 770 billion parameter model from roughly 1.5 terabytes down to about 214 gigabytes, a 7x reduction, with benchmarks that barely move.
The trick isn’t magic. It’s a smart constraint on how the rounding happens, and that constraint is what makes the compression math work out so cleanly.
TL;DR
- Ternary quantization means every weight in the model gets rounded to one of only three values (-1, 0, +1) instead of a long decimal like 0.42 or -1.7, which is what actually saves the space.
- Sherry forces exactly one zero into every group of four weights, rather than hoping zeros show up naturally, which guarantees a predictable, limited set of patterns per group.
- That constraint means each group of four weights has only 32 possible patterns, and 32 patterns fit exactly into 5 bits of storage, working out to about 1.25 bits per weight on average.
- Applied to a 770 billion parameter model, this shrank the weights from around 1.5 terabytes down to roughly 214 gigabytes while keeping benchmark scores close to the original.
- Tencent’s own documentation and paper are described as hard to follow, mixing English and Chinese text and dense diagrams, which may explain why the technique took months to get wider attention despite being available earlier.
- Running the full-size quantized model still requires serious hardware, on the order of multiple high-memory GPUs, so this isn’t something you spin up on a laptop.
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
How does Sherry quantization actually work?
Start with the problem: a large language model’s weights are stored as long, precise decimal numbers, things like 0.42 or -1.7 or 2.1. Multiply that precision by hundreds of billions of parameters and you get enormous file sizes. A 770 billion parameter model in full precision lands at around 1.5 terabytes, more storage than a single consumer machine can hold, let alone run.
Quantization is the general fix: round those messy decimals to something coarser so they take less space. Sherry takes this to an extreme by using only three possible values per weight, positive one, negative one, or zero. This is called ternary quantization. A number like -1.7 becomes -1. A near-zero value like 0.03 becomes 0. All the fine decimal detail gets thrown away.
The interesting part isn’t the three-value idea itself, other methods have flirted with ternary weights before. It’s how Sherry handles the zero. Instead of letting zeros appear wherever they naturally fall, Sherry forces exactly one zero into every group of four weights. Always one, no more, no less. That single rule turns an unpredictable rounding process into something with a fixed, countable structure.
Why does forcing one zero per group matter?
Once every group of four weights is guaranteed to contain exactly one zero, the number of possible patterns that group can take becomes limited and known in advance: 32 distinct combinations. That’s the key move. Without the constraint, you’d need to store which values landed where in a much more open-ended way. With it, you only need enough bits to identify which of the 32 fixed patterns a given group represents.
Thirty-two patterns fit exactly into 5 bits of storage (2 to the power of 5 equals 32). Spread across four weights, that works out to 1.25 bits per weight, with no wasted space. Compare that to the 16 or 32 bits typically used per weight in full precision formats, and the size reduction follows directly from the math: fewer bits per weight, multiplied by hundreds of billions of weights, equals a dramatically smaller file.
That’s the whole mechanism. Round weights to three values, guarantee one zero per group of four, pack each group into 5 bits. Apply it across a 770 billion parameter model and the 1.5 terabyte footprint collapses to roughly 214 gigabytes.
Why did it take months for Sherry to get attention?
Angel Slim wasn’t a brand-new announcement when the 1.5TB-to-214GB headline started circulating. The toolkit had been available and usable for months before it trended, including smaller quantized models that could already be run and tested on modest hardware.
Part of the delay likely comes down to documentation. Tencent’s technical materials for Angel Slim mix English and Chinese text in the same diagrams, and the accompanying research paper leans heavily on dense notation that assumes a strong machine learning background. For a technique with real practical value, that’s a meaningful barrier. A compression method that could let more people run massive models is much less useful if almost nobody can figure out how it works or how to apply it from the source material alone.
This is a recurring pattern in AI tooling: methods with real technical merit sit unnoticed until a headline number (in this case, “7x smaller”) makes the value obvious to a much wider audience, even though the underlying tool was already there.
Is Sherry quantization worth it for running large models?
For the specific case demonstrated, a 770 billion parameter model, the tradeoff looks strong: benchmark scores stayed close to the original while size dropped by a factor of seven. That’s a meaningful win for anyone trying to deploy a model of that scale without dedicating a full server rack to it.
That said, “smaller” doesn’t mean “runs on a laptop.” A 214GB model still requires substantial hardware, generally multiple GPUs with large VRAM pools (the kind of setup involving several high-memory GPUs used together) to load and run at reasonable speed. Sherry moves the model from “needs a rack” to “needs a serious multi-GPU workstation,” not from “needs a rack” to “runs on a phone.”
Whether it’s worth it depends on what you’re trying to do. If you’re already working with models at the hundreds-of-billions-of-parameters scale and are constrained by storage or memory, a 7x reduction with minimal accuracy loss is a significant practical gain. If you’re working with smaller models on consumer hardware, other quantization approaches (including smaller Angel Slim configurations) may be more directly relevant to your setup.
Frequently Asked Questions
What does “ternary” mean in ternary quantization?
Ternary means each weight in the model can only take one of three values: -1, 0, or +1. This is in contrast to full-precision weights, which are stored as long decimal numbers with much finer detail. Ternary quantization trades that precision for a massive reduction in storage size.
How much smaller does Sherry make a model?
In the case demonstrated, a 770 billion parameter model went from approximately 1.5 terabytes in full precision to roughly 214 gigabytes after Sherry quantization, a reduction of about 7 times, while benchmark performance stayed close to the original.
What hardware do you need to run a Sherry-quantized model?
Even after quantization, a model in the 214GB range needs substantial hardware to run, generally multiple GPUs with large VRAM capacity working together. It’s a significant step down from needing a full server rack, but it’s still well beyond what a single consumer laptop or desktop GPU can handle.
Is Angel Slim a new tool from Tencent?
No. Angel Slim is Tencent’s toolkit for compressing large models and had been available for months, including smaller quantized models runnable on modest hardware, before the 770 billion parameter compression result brought it wider attention.
Why is Sherry’s zero-forcing rule important?
By guaranteeing exactly one zero in every group of four weights, Sherry limits each group to 32 possible patterns. Since 32 patterns can be represented in exactly 5 bits, this allows the model to be packed at about 1.25 bits per weight with no wasted storage, which is the main source of the size reduction.