2-Bit vs FP8 Quantization: What the Escha-W2 Benchmarks Show
Escha-W2 compresses a 27B model to 2-bit and matches FP8 on GPQA, LiveCodeBench, and commonsense tests. Here's what that means for local inference.
Does 2-bit quantization actually hold up against FP8?
For one specific case, yes. Escha-W2, a 2-bit quantized build of Qwen3.8-27B from Escha Labs, was benchmarked against a same-backend FP8 reference of the same base model across three test categories: GPQA-Diamond, LiveCodeBench, and a commonsense reasoning suite. The 2-bit version trailed FP8 by exactly one question on GPQA-Diamond, beat it on commonsense reasoning, and edged ahead on LiveCodeBench, though that last gap falls inside the benchmark’s own noise. In practical terms, dropping from 8 bits to roughly 2.5 bits per weight cost almost nothing on quality while cutting the model’s footprint from tens of gigabytes to about 10 GB.
TL;DR
- Escha-W2 packs the full 27-billion-parameter Qwen3.8-27B into 10.15 GB of weights using a mixed 2/3-bit scheme that averages 2.469 bits per weight, with int8 kept for embeddings and the output head.
- On a head-to-head comparison against an FP8 version of the same base model, the quality gap was essentially zero: one question behind on GPQA-Diamond, ahead on commonsense reasoning, and ahead on LiveCodeBench within measurement noise.
- The compressed size means the entire model, its KV cache, and a 64k context window fit on a single 24 GB consumer GPU, with a tuned configuration stretching that to 128k context on the same card.
- This isn’t a dense transformer end to end. It’s a hybrid architecture where only 16 of 64 layers use full attention and the rest are gated-delta-net layers holding fixed recurrent state, which is why the KV cache costs a fraction of what a conventional dense 27B would need.
- Reasoning effort is a prompt, not a hard limit. The default
xhighsetting is what every benchmark on the card ran under, and it works by prepending an instruction to the model’s thinking process rather than capping token count. - Prefix caching doesn’t help growing conversations. It only reuses an exact, complete match of a previous prompt, so agent loops that append new tool output and re-send get no speedup from it.
- Verified hardware runs from an RTX 3090 up through an RTX 5090, with throughput ranging from tens of tokens per second at batch size one to several hundred tokens per second when serving many concurrent streams.
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
What is Escha-W2, exactly?
Escha-W2 is a quantized version of Qwen3.8-27B, a 27-billion-parameter model, released by Escha Labs. Quantization is the process of representing a model’s weights with fewer bits than the format it was trained or originally released in, trading some precision for a smaller file size and lower memory use. FP8 (8-bit floating point) is already a common compression target for serving large models efficiently. Escha-W2 goes much further, using a mixed 2-bit and 3-bit scheme per projection layer that works out to an average of 2.469 bits per weight, while keeping the embedding and output head layers at int8 to protect the parts of the model most sensitive to precision loss.
The result is a 10.15 GB weights file for a model whose parameter count would normally demand far more memory at higher precision. That size is what makes it possible to run the full 27B model, its key-value cache, and a substantial context window entirely within the 24 GB of VRAM found on consumer cards like the RTX 3090 and RTX 4090.
How does the 2-bit version compare to FP8 on real benchmarks?
The comparison ran across three categories chosen to stress different capabilities: GPQA-Diamond for graduate-level science reasoning, LiveCodeBench for code generation, and a commonsense reasoning suite. Both the 2-bit and FP8 versions were built from the same Qwen3.8-27B base and served on the same backend, which controls for infrastructure differences and isolates the quantization method as the variable being tested.
The outcome was close to a wash. FP8 edged ahead on GPQA-Diamond, but by a single question, which is a small enough margin that it could shift with a different random seed or slightly different test split. On commonsense reasoning, the 2-bit model came out ahead. On LiveCodeBench, the 2-bit model also came out ahead, though the margin sits within what the benchmark itself considers noise. Taken together, none of the three categories showed a clear, meaningful advantage for the higher-precision format. For a model that is roughly a quarter the size in bits per weight, that’s a notable result: it suggests that for this base model and this quantization method, most of the precision reduction from FP8 down to roughly 2.5 bits is not costing measurable task performance.
Why does a hybrid architecture matter for memory?
Part of what makes Escha-W2 practical to run isn’t just the bit-width of its weights but the underlying architecture of the base model. Only 16 of the model’s 64 layers use full attention, the mechanism that lets a model look back across its entire context window. The other 48 layers use a gated-delta-net design, a form of recurrent state that stays fixed in size no matter how long the context grows.
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
This matters because the key-value (KV) cache, the memory a model uses to avoid recomputing attention over previous tokens, is one of the biggest costs of serving long-context requests. With most layers skipping full attention entirely, the KV cache for this model works out to 64 KiB per token, a fraction of what a conventional dense 27B model with full attention on every layer would require. That’s the mechanical reason a 128k-token context can fit on a single 24 GB card at all: the architecture, not just the quantization, is doing a lot of the work.
What does this mean for running the model on consumer GPUs?
Verified configurations exist for the RTX 5090 (32 GB), RTX 4090 (24 GB), and RTX 3090 (24 GB), all running through an SGLang-based runtime built specifically for this quantization format. On a 24 GB card at the shipped default settings, the model serves a single user with a full 64k context window. Pushing the memory allocation and batch settings further enables much higher throughput: on an RTX 4090, tuning for concurrency reached 649 tokens per second across 16 simultaneous streams, up from 67 tokens per second at batch size one. On the RTX 5090, similar tuning reached 955 tokens per second at 16 streams.
Context length and concurrency draw from the same memory pool, so there’s a real tradeoff. Configuring for a single long-context stream (up to roughly 128k to 147k tokens depending on how much memory is allocated) leaves little room for serving multiple users at once, and vice versa. The documentation is explicit that raising the context length setting alone does not reserve any more memory. What actually determines whether requests succeed is the total token pool the server reports at startup, which needs to be large enough to cover concurrent streams multiplied by their context length. Configurations that get this wrong can start up without error and then silently truncate long prompts rather than rejecting them, which is a subtle failure mode worth checking for in any production deployment.
Is 2-bit quantization worth it compared to running a smaller model at full precision?
The benchmark comparison here answers a narrower but more useful question: for this specific base model, does going to 2-bit cost you accuracy compared to a more conventional 8-bit approach on the same architecture? Based on the three test categories reported, the answer is no, at least not meaningfully. That’s a different claim than saying 2-bit quantization is always safe. Quantization sensitivity varies by architecture, by which layers get compressed, and by task type, and the card itself notes that quality above 64k tokens of context is unvalidated for this particular checkpoint, since the long-context retrieval numbers reported elsewhere in the documentation come from a different model.
The practical case for a well-executed 2-bit build is memory. A 27B model that fits in 10 GB of weights opens up hardware that would otherwise be out of reach for that parameter count, including mid-range consumer GPUs rather than data-center cards. If the accuracy trade-off is genuinely negligible, as this comparison suggests, then the choice between 2-bit and FP8 becomes mostly a question of what hardware you have, not what quality you’re willing to sacrifice.
Frequently Asked Questions
What is FP8 quantization?
FP8 is an 8-bit floating point format used to store model weights with less memory than standard 16-bit or 32-bit formats, while retaining more numerical precision than very low-bit formats like 2-bit or 4-bit quantization.
How many bits per weight does Escha-W2 actually use?
It uses a mixed scheme averaging 2.469 bits per weight across most projection layers, with the embedding and output head layers kept at int8 for stability.
Can Escha-W2 run on a 16 GB GPU?
The model card lists 24 GB and 32 GB cards (RTX 3090, RTX 4090, RTX 5090) as verified. A 16 GB card is expected to work at reduced context length but is explicitly noted as untested.
Does prefix caching speed up long conversations with this model?
No. Prefix caching only reuses an exact, complete match of a previously seen prompt. A conversation that grows by appending new content, such as an agent loop adding tool results, gets no benefit and re-runs the full prefill each turn.
Is this model good for tasks beyond 64k tokens of context?
The architecture supports much longer context and the memory math checks out up to around 128k to 147k tokens on a 24 GB card, but retrieval accuracy at that length has not been validated for this specific checkpoint.
