How to Run Qwen3.8-27B-Escha-W2 Locally on a 24GB GPU
Guide to running Escha-W2, a 2-bit quantized Qwen3.8-27B, on a 24GB GPU with SGLang, covering VRAM tuning and 128k context setup.
What is Escha-W2, and why does it fit on a 24GB GPU?
Escha-W2 is a 2-bit quantized build of Qwen3.8-27B, published by Escha Labs, that squeezes all 27 billion parameters into roughly 10.15 GB of weights (2.469 bits per weight on average, using a mixed 2/3-bit scheme per projection plus int8 for the embedding and output head). That’s small enough that the model, its KV cache, and a 64k context window all fit on a single 24 GB card like an RTX 3090 or 4090, with room to spare. A tuned configuration pushes that same card to a 128k context window. The model is a hybrid architecture where only 16 of 64 layers use full attention, which is the main reason long context is affordable here.
TL;DR
- Escha-W2 compresses Qwen3.8-27B down to about 10.15 GB of weights using 2-bit quantization, small enough to run entirely on one 24 GB consumer GPU.
- The model uses a hybrid attention design where only 16 of 64 layers are full attention and the rest are gated-delta-net layers holding a fixed ~0.15 GB of recurrent state per stream, which is why the KV cache costs just 64 KiB per token instead of four times that.
- On a 24 GB card, the shipped defaults give a 64k context window at 18.3 GB peak VRAM, and raising
MEMto 0.88 withCTXLEN=131072extends that to 128k context at 21.8 GB. - Serving requires a specific runtime, escha-runtime-qwen3dense, an SGLang fork with custom decode kernels; installing plain SGLang from PyPI alongside it will cause conflicts.
- Key tuning knobs are
MEM(VRAM fraction),CTXLEN(context cap), andMAMBA_RATIO(memory reserved for recurrent state), and they interact tightly enough that changing one without the others can silently break things. - Prefix caching (
RADIX=1) does not speed up growing conversations on this architecture; it only helps exact, complete repeats of a prompt, so agentic loops that append tool results get no benefit from it. - Benchmarked against an FP8 reference on the same backend, Escha-W2 is roughly on par: ahead on commonsense reasoning, behind by one question on GPQA-Diamond, and ahead on LiveCodeBench within noise.
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
How do you set up Escha-W2 on a local GPU?
The setup has three moving pieces: a pinned PyTorch build, the Escha runtime wheel, and the model weights, which live in a separate repository from the runtime.
The install starts with Python 3.12 and a virtual environment, then PyTorch pinned to the 2.9.x series built for CUDA 12.8. This pin matters because the custom decode kernels are ABI-linked to a specific PyTorch build; letting pip resolve to a newer version like 2.11 will produce a runtime failure (undefined symbol errors) that’s confusing to debug after the fact.
Next comes the runtime itself, pulled from Hugging Face as EschaLabs/escha-runtime-qwen3dense. This package bundles a customized SGLang build along with its full dependency tree. Installing a separate sglang package from PyPI alongside it causes conflicts, since the two versions fight over the same import paths.
Finally, the model weights download separately from EschaLabs/Qwen3.8-27B-Escha-W2 into a flat folder. Once both pieces are in place, a single serve.sh script starts an OpenAI-compatible HTTP server on 127.0.0.1:30000.
Before serving, a three-part sanity check confirms CUDA is available, the custom kernel (escham_decode_gemv) is registered, and SGLang itself imports correctly. Skipping the SGLang import check specifically is a known trap: earlier testing found a box that passed a two-part check but couldn’t actually serve requests.
What GPUs and VRAM does it actually need?
The model card lists three verified GPUs: RTX 5090 (32 GB, Blackwell architecture), RTX 4090 (24 GB, Ada), and RTX 3090 (24 GB, Ampere). A 16 GB card should theoretically fit at reduced context, but that configuration hasn’t been tested.
On a 24 GB card at the shipped defaults (MEM=0.72, 64k context), peak VRAM usage sits at 18.3 GB, leaving meaningful headroom. Pushing to 128k context via MEM=0.88 raises peak usage to 21.8 GB, still under the 24 GB ceiling but with less margin for error.
Throughput scales differently depending on whether you’re optimizing for a single user or for concurrent requests. On an RTX 4090, the single-user default configuration produces about 67 tokens per second at batch size 1. A throughput-oriented configuration with more streams and a shorter 32k context reaches roughly 649 tokens per second across 16 concurrent streams. The RTX 5090, with more VRAM to work with, hits about 87 tokens per second at batch 1 and 955 tokens per second at 16 streams.
One hardware-specific quirk: on Ampere cards (RTX 3090 and similar), setting ESCHA_ROUTE=blackwell for single-user workloads produces a measured 1.72x speedup at batch size 1, with identical output quality. That gain disappears at batch sizes 2 and above, so it’s only worth setting for solo, low-concurrency use.
How do the MEM, CTXLEN, and MAMBA_RATIO settings interact?
These three settings share one pool of VRAM, and misunderstanding that relationship is the most common way to misconfigure the server.
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
MEM sets the fraction of total VRAM allocated to the combined weight and KV cache pool. Setting it too low causes an out-of-memory error during startup; setting it too high can cause a different OOM during CUDA graph capture. The fix in that second case is to lower MEM, not raise it.
CTXLEN sets the per-request context ceiling, but it’s a default, not a hard allocation. What actually determines whether a given context length works is the shared token pool the server prints at startup as max_total_num_tokens. That pool must be at least as large as concurrent streams multiplied by context length. Raising CTXLEN without also raising MEM produces a pool that’s too small for the context you asked for, and with the default TRUNCATE=1 setting, prompts get silently truncated rather than rejected with an error.
MAMBA_RATIO controls how much memory is reserved for the recurrent state used by the model’s gated-delta-net layers. Because this is a hybrid architecture, every concurrent stream holds recurrent state that doesn’t shrink as context grows, unlike standard KV cache. The default of 0.3 is tuned to leave enough room for both this recurrent state and KV cache at the default 64k context.
The practical guidance from testing: context length and concurrent stream count draw from the same budget. Shortening CTXLEN alone doesn’t free up capacity for more simultaneous streams. To actually raise the number of concurrent streams, you need to raise MAXREQ and MAXMAMBA together with MEM, then confirm the result by checking the #running-req figure in the server log against what you intended.
How far can you push the context window?
Long context works on this model because of its hybrid layer structure. Only 16 of 64 layers use full attention; the rest use gated-delta-net layers with fixed recurrent state that doesn’t grow with sequence length. The result is a KV cache cost of 64 KiB per token, roughly a quarter of what a conventional dense 27B model with full attention on every layer would require.
Measured on an RTX 4090, the token pool grows at approximately 366,600 tokens per 1.0 increment of MEM. At the shipped default (MEM=0.72), the pool holds 68,686 tokens, just above the 65,536 default CTXLEN, enough for exactly one full-length 64k stream. Pushing MEM to 0.88 with CTXLEN=131072 yields a pool of 141,431 tokens, comfortably covering a 128k context with headroom, at a peak VRAM cost of 21.8 GB. Going further, MEM=0.92 reaches a pool of 156,092 tokens, close to the practical ceiling for a 24 GB card; 160k does not fit.
A documented pitfall: some MEM/CTXLEN combinations produce a pool smaller than the requested context (for example MEM=0.84 with CTXLEN=131072), and the server starts without an error. With TRUNCATE=1 left on, over-long prompts get silently trimmed instead of triggering a failure. The only reliable check is comparing the startup-logged max_total_num_tokens against your actual concurrent-streams-times-context math. For agentic workloads, setting TRUNCATE=0 is safer, since it fails loudly on an over-long prompt instead of quietly cutting it off.
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
One caveat worth noting: quality above 64k tokens hasn’t been independently validated for this specific model. A 120,000-token prompt produces coherent, on-topic output in testing, which demonstrates the model functions at that length, but it isn’t the same as confirming retrieval accuracy holds across a long context.
Does prefix caching help with long conversations?
No, not for the typical agentic pattern of appending new content to an existing conversation. Prefix caching (RADIX=1) is off by default in this setup, and testing shows why: it only delivers a speedup when a new request is an exact, complete match for something already cached. In one test, sending the identical 120k-token prompt twice dropped wall time from 66.5 seconds to 1.4 seconds on the second call. But a prompt that shares a 116k-token prefix with a prior request, plus 4k tokens of new content, still took the full 65.7 seconds, reusing nothing.
This appears to be a consequence of how the recurrent state in the hybrid layers works: it seems to only be valid at the exact boundary where it was captured, so there’s no partial state to resume from mid-sequence. Practically, this means RADIX=1 is useful for retries, cache warming, or resampling a fixed prompt multiple times, but it does nothing for an agent loop that keeps appending tool outputs and resending. The real latency lever is keeping the working context small, not caching it.
Frequently Asked Questions
How much VRAM does Escha-W2 actually need?
The weights alone total about 10.15 GB. Running with the default 64k context window brings peak VRAM usage to roughly 18.3 GB on a 24 GB card. Extending to a 128k context window raises peak usage to about 21.8 GB, still within a 24 GB budget but with less margin.
Can I run Escha-W2 on a 16GB GPU?
The model card lists 16 GB as a card that “should fit” at reduced context length, but this configuration is explicitly noted as untested. The verified GPUs are the RTX 3090, RTX 4090, and RTX 5090, all with 24 GB or more.
Why do I need a special runtime instead of plain SGLang?
Escha-W2 uses custom decode kernels bundled in a fork called escha-runtime-qwen3dense. Installing standard SGLang from PyPI alongside it causes dependency conflicts, since the two versions compete over the same code paths.
What happens if I raise CTXLEN without raising MEM?
The context cap increases, but the shared token pool that actually backs it doesn’t grow unless MEM also increases. This can produce a pool smaller than your intended context, and with default settings the server will silently truncate long prompts rather than throwing an error.
Is thinking mode on by default, and how do I control its length?
Yes, thinking is on by default, and reasoning effort defaults to xhigh, which produces the most thorough (and slowest) responses. Effort is a prompt instruction, not a hard limit, so it’s a suggestion rather than a guarantee. For a hard cap on reasoning length, a thinking budget parameter forces the reasoning phase to end after a set number of tokens.
