Run Qwen3.8-27B-Escha-W2 on a 24GB GPU with SGLang
How to install and tune Escha-W2, a 2-bit quant of Qwen3.8-27B, on a 24GB consumer GPU using SGLang for long context or high throughput.
What is Qwen3.8-27B-Escha-W2?
Escha-W2 is a 2-bit quantized build of Qwen3.8-27B, published by Escha Labs, that fits the full 27-billion-parameter model into roughly 10.15 GB of weights. That’s small enough to load the model, its KV cache, and a 64k context window on a single 24 GB consumer GPU, with headroom to spare. With a tuned configuration, the same card can push all the way to 128k context. The model is served through a custom SGLang runtime built specifically for this quantization format.
TL;DR
- Escha-W2 quantizes Qwen3.8-27B to roughly 2.469 bits per weight (mixed 2/3-bit per projection, int8 embedding and head), bringing total download size to about 10.18 GB.
- A 24 GB card runs the shipped defaults out of the box: 64k context, one concurrent stream, around 18 GB of VRAM used.
- The model uses a hybrid attention architecture where only 16 of 64 layers do full attention and the rest are gated-delta-net, which is why the KV cache is small enough to allow 128k context on consumer hardware.
- Three tuning variables (
MEM,CTXLEN,MAMBA_RATIO) control the tradeoff between context length and concurrent request throughput, and they draw from the same shared memory pool. - Prefix caching (
RADIX=1) only helps exact repeated prompts, not growing agent conversations, because the recurrent state can’t be resumed mid-sequence. - RTX 3090 users should set
ESCHA_ROUTE=blackwellfor a measured 1.72x speedup at batch size 1, and RTX 5090 owners needATTN_BACKEND=tritoninstead of the default flashinfer backend. - The default reasoning mode (
xhigh) is a prompt injection, not a hard limit, so latency-sensitive applications should either switch toloweffort or use a thinking budget to force a cutoff.
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
Why does a 27B model fit in 10GB?
Standard 16-bit weights for a 27-billion-parameter model would require on the order of 54 GB, well past what any single consumer GPU can hold. Escha-W2 gets around this with aggressive quantization: most projections run at 2-bit precision, some at 3-bit, averaging out to about 2.469 bits per weight, while the embedding and output head stay at int8 for stability. That shrinks the weight footprint to 10.15 GB, small enough to share a 24 GB card with a meaningful KV cache and still leave room for CUDA graph capture.
The bigger reason long context is affordable, though, is architectural rather than a quantization trick. Only 16 of the model’s 64 layers perform full attention; the other 48 use gated-delta-net, a linear-attention-style mechanism that holds a fixed amount of recurrent state per stream (about 0.15 GB) regardless of context length. Because most layers don’t scale with sequence length, the KV cache works out to about 64 KiB per token, roughly a quarter of what a conventional dense 27B model with full attention on every layer would need. That’s the mechanical reason 128k context is reachable on a 24 GB card at all.
How do you install and run it?
The setup has three moving pieces: a pinned PyTorch build, a runtime wheel from Escha Labs (which bundles its own SGLang fork), and the weights themselves.
python3.12 -m venv .venv && source .venv/bin/activate
pip install -U pip wheel
pip install "torch==2.9.*" --index-url https://download.pytorch.org/whl/cu128
pip install -U "huggingface_hub[cli]"
hf download EschaLabs/escha-runtime-qwen3dense --include "sglang/*" --local-dir runtime
pip install ./runtime/sglang/escha-*.whl
hf download EschaLabs/Qwen3.8-27B-Escha-W2 --local-dir Qwen3.8-27B-Escha-W2
MODEL=./Qwen3.8-27B-Escha-W2 bash runtime/sglang/serve.shTwo details matter here. First, PyTorch has to be pinned to the 2.9.x line and installed before anything else, because the decode kernels are ABI-linked to that specific build. Installing a loose torch>=2.9 constraint can silently resolve to a newer, incompatible version, and the failure only shows up later as an obscure undefined symbol error. Second, don’t install sglang from PyPI separately. The runtime wheel already includes a modified SGLang build, and the two will conflict.
Before serving, a sanity check confirms CUDA is available, the custom decode kernel is registered, and SGLang actually imported: all three must print True. Once the server is running, it exposes an OpenAI-compatible endpoint at http://127.0.0.1:30000/v1, and any existing OpenAI client library or tool (including agent frameworks like opencode) can point at it with a placeholder API key, since the server does no authentication by default.
What do MEM, CTXLEN, and MAMBA_RATIO actually control?
These three environment variables, passed to serve.sh, govern how the GPU’s memory is split between model weights, KV cache, and the recurrent state used by the gated-delta-net layers.
MEM sets the fraction of total VRAM reserved for the weight and KV pool (default 0.72). Push it too high and CUDA graph capture runs out of memory; push it too low and the server refuses to start with a “not enough memory” error. The fix for an OOM during graph capture is to lower MEM, not raise it.
CTXLEN sets the per-request context ceiling (default 65,536) but doesn’t reserve memory on its own. What actually constrains things is a shared pool the server reports at startup as max_total_num_tokens. That pool needs to be at least as large as the number of concurrent streams multiplied by the context length each stream uses. Raise CTXLEN without also raising MEM, and the pool can end up too small, silently truncating long prompts if TRUNCATE is left at its default of 1.
MAMBA_RATIO (the --mamba-full-memory-ratio flag) governs how much memory is set aside for the recurrent state that every concurrent stream needs, independent of context length. The default of 0.3 is deliberately lower than SGLang’s usual 0.9, because this is a hybrid model where recurrent state doesn’t shrink even for short contexts.
Because context and concurrency draw from the same pool, there’s a real tradeoff: a config tuned for one very long stream (128k tokens) can’t simultaneously serve many short concurrent conversations, and vice versa.
How do you configure for long context vs. high throughput?
Escha Labs published three verified configurations, each measured on physical hardware:
Single-user, 24 GB, shipped defaults. Just MODEL=./Qwen3.8-27B-Escha-W2 bash sglang/serve.sh gets 64k context on an RTX 4090 at about 67 tokens/second at batch size 1, using 18.1 GB of VRAM. On an RTX 3090, the same setup with ESCHA_ROUTE=blackwell added goes from 23.6 to 40.7 tokens/second.
High throughput, 24 GB. Setting MEM=0.86, CTXLEN=32768, MAXREQ=32, MAXMAMBA=32, and expanding CUDA_GRAPH_BS to include larger batch sizes lifts an RTX 4090 to 649 tokens/second across 16 concurrent streams. The key insight is that MAXREQ and MAXMAMBA are what actually raise the stream ceiling; the shipped MEM/MAMBA_RATIO defaults otherwise cap a 24 GB card at 8 to 9 concurrent streams no matter what batch sizes are listed in CUDA_GRAPH_BS.
32 GB, RTX 5090. With ATTN_BACKEND=triton set (required on consumer Blackwell, since the default flashinfer backend isn’t supported for this hybrid architecture on sm_120), MEM=0.85, and similar concurrency settings, a 5090 hits 87.1 tokens/second at batch size 1 and 955 tokens/second at 16 streams.
For maximum context on a 24 GB card, the documented recipe is MEM=0.88, CTXLEN=131072, MAXREQ=1, MAXMAMBA=2. That configuration handled a 120,000-token prompt in 68 seconds with 22.1 GB peak VRAM out of 24, and represents close to the practical ceiling: MEM=0.92 (147,456 tokens) is workable, but MEM=0.94 overflows the pool.
Is prefix caching worth turning on?
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
Generally, no, unless the workload involves retries or repeated identical prompts. RADIX=1 enables prefix caching, but on this hybrid architecture it only reuses an exact, complete match of a previous prompt. Measured at 120k tokens, re-sending the identical prompt dropped wall time from 66.5 seconds to 1.4 seconds. But appending new content to an already-cached prefix, the pattern of a typical agent loop that adds a tool result each turn, reused nothing and took the same 65.7 seconds as a fresh prompt. The likely explanation is that the recurrent state used by the gated-delta-net layers is only valid at the exact point it was captured, so there’s no partial state to resume from. That makes RADIX=1 useful for cache warming or multi-sampling a fixed prompt, but not a latency lever for growing conversations. It also requires MAXREQ of at least 2, since prefix caching consumes a request slot itself.
Frequently Asked Questions
How much VRAM does Escha-W2 actually need?
The weights alone are about 10.15 GB. With the shipped defaults (64k context, single stream), total peak usage on an RTX 4090 is around 18.1 to 18.3 GB, leaving comfortable margin on a 24 GB card. Pushing to 128k context raises peak usage to about 22.1 GB.
Does this model support multimodal input?
No. The configuration file references a vision tower inherited from the base architecture, but the quantized checkpoint is text-only.
What GPUs has this been verified on?
Escha Labs tested on RTX 5090 (32 GB, sm_120), RTX 4090 (24 GB, sm_89), and RTX 3090 (24 GB, sm_86). A 16 GB card should work at reduced context but wasn’t tested.
Why does generation sometimes produce fluent but wrong answers?
This usually means the transformers library is below version 5.8, which loads this model’s architecture with a different attention path without raising an error. Upgrading transformers resolves it.
What does the reasoning_effort setting change?
It controls a short instruction prepended to the model’s internal reasoning: xhigh (the default) asks it to validate assumptions and prioritize correctness, medium adds no steering at all, and low asks for brief, direct reasoning. It’s a prompt, not an enforced limit, so a genuinely hard problem can still produce a long response even at low. For guaranteed latency, a thinking budget that forces a cutoff is the reliable option.
