Running GLM-5.3-UNCENSORED Locally: TabbyAPI, EXL3, and 8x A100s
How to serve the 273GiB GLM-5.3-UNCENSORED EXL3 quant on 8x A100 GPUs via TabbyAPI, with MTP drafting and a tool-calling parser fix.
What is GLM-5.3-UNCENSORED and what does it take to run it?
GLM-5.3-UNCENSORED is a weight-edited variant of Z.ai’s GLM-5.3, a 753B-parameter mixture-of-experts model (256 routed experts, 8 active per token, plus one shared expert) with MLA attention and a sparse DSA indexer. The “uncensored” version comes from dealignai, who published the edit as CRACK_SURGERY.json applied directly to the FP8 weights, no fine-tuning involved. A separate EXL3 quantization at 3.0 bits per weight brings the full checkpoint down to 273 GiB, small enough to fit across 8x A100 40GB GPUs using TabbyAPI as the inference server.
TL;DR
- GLM-5.3-UNCENSORED is a direct weight edit of Z.ai’s 753B-parameter GLM-5.3 MoE model, not a retrained fine-tune, documented in a
CRACK_SURGERY.jsondiff file from dealignai. - The EXL3 3.0bpw quant compresses the FP8 source to 273 GiB using mixed precision: 5 bpw for attention and shared experts, 4 bpw for dense MLPs, 3 bpw for routed experts, and 6 bpw for the lm_head.
- TabbyAPI with ExLlamaV3 is the serving stack used to run it, tested on 8x A100 40GB with automatic layer splitting (
gpu_split_auto) and a 98K-token shared KV cache. - The model ships with an MTP (multi-token prediction) layer that doubles as a speculative decoding draft model (
draft_mode: mtpin TabbyAPI config), speeding up generation without a separate draft checkpoint. - Quality loss from quantization is measurable but modest: KL divergence of about 0.09 against the FP8 source, with perplexity rising from 3.302 to 3.440 on wikitext-2.
- A critical tool-calling bug in TabbyAPI’s GLM parser causes string parameters (order IDs, zip codes) to get silently coerced into integers, breaking roughly 70% of tool calls on agentic benchmarks until patched.
- Agentic performance (tau2-bench airline and retail tasks) shows the quantized, uncensored model trailing stock GLM-5.3 FP8 by a small margin that isn’t statistically significant at the tested sample sizes.
Other agents ship a demo. Remy ships an app.
Real backend. Real database. Real auth. Real plumbing. Remy has it all.
How does the EXL3 quantization work?
EXL3 is ExLlamaV3’s quantization format, and this release uses a mixed-precision scheme rather than a flat bit count. The average is 3.04 bits per weight, but different parts of the network get different treatment based on sensitivity: attention layers and the shared expert run at 5 bpw, the dense MLP layers at 4 bpw, and the routed experts (the bulk of the parameter count, since only 8 of 256 activate per token) drop to 3 bpw. The output head (lm_head) stays at 6 bpw to preserve final token prediction quality, and the quant uses the mul1 codebook.
The model also retains an MTP (next-token prediction) layer from the original architecture, quantized separately at 4 bpw for experts and 6 bpw for attention and the shared expert. This layer wasn’t calibrated during conversion but is functional enough to serve as a speculative decoding draft inside TabbyAPI.
Conversion was done with ExLlamaV3 at a specific commit (d3739fd), using the default calibration set of 250 rows at 2048 tokens each, reading directly from the FP8 source checkpoint rather than from a higher-precision original.
How much does quantization hurt output quality?
The model card includes a direct diff against the FP8 source using KL divergence and perplexity on wikitext-2 test data (20 rows, 2048 tokens each). KL divergence between the quantized model and FP8 comes out to roughly 0.089 to 0.097 depending on direction, with a median per-token KL of 0.021 and a 90th percentile of 0.221. Perplexity rises from 3.302 (FP8) to 3.440 (quantized), a small but real increase.
One useful detail: for the 44% of tokens where the FP8 model was already highly confident (top probability at least 0.95), median KL divergence drops to 0.0011, meaning the quant barely changes the model’s behavior on “easy” predictions. Most of the divergence concentrates on harder, more ambiguous tokens, which is typical for aggressive quantization and not a red flag on its own.
How does it perform on agentic tasks?
The model was evaluated with tau2-bench, a benchmark that scores tool-calling agents on airline and retail customer service tasks, using Pass^1 scoring with an LLM-based user simulator and judge. The comparison point is stock GLM-5.3 (without the uncensoring edit) served at FP8 by Z.ai via OpenRouter, run through the same test harness.
Results were mixed but close. On airline tasks (50 tasks x 2 trials), the quantized uncensored model scored 0.640 against a baseline of 0.710, a gap of about 1.1 standard errors. On retail tasks (114 tasks), it scored 0.482 against 0.504, under half a standard error of difference. Neither gap clears the bar for statistical significance given the sample sizes, so the honest read is that the uncensoring edit plus 3-bit quantization together produce a possible small regression on airline-style tasks, not a confirmed one. It’s also worth noting this comparison mixes two variables (the weight edit and the quantization) against a single FP8 baseline, so it can’t isolate which change matters more.
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
What hardware do you actually need?
The tested configuration is 8x A100 40GB GPUs, using TabbyAPI’s gpu_split_auto option to let the server handle layer distribution across devices rather than manually specifying splits. With a 273 GiB model, you need enough combined VRAM to hold the weights plus room for KV cache, in this case a 98,304-token shared cache. Eight 40GB A100s give you 320GB total, which covers the model comfortably with headroom for context and the MTP draft layer.
This isn’t a consumer setup. It’s the kind of configuration available on multi-GPU cloud instances or dedicated inference servers, not a single workstation. Anyone without access to that scale of hardware would need a more aggressive quant, a smaller model, or an offloading strategy that trades speed for lower VRAM requirements.
What is the TabbyAPI configuration and what’s the tool-calling bug?
TabbyAPI, built on the ExLlamaV3 backend, is configured through a YAML file specifying the model name, backend, sequence length, cache size, and tool format. For this model, the working configuration sets backend: exllamav3, max_seq_len: 65536, cache_size: 98304, gpu_split_auto: true, tool_format: glm4_7, and reasoning: true, with a separate draft_model block enabling draft_mode: mtp to activate speculative decoding using the model’s own MTP layer instead of a standalone draft model.
The critical catch is in tool calling. GLM models write tool arguments as raw text inside tags like <arg_value>9523456873</arg_value>. As of TabbyAPI commit be74bf0, the glm4_5 parser JSON-decodes every one of these values without checking the tool’s declared schema. The practical effect: any parameter that looks like a number, such as an order ID, product ID, or zip code that’s defined as a string type, gets silently converted to an integer before it reaches your tool. On the tau2-bench retail benchmark, this caused roughly 70% of tool calls to fail outright. The fix is to patch the parser so it checks each parameter’s declared schema type and keeps string-typed fields as raw text rather than coercing them. The benchmark numbers above were collected with that fix applied, so anyone running this model for agentic or tool-calling workloads needs the same patch before trusting results out of the box.
Is it worth running locally instead of using a hosted API?
For teams with the hardware already available (multi-A100 clusters or similar), running GLM-5.3-UNCENSORED locally buys full control over moderation behavior, since the model’s refusal patterns have been directly edited rather than inherited from the vendor’s safety tuning. It also keeps data on-premises, which matters for tool-calling agents handling customer records or order data. The tradeoff is real engineering overhead: you’re responsible for patching inference server bugs like the tool-call parser issue, managing a 273 GiB model across multiple GPUs, and accepting a small, not-fully-confirmed performance dip compared to the vendor’s hosted FP8 version. For anyone without dedicated GPU infrastructure, the hosted stock model remains the simpler and likely more reliable choice.
Frequently Asked Questions
What does “uncensored” mean for this model?
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
It refers to a direct weight edit (documented as CRACK_SURGERY.json) applied to GLM-5.3’s FP8 weights by dealignai, intended to reduce refusal behavior without retraining the model. It is not an official Z.ai release.
How much VRAM do I need to run this model?
The tested setup uses 8x A100 40GB GPUs (320GB total) to hold the 273 GiB EXL3 quant plus a 98K-token KV cache and the MTP draft layer. Smaller setups would need a more aggressive quantization or offloading.
What is MTP drafting and why does it matter here?
MTP (multi-token prediction) is a layer included in the original GLM-5.3 architecture that can double as a speculative decoding draft model. Enabling draft_mode: mtp in TabbyAPI lets the server use this built-in layer to speed up generation instead of requiring a separate smaller draft model.
Why do tool calls fail without the parser fix?
TabbyAPI’s GLM tool-call parser converts every argument value through JSON decoding without checking the tool schema, so string fields that look numeric (order IDs, zip codes) get turned into integers. This broke about 70% of tool calls in retail-domain testing until the parser was patched to respect declared string types.
Is the uncensored quant worse than the official GLM-5.3?
On tau2-bench agentic tasks, it scored slightly lower than stock GLM-5.3 FP8 on both airline and retail domains, but the differences are within one standard error or close to it, meaning they aren’t statistically confirmed regressions at the tested sample sizes.
