How to Run Xing4.0-29B-A4B Locally with vLLM or SGLang
A practical guide to deploying Xing4.0-29B-A4B, a 29B MoE model with 4B active parameters, locally using vLLM, SGLang, or KTransformers.
What is Xing4.0-29B-A4B?
Xing4.0-29B-A4B is an open-weight mixture-of-experts (MoE) language model from China Telecom Artificial Intelligence Technology Co., Ltd., released as part of the Xing series (previously known as TeleChat). It has 29 billion total parameters but only activates about 4 billion per token, which keeps inference costs closer to a small dense model while retaining the capacity of a much larger one. It natively supports a 256K token context window, extensible to 512K, and is distributed in Hugging Face Transformers format so it works out of the box with vLLM, SGLang, and KTransformers.
TL;DR
- Sparse activation means Xing4.0-29B-A4B has 29B total parameters but only 4B active per token, so it needs less compute and memory bandwidth per forward pass than a dense model of similar total size.
- Long context is native, with a 256K token window out of the box and a documented extension path to 512K, which matters for agent workloads and large codebases.
- Three inference backends are officially supported: vLLM, SGLang, and KTransformers, all consuming the same Hugging Face-format weights without conversion.
- Benchmark results are competitive with similarly sized MoE models like Gemma4-26B-A4B and Qwen3.6-35B-A3B, with particularly strong scores on Terminal-Bench 2.1, Claw-Eval, and DeepresearchBII.
- The architecture is agent-oriented, combining mHC, MLA (multi-head latent attention), and MTP (multi-token prediction) to support multi-step planning and tool calling.
- The model was trained entirely on Huawei Ascend NPUs using the MindSpore framework, making it one of the first models at this scale trained outside the Nvidia CUDA ecosystem, though inference is framework-agnostic.
- Recommended sampling settings differ by task: temperature 1.0 for general reasoning, temperature 0.8 for coding and agent tasks, both with top_p 0.95 and repetition_penalty 1.05.
One coffee. One working app.
You bring the idea. Remy manages the project.
Why does the MoE design matter for local deployment?
The headline number for anyone planning hardware is not 29B, it’s 4B. Mixture-of-experts models route each token through a small subset of specialized sub-networks rather than the whole model. Xing4.0-29B-A4B uses 64 routed experts plus 1 shared expert, activating 4 experts per token. That means the compute cost of generating each token looks much closer to a 4B dense model than a 29B one, even though you still need enough memory to hold all 29B parameters (or a quantized version of them) on your GPU or in your NPU/CPU memory pool.
This is the same tradeoff behind models like Qwen3’s A3B variants and Gemma4-26B-A4B, which the Xing4.0 model card benchmarks itself against directly. For local hosting, this means you get faster token generation and lower latency per request than a dense 29B model would give you, at the cost of needing to fit the full parameter set in memory somewhere, whether that’s GPU VRAM, unified memory, or a hybrid CPU/GPU setup via KTransformers.
How do you serve Xing4.0-29B-A4B with vLLM?
vLLM is the most common starting point for serving Xing4.0-29B-A4B because it’s built for high-throughput, OpenAI-API-compatible serving of Hugging Face models. Since the model card confirms native Transformers-format compatibility, the general workflow looks like this:
- Pull the model weights from the Hugging Face repository (
XingChen-AGI/Xing4.0-29B-A4B). - Install a vLLM version that supports the model’s architecture (MLA attention plus the mHC/MTP components used by the Xing series). Check the model’s GitHub repository for the exact launch flags and any required vLLM patches, since MoE models with custom attention mechanisms sometimes need specific serving configurations.
- Launch vLLM’s OpenAI-compatible server, pointing it at the downloaded weights.
- Send requests using the standard OpenAI Python client, exactly as shown in the model’s quickstart example, just pointing
base_urlat your local server instead of a hosted endpoint.
Once running, you interact with it like any other OpenAI-compatible model: pass messages, set temperature and top_p, and optionally toggle the enable_thinking flag inside chat_template_kwargs to turn the model’s reasoning trace on or off.
How does SGLang serving differ from vLLM?
SGLang is the other primary serving option and is explicitly called out in the model card alongside vLLM. It’s designed for high-throughput structured generation and tends to be a common choice for agent-style workloads that involve tool calls, multi-turn state, or constrained output formats. Since Xing4.0-29B-A4B is described as “agent-oriented” and was evaluated on agent benchmarks like Tau3-Bench and Claw-Eval, SGLang’s support for structured output and efficient KV cache reuse can be a good fit if you’re building tool-calling pipelines on top of the model rather than doing simple chat completion.
The setup pattern mirrors vLLM: download the weights, launch SGLang’s server pointed at the model directory, and connect through the OpenAI-compatible API. Exact launch commands and any model-specific flags live in the model’s GitHub repository rather than the Hugging Face card itself, so check there before deploying in either framework.
What about KTransformers for constrained hardware?
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
KTransformers is the third supported framework, and it’s worth calling out separately because it targets a different deployment scenario: running large MoE models on hardware that can’t hold the full model in GPU memory. KTransformers specializes in hybrid CPU/GPU inference, offloading portions of the expert weights to CPU RAM while keeping hot paths on GPU. For a model like Xing4.0-29B-A4B, where only 4 of 64 routed experts fire per token, this kind of offloading strategy can make sense: you keep frequently activated experts and attention layers on GPU while parking less-frequently-used experts in system memory.
This makes KTransformers the practical option for people who want to run the model locally on a single consumer GPU without the VRAM to hold all 29B parameters, accepting a throughput tradeoff for feasibility. The exact configuration (how many experts to offload, memory pool sizing) depends on your hardware and is documented in the KTransformers project and the Xing4.0 GitHub repository rather than the model card.
What are the recommended inference settings?
The model card specifies different sampling parameters depending on your use case:
| Scenario | temperature | top_p | repetition_penalty |
|---|---|---|---|
| Complex reasoning / general tasks | 1.0 | 0.95 | 1.05 |
| Coding / agent tasks | 0.8 | 0.95 | 1.05 |
Lower temperature for coding and agent tasks is standard practice, since these workloads reward deterministic, syntactically correct output over creative variation. The enable_thinking flag in the chat template lets you toggle a visible reasoning trace, which is useful for debugging agent behavior but adds token overhead if you don’t need it in production.
Is Xing4.0-29B-A4B worth running over alternatives?
The model card’s own benchmark table positions Xing4.0-29B-A4B against Gemma4-26B-A4B and Qwen3.6-35B-A3B, three MoE models in a similar size class. Xing4.0 leads on Terminal-Bench 2.1 (57.50 vs. 30.00 for Gemma and 51.50 for Qwen), Claw-Eval (76.55), SWE-bench Multilingual context, and DeepresearchBII (60.80), while trailing slightly on IFBench and AA.LCR. On raw SWE-bench Verified, Qwen3.6-35B-A3B edges it out (76.00 vs. 75.00), and AIME2026 math scores are close across all three (88 to 93 range).
In practice, this means Xing4.0-29B-A4B is a strong candidate specifically for agentic coding and terminal-based tasks, areas where its architecture (mHC + MLA + MTP) was explicitly optimized. If your workload leans toward long-running agent sessions, tool use, or software engineering tasks over pure instruction-following or math, its benchmark profile favors that direction. Whether it’s “worth it” over Qwen or Gemma alternatives depends on whether your infrastructure already favors one ecosystem, and whether the specific benchmarks that matter to your use case line up with where Xing4.0 wins.
Frequently Asked Questions
How much VRAM does Xing4.0-29B-A4B need to run locally?
The model card doesn’t publish an explicit VRAM requirement, but with 29B total parameters, a full-precision or lightly quantized deployment typically needs high-end multi-GPU setups, while quantized formats or KTransformers’ CPU/GPU offloading can bring it within reach of a single consumer GPU. Check the model’s GitHub repository for concrete hardware recommendations and any published quantized checkpoints.
Can I fine-tune Xing4.0-29B-A4B on my own data?
Yes. The model supports fine-tuning through LLaMA-Factory and MindFormers, and the model card specifically calls out use cases like intent classification, table understanding, contract auditing, and knowledge-based QA as good fits for lightweight domain adaptation.
What’s the difference between the 256K and 512K context claims?
The model natively supports a 256K token context window. The 512K figure is described as an extension path, meaning it requires additional configuration or techniques beyond the out-of-the-box setup rather than being available by default.
Does Xing4.0-29B-A4B require Huawei Ascend hardware to run?
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
No. It was trained on Ascend 910C NPUs using MindSpore, but the released weights are in standard Hugging Face Transformers format, so inference works on standard GPU infrastructure through vLLM, SGLang, or KTransformers without needing Ascend hardware.
What is the “A4B” in the model name referring to?
“A4B” indicates the number of active parameters per token, roughly 4 billion, distinguishing it from the 29B total parameter count. This naming convention is shared with comparable MoE models like Qwen’s A3B variants.
