How to Deploy Ling-3.1-flash with SGLang and vLLM
A practical guide to self-hosting InclusionAI's Ling-3.1-flash MoE model with SGLang or vLLM, covering GPU requirements and server commands.
What is Ling-3.1-flash and what does it take to run it?
Ling-3.1-flash is an open-weight hybrid reasoning model from InclusionAI with 560 billion total parameters and 25 billion active per token, built on a mixture-of-experts (MoE) architecture with a 1 million token context window. Running it yourself means serving a model that needs multi-GPU tensor and expert parallelism, not a laptop-friendly checkpoint. InclusionAI publishes day-0 support for both SGLang and vLLM, with verified configurations for H200, GB300, H800, H100, and B200 clusters.
TL;DR
- Ling-3.1-flash is a 560B-parameter MoE model with 25B active parameters per token and a 1M-token context window, positioned as an upgrade over Ling-3.0-flash for agentic and real-world tasks.
- The architecture mixes 63 KDA layers and 9 gated MLA layers (a 7:1 ratio) across 512 routed experts, 1 shared expert, and 8 activated experts per token.
- SGLang is the recommended serving path, with a dedicated day-0 Docker image (
lmsysorg/sglang:dev-ling-3.1-flash) that bundles MTP speculative decoding and Hopper-specific decode optimizations. - A typical SGLang deployment uses 8x H200 GPUs with tensor parallel 8 and expert parallel 8, while 4x GB300 setups scale down to TP4/EP4.
- vLLM support exists as a separate branch and requires building from source with
uv, plus flags for expert parallelism, prefix caching, and MTP speculative decoding. - Benchmarks cited by InclusionAI include Terminal-Bench 4.0, SWE-bench Pro, FrontierSWE, GDPVal-AA V2.1, and others, aimed at coding agents, office tasks, and tool-calling workflows.
- InclusionAI also introduces LingCode, an internal coding-agent harness built on the open-source Pi agent, which it may open-source later.
What hardware do you need to self-host Ling-3.1-flash?
The model card specifies multiple verified hardware paths rather than a single minimum spec, because a 560B-parameter MoE model with only 25B active parameters per token has different memory and compute tradeoffs than a dense model of similar size. The documented configurations are:
- 8x H200 with tensor parallel size 8 and expert parallel size 8 (TP8 + EP8), the primary recipe shown in the quickstart.
- 4x GB300 using TP4 + EP4 instead, reflecting GB300’s higher per-GPU memory and interconnect bandwidth.
- Additional verified recipes for H800, H100, and B200 are maintained in InclusionAI’s SGLang cookbook, along with measured accuracy and speed numbers for each.
Because the model activates only 25B of its 560B parameters per forward pass, inference compute per token is far lower than a dense 560B model would require, but the full parameter set still needs to be loaded into GPU memory across the cluster, which is why multi-GPU, high-memory configurations (H200’s 141GB per GPU, for instance) are the baseline rather than an edge case.
How do you deploy Ling-3.1-flash with SGLang?
SGLang is InclusionAI’s recommended serving engine for this model, and it ships a day-0 Docker image with built-in support rather than requiring a manual source build. The deployment flow looks like this:
1. Pull the image
docker pull lmsysorg/sglang:dev-ling-3.1-flash2. Launch the server (example for 8x H200):
docker run --gpus all --network host --ipc=host --rm \
lmsysorg/sglang:dev-ling-3.1-flash \
python3 -m sglang.launch_server \
--model-path inclusionAI/Ling-3.1-flash \
--tp 8 --ep 8 \
--speculative-algorithm NEXTN \
--host 0.0.0.0 --port 30000The --tp 8 --ep 8 flags split the model across eight GPUs using both tensor parallelism and expert parallelism, which matters for MoE models since routed experts can be distributed independently from the dense layers. The --speculative-algorithm NEXTN flag turns on MTP (multi-token prediction) speculative decoding, which generates draft tokens ahead of the main forward pass to speed up decoding.
For a 4x GB300 setup, the only change is swapping to --tp 4 --ep 4.
3. Optional Hopper optimizations
Two environment variables enable roughly 10% faster batch-1 decode on Hopper-generation GPUs (H100/H200):
SGLANG_ENABLE_FP8_BLOCK_GEMV=1
SGLANG_ENABLE_FUSED_KDA_OUTPUT_QUANT=14. Query the server
Once running, the server exposes an OpenAI-compatible chat completions endpoint:
curl -s http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "Ling-3.1-flash",
"messages": [{"role": "user", "content": "What is the capital of France?"}],
"chat_template_kwargs": {"enable_thinking": true},
"stream": true,
"temperature": 1.0,
"top_k": 20,
"top_p": 0.95}'The recommended sampling parameters (temperature 1.0, top_k 20, top_p 0.95) and the enable_thinking flag in chat_template_kwargs control whether the model exposes its reasoning trace, which is relevant since Ling-3.1-flash is built as a hybrid reasoning model.
For hardware beyond the two examples above, InclusionAI maintains a cookbook with additional verified recipes for H800, H100, and B200, along with guidance on HiCache (Mooncake L3) caching, toggling reasoning on or off, and tool-calling setups.
How do you deploy Ling-3.1-flash with vLLM?
vLLM support currently lives on a dedicated branch rather than mainline, so deployment involves building from source:
1. Set up the environment
pip install uv
uv venv ~/my_ling_env
source ~/my_ling_env/bin/activate
git clone -b vllm https://github.com/vllm-project/vllm.git
cd vllm
VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=autoBuilt like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
Using uv for environment and package management speeds up the install considerably compared to plain pip, and VLLM_USE_PRECOMPILED=1 pulls precompiled kernels instead of building everything locally.
2. Launch the server
vllm serve "$MODEL_PATH" \
--port "$PORT" \
--trust-remote-code \
--served-model-name auto \
--tensor-parallel-size 8 \
--gpu-memory-utilization 0.85 \
--enable-prefix-caching \
--enable-expert-parallel \
--mamba-cache-mode align \
--enable-auto-tool-choice \
--tool-call-parser ling3 \
--reasoning-parser ling3 \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'This single command does a lot: --tensor-parallel-size 8 splits the model across 8 GPUs, --enable-expert-parallel handles the MoE routing, --enable-prefix-caching reuses KV cache across requests with shared prefixes, and the --tool-call-parser ling3 and --reasoning-parser ling3 flags wire up Ling-specific parsing for function calling and reasoning traces. The --speculative-config flag enables the same MTP-style speculative decoding available in SGLang, configured here for 3 speculative tokens.
3. Query the server
The client call mirrors the SGLang example, using the same recommended sampling settings and enable_thinking flag.
Is SGLang or vLLM the better choice for this model?
InclusionAI explicitly recommends SGLang first, and the quickstart documentation reflects that: the SGLang path has a ready-made Docker image with day-0 model support, built-in MTP speculative decoding, and Hopper-specific kernel optimizations out of the box. It also comes with a more extensive cookbook covering five GPU types and caching strategies.
vLLM support exists and is documented, but it requires building from a feature branch rather than installing a stable release, which adds setup friction and some risk of the branch diverging from vLLM’s mainline over time. For teams already standardized on vLLM for other models, it’s a workable path, but teams starting fresh with Ling-3.1-flash will likely find the SGLang route faster to get running and more actively supported for this specific model.
What is Ling-3.1-flash actually good at?
InclusionAI positions Ling-3.1-flash as an upgrade over Ling-3.0-flash aimed at real-world agentic and professional tasks rather than pure chat benchmarks. The model card cites strong results on coding and agentic benchmarks including Terminal-Bench 4.0, SWE-bench Pro, FrontierSWE, GDPVal-AA V2.1, Automationbench, MCP-Atlas, and SkillsBench. It’s also reported to work well inside existing agent frameworks such as Claude Code, Kilo Code, Qwen Code, Hermes Agent, and OpenClaw. Beyond coding, InclusionAI reports gains on HealthBench Professional and in specialized domains like finance and materials science.
The company also built LingCode, an internal coding-agent harness layered on top of the open-source Pi agent, designed to handle long-running coding sessions involving hundreds of tool calls and repeated context compaction. InclusionAI says it is considering open-sourcing LingCode, but as of this release it remains an internal evaluation tool rather than a public artifact.
Frequently Asked Questions
How many GPUs does Ling-3.1-flash need to run?
InclusionAI’s documented configurations use 8x H200 GPUs (TP8 + EP8) or 4x GB300 GPUs (TP4 + EP4) for SGLang, with additional verified setups for H800, H100, and B200 in their cookbook. vLLM’s example configuration also uses 8-way tensor parallelism.
Is Ling-3.1-flash open source?
Yes, it’s released as an open-weight model by InclusionAI, available on Hugging Face, ModelScope, and through OpenRouter for hosted inference without self-deployment.
What’s the difference between Ling-3.1-flash and Ling-3.0-flash?
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
Ling-3.1-flash builds on Ling-3.0-flash’s architecture with improvements aimed at real-world tasks like agentic workflows, search, office productivity, and software engineering, based on the model’s own release notes.
Does Ling-3.1-flash support tool calling and reasoning traces?
Yes. Both the SGLang and vLLM deployment examples include a chat_template_kwargs flag called enable_thinking to expose reasoning, and vLLM’s server command includes dedicated ling3 parsers for tool calls and reasoning output.
Can I run Ling-3.1-flash without a Docker setup?
The SGLang quickstart is built around a prebuilt Docker image, which is the simplest path. vLLM requires a source build in a Python virtual environment instead, which avoids Docker but adds manual build steps.
