How to Run dots3-note Locally on vLLM or SGLang
A hardware and setup guide to deploying dots3-note-prev-fp8 locally with vLLM or SGLang, covering FP8 quantization and 8-GPU serving.
What is dots3-note and what does it take to run it locally?
dots3-note preview is an open-weight multimodal Mixture-of-Experts (MoE) model released by dots studio, the first entry in the dots3 family. It has 280 billion total parameters with only 16 billion activated per token, supports context lengths up to 512K tokens, and takes text, image, video, and audio as input, producing text output. Running it locally in FP8 requires a single 8-GPU node (H100-class hardware is the documented target) served through vLLM or SGLang. The BF16 checkpoint needs even more memory, so FP8 is the practical default for anyone deploying outside a research cluster.
TL;DR
- dots3-note preview is a 280B-parameter MoE model with 16B activated parameters per token, built for text, image, video, and audio understanding with text-only output.
- FP8 quantization (dots3-note-prev-fp8) is the recommended checkpoint for local deployment and is explicitly designed to run on one 8-GPU node.
- vLLM and SGLang both offer native or near-native support, with SGLang also shipping a ready-made Docker image (
lmsysorg/sglang:dev-dots3-note). - Speculative decoding via MTP/NEXTN can cut time-per-output-token (TPOT) by more than 50% in SGLang, and vLLM supports a similar three-token MTP config.
- Context length is tunable: the model supports up to 512K tokens, but real deployments should scale this down based on available VRAM, concurrency needs, and which modalities are in use.
- Transformers support exists for single-node, non-distributed testing, but production-style multimodal serving is meant to go through vLLM or SGLang.
- Licensing is permissive: weights and modeling code are released under Apache 2.0, though the serving frameworks themselves carry their own separate licenses.
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
What are the hardware requirements for running dots3-note?
The model card is explicit that the FP8 checkpoint is meant for a single node with 8 GPUs, and the vLLM example specifically targets 8x NVIDIA H100 GPUs using tensor parallelism (TP=8) and expert parallelism (EP=8). This lines up with the model’s architecture: 280B total parameters spread across 256 routed experts plus one shared expert (top-8 routing), a vision encoder that adds another 7B parameters (1.2B activated), and an 800M-parameter dense audio encoder. Even with only 16B parameters activated per forward pass, the full MoE weight set has to sit in GPU memory across the cluster, which is why 8-way parallelism is the documented baseline rather than an option.
BF16 serving is supported but requires substantially more memory than FP8, and the model card does not offer a smaller-hardware BF16 path. For anyone without access to 8 high-memory GPUs, FP8 is not just recommended, it’s effectively the only realistic route to running this model outside of a research lab.
Context length is another lever. The architecture supports up to 512K tokens, but the deployment examples default to smaller windows (524288 tokens in the SGLang docker command, 262144 in the vLLM example). Memory usage scales with both context length and concurrency, so production setups should size the context window to the actual workload rather than defaulting to the maximum.
How do you deploy dots3-note with SGLang?
SGLang is one of the two primary serving paths, and dots studio ships a dedicated Docker image, lmsysorg/sglang:dev-dots3-note, that pulls the checkpoint from Hugging Face on first run. A one-node launch looks like this:
docker run --gpus all --ipc=host -p 8000:8000 \
lmsysorg/sglang:dev-dots3-note \
sglang serve \
--model-path dots-studio/dots3-note-prev-fp8 \
--served-model-name dots3-note-prev \
--host 0.0.0.0 \
--port 8000 \
--context-length 524288 \
--enable-dp-attention \
--dp-size 8 \
--tp-size 8 \
--ep-size 8 \
--moe-dense-tp-size 1 \
--page-size 64 \
--trust-remote-code \
--attention-backend fa3 \
--moe-a2a-backend deepep \
--enable-multimodal \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--speculative-draft-model-path dots-studio/dots3-note-prev-fp8A few details matter here. The fa3 attention backend handles prefill, decode, and draft attention when speculative decoding is active. The --speculative-algorithm NEXTN flag enables MTP-based speculative decoding, which the model card states can reduce TPOT (time per output token) by more than 50%, a meaningful latency win given the model’s size. Prefill CUDA graph support is not yet available, which is worth knowing if you’re benchmarking throughput against other models that do support it.
Native SGLang support is tracked under a pull request (SGLang #33829) that was still under review at the time of the model’s release, so until it merges, the documented path is either the dev Docker image or installing from the PR branch directly. Optional flags let you strip the deployment down to language-only mode (--language-only) or enable OpenAI-style tool calling (--tool-call-parser dots).
How do you deploy dots3-note with vLLM?
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
vLLM has native support for dots3-note-prev on its main branch, so a recent nightly build is the recommended path until it lands in a stable release. The documented example serves the FP8 checkpoint across 8 H100 GPUs:
vllm serve dots-studio/dots3-note-prev-fp8 \
--served-model-name dots3-note-prev \
--host 0.0.0.0 \
--tensor-parallel-size 8 \
--enable-expert-parallel \
--moe-backend deep_gemm \
--max-model-len 262144This mirrors the SGLang setup in spirit: tensor parallelism across all 8 GPUs, expert parallelism enabled for the MoE layers, and a context length set below the theoretical 512K max to keep memory usage manageable. The deep_gemm MoE backend is specified explicitly, which matters for throughput on MoE architectures like this one.
vLLM also supports speculative decoding through a JSON config flag:
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'and tool calling via --enable-auto-tool-choice --tool-call-parser dots. As with SGLang, there’s a --language-model-only flag for text-only workloads that skips loading the vision and audio encoders entirely, useful if you don’t need multimodal input and want to save memory or startup time.
Is Transformers a viable option for local deployment?
Transformers support exists (tracked under Transformers PR #47844) and works for straightforward single-GPU or single-node testing, but it’s not positioned as the production serving path. The setup requires installing a PyTorch and torchvision pairing compatible with your NVIDIA driver, plus torchcodec and FFmpeg if you want audio and video input support. The install command pulls Transformers directly from the PR branch:
pip install accelerate pillow torchcodec kernels==0.16.0 "transformers @ git+https://github.com/huggingface/transformers.git@refs/pull/47844/head"From there, a minimal inference script uses AutoModelForMultimodalLM and AutoProcessor with device_map="auto" to load the FP8 checkpoint. This is useful for quick local testing or debugging model behavior, but the model card is direct about the fact that multi-GPU OpenAI-compatible serving should go through vLLM or SGLang, not raw Transformers.
Is running dots3-note locally worth it?
For teams that already operate multi-GPU inference infrastructure, dots3-note offers a genuinely capable multimodal MoE model under an Apache 2.0 license, with real speculative decoding support and documented serving recipes for two major inference engines. That’s a meaningful amount of engineering already done for you. The catch is hardware: this is not a model you run on a single consumer GPU or even a single high-end workstation card. The 8-GPU requirement for the FP8 checkpoint puts it in the same deployment tier as other large open MoE models, meaning it’s realistic for companies with cloud GPU budgets or on-prem clusters, and out of reach for most individual developers experimenting locally.
If the hardware is available, the setup itself is well documented: known flags, a ready Docker image for SGLang, and a native vLLM recipe. The tradeoffs to weigh are mostly about context length and modality support versus memory, both of which are tunable rather than fixed.
Frequently Asked Questions
What GPUs do I need to run dots3-note-prev-fp8?
The documented configuration uses one node with 8 GPUs, with the vLLM example specifically using 8x NVIDIA H100 GPUs and tensor parallelism plus expert parallelism set to 8.
Does dots3-note support audio and video, or just text and images?
It supports all four: text, image, video, and audio as input, with text as the only output format. Video inputs also carry their audio track when available.
Remy doesn't write the code. It manages the agents who do.
Remy runs the project. The specialists do the work. You work with the PM, not the implementers.
What’s the difference between dots3-note-prev and dots3-note-prev-fp8?
dots3-note-prev is the full-precision (BF16-capable) checkpoint, while dots3-note-prev-fp8 is the FP8-quantized version. FP8 is the recommended deployment format because it fits within a single 8-GPU node, while BF16 requires more memory.
Can speculative decoding actually improve performance for this model?
Yes. The model card states that enabling MTP/NEXTN speculative decoding in SGLang can reduce time-per-output-token by more than 50%. Both SGLang and vLLM expose flags to turn this on.
Do I need the full 512K context window in production?
Not necessarily. The architecture supports up to 512K tokens, but the deployment examples default to smaller windows, and the model card recommends tuning context length based on available memory, concurrency, and which modalities you’re using.
