dots3-note Preview: Inside the 280B Multimodal MoE Model
dots3-note preview is a 280B-parameter, 16B-active multimodal MoE model with 512K context. Here's what it is and how it works.
What is dots3-note preview?
dots3-note preview is an open-weight multimodal model released by dots studio, the first entry in what the company calls the dots3 family. It’s a Mixture-of-Experts (MoE) model with 280 billion total parameters, but only 16 billion of those are active on any given forward pass. It handles text, images, video, and audio as input and generates text as output, with a context window stretching to 512K tokens. The model card describes it as the lightweight member of the dots3 lineup, meant to trade some raw capability for lower latency and inference cost.
TL;DR
- dots3-note preview is a 280B-parameter MoE model with only 16B parameters active per token, keeping inference costs closer to a much smaller dense model.
- It accepts four input modalities (text, image, video, audio) and outputs text, with video inputs also carrying their audio track when present.
- The 512K context window combined with a 152K vocabulary positions it for long-document, long-video, and extended agent-style workloads.
- Architecture details show a hybrid attention design (13 DSA layers plus 33 sliding-window attention layers) and 256 routed experts plus one shared expert, top-8 routing.
- A separate MoE vision encoder (7B total, 1.2B activated) and a dense 800M audio encoder handle non-text modalities before feeding into the main language model.
- The model ships in both BF16 and FP8, with FP8 recommended for practical deployment on an 8-GPU node.
- It’s licensed under Apache 2.0 and already has day-one or near-day-one support paths through Transformers, SGLang, and vLLM.
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
How big is dots3-note preview and why does “active parameters” matter?
The headline number is 280B total parameters, but the number that actually determines compute cost per token is 16B activated parameters. This is the standard MoE trade-off: the model stores a huge pool of specialized “expert” sub-networks (256 routed experts plus one shared expert in this case), but a router selects only the top 8 experts to process each token. That means dots3-note preview computes roughly like a 16B dense model while having the storage and specialization capacity of a much larger one.
The architecture also includes a Multi-Token Prediction (MTP) component, a single shared layer with 1.13B parameters, which is used for speculative decoding to speed up generation. The core language model has 1 dense layer plus 45 MoE layers, a hidden size of 5120, and an FFN hidden size of 13824 for the dense layer (1536 per expert). Vocabulary size sits at 152K tokens.
What does the hybrid attention setup actually do?
dots3-note preview mixes two attention mechanisms across its 45 layers: 13 layers use DSA (a sparse attention pattern that only attends to the top 2048 most relevant tokens) and 33 layers use sliding window attention (SWA), roughly a 1:3 ratio. This combination is a common way for long-context models to keep compute manageable at 512K tokens without abandoning the ability to reference distant context when it matters. Full dense attention over 512K tokens is expensive; sparse and windowed attention patterns cut that cost while preserving most of the useful signal, particularly for tasks like long-document QA or extended agent trajectories where relevant information is scattered but not uniformly distributed.
How does dots3-note preview handle images, video, and audio?
Multimodal input is handled by two dedicated encoders that sit in front of the main language model. Images and video frames go through a MoE Vision Transformer (ViT) with 7B total parameters and 1.2B activated, following the same sparse-activation logic as the main model. Audio goes through a separate dense encoder with 800M parameters. Notably, video inputs carry their audio track along with the visual frames when available, so a single video clip can be understood as a combined audio-visual signal rather than silent frames.
On the input side, the model card lists four supported modalities: text, image, video, and audio. Output is text only, there’s no image or audio generation. This makes dots3-note preview a multimodal understanding model rather than a generation model, positioned for tasks like transcribing audio, describing video content, reading charts and documents, and answering questions grounded in visual or auditory input.
What tasks is dots3-note preview built for?
Other agents ship a demo. Remy ships an app.
Real backend. Real database. Real auth. Real plumbing. Remy has it all.
The model card lists a fairly broad task set: general knowledge and instruction following, mathematical and logical reasoning, tool use and multi-step agent workflows, interactive tasks requiring memory updates and adaptation, code generation, and understanding of images, documents, charts, audio, and video, on top of long-context processing. That combination (agentic tool use plus multimodal understanding plus long context) suggests the model is aimed at agent-style applications that need to read documents, watch video, listen to audio, and act on that information across multi-step interactions, not just answer one-shot questions.
The model also supports a “thinking” mode via an enable_thinking flag in its chat template, letting a deployer choose between an extended reasoning trace or a direct response depending on latency needs.
Is dots3-note preview practical to deploy?
Running the full BF16 checkpoint requires substantial GPU memory given the 280B total parameter count, even though only 16B are active. dots studio recommends the FP8-quantized checkpoint (dots3-note-prev-fp8) as the practical deployment path, targeting a single 8-GPU node. Serving recipes are provided for three frameworks:
- vLLM, with native support on the
mainbranch, using tensor parallel size 8 and expert parallel enabled. - SGLang, via a dedicated Docker image (
lmsysorg/sglang:dev-dots3-note) or a pending pull request, supporting the full 524288-token context length with data, tensor, and expert parallelism all set to 8. - Transformers, for simpler single-process local inference, via a pull request that adds
AutoModelForMultimodalLMsupport.
Both vLLM and SGLang paths support speculative decoding using the model’s MTP layer, which the documentation says can cut time-per-output-token by more than 50% when enabled. Both also support OpenAI-compatible tool calling through a dedicated “dots” tool-call parser, which matters for anyone building agent pipelines that expect standard function-calling behavior.
How does dots3-note preview compare to other open multimodal models?
The model card includes benchmark charts covering general reasoning and agent tasks, plus multimodal understanding, though exact competitor names and scores live in the linked benchmark images rather than as inline text. What is clear from the specs alone is the positioning: dots3-note preview is explicitly framed as the smallest, fastest model in a planned family of dots3 models, implying that larger, more capable (and more expensive to run) variants are expected to follow. For teams evaluating it today, the practical comparison points are the 16B active parameter footprint (competitive with mid-size dense models on inference cost) against the 512K context window and four-modality input support, which few open models combine at this parameter scale.
The model is released under Apache 2.0, a permissive license that allows commercial use and modification, and it’s available for free trial through OpenRouter as well as direct download from Hugging Face and ModelScope.
Frequently Asked Questions
What does “16B activated” mean for a 280B parameter model?
It means that although the model stores 280 billion parameters across all its experts, only about 16 billion are used to process any single token, thanks to Mixture-of-Experts routing. This keeps inference speed and memory bandwidth closer to a 16B dense model while retaining a much larger overall capacity.
Can dots3-note preview generate images or audio?
No. It accepts text, image, video, and audio as input but only produces text as output. It’s designed for multimodal understanding, not generation.
What context length does dots3-note preview support?
Up to 512K tokens (524288 tokens exactly, as referenced in the SGLang deployment example), though actual usable context in deployment depends on available GPU memory and concurrency settings.
What hardware is needed to run it?
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
dots studio recommends the FP8 checkpoint on a single node with 8 GPUs, using tensor and expert parallelism. The BF16 checkpoint requires more memory and hasn’t been given a specific minimum hardware target in the documentation.
Is dots3-note preview free to use?
The weights are released under the Apache 2.0 license, allowing free download, modification, and commercial use. There’s also a free hosted version available through OpenRouter for testing without local deployment.
