How to Run Agnes-3.0-Flash Preview Locally: Hardware Requirements
Agnes-3.0-Flash Preview needs an H100 or H200 GPU and about 66GB disk space. Here's how to set it up with SGLang or Transformers.
What is Agnes-3.0-Flash Preview?
Agnes-3.0-Flash Preview is an open-weight, 33B-parameter multimodal model released by Agnes AI under Apache 2.0. It’s a hybrid-attention decoder built for reasoning, coding, and instruction-following work, with a 262,144-token context window and support for text, image, and video inputs. It’s distinct from the production Agnes 3.0 Flash model served through the Agnes AI API, which uses a different checkpoint and a 1M-token context window. If you’re comparing benchmark numbers, make sure you know which version you’re looking at, because the two are not interchangeable and the API model’s scores don’t apply to these open weights.
TL;DR
- The Preview checkpoint runs on a single high-end datacenter GPU, specifically one NVIDIA H200 (141GB) or H100 (80GB), rather than requiring a multi-GPU cluster.
- Disk footprint lands around 66GB for the bf16 weights, so plan storage accordingly before downloading.
- The architecture is hybrid, mixing 54 gated delta-rule (recurrent) layers with 18 global attention layers in a 3:1 pattern, which keeps the KV cache smaller than a dense model of similar size.
- Two serving paths are documented: Hugging Face Transformers (5.12+) for quick local inference, and SGLang via a patched Docker image for production-style OpenAI-compatible serving.
- Reasoning effort is adjustable at inference time (
high,medium,low, or off), letting you trade latency for depth of reasoning per request. - Tool calling and vision/video inputs are built into the chat template, so multimodal and agentic workflows don’t require extra glue code beyond
trust_remote_code=True.
What hardware do you actually need?
The model card specifies a single GPU as the baseline: an NVIDIA H200 with 141GB of memory, or an H100 with 80GB, running at bf16 precision. That’s notable for a 33B model, since many models in this size class get served across multiple GPUs once you factor in KV cache overhead for long-context workloads. Here, tensor parallelism is optional rather than mandatory: --tp 1 runs on one GPU, and --tp 2 is recommended only if you want to push toward the model’s full 262K context window with high concurrency.
Host memory recommendations sit at 128GB or more. That’s separate from GPU VRAM and covers the overhead of loading weights, running the tokenizer and processor, and handling batched requests without starving the system.
Disk space is straightforward: budget about 66GB for the bf16 checkpoint. That number will roughly double if you keep a local cache plus a separate download in progress, so plan for headroom on whatever volume you’re pulling the weights onto.
Actual usable context and concurrency depend on how much of your VRAM budget goes to the KV cache versus the model weights themselves. Because only 18 of the model’s 72 layers hold a KV cache that grows with sequence length (the other 54 use fixed-size recurrent state), the memory cost of long context is lower than it would be for a fully dense transformer of the same parameter count. Still, the model card is explicit that you should validate context length and concurrency on your actual target workload rather than assuming the full 262K window fits comfortably in every configuration.
Why does the architecture affect hardware sizing?
Agnes-3.0-Flash Preview uses what the model card calls a hybrid-attention decoder. Out of 72 total decoder layers, 54 run a gated delta rule, a recurrent mechanism where the per-layer state doesn’t grow with sequence length. The remaining 18 layers use standard global attention with grouped-query attention (24 query heads, 4 KV heads, a 6:1 ratio) and are the only layers that accumulate a KV cache as context grows.
This 3:1 ratio (three recurrent layers for every one attention layer) is the architectural reason a 33B model with a 262K context window can plausibly run on one H100 or H200 instead of requiring the multi-GPU setups typically associated with long-context serving. A fully dense attention stack at this context length would push KV cache memory much higher. The delta-rule layers instead carry their recurrent state in fp32, decoupled from sequence length, which caps a major source of memory growth.
Other architectural specifics from the model card: hidden size of 5120, SwiGLU feed-forward layers with an intermediate size of 17408 (plus a smaller parallel SwiGLU branch in every layer), a vocabulary of 248,320 tokens, and a vision tower with 27 layers feeding into the main model via a 2x2 spatial merge. None of these change your hardware plan directly, but they explain why the params-to-VRAM ratio looks the way it does.
How do you set it up with Transformers?
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
The fastest path to a working local instance is Hugging Face Transformers, version 5.12 or newer (tested against 5.12.1). Because Agnes-3.0-Flash Preview ships its own custom model code, every load call needs trust_remote_code=True, both for the model and for the processor if you’re handling images or video.
A minimal setup looks like this:
pip install "transformers>=5.12" torch torchvision acceleratefrom transformers import AutoModelForCausalLM, AutoTokenizer
path = "Agnes-AI/Agnes-3.0-Flash"
tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(
path, dtype="bfloat16", device_map="auto", trust_remote_code=True
)device_map="auto" handles placement on your GPU. For image or video inputs, load AutoProcessor the same way with trust_remote_code=True, since the bundled processor depends on torchvision.
The chat template also exposes reasoning effort as a parameter. Passing reasoning_effort="medium" (or "low", or enable_thinking=False) at generation time lets you dial down latency for simpler queries without switching models.
How do you serve it with SGLang?
For anything beyond single-request local testing, the model card documents an SGLang path using a patched Docker image. The serve.sh script overlays three files onto the stock sglang package inside a public nightly image, nothing more invasive than that:
docker run --gpus all --shm-size 64g -p 30001:8080 \
-v /path/to/agnes-3.0-flash:/model \
lmsysorg/sglang:nightly-dev-20260908-20ca564b \
bash /agnes-3.0-flash/serve.sh --served-model-name Agnes-3.0-FlashExtra arguments after serve.sh get forwarded straight to sglang, so --tp 2 works the same way --served-model-name does. The container listens on port 8080 internally, mapped to 30001 in the example above.
Once running, it exposes an OpenAI-compatible endpoint:
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:30001/v1")
response = client.chat.completions.create(
model="Agnes-3.0-Flash",
messages=[{"role": "user", "content": "Design a fault-tolerant event processing architecture."}],
temperature=1.0,
max_tokens=2000,
)Streaming, tool definitions, and reasoning_effort all pass through the same API surface. By default the server returns tool calls as raw text in the format <tool_call><function=...><parameter=...>; getting structured tool_calls objects back requires configuring sglang with a tool-call parser that matches this format, and optionally a reasoning parser to split out the thinking span into reasoning_content.
Is it worth running locally versus using the API?
That depends on what you need. The open-weight Preview checkpoint gives you a 262K context window, full control over deployment, and no dependency on an external API, at the cost of needing an H100 or H200-class GPU and managing your own serving stack. The production API model has a much larger 1M-token context window and is presumably easier to integrate without hardware investment, but it’s a different checkpoint with different behavior, and its benchmark numbers don’t carry over to the Preview weights.
If your use case needs tool calling, multimodal input, and adjustable reasoning effort in a self-hosted environment, the Preview checkpoint’s hardware bar (one GPU, one Docker command) is low enough to be practical for teams that already have H100 or H200 access, which is increasingly common at cloud and colo providers.
Frequently Asked Questions
What GPU do I need to run Agnes-3.0-Flash Preview?
The model card recommends one NVIDIA H200 (141GB) or H100 (80GB) GPU at bf16 precision. Tensor parallelism with --tp 2 is optional and mainly useful if you want maximum context length and concurrency.
How much disk space does Agnes-3.0-Flash Preview need?
About 66GB for the bf16 checkpoint. Leave extra headroom on the download volume for temporary files during the download and load process.
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
Is Agnes-3.0-Flash Preview the same as the model on Artificial Analysis?
No. The Artificial Analysis listing covers the production/API version of Agnes 3.0 Flash, which uses a different checkpoint and a 1M-token context window. The open-weight Preview checkpoint covered here has 33B parameters and a 262,144-token context window, and its benchmark results are separate from the API model’s.
Can I run this model without a GPU?
The documented hardware requirements assume a single H100 or H200-class GPU at bf16. The model card doesn’t provide CPU-only or lower-VRAM quantized configurations, so running without a capable GPU isn’t part of the documented setup.
Does it support tool calling and image input out of the box?
Yes. The chat template renders tool definitions and parses model-issued tool calls in a <tool_call> format, and the bundled processor (loaded with trust_remote_code=True) handles image and video inputs alongside text.
