How to Run Recursive Flint Locally with SGLang
A practical guide to self-hosting Recursive Flint's 40B MoE coding model with SGLang, including hardware needs and setup steps.
What is Recursive Flint?
Flint is a mixture-of-experts language model built by Recursive for coding agents. It has 39.8 billion total parameters but activates only about 2.5 billion per token, which keeps inference cheaper than a dense model of similar scale would be. It’s meant for the kind of work coding agents actually do: reading unfamiliar codebases, fixing bugs, implementing features, and writing tests. It handles text input and output, supports multi-turn reasoning, and can make tool calls in an OpenAI-compatible format. Recursive reports 54.9% on Terminal-Bench 2.1, 89.7% on SWE-bench Pro v2, and 74.2% on SWE-bench Verified. The weights are released under Apache 2.0, and Recursive ships an SGLang-based serving kit alongside the checkpoint so people can run it on their own hardware instead of calling a hosted API.
TL;DR
- Flint is a 40B-class MoE model with roughly 2.5B active parameters per token, making it a coding-focused agent model rather than a general chatbot.
- It needs a single H200 or B300 GPU per replica to run the BF16 checkpoint through the official SGLang kit; no multi-GPU setup is required for one replica.
- The download is about 80GB, pulled directly from Hugging Face with the
huggingface_hubCLI before you build or run anything. - Serving is Docker-based: you build an image from the provided
sglang/directory and launch it with a hardware-specific serve script (serve_h200.shorserve_b300.sh). - The API is OpenAI-compatible, so existing tooling built against the OpenAI SDK or client libraries can point at the local server with minimal changes.
- Recommended sampling settings are temperature 1.0, top-p 0.95, top-k 20, and an output budget of at least 32,768 tokens for hard problems since reasoning counts against that budget.
- Quantized variants exist separately for NVIDIA and Apple Silicon, but the repository covered here distributes the BF16 checkpoint and Transformers support isn’t included yet.
What hardware do you need to run Flint locally?
Recursive’s documentation specifies that Flint runs on a single H200 or B300 GPU per replica using the BF16 checkpoint and the official SGLang kit. That’s a meaningful detail: despite having nearly 40 billion total parameters, the MoE architecture (256 routed experts, 6 selected per token, plus one always-active shared expert) means you don’t need to fit the entire model’s compute cost into active memory the way you would with a dense 40B model. Still, all the weights have to live somewhere, and the download itself is around 80GB, so you need storage and VRAM headroom to match.
H200 and B300 are both data-center-class GPUs with large high-bandwidth memory pools, not consumer cards. If you don’t have access to one of these, the BF16 path described in the main repository isn’t realistic for you right now. Recursive does prepare quantized variants separately for NVIDIA and Apple Silicon runtimes, which is the likely path for anyone trying to run Flint on smaller or consumer-grade hardware, though those come with their own separate setup and aren’t part of the SGLang kit discussed here.
How do you download and serve Flint with SGLang?
The setup follows a fairly standard pattern for self-hosted open-weight models, with a few specifics worth calling out.
First, install the required Python packages and pull the model from Hugging Face:
pip install -U huggingface_hub openai
hf download RecursiveSI/flint --local-dir flint && cd flint # about 80 GBNext, build the Docker image using the sglang/ directory bundled in the repository, then run it against your GPU:
docker build -t flint-sglang sglang
docker run -d --name flint --gpus '"device=0"' --ipc=host -p 127.0.0.1:30000:30000 \
-v "$PWD:/model" -e MODEL=/model flint-sglang bash /model/sglang/serve_b300.sh # H200: serve_h200.shNote the two serve scripts: serve_b300.sh and serve_h200.sh. You pick the one matching your GPU. The container exposes an API on port 30000, bound to localhost by default.
Because loading an 80GB checkpoint onto a GPU takes time, you’ll want to poll the health endpoint before sending real traffic:
until curl -sf localhost:30000/health_generate >/dev/null; do sleep 15; done # progress: docker logs -f flintOnce the server reports healthy, you can run the included example client to confirm reasoning and tool-calling work end to end:
python3 sglang/example_client.py --base-url http://localhost:30000/v1From there, Flint behaves like any other OpenAI-compatible endpoint. A minimal Python call looks like this:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:30000/v1", api_key="unused")
response = client.chat.completions.create(
model="flint",
messages=[{"role": "user", "content": "Write a Python function that merges overlapping intervals."}],
max_tokens=32768,
)
print(response.choices[0].message.content)The sglang/README.md file in the repository covers additional detail on requirements, serve-script options, multi-GPU deployments, and the full API surface, which is worth reading before putting this into any production-adjacent workflow.
What generation settings should you use?
Other agents start typing. Remy starts asking.
Scoping, trade-offs, edge cases — the real work. Before a line of code.
Recursive recommends temperature 1.0 with top-p 0.95 and top-k 20, the same settings used in their own benchmark evaluations. For difficult problems, they suggest reserving at least 32,768 tokens of output budget, since Flint’s reasoning tokens count against the same budget as the final answer. If a request to the server omits sampling parameters, the server applies these defaults automatically, so you don’t strictly need to set them yourself, but it’s worth knowing what’s happening under the hood if outputs seem off.
One detail that matters for multi-turn use: Flint returns a reasoning_content field separately from the final response, and Recursive’s guidance is to preserve that content in conversation history across turns. Dropping it is likely to degrade performance on tasks that build on prior reasoning steps.
Is self-hosting Flint worth it compared to a hosted API?
The honest answer depends on your hardware access and workload volume. Self-hosting removes per-token API costs and gives you full control over data handling, which matters for teams working with sensitive codebases. But the hardware requirement (a single H200 or B300 GPU) is not cheap or easy to access for most individual developers. These are GPUs typically found in cloud data-center instances or dedicated AI infrastructure, not local workstations.
If you already have access to this class of hardware, for example through a cloud GPU rental or existing infrastructure, running Flint locally via SGLang gives you a coding-agent-grade model with a documented 262,144-token context window and no dependency on an external API’s uptime or rate limits. If you don’t have that hardware, waiting for or seeking out the quantized NVIDIA or Apple Silicon variants is the more realistic path, though those aren’t covered by the SGLang kit detailed in Flint’s main repository.
What are the known limitations?
Recursive is explicit that Flint’s supervised fine-tuning included safety-oriented examples, but the model can still generate incorrect code, unsafe tool calls, or harmful content. The structured format of tool calls doesn’t prevent the model from choosing a wrong or unsafe action, so any tool-using deployment needs its own permission boundaries and execution limits independent of the model itself. Long reasoning chains can also eat into the output budget before the model finishes an answer, which is part of why the recommended 32,768-token minimum exists for harder problems. Recursive also notes that results can shift depending on checkpoint precision and the inference runtime used, so the BF16 SGLang path and any future quantized variant should be evaluated separately rather than assumed equivalent.
Frequently Asked Questions
What GPU do I need to run Flint locally?
Recursive’s documentation specifies a single H200 or B300 GPU per replica for the BF16 checkpoint through the SGLang kit. These are data-center GPUs, not consumer hardware.
How big is the Flint download?
The full checkpoint, including configuration, tokenizer, and chat template, is approximately 80GB when pulled from Hugging Face.
Does Flint support tool calling?
Yes. Flint supports OpenAI-compatible tool calls and is designed for agentic coding tasks that involve multi-turn tool use alongside reasoning.
Can I run Flint on a Mac or consumer NVIDIA card?
Not through the BF16 SGLang kit described here. Recursive prepares separate quantized variants for NVIDIA and Apple Silicon runtimes, though setup details for those differ from the H200/B300 path.
What context window does Flint support?
The documented context configuration is 262,144 tokens, which covers both the prompt and the generated output combined.
