Recursive Flint Benchmarks: SWE-bench and Terminal-Bench Scores Explained
Flint scores 74.2% on SWE-bench Verified and 89.7% on SWE-bench Pro v2. Here's what those coding agent benchmarks measure and why.
What are Flint’s benchmark scores?
Recursive Flint, an open-weight mixture-of-experts model built for coding agents, scores 74.2% on SWE-bench Verified, 89.7% on SWE-bench Pro v2, and 54.9% on Terminal-Bench 2.1. These numbers come from Recursive’s own release evaluations, run with mini-swe-agent for the SWE-bench suites and Terminus 2 for Terminal-Bench, using temperature 1.0, top-p 0.95, and top-k 20, with scores averaged over at least five attempts (pass@1).
TL;DR
- Flint is a 39.8 billion parameter mixture-of-experts model with roughly 2.5 billion active parameters per token, built specifically for coding agent workloads like bug fixing, feature implementation, and test writing.
- Its SWE-bench Verified score of 74.2% places it among models capable of resolving a majority of real-world GitHub issue resolution tasks drawn from human-filtered problems.
- The SWE-bench Pro v2 score of 89.7% is notably higher than its Verified score, which reflects differences in task composition and difficulty between the two benchmark versions rather than a contradiction.
- On Terminal-Bench 2.1, Flint scores 54.9%, and Recursive explicitly counts any solution reached through web retrieval as a failure, a stricter standard than some developer-reported comparison scores apply.
- Flint runs on a single B300 or H200 GPU per replica using an SGLang serving kit, with the BF16 checkpoint weighing in around 80 GB.
- The model supports a 262,144 token context window shared between prompt and output, and recommends an output budget of at least 32,768 tokens for hard problems.
- Flint ships under the Apache 2.0 license, making both the weights and commercial use unrestricted compared to closed, API-only coding models.
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
How does SWE-bench measure coding ability?
SWE-bench is a benchmark that tests whether a model can resolve real software engineering issues pulled from actual GitHub repositories. A model is given an issue description and a codebase snapshot, and it has to produce a patch that makes the associated test suite pass. This is a meaningfully different task from generating a short code snippet from a prompt: it requires understanding an unfamiliar codebase, locating the relevant files, and making changes that don’t break anything else.
SWE-bench Verified is a human-filtered subset of the original SWE-bench dataset, curated to remove ambiguous or unsolvable tasks. It has become one of the standard references for coding agents because it strips out noise that made early SWE-bench scores hard to compare across models. Flint’s 74.2% on this benchmark means it successfully resolved close to three out of four issues in that curated set, using the mini-swe-agent harness to drive tool calls and file edits.
SWE-bench Pro v2 is a separate, newer iteration of the benchmark. The “Pro” naming and version number matter here: different versions of SWE-bench can draw from different repositories, impose different task constraints, or weight difficulty differently, so a score of 89.7% on Pro v2 is not directly comparable to the 74.2% on Verified. They’re measuring related but distinct things, and Recursive’s model card presents them side by side without claiming they’re interchangeable.
What does the Terminal-Bench 2.1 score tell you?
Terminal-Bench evaluates how well a model operates inside a terminal environment, using tools, running commands, and completing multi-step tasks that require shell interaction rather than just writing code in isolation. This is closer to how a coding agent actually works in practice: it needs to navigate a filesystem, run tests, interpret error output, and decide what to do next.
Flint’s 54.9% score on Terminal-Bench 2.1 was produced using Terminus 2, the harness version tied to that benchmark release. One detail worth flagging: Recursive’s evaluation methodology treats any solution a model reaches via web retrieval as unsuccessful, even if the end result is correct. That’s a stricter bar than some other developer-reported scores use, since not every lab applies the same penalty for retrieval-assisted answers. This means direct comparisons between Flint’s Terminal-Bench number and a competing model’s self-reported score should be read with that caveat in mind. The model card itself flags this explicitly rather than letting the number stand without context.
How does Flint’s architecture support these scores?
Flint is a decoder-only mixture-of-experts transformer with 49 layers: two dense layers followed by 47 MoE layers. Each MoE layer has 256 routed experts, of which 6 are selected per token, plus a shared expert that’s always active. This routing design is what keeps the active parameter count low (about 2.5 billion per token, including the output head) relative to the total parameter count (39.8 billion), which is the core efficiency trade-off that mixture-of-experts architectures are built around: a large total capacity for knowledge and specialization, but a much smaller compute cost per generated token.
Remy doesn't write the code. It manages the agents who do.
Remy runs the project. The specialists do the work. You work with the PM, not the implementers.
The attention mechanism mixes 35 sliding-window layers (each looking at the current token plus 512 preceding tokens) with 14 full-attention layers that have no positional encoding. This hybrid approach is a common technique for extending context length without paying the full computational cost of global attention at every layer. Flint’s documented context configuration is 262,144 tokens, shared between prompt and generated output, which matters for coding agents that need to hold large codebases, long tool-call histories, and extended reasoning traces in context at once.
The model also uses per-head QK RMSNorm and learned per-head attention sinks, normalization techniques aimed at training stability and inference consistency at long context lengths. None of this is unusual by itself, but it is the kind of architectural detail that tends to correlate with how well a model holds up on agentic, multi-turn tasks like those in Terminal-Bench, where losing track of earlier context can derail a whole task.
Is Flint worth deploying for coding agent workloads?
Whether Flint is a good fit depends heavily on hardware access and what kind of coding tasks you’re automating. On the capability side, a 74.2% SWE-bench Verified score is competitive with models designed for similar workloads, and the Terminal-Bench number comes with an unusually transparent methodology note about retrieval-based cheating, which is a point in favor of trusting the number at face value.
On the deployment side, Flint requires a single B300 or H200 GPU per replica, served through Recursive’s SGLang kit, with the BF16 checkpoint around 80 GB to download. That’s enterprise-grade hardware, not something you run on a laptop. Quantized variants for NVIDIA and Apple Silicon runtimes are prepared separately, which should lower the hardware bar, though the model card notes that Transformers modeling code isn’t included yet, only the SGLang serving path.
The Apache 2.0 license is a meaningful factor for teams that want to self-host, fine-tune, or avoid API dependency on a third-party provider. Recursive also recommends a generous output budget (at least 32,768 tokens for hard problems) because long reasoning traces can otherwise get cut off before the model finishes. Teams evaluating Flint should treat the published benchmark numbers as a starting point and run their own evaluation against their specific checkpoint and inference runtime, since the model card itself notes that results can shift with precision and runtime choices.
Frequently Asked Questions
What is Recursive Flint?
Flint is an open-weight mixture-of-experts language model from Recursive, built specifically for coding agents. It has 39.8 billion total parameters with about 2.5 billion active per token, and it’s designed for tasks like investigating codebases, fixing bugs, implementing features, and writing tests.
What is the difference between SWE-bench Verified and SWE-bench Pro v2?
SWE-bench Verified is a human-filtered subset of the original SWE-bench dataset, used widely as a standard coding agent benchmark. SWE-bench Pro v2 is a separate, newer benchmark version with its own task set and difficulty profile. Flint scores differently on each (74.2% versus 89.7%) because they test different, though related, things, not because one score is more “correct” than the other.
Why is Flint’s Terminal-Bench score lower than its SWE-bench scores?
Terminal-Bench tests different skills: multi-step terminal and tool-use tasks rather than single-issue code patching. Flint’s 54.9% score also comes from an evaluation that counts web-retrieval-based solutions as failures, a stricter standard that may not match how other developers report their own Terminal-Bench numbers.
What hardware does Flint require to run?
Flint runs on a single B300 or H200 GPU per replica using Recursive’s SGLang serving kit. The BF16 checkpoint is about 80 GB to download. Quantized versions for NVIDIA and Apple Silicon hardware are prepared separately for teams without access to that tier of GPU.
Is Flint free to use commercially?
Flint’s weights are released under the Apache 2.0 license, which permits commercial use and modification. Note that hosted API terms and data practices, if you use a hosted version rather than self-hosting, are governed separately from the weight license itself.
