Skip to main content
MindStudio
Pricing
BlogAbout
My Workspace
Inkling 2 Mini benchmarksSWEBench InklingGemma 4 vs Inkling

Inkling 2 Mini Benchmarks: How It Stacks Up Against Rivals

Inkling 2 Mini's SWEBench, Terminal Bench and FrontierCode scores compared against Gemma 4 31B, Qwen3.8-27B and Nemotron 3.5.

Edited by Luis Chavez-Mattos, Director of Product RSS
Inkling 2 Mini Benchmarks: How It Stacks Up Against Rivals

What is Inkling 2 Mini and how does it benchmark against similar models?

Inkling 2 Mini is an open-weight multimodal model from Thinking Machines Lab, released November 20, 2026 under Apache 2.0, with 26.4B total parameters and 3.4B active per forward pass (5.4B effective active parameters once its looped middle layers are counted). On published evaluations, it leads or ties for the top score on nearly every coding and agentic benchmark in its comparison set, including SWEBench Verified (79.2%), SWEBench Pro v2 Hard (77.9%) and DeepSWE v1.1 (48.2%), outperforming Gemma 4 31B, Qwen3.8-27B, Nemotron 3.5 Super VL, and a model referred to as Muse Glimmer.

TL;DR

  • Inkling 2 Mini punches above its active parameter count, scoring at or near the top of its comparison group on coding benchmarks despite running with only 3.4B active parameters per token (5.4B effective, counting its looped layers).
  • SWEBench Verified hits 79.2% using a bash-only tool setup, the highest score reported among the five models in its evaluation table.
  • Qwen3.8-27B is the closest competitor, beating Inkling 2 Mini on Terminal Bench v2.1 (73.0% vs 71.6%) and SWEBench Pro v2 Hard by a smaller margin, and matching it exactly on the AA Index.
  • Gemma 4 31B trails badly on agentic coding tasks, scoring 0.0% on both FrontierCode v2.0 and Terminal Bench v4.0, suggesting it wasn’t tuned for tool-use-heavy agent loops.
  • Nemotron 3.5 Super VL has the largest total parameter count (125B) but the lowest active count (12B), and it still underperforms Inkling 2 Mini on most coding evals, showing total size isn’t predictive here.
  • The architecture behind the scores is unusual: a 24-layer transformer where the middle 12 layers run three times with shared weights, effectively giving 48 layer applications per token without tripling the parameter count.
  • It’s built for local deployment, shipping as a roughly 20GB checkpoint in mixed NVFP4/MXFP8 precision that runs on a single DGX Spark or an Apple silicon machine with 32GB+ memory.
The enterprise platform for building software with AI

Build an onboarding app for 4,000 new hires.

Build ↵
Self-hosted
SOC 2
Role-based access
Audit log

How does Inkling 2 Mini’s architecture enable its benchmark results?

The model uses a decoder-only transformer with sparse Mixture-of-Experts feed-forward layers, 128 experts per layer with 8 routed per token. What’s unusual is the “looped” design: of the 24 layers of weights, the middle 12 run three times before passing to the output layers. That gives the model 48 effective layer applications per token while only storing 24 layers worth of weights on disk, which is part of why it reports 26.4B total parameters but only 3.4B active (climbing to roughly 5.4B effective active parameters once the repeated passes are counted).

This matters for benchmark comparisons because Inkling 2 Mini is going up against dense and MoE models with more active parameters per token. Gemma 4 31B and Qwen3.8-27B both run with all or most of their parameters active (31B and 27B respectively), while Inkling 2 Mini’s effective active compute is a fraction of that. Getting SWEBench and Terminal Bench scores that match or beat those larger-active models suggests the looped-layer approach is extracting more reasoning depth per parameter, at least on the published evals.

Inputs are multimodal: text and images share a projection into the same hidden space before the decoder processes them jointly, and the model supports a context window up to 1M tokens.

How does Inkling 2 Mini perform on coding and agentic benchmarks?

The model card reports five coding-focused evaluations, each run with a specific agent harness noted alongside the score:

  • FrontierCode v2.0 (via stirrup v0.2.1): Inkling 2 Mini scores 15.1%, narrowly ahead of Qwen3.8-27B’s 14.9%. Gemma 4 31B scores 0.0%, and Muse Glimmer scores 1.5%.
  • DeepSWE v1.1 (via mini-swe-agent v2.4.6): Inkling 2 Mini leads at 48.2%, with Qwen3.8-27B second at 42.2% and Nemotron 3.5 Super VL at 24.7%.
  • Terminal Bench v4.0 (via mini-swe-agent): Inkling 2 Mini and Qwen3.8-27B tie at 5.6%, while Gemma 4 31B and Nemotron 3.5 Super VL both score 0.0%.
  • Terminal Bench v2.1 (run inside a harness called “pi v0.83.0”): this is the one benchmark where Qwen3.8-27B edges out Inkling 2 Mini, 73.0% to 71.6%. Nemotron 3.5 Super VL scores 60.7%.
  • SWEBench Pro v2 Hard (via mini-swe-agent): Inkling 2 Mini scores 77.9%, well ahead of Qwen3.8-27B’s 73.8% and more than double Gemma 4 31B’s 14.2%.

The pattern across these five evals is consistent: Inkling 2 Mini and Qwen3.8-27B trade the top spot depending on the specific benchmark and harness, while Gemma 4 31B consistently posts the weakest agentic coding numbers in the group, including two outright zeroes.

How does it do on SWEBench specifically?

SWEBench, in its various forms, is where Inkling 2 Mini shows its biggest separation from the field. On SWEBench Multilingual (tested with a bash-only tool configuration), it scores 73.7%, ahead of Qwen3.8-27B (70.0%) and Nemotron 3.5 Super VL (67.7%). On SWEBench Verified, also bash-only, it reaches 79.2%, the highest reported score in the table (Muse Glimmer’s score is listed but the exact figure is cut off in the available data; Gemma 4 31B comes in at 46.1%).

Remy AI Engineer Certification

Companies want to build their own software.

Few people can build it right.

Become the one they trust.

  • Access
  • Audit trails
  • Security
  • Compliance

The repeated use of “bash only” as a harness note is worth flagging for anyone trying to reproduce these numbers: the model wasn’t given a wider toolset (like structured file-editing APIs or IDE integration) and still produced strong results, which implies reasonably capable raw code generation and shell-command planning rather than dependence on a feature-rich agent scaffold.

What about non-coding benchmarks like AA Index and Vals Index?

Outside of coding, the model card includes two “third-party index” scores. On the AA Index (v5.0), Inkling 2 Mini ties exactly with Qwen3.8-27B at 20.1, both well ahead of Nemotron 3.5 Super VL (13.6), Muse Glimmer (6.3) and Gemma 4 31B (4.4). On the Vals Index (v2.2), Inkling 2 Mini posts 46.5%, with Nemotron 3.5 Super VL as the next-best at 29.8%; no scores are listed for Gemma 4 31B, Muse Glimmer or Qwen3.8-27B on this particular index.

The exact tie with Qwen3.8-27B on the AA Index is notable given the two models differ in active parameter count (Inkling 2 Mini’s 3.4B/5.4B effective vs Qwen3.8-27B running closer to its full 27B), reinforcing that raw active parameter count isn’t the only variable driving these results.

Is Inkling 2 Mini worth using over Gemma 4 or Qwen3.8?

For anyone building coding agents or terminal-automation tools, the benchmark data favors Inkling 2 Mini or Qwen3.8-27B over Gemma 4 31B, which posts multiple 0.0% scores on agentic tasks in this comparison. Between Inkling 2 Mini and Qwen3.8-27B, the choice looks close: Inkling 2 Mini wins on SWEBench variants and DeepSWE, Qwen3.8-27B wins on Terminal Bench v2.1, and the two tie on Terminal Bench v4.0 and the AA Index.

Deployment footprint may end up being the deciding factor for many teams. Inkling 2 Mini ships as a roughly 20GB checkpoint in mixed NVFP4/MXFP8 precision, runs on a single NVIDIA DGX Spark or an Apple silicon machine with 32GB or more of memory, and has out-of-box support in vLLM, SGLang, llama.cpp, Ollama and Unsloth (including GGUF quantizations). That combination of competitive coding benchmarks and modest hardware requirements is the practical argument for trying it, separate from the raw score comparisons.

Frequently Asked Questions

What is Inkling 2 Mini’s parameter count?

Inkling 2 Mini has 26.4B total parameters with 3.4B active per token. Because its middle 12 layers run three times with shared weights, the effective active parameter count rises to about 5.4B per token.

How does Inkling 2 Mini compare to Qwen3.8-27B on benchmarks?

The two models trade wins depending on the test. Inkling 2 Mini leads on SWEBench Verified, SWEBench Pro v2 Hard, SWEBench Multilingual and DeepSWE v1.1. Qwen3.8-27B leads on Terminal Bench v2.1, and the two tie on Terminal Bench v4.0, FrontierCode v2.0 (near-tie) and the AA Index.

Why does Gemma 4 31B score 0.0% on some benchmarks?

The model card doesn’t explain the exact cause, but Gemma 4 31B scores 0.0% on both FrontierCode v2.0 and Terminal Bench v4.0, both of which are agentic, tool-use-driven evaluations run through the mini-swe-agent harness. This suggests the model struggles with the specific agent loop or tool-calling format used in those tests, rather than a general capability gap.

What hardware is needed to run Inkling 2 Mini locally?

Live class · Wed, Dec 2

$40,000+/mo

in SaaS spend the Remy team cut by building its own apps with AI.

We'll show you how.

Bring an idea.
Build it live.
Leave certified.

The checkpoint is about 20GB in mixed NVFP4/MXFP8 precision. It’s supported on a single NVIDIA DGX Spark, Apple silicon machines with 32GB or more of memory, and other GPU setups with roughly 32GB or more of VRAM recommended.

What does “bash only” mean in the SWEBench results?

It indicates the model was tested using only a basic shell/terminal tool interface, without a richer agent scaffold like structured file-editing or IDE-style tools. Scoring well under that constraint suggests strong native code-reasoning ability rather than reliance on a heavily engineered agent harness.

Editorial standards

Presented by MindStudio

No spam. Unsubscribe anytime.