Nanbeige 4.3 33B: Benchmark Charts vs. a Real Debugging Test
Nanbeige 4.3 33B tops its own benchmark charts and claims Claude-level agentic skill, but a hands-on coding test tells a different story.
What is Nanbeige 4.3 33B and why does it matter?
Nanbeige 4.3 33B is an open-weight language model from Nanbeige LLM Lab, the AI research team inside Boss Zhipin, a major Chinese hiring platform. It’s a 33-billion-parameter dense model, fine-tuned from a matching Nanbeige 4.3 33B Base model, and released under an Apache 2.0 license on Hugging Face. The lab positions it as a “teacher” model meant to anchor a future distillation pipeline for smaller Nanbeige models, starting with a planned 3B version. On paper, it claims state-of-the-art agentic performance for its size class, specifically in coding and professional “cowork” tasks, and the lab’s own marketing frames it as approaching Claude-level capability. A hands-on test using a real debugging scenario found it struggling badly on tasks it should have handled with ease.
TL;DR
- Nanbeige 4.3 33B is a 33B dense model from Boss Zhipin’s AI lab, fine-tuned from Nanbeige 4.3 33B Base and released under Apache 2.0.
- The model’s own benchmark charts show it beating Qwen3.8-27B, Muse-Glimmer-30B, and Gemma4-31B across most agentic coding and cowork benchmarks, including Terminal-Bench, DeepSWE, and JobBench.
- A real-world debugging test built around a restaurant costing bug (a yield-percentage calculation error) took over an hour and ended with the model finding 14-15 unrelated “bugs” while missing the actual one.
- A simple single-file HTML animation task (swaying paper banners) caused the model to loop in extended “thinking” for 45+ minutes before hitting its max output token limit without finishing.
- Running the model locally via vLLM required close to 88GB of VRAM at full context length, with an estimated 60-62GB floor even with a shorter context.
- The gap between chart performance and practical reliability highlights a recurring issue with benchmark-driven marketing for mid-size open models: strong numbers don’t always translate to usable agentic behavior.
- The model supports a 512K token context window and ships with recommended settings for both agentic/tool-use and reasoning/chat scenarios.
What does Nanbeige claim about its performance?
Nanbeige’s model card presents a dense comparison table pitting Nanbeige 4.3 33B against three similarly sized open models: Qwen3.8-27B, Muse-Glimmer-30B, and Gemma4-31B. All four were evaluated at their highest thinking effort using the same inference settings and harness.
On agentic coding benchmarks, Nanbeige 4.3 33B leads on most metrics. It scores 80.9 on Terminal-Bench 2.1 versus 73.0 for Qwen3.8-27B, 61.8 for Muse-Glimmer-30B, and 47.2 for Gemma4-31B. On Terminal-Bench 4.0 the gap widens further: Nanbeige posts 23.8 against single digits or near-zero scores for the other three. DeepSWE tells a similar story, with Nanbeige at 65.5 versus 42.5 for its closest competitor. The one coding benchmark where it doesn’t lead is SWE-Bench Pro, where Qwen3.8-27B edges it out, 61.7 to 60.3.
The “agentic cowork” category, which covers tasks mapped to 35 real occupations across IT, engineering, finance, healthcare, and law, shows the same pattern. Nanbeige leads on Automation Bench, JobBench, Claw-Eval, Toolathlon-Verified, DeepPlanning, EnterpriseOps-Gym, and a dollar-denominated “YC-bench” metric, trailing only on GDPval, where Qwen3.8-27B comes out slightly ahead (86.4 vs 84.8). General reasoning benchmarks (GPQA Diamond, AIME 2026, AA-LCR) are closer across the board, with Gemma4-31B actually topping GPQA Diamond.
Taken together, the charts paint Nanbeige 4.3 33B as a clear leader in its weight class, particularly on tasks tied to terminal use, software engineering, and structured job simulations.
How did it perform in a real coding and debugging test?
A hands-on test documented by AI reviewer Farhat Mirza put that claim under pressure with a concrete, real-world scenario: a fictional restaurant’s kitchen costing system had a bug in how it calculated meat yield. Exotic cuts like crocodile tail, wallaby shank, and rabbit lose a large percentage of their weight during trimming and boning (the restaurant’s own figures: 58%, 46%, and 44% yield respectively), so the system needs to buy more raw meat than it ultimately serves. The bug caused the dashboard to show implausibly low food costs because the purchasing calculation was inverted. A specific example: a 220g wallaby shank portion at 46% yield should require buying roughly 478g of raw meat, but the system computed only 101g needed, undercounting cost and overstating margin.
The model was asked to find and fix this single, well-defined bug in a vLLM-served local deployment. After more than an hour of visible “thinking” (repeatedly compacting context, making small moves, then thinking at length again), it never isolated the actual issue. Instead it reported 14 to 15 separate “bugs” it believed it had found, none of which was the real problem. The test was called off as a failure.
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
A second, much simpler test asked the model to generate a single HTML file simulating swaying paper banners with basic colors and animated doves, a task well within reach of most current coding-capable models. After running for 45 minutes, the model hit its maximum output token limit and context length without producing a finished file, having spent the bulk of that time in extended internal reasoning rather than converging on working HTML, CSS, and JavaScript.
Is Nanbeige 4.3 33B worth running locally?
For teams evaluating it purely on hardware terms, the model is accessible but not light. Serving it through vLLM required close to 88GB of VRAM at the model’s full supported context, with a likely floor around 60-62GB of VRAM at full precision even with a reduced context window. The model card lists support for up to 524,288 tokens (512K) of context, and recommends different temperature, top_p, and max-new-token settings depending on whether it’s being used for agentic tool-use work or general reasoning and chat.
Whether it’s “worth it” depends heavily on the use case. The benchmark table suggests genuine strength on structured, well-scoped agentic tasks resembling the exact test suites it was tuned against: terminal commands, defined software engineering problems, and job-simulation benchmarks with clear rubrics. The hands-on test, by contrast, involved an open-ended, realistic debugging scenario with a single root cause buried in business logic, plus a simple creative coding task. Both are the kind of work developers do every day, and both produced extended, unproductive reasoning loops rather than a working fix or output.
That gap matters for anyone choosing a model to run coding agents against actual codebases rather than benchmark suites. A model that excels at Terminal-Bench and DeepSWE but can’t isolate a single logic bug in an hour, or finish a basic HTML animation, is not yet a reliable substitute for frontier closed models in production debugging workflows.
How does it compare to Claude-level claims?
Nanbeige LLM Lab has marketed this release (and prior Nanbeige models) around the idea of matching larger, proprietary models with a smaller open one. The “Claude-level” framing referenced in the model’s promotional materials sets a high bar: Claude models, including smaller variants like Haiku, are generally regarded as strong, fast, and reliable for agentic coding tasks. In the documented test, even without a side-by-side run against Claude Haiku, the reviewer’s assessment was blunt: the thinking process observed in Nanbeige 4.3 33B fell well short of what’s expected from a Claude-class model, let alone matched it. The model’s inability to complete either test case, despite roughly three hours of combined runtime, undercuts the positioning suggested by its own charts and marketing.
Frequently Asked Questions
What is Nanbeige 4.3 33B built for?
It’s designed as a flagship “teacher” model for agentic coding and professional cowork tasks, and it’s also intended to anchor a distillation pipeline for smaller Nanbeige models planned for the future, including a 3B variant.
How big is Nanbeige 4.3 33B and what license does it use?
It’s a 33-billion-parameter dense model, released under the Apache 2.0 license, fine-tuned from the companion Nanbeige4.3-33B-Base model.
How much VRAM does Nanbeige 4.3 33B need to run?
A local vLLM deployment at full supported context length consumed close to 88GB of VRAM. A shorter context would likely bring full-precision requirements down to roughly 60-62GB.
Did Nanbeige 4.3 33B pass real-world coding tests?
Remy is new. The platform isn't.
Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.
No. In a hands-on test, it failed to find a specific yield-calculation bug in a restaurant costing system after more than an hour, instead flagging over a dozen unrelated issues, and it also failed to complete a simple single-file HTML animation task within 45 minutes.
Should developers trust benchmark charts alone when picking a model?
Not exclusively. Nanbeige 4.3 33B’s official benchmarks show strong results against similarly sized open models like Qwen3.8-27B, Muse-Glimmer-30B, and Gemma4-31B, but a hands-on debugging and coding test produced results well below what those charts would predict, which is a useful reminder to test models on realistic, open-ended tasks before relying on them for production work.
