Alpie Seed 9B Tested: Does It Live Up to Its Benchmarks?
Hands-on tests of Alpie Seed 9B show failed coding, vision, and physics tasks despite benchmark charts claiming it beats 20B and 30B models.
What is Alpie Seed 9B?
Alpie Seed 9B is a 9 billion parameter vision-language model built by 169Pi on a Qwen3.5 base, post-trained with supervised fine-tuning and GRPO. It accepts both text and images, supports a thinking mode, offers a 262K token context window, and ships under an Apache 2.0 license. The model is distributed in GGUF format (including Q6_K and Q8_0 quantizations with a companion mmproj vision file for ROCm/Strix Halo hardware), and VRAM consumption in testing landed around 9GB, in line with other models of this size. The pitch, according to the model’s own benchmark charts, is that it outperforms other 9B models and even larger 13B, 20B, and 30B competitors on knowledge, hard science, competition math, and real coding benchmarks.
TL;DR
- Benchmark claims don’t hold up: independent testing found Alpie Seed 9B failing basic coding, vision, and physics tasks despite charts showing it topping every category against much larger models.
- Agentic coding test failed: given a Flask/Postgres/Docker app with one planted bug (a milk-per-producer average divided by total herd size instead of milking yaks), the model reported three “bugs” that didn’t exist and left the real one untouched.
- A patch it proposed could have broken the app: one suggested fix swapped a frontend API address for a Docker service name that a browser on the host couldn’t resolve, though the edit was never actually applied.
- Vision-to-code generation was weak: asked to turn a hand sketch of a doner kebab into an animated simulation, the model produced an animation where the tomato, meat, and heat source were barely recognizable as the intended objects.
- Physics derivation contained real errors: asked to derive a physics identity (the zeta-four / Stefan-Boltzmann style result) step by step, the model produced a derivation with a wrong units step and an arithmetic answer off by roughly a factor of 100.
- The Qwen base still shows through: despite the failures, the model recalled correct textbook constants and concepts at points, suggesting the underlying Qwen3.5 base retains general competence even when Alpie’s post-training doesn’t reliably improve it.
- Small model hype needs scrutiny: the case illustrates a broader problem in the 4B-9B model space, where demand for locally runnable models is high and benchmark charts can be used to create an impression that doesn’t match real-world use.
Enterprise-ready- Self-hosted
- SOC 2
- HIPAA
- Audit log
- Role-based access
How was Alpie Seed 9B tested?
The model was served locally through llama.cpp using its GGUF release, running at roughly 9GB of VRAM, a footprint typical for a 9B class model. It was wired into an agent harness (Hermes) so it could call tools, read files, and make edits autonomously rather than just answering prompts in a chat window.
The core test was an agentic coding task: a working three-tier Docker application (Postgres database, Flask API, dashboard frontend) simulating a yak herd management system for a dairy cooperative. The app had one deliberately planted bug with no comments or hints pointing to it. The dashboard showed 15 yaks, only 8 of which were milking, producing about 31.3 liters of milk total that day. The bug: the average milk-per-producer calculation divided total milk by all 15 animals (including non-milking males and calves) instead of just the 8 milking yaks, producing a reported average of 2.09 liters that was mathematically wrong.
Beyond coding, two more tests probed different capabilities: a vision task asking the model to interpret a hand-drawn sketch of a doner kebab and generate an animated simulation of it, and a reasoning/math task asking it to derive a physics result from scratch, including proving an identity, solving an equation numerically, and checking the arithmetic.
What went wrong with the coding test?
On the first attempt, the model’s tool call failed outright, with the agent harness reporting the goal “unachievable” because it couldn’t locate any code, despite the working directory already being correctly configured and pointed at the application folder. A second attempt did succeed in making the tool call and locating the files.
From there, the model reported finding three bugs, even though only one existed. It then spent a brief period “thinking” before declaring everything fixed. When the application was refreshed, nothing had changed: the actual averaging bug remained untouched. Worse, one of the three “fixes” the model proposed would have changed the frontend’s API address to a Docker internal service name, something a browser running on the host machine cannot resolve. Had that edit actually been applied, it risked breaking a dashboard that was otherwise functioning correctly. The model’s patch wasn’t applied in the end, but the fact that it was proposed at all points to a lack of grounding in how the Docker networking in the test app actually worked.
How did it do on vision and reasoning tasks?
The vision test asked the model to take a simple hand sketch of a doner kebab and turn it into an animated simulation capturing the sketch’s intended features; a rotating cone of meat, a slicing motion, a heat source, garnish. The resulting animation was described as barely resembling the intended scene: a tomato, a slab, something resembling fire, and meat that appeared to be slicing off a cone, but with proportions and relationships between elements “totally off.” The thinking trace that preceded the output showed little evidence of depth or self-checking; the model didn’t appear to verify its own visual output against the prompt.
The reasoning test asked the model to derive a known physics result (involving a zeta-function identity and a Stefan-Boltzmann-style constant) step by step, including numerical verification. The model returned an answer quickly, which itself was a warning sign given the complexity of the task. The derivation contained a line that broke dimensional consistency (making the units wrong) and an arithmetic result for the physical constant that was off by a factor of about 100. Interestingly, the model still appeared to recall textbook-correct values and concepts at various points, which points to the underlying Qwen3.5 base model retaining general physics knowledge, even though Alpie’s post-training didn’t translate into a reliable, verified derivation.
Do Alpie Seed 9B’s benchmark claims hold up?
Not based on this testing. The model’s published benchmark charts show it placing at the top of comparison tables against other 9B models and even against notably larger 13B, 20B, and 30B models across categories including knowledge, hard science questions, competition math, and coding. In practice, across an agentic coding task, a vision-to-animation task, and a step-by-step physics derivation, the model failed all three in ways that ranged from minor (a weak animation) to potentially damaging (a proposed code edit that could have broken a working application).
This gap between leaderboard position and real-world task performance is not unique to Alpie, but it is a useful reminder that benchmark charts for niche or newly released models deserve scrutiny, particularly when a small model claims to outperform models several times its size across nearly every category simultaneously.
Is Alpie Seed 9B worth using?
For users specifically looking for a lightweight, locally runnable vision-language model under Apache 2.0 licensing, Alpie Seed 9B is technically functional. It runs at a modest ~9GB VRAM footprint, supports a large 262K context window, handles multiple Indian languages alongside English, and is available in ready-to-use GGUF quantizations. The thinking mode and tool-calling support also work mechanically.
But based on hands-on testing, it struggles with tasks it specifically markets itself for, including coding assistance and precise reasoning. Users evaluating it for agentic coding, vision-grounded tasks, or technical derivations should verify outputs carefully rather than trusting benchmark position alone, and should be prepared for the model to confidently report incorrect fixes or results.
Frequently Asked Questions
What is Alpie Seed 9B built on?
It’s a 9 billion parameter vision-language model from 169Pi, built on a Qwen3.5 base and post-trained using supervised fine-tuning and GRPO (a reinforcement learning technique). It takes both text and image inputs and supports a thinking/reasoning mode.
How much VRAM does Alpie Seed 9B need?
Running the GGUF version through llama.cpp consumed roughly 9GB of VRAM in testing, which is standard for a 9B-class model and puts it within reach of consumer GPUs with 8-12GB of memory depending on quantization.
Did Alpie Seed 9B actually fix the planted code bug?
No. In testing, it misreported three nonexistent bugs instead of the single real one (an incorrect average calculation dividing by all animals instead of only milk-producing ones), and declared the issue resolved when it was not.
Is Alpie Seed 9B open source?
Yes. It’s released under an Apache 2.0 license and distributed via Hugging Face in GGUF quantizations (including Q6_K and Q8_0 variants with a dedicated vision projector file), making it freely usable and modifiable.
Why do small 9B models get this much benchmark hype?
There’s strong demand from developers who want capable models that run locally and privately on consumer-grade GPUs (8-24GB VRAM). That demand creates pressure and incentive for labs to publish favorable benchmark comparisons, which is why independent, task-based testing matters before trusting a leaderboard position.