Ornith 1.5 9B: Local Test Results Expose a Benchmark Gap
A hands-on local test of Ornith 1.5 9B on a single GPU reveals agentic and coding tasks breaking down despite strong benchmark claims.
What is Ornith 1.5 9B and does it live up to its benchmarks?
Ornith 1.5 9B is a dense, 9-billion-parameter language model built by ornith-ai, designed to run on a single GPU or even mobile hardware in quantized GGUF format. It’s the smallest member of the Ornith 1.5 family, which also includes a larger mixture-of-experts variant. On paper, the 9B model reports benchmark scores that rival or beat Qwen 3.5’s similarly sized model and even close in on Qwen’s 35B mixture-of-experts model on several agentic and reasoning tests. In a hands-on local test, however, the model struggled badly with real-world agentic tasks, including basic cloud provisioning and simple code generation, exposing a wide gap between its published numbers and practical usability.
TL;DR
- Ornith 1.5 9B is a dense (non mixture-of-experts) model from ornith-ai, built on a Qwen3.5-based architecture and released in safetensors format on Hugging Face, with a GGUF variant aimed at mobile and single-GPU deployment.
- The model is trained with a self-improving reinforcement learning loop, where it generates its own tasks, scaffolds, and solutions rather than relying on a fixed human-designed training harness, the same approach used across the broader Ornith 1.5 lineup.
- Published benchmarks claim the 9B model outperforms same-size peers like Qwen 3.5 and approaches the performance of Qwen’s much larger 35B mixture-of-experts model on agentic and reasoning tasks.
- In a local test on a single Nvidia A100 GPU served through vLLM, the model was tasked with using AWS CLI to provision a free-tier EC2 Ubuntu instance with a scoped IAM user, and it failed to complete the task after burning through tens of thousands of tokens.
- The model got confused selecting the correct Ubuntu AMI ID among thousands of public images, made repeated typos in CLI commands, and eventually produced nonsensical output, including a stretch of hallucinated Chinese text.
- A second test asking for a simple self-contained HTML file (a rotisserie chicken animation) also produced garbled, low-quality output, in contrast to the mixture-of-experts variant, which handled a similar task cleanly.
- The results suggest that for this dense 9B model, benchmark scores did not translate into reliable agentic or coding performance, reinforcing the need for independent, task-based testing rather than relying on published leaderboards alone.
Everyone else built a construction worker.
We built the contractor.
One file at a time.
UI, API, database, deploy.
How was Ornith 1.5 9B tested locally?
The test ran on an Ubuntu machine with a single Nvidia A100 GPU (80GB VRAM), using vLLM to download and serve the model. VRAM consumption landed just over 72GB, driven largely by the KV cache tied to context length. Lowering the configured context window reduces that footprint, which matters for anyone trying to run this model on smaller cards than a data-center-grade A100.
The model was wired up with a Hermes-style agent scaffold so it could make tool calls, and given two separate real-world tasks:
- Provision a free-tier Ubuntu EC2 instance on a real AWS account, using AWS CLI, with a scoped IAM user limited to EC2 permissions (not admin). The task required the model to find the right Ubuntu AMI, configure a security group, set up networking in the correct VPC and subnet, and launch the instance, all without a pre-built harness.
- Generate a self-contained HTML file rendering a rotisserie chicken animation over an open flame, a simple front-end coding task with no external dependencies.
Both tasks were designed to be achievable by a competent human in a few minutes, either through a handful of CLI commands or a few clicks in the AWS console.
What went wrong in the AWS provisioning task?
The model authenticated correctly with AWS CLI and began issuing commands, but got stuck almost immediately trying to identify the correct Ubuntu AMI ID. Because AWS hosts thousands of public images, the model had no reliable way to narrow the search, and it started making typos in its own CLI commands while sifting through the results.
After roughly seven minutes and around 43,000 tokens consumed, the model still hadn’t completed the first step of the task. At that point it stalled, effectively giving up on the approach it had chosen. The human tester intervened, manually supplying the correct AMI ID to unblock it. Even with that assist, the model hit another snag: it appeared unable to properly invoke its shell tool, then proposed switching to a browser-based tool instead, only to find no browser available in the environment. It looped back to the terminal, and when steered again, began outputting hallucinated text in Chinese rather than a valid command. The task was called as a failure and canceled.
Why does this matter for agentic AI claims?
The gap between benchmark performance and real-world execution is the central issue here. Ornith 1.5 9B’s published scores suggest it should be competitive with, or close to, models several times its size on agentic benchmarks. But agentic benchmarks typically test narrower, more structured scenarios than an open-ended task like “figure out which AMI to use out of thousands of options, then wire up networking correctly.” Real cloud infrastructure work involves ambiguity, large unstructured data (like AMI listings), and multi-step error recovery, exactly the conditions where a 9B dense model appears to run out of capacity.
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
This is a recurring pattern in the field: strong leaderboard numbers don’t guarantee that a model can autonomously drive a CLI tool, recover from a wrong turn, or avoid hallucinating when stuck. It’s a useful reminder that benchmark-topping small models still need to be stress-tested against messy, real-world workflows before being trusted with production tasks, especially ones involving live cloud credentials.
How did the coding test go, and how does the 9B compare to the larger Ornith model?
The second test, generating a single HTML file for a rotisserie chicken animation, was intentionally more modest in scope than the AWS task. Even here, the model struggled to produce a clean, working script. The output during the run showed signs of the same instability seen in the agentic test: garbled generation and low-quality results rather than a functional file.
This stood in contrast to the larger Ornith 1.5 mixture-of-experts model, tested separately, which handled a comparable coding task without the same breakdown. The dense 9B model’s issues weren’t limited to complex multi-step agentic workflows: they also showed up in a simple, self-contained generation task, pointing to a broader quality gap at this smaller size rather than an issue specific to tool use or long-horizon planning.
Is Ornith 1.5 9B worth using right now?
Based on this local test, the 9B variant is not yet reliable for agentic tasks involving real infrastructure, nor for straightforward code generation. The self-improving training approach behind the Ornith 1.5 family (models generating their own tasks and solutions via reinforcement learning rather than a fixed human harness) is an interesting technique, and it may explain strong scores on the specific benchmarks it was evaluated against. But those scores did not carry over into the kind of open-ended, real-world execution developers actually care about.
Anyone considering the model for agentic use cases, especially anything touching live cloud accounts or production code, should run their own task-specific tests rather than relying on benchmark tables. The mixture-of-experts sibling in the same family performed noticeably better in similar testing, which suggests parameter count and architecture still matter more than training methodology alone at this stage.
Frequently Asked Questions
What is Ornith 1.5 9B built on?
It’s a dense, non mixture-of-experts model from ornith-ai, based on a Qwen3.5-style architecture, released in safetensors format with a GGUF variant for lightweight and mobile deployment.
How much VRAM does Ornith 1.5 9B need to run locally?
In testing on an Nvidia A100 with vLLM, the model consumed just over 72GB of VRAM, largely due to KV cache sized by context length. Reducing the configured context window lowers VRAM usage, making it feasible on smaller single-GPU setups.
Did Ornith 1.5 9B complete the AWS provisioning task?
No. It failed to launch an EC2 Ubuntu instance even with manual steering and a supplied AMI ID, eventually producing hallucinated output and getting stuck switching between shell and browser tools.
How does the 9B model compare to the Ornith 1.5 mixture-of-experts model?
The larger mixture-of-experts variant performed noticeably better on similar coding tasks in separate testing, suggesting the dense 9B model’s benchmark scores don’t reflect its real-world execution quality.
Should benchmark scores be trusted for small agentic models?
Not on their own. This test showed a dense 9B model with competitive published benchmarks failing basic real-world agentic and coding tasks, reinforcing the need for hands-on, task-specific evaluation before deployment.