Holo4 Benchmarks: How H Company's Agent Model Stacks Up vs Frontier AI
Holo4's OSWorld scores and cost-per-task compared to GPT-5.5 and Claude Opus, plus why the dense model beats the MoE variant.
What is Holo4, and why does it matter for agent benchmarks?
Holo4 is an open-weight agent model from the Paris-based lab H Company, built to handle three different ways of acting on a computer: clicking through graphical interfaces, writing and running code, and calling tools through MCP servers or APIs. Instead of needing separate models stitched together for visual tasks and API tasks, Holo4 is trained to pick whichever approach works best for a given step, and benchmark results show it landing close to frontier closed models on OSWorld while costing a fraction as much to run.
TL;DR
- Holo4 unifies three action modes (GUI clicking, code execution, and tool/MCP calls) into one model trained to choose between them rather than relying on separate specialized systems.
- The dense 27B variant beats the mixture-of-experts version by a wide margin, scoring more than double on OSWorld compared to the 35B MoE model with 3B active parameters.
- On OSWorld, the 27B dense model edges out the older GPT-5.5 despite being small enough to run on a single workstation GPU.
- Claude Opus and other top frontier models still lead on raw OSWorld accuracy, but H Company reports they cost roughly seven times more per task than the Holo4 27B.
- Training relies on an “agentic task factory”: automated environments with checkable tasks, supervised fine-tuning on about 127 billion tokens, and reinforcement learning that rewards whichever lane (GUI, code, or tool) solves the task most efficiently.
- The same post-training recipe was applied across three base models (Qwen3-VL 8B/27B, Qwen3 30B-A3B, and NVIDIA’s Nemotron 3 Nano), suggesting H Company is building a repeatable pipeline rather than a one-off model.
- Full weights are available on Hugging Face in multiple quantizations, with the bf16 version around 54GB, making local deployment realistic on high-memory consumer or workstation GPUs.
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
How does Holo4 compare to GPT-5.5 and Claude Opus on OSWorld?
OSWorld is a benchmark that tests whether an agent can complete real desktop tasks: opening applications, navigating file systems, clicking through unfamiliar interfaces, and completing multi-step workflows that mirror actual computer use. It’s become one of the standard yardsticks for computer-use agents because it doesn’t let a model cheat by calling a clean API. It has to actually look at a screen and act.
On this benchmark, Holo4’s 27B dense model scores close to the frontier pack. According to the comparisons shown by H Company, the 27B model actually outperforms what was, at the time, the GPT-5.5 model on OSWorld. That’s a notable result for an open-weight model under 30 billion parameters going up against closed frontier systems with far more compute behind them.
Claude Opus and other current frontier models still score higher in absolute terms. But the accuracy gap has to be read alongside cost. H Company’s benchmarks put the cost of running models like Opus on these tasks at around seven times what it costs to run the Holo4 27B. For anyone running agents at scale, that cost multiplier matters more than a few extra benchmark points, especially for tasks that don’t need frontier-level reasoning, just reliable clicking and navigation.
It’s also worth noting the benchmark landscape moves fast. H Company’s comparisons were made against frontier models that were current within weeks of Holo4’s release, and some of those models have already been superseded. The useful signal here isn’t “Holo4 beats GPT-5.5 forever,” it’s that a sub-30B open model can get within striking distance of frontier systems on a hard agentic benchmark while running locally.
Why does the dense model outperform the mixture-of-experts version?
H Company released two main Holo4 variants: a 27B dense model built on Qwen3-VL 8B/27B, and a 35B mixture-of-experts model with only 3B active parameters, built on Qwen3 30B-A3B. Both went through the same training recipe, which makes the performance gap between them instructive.
On OSWorld, the MoE model scores less than half what the dense model achieves. The gap narrows on some other benchmarks but doesn’t disappear. The likely explanation comes down to active parameters during image processing: the dense model uses all 27 billion parameters on every forward pass, while the MoE model only activates about 3 billion at a time. For tasks that depend on extracting fine-grained detail from screenshots (reading small text, locating a specific button among many, interpreting a cluttered UI), having more active capacity per token seems to matter more than total parameter count.
Seven tools to build an app. Or just Remy.
Editor, preview, AI agents, deploy — all in one tab. Nothing to install.
This is a useful data point for anyone choosing between dense and MoE architectures for vision-heavy agent work. MoE models are generally cheaper to run because fewer parameters activate per token, but that efficiency doesn’t necessarily transfer cleanly to tasks that are heavily visual. If the job is mostly tool calling and code generation, the MoE model’s lower active parameter count may be less of a handicap. If the job involves a lot of screenshot interpretation, the dense model looks like the safer choice based on these results.
How was Holo4 trained to switch between clicking, coding, and calling tools?
The training approach is arguably the more interesting part of the release, separate from the raw scores. H Company built what they call an “agentic task factory”: a set of pipelines that generate interactive environments and tasks with automatically checkable outcomes. These include screenshots of real websites, open-source software running in sandboxes, specific MCP servers, and full desktop environments.
Some of these environments are deliberately hybrid. The same underlying task, say, extracting data from a website, can be completed either by clicking through the UI or by calling an available API or MCP tool. Because both paths are available in the same environment, reinforcement learning can reward the model for choosing whichever path actually gets the job done faster or more reliably, rather than training it to always default to one mode.
The pipeline itself runs in two stages: supervised fine-tuning on roughly 127 billion tokens, followed by reinforcement learning that produces two separate specialized experts, which are then merged back into a single model. On top of the model training, H Company also rebuilt their agent harness (the code layer that executes whatever action the model decides on) based on analysis of failure cases, with engineers reviewing what broke and adjusting the harness accordingly. One outcome highlighted from this process was more reliable long-horizon memory, allowing the model to track state across hundreds of steps.
This same recipe was then applied to multiple base models: the Qwen3-VL dense and MoE variants, and separately to NVIDIA’s Nemotron 3 Nano Omni model, released as Holotron Nano. Reusing one training pipeline across different base architectures suggests H Company is optimizing their post-training process itself as the core asset, something they can reapply to future base models like the upcoming Qwen4 27B that Alibaba has confirmed is in development.
Is Holo4 worth running locally instead of a frontier API?
For teams running agentic workflows at volume, Holo4 offers a meaningfully different cost structure than calling a frontier model’s API for every screenshot and tool call. Vision-heavy agent tasks are expensive specifically because image tokens are costly, and screenshots consume a lot of context window on top of that. Running a 27B model locally on a single high-memory GPU sidesteps both the per-token image cost and the context window pressure that comes with routing everything through hosted frontier APIs.
The tradeoff is accuracy: Claude Opus and similar frontier models still score higher on raw OSWorld performance. Whether that gap matters depends on the task. For straightforward, repeatable automation (data entry, report pulling, form navigation) the accuracy difference may not justify a sevenfold cost increase. For tasks requiring deeper reasoning or handling genuinely novel interfaces, the frontier model’s higher ceiling might still be worth the premium.
Remy doesn't write the code. It manages the agents who do.
Remy runs the project. The specialists do the work. You work with the PM, not the implementers.
There’s also a practical architecture question worth considering: Holo4 doesn’t have to be an either/or choice against a frontier model. H Company’s own harness (called Hi Agents) is designed to let a larger model handle planning and orchestration while delegating screen-heavy GUI work to Holo4. That hybrid setup, frontier model for reasoning, Holo4 for clicking, lets teams keep expensive frontier calls to a minimum while still getting reliable visual task completion.
Frequently Asked Questions
What is OSWorld and why is it used to benchmark agent models?
OSWorld is a benchmark that tests whether an AI agent can complete real desktop computer tasks, like navigating applications and file systems, by actually interacting with a screen rather than calling clean APIs. It’s widely used because it reflects the messy reality of real software environments.
How big is Holo4 and what hardware does it need?
Holo4 comes in two main sizes: a 27B dense model and a 35B mixture-of-experts model with 3B active parameters. The full bf16 precision version is around 54GB, making it feasible to run on a single high-memory workstation GPU, with smaller quantized versions available for less hardware.
Does Holo4 beat Claude Opus or GPT-5.5?
On OSWorld, Holo4’s 27B dense model outperformed the GPT-5.5 model it was compared against at release time, but Claude Opus and other current frontier models still scored higher in raw accuracy. H Company’s figures show those frontier models cost roughly seven times more per task to run.
Why does the mixture-of-experts version score lower than the dense model?
The MoE variant only activates about 3 billion parameters per forward pass compared to 27 billion in the dense model. For vision-heavy tasks like interpreting screenshots, that lower active capacity appears to hurt performance more than it does for text or tool-calling tasks.
Can Holo4 be combined with a larger model instead of replacing it?
Yes. H Company’s own agent harness supports using a larger model for planning and orchestration while routing GUI-heavy, screenshot-based steps to Holo4, which can reduce costs without giving up a stronger model’s reasoning for the parts of a task that need it.