How to Run IBM Granite 4.2 Locally with vLLM
Download and serve IBM's Granite 4.2 models locally with vLLM. VRAM needs and setup steps for the 3B, 8B and 30B reasoning variants.
What is Granite 4.2 and why run it locally?
Granite 4.2 is IBM’s latest family of open weight language models, released under the Apache 2.0 license, with variants at 3 billion, 8 billion, and 30 billion parameters. Unlike a lot of small models, Granite 4.2 bakes in chain of thought reasoning, native tool calling, and a 128K context window, even at the 3B size. Running it locally through vLLM means you get an OpenAI-compatible API endpoint on your own hardware, with full control over context length, quantization, and KV cache sizing instead of depending on a hosted API.
TL;DR
- Granite 4.2 ships in three sizes, 3B, 8B, and 30B, all Apache 2.0 licensed with open weights and a published training recipe.
- The 3B model supports chain of thought reasoning and native tool calling but skips agentic training stages, so it struggles with multi-step tool use.
- The 8B and 30B models get extra agentic training on tasks like fixing code repos, running shell commands, and browsing the web, making them better suited for harnesses like Hermes agent, OpenHands, or OpenCode.
- On a 48GB VRAM system, the 3B model used about 4.5GB of VRAM, leaving plenty of room to spare or to extend the context window.
- The 8B model consumed over 45GB of VRAM in one test run, though that included a large KV cache allocation that can be reduced.
- Serving either model requires vLLM plus a reasoning parser plugin provided alongside the model card on Hugging Face.
- In hands-on testing, both the 3B and 8B models made factual and execution errors on complex multi-part prompts, so real world reliability varies by task type.
How do you download Granite 4.2 from Hugging Face?
Granite 4.2 models are distributed through Hugging Face like most open weight releases. The practical first step is installing the Hugging Face CLI, which handles authentication and the actual file transfer for model weights. Once installed, you point the CLI at the specific Granite 4.2 repository (3B, 8B, or 30B, depending on which variant you want) and it pulls down the weights and config files.
One detail specific to Granite 4.2: IBM provides a Python file alongside the model card that acts as a reasoning parser plugin. This isn’t optional if you want the chain of thought output structured correctly. You download it separately and reference it when launching vLLM, so budget an extra step here that you wouldn’t hit with a plain instruction-tuned model.
How do you serve Granite 4.2 with vLLM?
vLLM is a serving engine built for high-throughput LLM inference, and it’s become a common way to self-host open weight models with an OpenAI-compatible API. For Granite 4.2, the setup follows the standard vLLM pattern: point it at the downloaded model directory, load the reasoning parser plugin IBM supplies, and start the server.
Once running, vLLM exposes an endpoint you can wire into agent harnesses. The demonstrated setup used Hermes agent as the front end, but IBM’s documentation also lists compatibility with OpenHands and OpenCode, since Granite 4.2 uses the OpenAI-compatible tool call format natively. If your workflow already runs on any of these, swapping in Granite 4.2 is mostly a matter of pointing the harness at your local vLLM endpoint.
What are the VRAM requirements for each Granite 4.2 size?
VRAM usage depends heavily on context window and KV cache settings, but here’s what was observed on a 48GB VRAM Ubuntu system:
- 3B model: roughly 4.5GB of VRAM at default settings. This leaves enormous headroom, meaning the 3B model is realistically usable on consumer GPUs or even some laptop GPUs, not just data-center cards.
- 8B model: over 45GB of VRAM in the tested configuration, largely because of a large KV cache allocation. Reducing the context window brings this down substantially, so the 8B model doesn’t strictly require a 48GB card, but it will need a meaningfully larger GPU than the 3B variant if you want long context.
- 30B model: not benchmarked for VRAM in this round of testing, but given the training recipe and parameter count, expect it to need noticeably more memory than the 8B model, likely requiring a data-center-class GPU or multi-GPU setup for comfortable serving with a large context window.
If VRAM is tight, the simplest lever is trimming the context window rather than the model itself, since KV cache size scales with how much context you allow.
How does the training recipe differ across sizes?
Remy doesn't write the code. It manages the agents who do.
Remy runs the project. The specialists do the work. You work with the PM, not the implementers.
IBM published the training recipe for Granite 4.2, and it explains a lot about what each size is actually good at. All three models start from a base model and go through reinforcement learning on verifiable tasks like math and code, followed by a scale boost phase. That’s where the 3B model’s pipeline stops before final alignment.
The 8B and 30B models get additional rounds of training on real-world agentic tasks: fixing code repositories, running shell commands, and browsing the web. Every model size finishes with a final alignment pass for safety and preference tuning, but only the 8B and 30B models are explicitly trained to operate tools before that final stage.
The practical implication: if you need a model for straightforward code generation, math, or general language tasks, the 3B model’s training matches that use case. If you’re building an agent that needs to chain multiple tool calls together reliably, the 8B or 30B models are the ones designed for it.
Is Granite 4.2 worth running locally?
It depends on your task. In testing, the 3B model handled a complex prompt (generating a self-contained HTML file with tabs for every continent, listing countries and their national drinks in native script) by producing reasoning that looked coherent, but it failed to actually write the file to disk, and the content itself only covered four or five countries per continent instead of a comprehensive list. The HTML also didn’t render correctly, with no working tabs or styling.
The 8B model did better on the same prompt: it successfully created the file, and the output was more structured, though it still only listed a handful of countries per continent rather than being exhaustive. This lines up with IBM’s own framing: the 3B model is designed for general-purpose tasks, not complex agentic or highly detailed knowledge retrieval, while the 8B and 30B models are built for tool use and more demanding multi-step work.
A separate reasoning test asked the model to choose between three fridges of chicken with different freshness levels while framed around financial hardship, testing whether it would prioritize food safety over cost. The 8B model in this test chose the cheapest option despite it being mostly spoiled, showing that even models good at surface-level reasoning can be swayed by framing rather than sticking to the objectively safer choice.
The takeaway: Granite 4.2 is genuinely lightweight to run, fully open, and well documented, but treat benchmark claims and marketing framing with some skepticism until you’ve tested it against your own specific tasks.
Frequently Asked Questions
What license is Granite 4.2 released under?
Granite 4.2 is released under Apache 2.0, meaning the weights are fully open and usable commercially without the restrictions attached to more limited “open” licenses.
Can Granite 4.2 3B run on a laptop GPU?
Based on observed VRAM usage of around 4.5GB, the 3B model is light enough to run on many consumer and laptop GPUs, especially with a modest context window.
Does Granite 4.2 support tool calling out of the box?
Yes, all sizes support native tool calling using an OpenAI-compatible format, and IBM documents compatibility with harnesses like Hermes agent, OpenHands, and OpenCode. The 8B and 30B models received additional agentic training that makes them more reliable at it.
Which Granite 4.2 size should I use for agent workflows?
One coffee. One working app.
You bring the idea. Remy manages the project.
The 8B or 30B models are better suited for agentic, multi-step tool-use workflows since they went through extra training rounds on tasks like code repo fixes, shell commands, and web browsing. The 3B model skips those stages.
How much VRAM does the 8B model actually need?
In one test it consumed over 45GB due to a large KV cache, but that figure can be reduced significantly by lowering the context window, so it doesn’t strictly require a 48GB GPU for shorter-context use cases.