Tool LLM 3 Pro: A 3.66B Logic Model That Catches Contradictions Locally
Tool LLM 3 Pro is a 3.66B Granite-based logic specialist run locally with llama.cpp. Here's how it performs on contradiction detection and reasoning.
What is Tool LLM 3 Pro?
Tool LLM 3 Pro is a 3.66 billion parameter model from Web AI, built on IBM’s Granite 4.2 architecture and post-trained specifically for formal logic. It isn’t a general chatbot. It’s a narrow specialist designed to read a set of statements, reason through them step by step, and determine whether they’re consistent, contradictory, or simply underdetermined (the “cannot tell” case that trips up a lot of larger models). The practical use case that makes this interesting for people who build software: catching contradictions in requirement documents, specs, and policy text before they turn into bugs or disputes.
TL;DR
- Tool LLM 3 Pro is a 3.66B parameter model built on IBM’s Granite 4.2, fine-tuned exclusively for formal logic tasks rather than general chat or coding.
- In hands-on testing, the model correctly identified three planted contradictions in a grocery delivery app requirement spec, citing exact clause numbers, and flagged one additional defensible conflict rather than a false positive.
- On a return-policy test with ten customer cases, it scored 8 out of 10, correctly handling ambiguous “cannot tell” cases and even catching its own reasoning mistake mid-answer.
- The model runs locally via llama.cpp using roughly 9GB of VRAM, making it practical for on-device or private deployment on consumer hardware.
- On benchmark comparisons, it outperforms much larger models like Qwen3 8B and a 120B GPT-OSS variant on in-domain formal logic tasks, but falls behind on held-out, more general reasoning benchmarks.
- The model struggles with backward-chaining logic, tending to either give up with “cannot tell” or incorrectly reverse one-directional rules.
- Training combined LoRA supervised fine-tuning, checkpoint merging, and reinforcement learning against a programmatic verifier, meaning answers were rewarded for being provably correct rather than merely plausible-sounding.
- Patient records✓ Granted
- Surgery schedule✓ Granted
- Billing export✕ Denied
How was Tool LLM 3 Pro trained?
Web AI started with IBM’s Granite 4.2 (the 3B-class variant) and converted it into a logic specialist through a multi-stage post-training pipeline. The process used LoRA-based supervised fine-tuning, checkpoint merging across training runs, and reinforcement learning guided by a programmatic verifier rather than a human preference model or a general-purpose reward signal.
That last detail matters. Instead of optimizing for answers that sound convincing, the training rewarded answers that a logic checker could actually verify as correct. For a model meant to do formal reasoning rather than conversation, that’s a meaningfully different training objective than what goes into most instruction-tuned LLMs, and it shows up in how literal and clause-specific the model’s outputs are.
How does it perform on contradiction detection?
In testing, the model was given a realistic requirement specification for a grocery delivery app, the kind of document software teams write before any code gets built. Three contradictions had been deliberately planted in the text, along with some decoy clauses designed to look related without actually conflicting.
The model found all three planted contradictions, citing the correct clause numbers (for example, flagging a conflict between clause 2.1 and clause 9.1) along with clear reasoning for each. It also raised one additional flag, a conflict between a locked-order clause and a substitution-change clause, which wasn’t one of the planted bugs but is arguably a legitimate catch rather than a false positive. None of the decoy clauses were wrongly flagged.
A second test used a real-world electronic store return policy with ten customer scenarios, several of which were designed to have no clear answer (the correct response being “cannot tell” rather than a forced yes or no). The model scored 8 out of 10, correctly applying the ambiguous cases and, notably, catching and correcting its own mistake on one case mid-reasoning before finalizing its answer.
A third test used a 12-clause festival ticketing specification with three hidden contradictions and several decoy clauses designed to look related without conflicting. The model found all three contradictions with correct reasoning and flagged zero decoys, a perfect score.
How does it compare to larger models on benchmarks?
Benchmark comparisons split into two tracks. On “Track A,” in-domain formal logic tasks the model was specifically trained for, Tool LLM 3 Pro leads on overall gate score and strict test accuracy, and posts the best average across six evaluation lanes, beating models more than twice and even over thirty times its size, including Qwen3 8B, a 3.54B LFM model, and a 120B-parameter GPT-OSS model.
On “Track B,” held-out benchmarks outside its specialist training domain, the picture flips. GPT-OSS and Qwen3 8B pull ahead, landing in the middle of the pack while Tool LLM 3 Pro falls behind. The takeaway is straightforward: this is a small model that wins decisively on its home turf of formal logic but doesn’t generalize to broader reasoning or knowledge tasks the way larger general-purpose models do.
Where does the model break down?
Two additional stress tests revealed a consistent weakness. One test used invented chemistry rules (so the model couldn’t rely on memorized science) paired with eight questions requiring multi-step deduction. It scored 4 out of 8, handling straightforward forward-chain deduction well but answering “cannot tell” on every question that required working backward from a negative fact. Rather than getting those wrong, it gave up before completing the deduction.
A second test used invented rules for a masked ball scenario (who is wearing a mask, who isn’t, based on invented social rules) with eight multi-step deduction questions. The model scored 5 out of 8, again nailing the forward-chaining questions but struggling with backward chains. In one case it correctly declined to answer when the evidence was genuinely insufficient. In another, it got the answer wrong by assuming a rule worked in both directions when it was only defined to work one way (“everyone who waltzes wears a rose” doesn’t mean everyone wearing a rose waltzes).
The pattern across both tests: strong forward reasoning, weak backward reasoning, and a tendency to either bail out with “cannot tell” or incorrectly reverse a one-directional rule rather than push through a harder deduction.
Is Tool LLM 3 Pro worth running locally?
For a narrow use case, yes. The model runs locally through llama.cpp at around 9GB of VRAM, which fits on a single consumer or prosumer GPU and keeps the entire workflow on-device, useful for privacy-sensitive document review or air-gapped environments. For teams that specifically need to check whether a conclusion follows from a set of rules, or need to scan specs, contracts, and policy documents for internal contradictions, it performs well above its size class.
It is not a general-purpose assistant. The developer’s own licensing reflects this: it’s positioned for research and non-commercial use, with a separate commercial agreement required for revenue-generating deployment. It also isn’t competitive with larger closed models or general open models like GLM variants on broad reasoning, coding, or everyday productivity tasks, and testing confirmed it shouldn’t be pushed outside its lane. Within formal logic and contradiction detection specifically, though, it punches well above its 3.66B parameter count.
Frequently Asked Questions
What is Tool LLM 3 Pro built on?
It’s built on IBM’s Granite 4.2 architecture at roughly the 3B parameter scale, then specialized through post-training into a 3.66B parameter formal logic model.
How much VRAM does Tool LLM 3 Pro need to run?
Running locally through llama.cpp, the model consumed approximately 9GB of VRAM, making it feasible on a single consumer-grade GPU.
Is Tool LLM 3 Pro better than larger models like GPT-OSS or Qwen3?
Only on in-domain formal logic tasks, where it outperforms much larger models including a 120B GPT-OSS variant and Qwen3 8B. On general held-out benchmarks outside its specialty, those larger models perform better.
What are the best use cases for this model?
Checking whether a conclusion logically follows from stated rules, detecting contradictions in specifications or policy documents, and private on-device logic verification. It is not suited for coding, general chat, or productivity tasks.
Can Tool LLM 3 Pro be used commercially?
The license covers research and non-commercial use. Commercial, revenue-generating deployment requires a separate agreement with Web AI.
