Thomson-1 Benchmark: Can Thomson Reuters' AI Actually Review Contracts?
An independent test of Thomson Reuters' Thomson-1 model on NDA red flags, query sufficiency, and tax citation accuracy reveals how it handles ambiguity.
What is Thomson-1 and why does it matter?
Thomson-1 is an open weight language model built by Thomson Reuters, the company behind Westlaw and a large share of the legal research infrastructure lawyers use daily. It starts from Cohere’s open 35 billion parameter mixture of experts model and goes through a process Thomson Reuters calls continual learning: realigning the model’s values, injecting decades of proprietary legal, tax, and news data, and then training it on how legal and tax professionals actually work. That distinction matters because most “legal AI” products are general purpose chatbots with a legal prompt wrapped around them. Thomson-1 is trained specifically for the domain, and because it ships as open weights, anyone with enough hardware can run it locally instead of routing sensitive documents through a third-party API.
An independent test walked Thomson-1 through three scenarios that map to real risks in legal AI: whether it can catch planted problems in a contract, whether it knows when it doesn’t have enough information to answer, and whether it hallucinates when asked for citable facts. Each scenario targets a different failure mode that generic models are known for.
TL;DR
- A tester wrote a mutual NDA with seven deliberately planted red flags (a one-sided indemnity clause, a hidden global non-compete, and others), and Thomson-1 found all seven plus an eighth issue that wasn’t intentionally planted.
- When given a vague prompt with no contract attached (“Can I get out of this contract?”), the model refused to guess and instead asked six targeted clarifying questions, a behavior Thomson Reuters explicitly benchmarks as “query sufficiency.”
- On a factual question about current US capital gains tax rates, the model flagged its own knowledge cutoff, cited real IRC sections, reasoned through the TCJA sunset provisions instead of inventing a number, and self-corrected a mid-answer error.
- The model is built through a three stage pipeline: value alignment via constitutional DPO, knowledge injection through continued pre-training on proprietary legal and tax data merged back into the base model, and a behavior and agentic stage with two rounds of DPO plus two rounds of reinforcement learning, one specifically for end-to-end deep research.
- Locally, the model consumed roughly 87GB of VRAM during testing when served through vLLM, with a KV cache configuration pushing usage that high.
- The long clarifying-question response took about 3,300 tokens in 21 seconds, reflecting a model that reasons at length before committing to an answer, which is arguably appropriate for legal work.
- The underlying architecture is Cohere’s 35 billion parameter mixture of experts model, not a Thomson Reuters model built from scratch.
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
How was the NDA red-flag test set up?
The test used a mutual NDA written specifically for the evaluation, with seven red flags buried inside standard-looking contract language. These included a one-sided indemnity clause that shifts all risk onto one party and a global non-compete clause hidden inside language that wouldn’t obviously read as a non-compete on a fast pass. The instruction given to the model was simple: review the contract like a lawyer would and flag the riskiest clauses for the receiving party.
This kind of test mirrors what a junior associate or paralegal does on a first-pass review: skim a long document, catch the clauses that create outsized risk, and flag them before a senior lawyer spends billable time on the same document. Rushed reads are exactly where these clauses get missed in practice, and missed clauses are what turn into real financial exposure later.
What happened when Thomson-1 reviewed the contract?
The model identified all seven planted red flags and surfaced an eighth issue buried in the wording of an obligation clause that wasn’t intentionally planted by the tester. It produced a full walkthrough of its reasoning followed by a summary, structured the way a lawyer’s review memo might read. That combination, catching every planted issue plus something extra, is the kind of result that would be notable from any model, let alone one running fully on local hardware without calling out to a cloud API.
How does query sufficiency behavior work?
Query sufficiency is the term Thomson Reuters uses for a model’s ability to recognize when it doesn’t have enough information to give a responsible answer, rather than filling gaps with generic advice. It’s a direct response to a known failure mode in general purpose chatbots: when a user asks something ambiguous, a model optimized purely to sound helpful will often guess at the missing context and answer anyway.
The test for this was deliberately underspecified: a single line asking “Can I get out of this contract?” with no contract attached, no jurisdiction named, and no other context. Thomson-1 responded by asking six clarifying questions before offering anything specific. It declined to assume a jurisdiction or contract type, and it explicitly warned against unilaterally stopping performance on a contract without a valid legal basis, which is sound practical advice given how much liability can turn on that exact decision. The response ran long, about 3,300 tokens over 21 seconds, but for a domain where a wrong guess can cost real money, a slower, more thorough answer is arguably the correct tradeoff over a fast, confident, and possibly wrong one.
Does Thomson-1 hallucinate on factual legal questions?
- ✕a coding agent
- ✕no-code
- ✕vibe coding
- ✕a faster Cursor
The one that tells the coding agents what to build.
This is where legal AI tools most often fail in the real world: confidently citing a case, statute, or number that doesn’t exist. Thomson Reuters’ model card claims accurate citations and reduced hallucination through what it calls agentic deep research. To test the raw model’s training knowledge (without any external retrieval or tool use layered on top), the evaluation asked a factual, checkable question: current US capital gains tax rates, with a request to cite sources.
Thomson-1 flagged its own knowledge cutoff rather than asserting a current number with false confidence. It cited real IRC sections and reasoned through the sunset provisions in the Tax Cuts and Jobs Act (TCJA) instead of inventing a specific 2026 figure it couldn’t actually verify. Notably, it also caught and corrected its own error mid-answer regarding the pre-TCJA top tax rate. That kind of self-correction, visible in the model’s reasoning chain rather than hidden, is a meaningfully different behavior from models that state incorrect figures with the same confidence as correct ones.
What’s actually inside the Thomson-1 training pipeline?
Thomson Reuters describes a three-stage build process. The first stage, called value focus, realigns the open weight base model’s values using constitutional DPO (direct preference optimization) before any domain-specific knowledge gets added. The second stage, knowledge focus, uses continued pre-training to inject Thomson Reuters’ proprietary legal, tax, and news data into the model, then merges that back with the original weights to avoid degrading the model’s general reasoning capability. The third stage, behavior and agentic focus, applies two more rounds of DPO plus two rounds of reinforcement learning, including a stage specifically described as agentic end-to-end deep research trained against an actual research harness.
That third stage is the one that explains the hallucination-resistance seen in testing. Reduced hallucination here isn’t a prompting trick layered on top of a generic model. It’s the product of a dedicated reinforcement learning stage built around a research process, which is a structurally different approach than telling a chatbot to “always cite your sources” in a system prompt.
Is Thomson-1 worth running locally?
For teams already doing legal-domain fine-tuning, the practical answer that emerged from testing is that Thomson-1 may remove the need for further fine-tuning in most cases, with custom tuning reserved for unusual jurisdictions or highly client-specific requirements. Running it locally required a vLLM setup and around 87GB of VRAM in the tested configuration (with KV cache settings pushing usage that high), which puts it out of reach of consumer hardware but well within range of a single high-memory GPU server or a rented GPU instance. Because the weights are open, there’s no per-query API cost and no requirement to send confidential contract text to a third-party service, which matters for firms handling privileged material.
Frequently Asked Questions
What is Thomson-1?
Thomson-1 is an open weight AI model built by Thomson Reuters for legal and tax work. It’s based on Cohere’s 35 billion parameter mixture of experts model and further trained on Thomson Reuters’ proprietary legal, tax, and news data through a multi-stage alignment and reinforcement learning process.
What is query sufficiency in legal AI?
Query sufficiency refers to a model’s ability to recognize when a question lacks enough context to answer responsibly, and to ask clarifying questions instead of guessing. Thomson Reuters benchmarks this directly as part of Thomson-1’s evaluation, and it’s a meaningful safeguard against the generic-but-wrong answers that plague less specialized models.
Does Thomson-1 hallucinate legal citations?
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
In a tested example on capital gains tax rates, the model avoided hallucinating by flagging its knowledge cutoff, citing real IRC sections, and reasoning through applicable sunset provisions instead of inventing a number. It also self-corrected an error mid-response. This isn’t a guarantee against all hallucination, but it reflects the reinforcement learning stage Thomson Reuters built specifically for deep research behavior.
Can Thomson-1 run on consumer hardware?
Not easily. Testing showed VRAM consumption around 87GB when served through vLLM, which requires a high-memory GPU or a rented cloud GPU instance rather than a typical consumer graphics card.
How is Thomson-1 different from a general purpose model with a legal prompt?
Thomson-1 goes through continued pre-training on proprietary legal and tax data and dedicated reinforcement learning for agentic research behavior, rather than relying on prompt engineering alone. That structural difference is what showed up in testing as better red-flag detection, appropriate refusal to guess under ambiguity, and more disciplined citation behavior.