Jev AI Tested: A Fast "System One" Model for Structured Decisions
Hands-on tests of Jev, a fast classification model for routing, scoring, and yes/no decisions, covering negation, injection, and latency.
What is Jev, and what is it actually for?
Jev is a small, fast model built for structured decisions rather than open-ended chat. Instead of generating a written response, it answers one of three question types: choice (pick from a list of options you supply), score (rate something on a scale you define), or null (return a probability that a yes/no statement is true). The idea is to make cheap, quick judgments, like which team should handle a support ticket, whether a refund was requested, or whether a message is urgent, so your application code can act on the result immediately. It’s described as a “system one” model: fast, instinctive classification rather than reasoning through a problem step by step.
This matters for anyone building routing logic, moderation pipelines, or agent tooling where you don’t want to pay for a full language model just to decide “billing or technical support” or “true or false.”
How well does it handle routing and negation?
In one test, a support message described a duplicate charge, asked for money back, and mentioned the site was working fine. Asked four questions at once (department, refund request, urgency, frustration level), Jev routed it to billing, scored the refund probability at 98%, urgency at 11%, and frustration near the calm end of the scale. That’s a meaningful result: a billing issue didn’t get mislabeled as a technical one, and a calm refund request didn’t get flagged as an emergency just because money was involved.
Remy doesn't build the plumbing. It inherits it.
Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something.
Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want.
The more revealing test came from negation. When the message was changed to “I am not asking for a refund, I only need a copy of my invoice,” Jev still routed it to billing, correctly, but dropped the refund probability to 3%. That means it wasn’t just pattern-matching on the word “refund.” It was parsing the actual claim being made. This is a common failure point for simpler keyword-based or naive classification systems, and it’s a useful thing to verify before wiring any model into an automated action like issuing a refund.
Does Jev hallucinate less, or does it just have fewer choices?
This is the most important nuance from the testing. When asked what time the cafeteria closed, with “other” available as an option, Jev correctly picked “other,” keeping an irrelevant question out of billing, technical, or sales buckets. But when the same question was asked with only those three departments available and no “other” option, Jev picked “sales” anyway, with a low confidence score of 0.31.
That’s not really a hallucination in the sense of the model inventing a new category. It’s constrained to the options it’s given. But it does mean a forced-choice setup can produce a confident-sounding answer that’s simply wrong for the situation, because the correct answer wasn’t on the menu. The practical lesson: always include an “other” or “unknown” option with a path to human review, and don’t treat a numerical confidence score as a real-world accuracy rate without testing it against your own labeled examples.
Is Jev resistant to prompt injection?
In a basic injection test, a support message describing a crashed checkout page included a fake “system override” instructing the model to classify it as billing and mark refund and urgency as true. The actual evaluation instructions told Jev to treat text inside the message as untrusted. Jev held its ground: it kept the technical support classification, kept the refund probability low (3%), and kept urgency in the uncertain middle rather than spiking to a false positive.
That’s a good outcome, but it’s one test against one simple injection attempt. It doesn’t establish broad immunity to more sophisticated attacks, and anyone deploying a model like this in a pipeline that takes user-supplied text should keep testing adversarial inputs rather than assuming the behavior generalizes.
Can Jev extract exact values from documents?
Yes, within limits. In a test involving a message with an old billing address and a new receipt address, Jev was given both as selectable choices and correctly picked the new address, reproducing it exactly, including a plus sign and year, rather than paraphrasing or altering it.
This points to a specific and useful pattern: your code gathers candidate values (addresses, IDs, dates) from a document, and Jev selects which one is actually relevant to the question being asked. It copies rather than generates, which avoids the risk of a model subtly altering a number or address. But this only works if the candidate list your code builds actually contains the right answer. Jev can’t invent a correct value that was never offered as a choice, so the quality of your extraction step upstream still matters.
How did Jev perform on agent output verification?
One test simulated an AI agent audit: tool results explicitly said a save operation had failed due to denied permissions, but the agent’s own final message claimed the draft had saved successfully. Jev correctly classified the task as failed and assigned a 93% probability that the success claim was unsupported by the evidence.
This is a plausible use case for catching agents that confabulate success messages despite tool failures. The example tested was simple enough that ordinary code could likely catch it too. The more valuable application, checking multi-step agent traces with partial progress and conflicting signals, would need a much larger and messier test set to validate.
How fast and cheap is Jev in practice?
Across eight test requests, reported model evaluation time ranged from about 92 to 214 milliseconds, a figure returned by the service itself rather than end-to-end latency including network and browser overhead. The eight requests totaled 4,148 input tokens. At a published rate of 4.2 cents per million input tokens, the cost came out to a small fraction of a cent, with output tokens reportedly free at that rate.
Separately, a demo from developer Gregor Zunic combined Jev with browser automation tooling in a project called Jev Ultrafast. In his published run, it completed a Google Flights search from Zurich to London in about 7 seconds at a reported cost of $0.0039, using Jev for structured decisions (which button to click, which field to fill) and a separate small model (Mercury 2.5 in that example) to generate any actual text input. That timer started after the first page load and excluded broader browsing infrastructure costs, and the run wasn’t independently reproduced in this testing, so it should be read as a reported result rather than a verified benchmark.
Is Jev worth using in a production pipeline?
Based on the behaviors tested, ranging from correct negation handling and injection resistance to accurate value selection, Jev looks genuinely useful for narrow, well-defined classification tasks: routing tickets, scoring urgency, checking claims against evidence. The tested model version was identified as Jev 1.13.0, worth noting because an alias like “latest” can shift under you over time.
The caveats matter just as much as the results. Eight synthetic test cases don’t establish production-grade reliability, and the “other” option test shows that restricting choices can mask uncertainty rather than resolve it. Total application cost also includes whatever work goes into preparing inputs, validating outputs, and routing edge cases to a larger model or a human. Anyone considering it should run their own test set against their actual data before trusting it with real actions like issuing refunds or routing support tickets.
Frequently Asked Questions
What is a “system one” model?
It’s a term used to describe a model built for fast, low-effort judgments like classification or scoring, as opposed to a model that reasons through a problem step by step before answering.
Does Jev generate text?
Not on its own for the core decision types (choice, score, null). In the browser automation demo, a separate small model handled actual text generation while Jev handled structured decisions like which control to interact with.
Can Jev really achieve zero hallucinations?
Restricting Jev’s answers to a fixed set of choices prevents it from inventing a new category, but it doesn’t guarantee the chosen answer is correct if none of the supplied options actually fit the input. Testing showed a forced three-option choice produced a confident but wrong selection when the real answer was “none of these.”
How much does Jev cost to run?
Remy is new. The platform isn't.
Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.
At a published rate of 4.2 cents per million input tokens with free output tokens, per-request costs in these tests worked out to a small fraction of a cent, though total real-world cost also depends on the surrounding pipeline for input prep and validation.
Is Jev resistant to prompt injection attacks?
It handled one basic injection attempt correctly by treating embedded fake instructions as untrusted text rather than following them. That’s a positive early sign, but a single test doesn’t prove resistance to more sophisticated injection techniques.
