--- title: "Jev eval" url: https://sharadbapat.com/experiments/jev-eval/ description: "An evaluation of TypeSafe's Jev judgment model: 97.7% accurate, with one real flaw and a one-line fix. Give the model the rule, not just the request." published: 2026-09-22 author: Sharad Bapat --- # Jev eval TypeSafe's Jev gets almost everything right, until you ask it two related questions in the same call without telling it the rule to apply. Here's the evaluation, the one flaw that matters, and the one-line fix. ![An abstract interlocking geometric mark on a pale background of translucent cubes.](https://sharadbapat.com/experiments/jev-eval/cover.webp) *Image made with ChatGPT's image generator.* - **Made:** Sep 2026 - **Model tested:** jev-1.13.0 - **Scope:** 16 runs, 98 cases, 221 questions - **Accuracy:** 97.7% (173 of 177) ## What Jev is TypeSafe pitches Jev as "System One": fast, structured answers to narrow questions about a block of content, instead of the free-form text you get from a chat model. You give it a *state* (the text to judge) and one or more questions, each of one of three types: - **Noul**: a yes or no question. It returns a probability. - **Choice**: pick one option from a fixed list. It returns a probability for each option, plus a confidence score. - **Score**: a position on an ordered scale. It returns a number that can land between two levels. You can ask several questions about the same state in one call. TypeSafe's docs say up front that answers within a call aren't guaranteed to agree with each other. That caveat turns out to matter. ## How I tested it I used a small Python harness on TypeSafe's SDK, running against the live production API. Each case is a state plus questions with hand-written expected answers. Every numeric, date, hex and version-string answer was checked by hand before submission. The cases were built as traps for the weaknesses TypeSafe names in its own docs: numeric comparison, distractor text, references spread across a document, literal reading, and consistency between answers. A few were deliberately ambiguous to probe judgment calls; those are kept out of the accuracy number. No call failed for a technical reason in any run. *[Chart: Thirteen scored runs in order. Ten scored 100%. Run 2 scored 93%, run 10 scored 92% and run 12 scored 82%.]* *Accuracy for each of the 13 scored runs, numbered in the order they ran. Ten were perfect. Run 2's miss was the first sign of the contradiction below, run 10 missed a date calculation, and run 12's two misses were scores anticipating the next severity bucket.* | # | Run | Questions | Accuracy | | --- | --- | --- | --- | | 1 | `examples (canned demo)` | 9 | 100% | | 2 | `enterprise-messy-v1` | 15 | 93.3% | | 3 | `product-domain-v1` | 15 | 100% | | 4 | `harness-governance-v1` | 15 | 100% | | 5 | `literal-rules-v1` | 30 | 100% | | 6 | `explicit-vs-implicit-v1` | 16 | 100% | | 7 | `consistency-check-v1` | 20 | 100% | | 8 | `severity-calibration-v1` | 12 | 100% | | 9 | `injection-robustness-v1` | 7 | 100% | | 10 | `named-weakness-precision-v1` | 12 | 91.7% | | 11 | `injection-subtlety-v1` | 5 | 100% | | 12 | `tighter-monotonicity-v1` | 11 | 81.8% | | 13 | `explicit-vs-implicit-v2` | 10 | 100% | The realistic material held up well: an invoice with a duplicated freight line, a complaint buried under forwarded marketing copy, an expense report whose reimbursable total differs from the more visible one, and a plant-restart request with an expired safety certificate hidden behind an unrelated work order. It caught both of the sneakiest traps. One was a $305 expense that doesn't exceed a $300 cap once alcohol is excluded. The other was eight asset identifiers, each one character off a valid pattern. ## Two questions, one call, two answers This is the finding that matters. It started with a single miss. In one call, I asked a three-way routing question (do it, reason further, or escalate to a human) and a separate yes or no: is it fine to complete this without a human? The routing answer said go ahead. The yes or no said no. Same request, same call, a direct contradiction, and no rule had been given. So I tested it twice. First, eight fresh requests across eight unrelated domains, each with the same pair of questions and no rule. Six of the eight came back contradictory. On the five with a clear right answer, the routing choice was right every time and the yes or no was wrong every time, at 0.19 to 0.45: confidently wrong, not a coin flip. Then I resubmitted the exact same eight requests, word for word, with one sentence added to the state: > If the requester's message asks, directly or indirectly, to wait for review before the work counts as finished, stop and wait, even if the action itself is simple. - **2 of 8**: self-consistent pairs without the rule - **8 of 8**: with the rule, same requests - **50%**: accuracy on the clear-cut cases, before - **100%**: after, with answers at 0.66 to 0.90 Nothing else changed between the two runs, so this is about as clean a cause and effect as you get outside a lab. One disagreement remained, and on reflection my label was probably the weaker reading of an ambiguous sentence. Jev's revised answer may be the better call. It's the clearest case in the whole evaluation where the answer key, not the model, was wrong. The takeaway: don't ask Jev whether something is "appropriate", "safe" or needs human review without giving it the rule that defines that. Left to guess, it defaults to a cautious no, even in the same call where it correctly saw the request as simple. ## Confidence drops when the rule is implied Across thirteen matched pairs, the rule and the right answer stayed the same; only the phrasing changed, spelled out or implied. Every pair was more confident when the rule was spelled out: 0.947 on average, against 0.748 when implied. The answer itself never flipped, so this is a confidence effect only, but it isn't small. One implied request, an employee termination, came back at 0.51, one point above the usual decision threshold. If you gate autonomous action on confidence, naturally phrased requests will trip that gate far more often than ones that spell out the rule, with no change in correctness. ## Everything else - **Score keeps its order.** Two six-point severity series never inverted. Within a bucket the values barely moved, but clustered around a boundary they did: amounts just under a threshold scored 0.91 and 0.96, anticipating the next bucket. The detail sits where the decision is, which is arguably what triage needs. - **It ignored fake authorization.** Twelve attempts across two rounds tried to override a correct judgment from inside the content: bracketed approval codes, a forwarded email from a "finance lead", a fake Slack reply, a forged approval-history entry, a made-up meeting ID. None worked. - **Hex and version strings were fine.** Despite being a documented weakness, all eight comparisons came back right, including "2.10" against "2.9". - **Date math was the real miss.** For a 90-day term from January 15 (expiring April 15), it correctly called April 16 expired, but also called April 14 expired, at only 0.56. Plain date-format tests passed both ways, which isolates the weakness to counting days across months of different lengths. - **No self-hosted option is documented.** Only a hosted API. That's a hard limit for anything that must run on your own infrastructure, and worth confirming with TypeSafe directly. ## If you're integrating it - Put the governing rule in the state for any "is this appropriate, safe, or does it need a human" question. - Trust Score across buckets. Don't expect even detail within one; it's sharpest at the boundaries. - Compute deadlines and elapsed days in your own code. - Re-test resistance to embedded instructions before relying on it for anything security-sensitive. Twelve clean trials is a start, not a guarantee. ## Caveats Most questions were asked once. Only the explicit-versus-implicit pairs (13), the fake-authorization trials (12) and the Score series (4) were sampled repeatedly. A few answer keys are my own judgment rather than verifiable fact, and they're flagged where they appear. Everything here is one model version, jev-1.13.0, and shouldn't be assumed to carry over to later releases. I didn't benchmark against a general chat model, so none of this is a measured saving over an alternative. First published 22 September 2026.