Latch

The question

When a model writes text for you, it decides what varies. I wanted to flip that: write down a contract of the facts that must not move, let the wording change as much as it likes, and check every output against the contract.

The locked parts are specific:

• the entities
• the values and their operators ("at least 40", not "about 40")
• who acts on whom
• scope
• whether something must or may happen
• the order of events

The question was whether that could be made materially more reliable than careful prompting alone.

How it works

It started as a generation harness. A contract goes to a model, which renders it many times, and a different model checks each rendering claim by claim. I ran the models through their command-line tools (Claude Code, Codex and Antigravity) rather than through an API.

That harness produced the first surprise. When two models disagreed about whether a claim survived, I read every disagreement by hand. One model, acting as the checker, flagged about ten times as many false violations as the other. It even flagged the unmodified reference text itself. A model marking another model's homework needed checking too.

So the work became a verifier. Given a reference and a rewrite, it tries to prove each locked dimension was kept, or prove it wasn't. When it can't prove either, it says so and abstains rather than guessing. Abstaining is allowed. A confident wrong "yes" is the failure that matters.

How I tested it

• The rules were frozen first. I fixed the metrics, gates and datasets, and checksummed them, before scoring any design. No design was ever patched after seeing its own results. Every fix became a new design, scored separately.
• One change per design. Each design changed exactly one thing from the one before, so every result has a single cause.
• The main number is the false-accept rate. Of the rewrites that broke a locked fact, it's the share the verifier passed. The gate applied to the upper end of the rate's 95% Wilson interval, not the point estimate: it had to be 1% or lower. A second gate required at least 70% of cases to be decided correctly, so abstaining on everything couldn't pass.
• The data.
  • 440 cases in three rounds of my own holdout sets.
  • 3,200 cases sampled from HANS and PAWS, two public benchmarks built to trick language models.
  • Two blind sets of 900 cases each, built later, set in two fictional domains the verifier had never seen.

The ladder

Rows A to C2 v2 are measured on my holdout sets. Rows marked "blind" are measured on the unseen sets. Design B fixed the templated role swaps in HANS but barely changed results on real prose. It also added twelve new false rejections, so it didn't earn its place.

The biggest single lesson is the jump from C to C2. The model and the representation were identical; only the rule for combining evidence changed. Under C2, the deterministic checks could reject a rewrite on their own but never accept one on their own, and every locked dimension needed positive proof. False accepts fell about tenfold. How you combine evidence mattered more than which representation you pick.

What didn't work

• No design cleared the safety gate on any test set. C2 v2 hit 1.0% on my holdouts, but the upper end of its interval was 3.55%.
• The first blind test didn't test anything. The verifier decided what was in scope by matching category names from my own benchmarks, so only 25 of 900 blind cases counted. C3 replaced that with a check based on meaning, and scope then generalised cleanly, to 895 of 900.
• One good result wasn't a trend. C3's 0.67% on the first blind set was the best number the project produced. On the second, independent set it was 1.34%, with an upper bound of 2.62%. Every miss traced to a limitation I had already written down:
  • sentences built on "is"
  • phrases about location and manner, which weren't checked
  • a missing check on which way a limit points, such as "at least" versus "at most"
  • pronouns that refer back to something earlier
• The last fix made it worse. C4 fixed exactly the case that motivated it. But it credited a dimension as proven when any single matching role proved it. False accepts rose four to five times, concentrated on the cases that matter most.
• Coverage never came close. Being careful meant abstaining on most cases. The best designs decided roughly 30 to 48% of cases correctly, against a bar of 70%.
• The model behind every good result was licensed for research only. Even a passing score couldn't have shipped as it was.

The pattern is the finding. Three designs in a row fixed the problem they targeted and exposed a new, equally serious one a layer deeper. That isn't a sequence converging on a working system. It's a reason to stop rather than build C5.

What's worth keeping

• The two blind datasets. Each case's correct answer was fixed by how the case was built, not by any verifier.
• The measurement method. False accepts are judged on the upper bound of their interval, and coverage is always reported beside correctness.
• A vocabulary for "locked meaning". It covers what the term can mean, and the difference between what a contract asks for and what a design can prove.
• The deterministic pieces: literal values, negation and date normalisation. They're reliable on their own and don't need a product around them.
• A precise list of the failure modes, for whoever tries this next.

When to reopen it

Only if something genuinely new appears. That could be a much better way to represent who does what to whom, or a way to prove preservation without recreating the evidence-combining problem that sank C4. I won't reopen it out of an obligation to finish what I started.

The most durable result was the discipline itself: letting the evidence kill an attractive idea instead of rescuing it.