NDA clause model

The question

Reading an NDA is mostly checking the same things every time. Can the other side share it with advisers? Must it be returned at the end? Does anything survive termination? I wanted to know whether a model small enough to run on a laptop, with nothing sent to a cloud service, could do that checking reliably, and show its working.

How it works

• The data. ContractNLI (Koreeda and Manning, 2021; CC BY 4.0): 607 real NDAs from public sources, each annotated for 17 fixed clause hypotheses, split 423 for training, 61 for development and 123 for test. The split is the published one, not redrawn.
• Three answers per clause. For each hypothesis ("Receiving Party may share some Confidential Information with some third-parties", for example) the answer is Entailment (the contract says so), Contradiction (it says the opposite) or NotMentioned.
• Evidence, not just a verdict. The model has two heads on one DeBERTa-v3-base backbone: one gives the three-way verdict, the other picks which sentences support it. Long contracts are packed into 512-token windows.
• The rare class, weighted. Contradiction is only 11% of the labels, so the loss weights it up rather than oversampling the evaluation data.
• Small and local. 184.4 million parameters, 703 MB on disk. Training ran for a few hours on one rented GPU, a few dollars in all. Inference runs on a CPU.

The sealed test

The 123 test contracts were never touched while the model was being built. Every choice was made on the development set. At the end, all four candidate checkpoints were run on the test set together, in one pass, and all four are reported below, so the winner can't have been picked after seeing the results. The one adopted, seed 9, was already the front-runner on the development set.

The comparison is a general-purpose NLI model (MiniLM) used as it comes, with no training on contracts, on a 340-row development sample. The test score was also slightly higher than the development score (0.836 accuracy, 0.763 macro F1), so choosing a checkpoint on the development set didn't overfit it.

Where it's strong, and where it isn't

Contradiction, the rare class, is the weakest, and it's also the one that matters most in practice: it's where a contract says the opposite of what you'd expect. Clause by clause, accuracy runs from 100% for "no reverse engineering" to 57% for "may keep some confidential information after returning it".

All 17 clauses

Clause	Accuracy	95% interval
No reverse engineering	1.000	0.970–1.000
No solicitation	0.967	0.919–0.987
May develop similar information	0.959	0.908–0.983
May acquire similar information	0.943	0.887–0.972
Notice on compelled disclosure	0.943	0.887–0.972
No licensing	0.919	0.857–0.955
Confidentiality of the agreement itself	0.902	0.837–0.943
Limited use	0.894	0.828–0.937
Sharing with employees	0.854	0.781–0.905
Includes verbally conveyed information	0.837	0.762–0.892
Survival of obligations	0.837	0.762–0.892
Return of confidential information	0.805	0.726–0.865
Only technical information	0.797	0.717–0.858
May make copies	0.748	0.665–0.816
Must be explicitly identified	0.724	0.639–0.795
Sharing with third parties	0.667	0.579–0.744
May keep some after return or destruction	0.569	0.481–0.653

What didn't work

• Training wasn't stable at first. Before the random seed was fixed, runs sometimes collapsed into predicting one answer for everything. With seeding, three of four runs succeeded; the fourth still collapsed. The model is sensitive to its starting point.
• One clause found a shortcut. An early model answered Contradiction for every one of the 61 development contracts on "may keep some after return", whatever they said. It had learned that clause's most common training answer, which happens to be the rare answer in development and test. That early model scored 20% on this clause in development. The adopted one scores 57% in the sealed test, and it is still the weakest clause.
• Shrinking it failed, twice. Standard 8-bit dynamic quantisation cut accuracy on a development sample from 83% to 49%. Quantising only the backbone made it worse: 32%, against 82% for the full model on the same sample. It also saved less than it promises: the model only got 1.5 times smaller, because the 128,000-word vocabulary table (98 million of the 184 million parameters) isn't quantised at all. The likely cause is DeBERTa's disentangled attention, which sums several separate projections so small errors compound, but that wasn't verified. The model ships at full size.
• I had the size wrong. I described it as 86 million parameters for most of the project, a figure that leaves out the vocabulary table. Measured from the checkpoint, it's 184.4 million.

Limits

• The contracts are public ones, from documents found by web search and from securities filings, and all in English. Private deal paperwork may look different.
• It checks 17 fixed clauses, not anything you might ask about a contract.
• The local demo splits sentences with a simple rule, which is rougher than the dataset's own annotation.
• It is a reading aid, not legal advice.