NDA clause model

Can a small model read a non-disclosure agreement and say, for 17 standard clauses, whether the contract includes each one, rules it out or never mentions it, and quote the sentences that show it? I trained one on public contracts and tested it once on a sealed set it had never seen. It scored 84.5%.

Made
Built with
DeBERTa-v3-base, PyTorch, one rented GPU
Status
Concluded; runs locally on a CPU

The question

Reading an NDA is mostly checking the same things every time. Can the other side share it with advisers? Must it be returned at the end? Does anything survive termination? I wanted to know whether a model small enough to run on a laptop, with nothing sent to a cloud service, could do that checking reliably, and show its working.

How it works

The data is ContractNLI (Koreeda and Manning, 2021; CC BY 4.0): 607 real NDAs from public sources, each annotated for 17 fixed clause hypotheses. I used the published split of 423 for training, 61 for development and 123 for test.

For each hypothesis ("Receiving Party may share some Confidential Information with some third-parties", for example) the answer is Entailment (the contract says so), Contradiction (it says the opposite) or NotMentioned.

The model has two heads on one DeBERTa-v3-base backbone. One gives the three-way verdict and the other picks which sentences support it, so every answer comes with its evidence. Long contracts are packed into 512-token windows. Contradiction is only 11% of the labels, so the loss gives it more weight; nothing is oversampled.

It's small: 184.4 million parameters, 703 MB on disk. Training ran for a few hours on one rented GPU, a few dollars in all, and inference runs on a CPU.

The sealed test

The 123 test contracts were never touched while the model was being built. Every choice was made on the development set. At the end, all four candidate checkpoints were run on the test set together, in one pass, and all four are reported below, so the winner can't have been picked after seeing the results. The one adopted, seed 9, was already the front-runner on the development set.

84.5%verdicts correct, 2,091 clause checks
0.741F1 for the quoted evidence
0.788macro F1 across the three answers
31%the same checks by an off-the-shelf NLI model, untuned

The comparison is a general-purpose NLI model (MiniLM) used as it comes, with no training on contracts, on a 340-row development sample. The test score was also slightly higher than the development score (0.836 accuracy, 0.763 macro F1), so choosing a checkpoint on the development set didn't overfit it.

All four candidates on the sealed test
CandidateAccuracyMacro F1Evidence F1
Seed 20.8280.7690.670
Seed 70.7970.7210.559
Seed 80.8170.7710.725
Seed 9 (adopted)0.8450.7880.741

By answer and by clause

Seed 9 on the sealed test, by answer
AnswerPrecisionRecallF1
Entailment0.8240.9340.876
NotMentioned0.9240.8100.863
Contradiction0.6500.6000.624

Contradiction, the rare class, is the weakest, and it's also the one that matters most in practice: it's where a contract says the opposite of what you'd expect. Clause by clause, accuracy runs from 100% for "no reverse engineering" to 57% for "may keep some confidential information after returning it".

All 17 clauses
ClauseAccuracy95% interval
No reverse engineering1.0000.970–1.000
No solicitation0.9670.919–0.987
May develop similar information0.9590.908–0.983
May acquire similar information0.9430.887–0.972
Notice on compelled disclosure0.9430.887–0.972
No licensing0.9190.857–0.955
Confidentiality of the agreement itself0.9020.837–0.943
Limited use0.8940.828–0.937
Sharing with employees0.8540.781–0.905
Includes verbally conveyed information0.8370.762–0.892
Survival of obligations0.8370.762–0.892
Return of confidential information0.8050.726–0.865
Only technical information0.7970.717–0.858
May make copies0.7480.665–0.816
Must be explicitly identified0.7240.639–0.795
Sharing with third parties0.6670.579–0.744
May keep some after return or destruction0.5690.481–0.653

What didn't work

Training wasn't stable at first. Before the random seed was fixed, runs sometimes collapsed into predicting one answer for everything. With seeding, three of four runs succeeded and the fourth still collapsed, so the model is sensitive to its starting point.

One clause found a shortcut. An early model answered Contradiction for every one of the 61 development contracts on "may keep some after return", whatever they said. It had learned that clause's most common training answer, which happens to be the rare answer in development and test. That early model scored 20% on this clause in development. The adopted one scores 57% in the sealed test, and it's still the weakest clause.

Shrinking it failed, twice. Standard 8-bit dynamic quantisation cut accuracy on a development sample from 83% to 49%. Quantising only the backbone made it worse: 32%, against 82% for the full model on the same sample. It also saved less than it promises. The model only got 1.5 times smaller, because the 128,000-word vocabulary table (98 million of the 184 million parameters) isn't quantised at all. My guess is DeBERTa's disentangled attention, which sums several separate projections so small errors compound, but I didn't verify that. The model ships at full size.

I also had the size wrong. For most of the project I described it as 86 million parameters, a figure that leaves out the vocabulary table. Measured from the checkpoint, it's 184.4 million.

Limits

The contracts are public ones, from documents found by web search and from securities filings, and all in English. Private deal paperwork may look different. It checks 17 fixed clauses and can't answer arbitrary questions about a contract. The local demo splits sentences with a simple rule, which is rougher than the dataset's own annotation. And its output isn't legal advice.