---
title: "NDA clause model"
url: https://sharadbapat.com/experiments/nda-clause-model/
description: "A 184M-parameter model that reads an NDA, rules on 17 standard clauses and quotes its evidence. Tested once on a sealed set: 84.5% accuracy."
published: 2026-09-04
author: Sharad Bapat
---

# NDA clause model

Can a small model read a non-disclosure agreement and say, for 17 standard clauses, whether the contract includes each one, rules it out or never mentions it, and quote the sentences that show it? Trained on public contracts, then tested once on a sealed set it had never seen: 84.5% accuracy.

- **Made:** Sep 2026
- **Written by:** Claude Code, directed by me
- **Built with:** DeBERTa-v3-base, PyTorch, one rented GPU
- **Status:** Concluded; runs locally on a CPU

## The question

Reading an NDA is mostly checking the same things every time. Can the other side share it with advisers? Must it be returned at the end? Does anything survive termination? I wanted to know whether a model small enough to run on a laptop, with nothing sent to a cloud service, could do that checking reliably, and show its working.

## How it works

- **The data.** [ContractNLI](https://stanfordnlp.github.io/contract-nli/) (Koreeda and Manning, 2021; CC BY 4.0): 607 real NDAs from public sources, each annotated for 17 fixed clause hypotheses, split 423 for training, 61 for development and 123 for test. The split is the published one, not redrawn.
- **Three answers per clause.** For each hypothesis ("Receiving Party may share some Confidential Information with some third-parties", for example) the answer is Entailment (the contract says so), Contradiction (it says the opposite) or NotMentioned.
- **Evidence, not just a verdict.** The model has two heads on one DeBERTa-v3-base backbone: one gives the three-way verdict, the other picks which sentences support it. Long contracts are packed into 512-token windows.
- **The rare class, weighted.** Contradiction is only 11% of the labels, so the loss weights it up rather than oversampling the evaluation data.
- **Small and local.** 184.4 million parameters, 703 MB on disk. Training ran for a few hours on one rented GPU, a few dollars in all. Inference runs on a CPU.

## The sealed test

The 123 test contracts were never touched while the model was being built. Every choice was made on the development set. At the end, all four candidate checkpoints were run on the test set together, in one pass, and all four are reported below, so the winner can't have been picked after seeing the results. The one adopted, seed 9, was already the front-runner on the development set.

- **84.5%**: verdicts correct, 2,091 clause checks
- **0.741**: F1 for the quoted evidence
- **0.788**: macro F1 across the three answers
- **31%**: the same checks by an off-the-shelf NLI model, untuned

The comparison is a general-purpose NLI model (MiniLM) used as it comes, with no training on contracts, on a 340-row development sample. The test score was also slightly higher than the development score (0.836 accuracy, 0.763 macro F1), so choosing a checkpoint on the development set didn't overfit it.

| Candidate | Accuracy | Macro F1 | Evidence F1 |
| --- | --- | --- | --- |
| Seed 2 | 0.828 | 0.769 | 0.670 |
| Seed 7 | 0.797 | 0.721 | 0.559 |
| Seed 8 | 0.817 | 0.771 | 0.725 |
| **Seed 9** (adopted) | **0.845** | **0.788** | **0.741** |

## Where it's strong, and where it isn't

| Answer | Precision | Recall | F1 |
| --- | --- | --- | --- |
| Entailment | 0.824 | 0.934 | 0.876 |
| NotMentioned | 0.924 | 0.810 | 0.863 |
| Contradiction | 0.650 | 0.600 | 0.624 |

Contradiction, the rare class, is the weakest, and it's also the one that matters most in practice: it's where a contract says the opposite of what you'd expect. Clause by clause, accuracy runs from 100% for "no reverse engineering" to 57% for "may keep some confidential information after returning it".

| Clause | Accuracy | 95% interval |
| --- | --- | --- |
| No reverse engineering | 1.000 | 0.970–1.000 |
| No solicitation | 0.967 | 0.919–0.987 |
| May develop similar information | 0.959 | 0.908–0.983 |
| May acquire similar information | 0.943 | 0.887–0.972 |
| Notice on compelled disclosure | 0.943 | 0.887–0.972 |
| No licensing | 0.919 | 0.857–0.955 |
| Confidentiality of the agreement itself | 0.902 | 0.837–0.943 |
| Limited use | 0.894 | 0.828–0.937 |
| Sharing with employees | 0.854 | 0.781–0.905 |
| Includes verbally conveyed information | 0.837 | 0.762–0.892 |
| Survival of obligations | 0.837 | 0.762–0.892 |
| Return of confidential information | 0.805 | 0.726–0.865 |
| Only technical information | 0.797 | 0.717–0.858 |
| May make copies | 0.748 | 0.665–0.816 |
| Must be explicitly identified | 0.724 | 0.639–0.795 |
| Sharing with third parties | 0.667 | 0.579–0.744 |
| May keep some after return or destruction | 0.569 | 0.481–0.653 |

## What didn't work

- **Training wasn't stable at first.** Before the random seed was fixed, runs sometimes collapsed into predicting one answer for everything. With seeding, three of four runs succeeded; the fourth still collapsed. The model is sensitive to its starting point.
- **One clause found a shortcut.** An early model answered Contradiction for every one of the 61 development contracts on "may keep some after return", whatever they said. It had learned that clause's most common training answer, which happens to be the rare answer in development and test. That early model scored 20% on this clause in development. The adopted one scores 57% in the sealed test, and it is still the weakest clause.
- **Shrinking it failed, twice.** Standard 8-bit dynamic quantisation cut accuracy on a development sample from 83% to 49%. Quantising only the backbone made it worse: 32%, against 82% for the full model on the same sample. It also saved less than it promises: the model only got 1.5 times smaller, because the 128,000-word vocabulary table (98 million of the 184 million parameters) isn't quantised at all. The likely cause is DeBERTa's disentangled attention, which sums several separate projections so small errors compound, but that wasn't verified. The model ships at full size.
- **I had the size wrong.** I described it as 86 million parameters for most of the project, a figure that leaves out the vocabulary table. Measured from the checkpoint, it's 184.4 million.

## Limits

- The contracts are public ones, from documents found by web search and from securities filings, and all in English. Private deal paperwork may look different.
- It checks 17 fixed clauses, not anything you might ask about a contract.
- The local demo splits sentences with a simple rule, which is rougher than the dataset's own annotation.
- It is a reading aid, not legal advice.