Tutorials

Claude Code Overlooked $800,000 in Contracts, and Here Is How to Check Yours

Claude Code missed $800,000 and a deadline due to messy contract data. Check linked amendments, renewal clauses, and preprocessing before AI review.

Key Takeaways

  • Claude Code produced plausible but incorrect contract answers because the underlying document structure was incomplete, not because the model simply guessed.
  • The system missed an $800,000 contract value difference by failing to connect an amendment to its parent agreement.
  • It also reported the wrong action deadline by relying on a termination clause instead of deriving the earlier date from an automatic renewal clause.
  • Common contract-data traps include orphan amendments, restated totals that can be double-counted, and preprocessing failures such as timeouts or broken page tables.
  • A CUAD-based check shows these issues are measurable: many contracts have automatic renewal with short notice periods, amendments can appear without parent documents, and “amended and restated” language requires careful handling.

Meta description: Claude Code missed an $800,000 total and a renewal deadline in AlixPartners' contract test. See why messy contract data fools AI agents and how to check.

Tags: Claude Code, contract data, AlixPartners, AI agents

Featured image: Abstract friendly illustration of a contract pipeline with a hidden renewal clock, no real people, no logos, and no text.

For teams feeding contract files to Claude Code, an AlixPartners talk is a warning: a plausible answer can still be wrong when the underlying document structure is incomplete. On November 8, the AI Engineer channel posted a session by Natalia Connolly and Kevin Madura of AlixPartners about a healthcare company buying a smaller one and needing to review more than 40,000 contract documents. Using a synthetic replica of the deal, the speakers showed how Claude Code produced answers that looked finished, then checked those answers against the data warehouse. Spend and contract counts were correct. Two important results were not: a total contract value was off by about $800,000, and an action deadline was a month late.

The answers looked complete

The failure was not a missing number. The total contract value was wrong by about $800,000 because Claude Code could not find an amendment that had never been linked to its parent contract. The second error was a deadline. The agent reported December 1 as the date to act. The correct date was November 1, because the contract renews by itself after that point. The agent took the date from a termination clause instead of deriving it from the renewal clause.

The talk frames the problem as a data-structure problem, not a model failure alone. If an amendment sits in the same repository but is not connected to the agreement it changes, a system reading only linked records can produce a clean total that is still wrong. If a renewal clause contains the real timing trap, a system reading only the termination date can miss the earlier notice deadline.

Why messy contract data fooled Claude Code

Contract files often hide several traps. An orphan amendment can appear valid by itself while missing the parent agreement. A master agreement and two amendments may each state a total value. Adding them gives about $17 million, but the amendments restate the total, so the latest value is $7 million.

Preprocessing can also fail quietly. A long contract can hang, a timeout can be caught, and the document can never reach the database. A rate table can break across pages, with page two starting as bare numbers and no header. Even an access-denied error can reach the agent looking like a zero invoice.

The practical lesson is that a model can sound right while using an incomplete map of the data. Claude Code was not simply guessing; it was answering from what the pipeline gave it.

The script checks 510 CUAD contracts

The speakers also point to a dataset that makes the problem measurable: the Contract Understanding Atticus Dataset, or CUAD. It holds 510 contracts from public SEC filings, with 41 expert-labeled fields each, including "Renewal Term" and "Notice Period To Terminate Renewal." A contract that renews automatically unless notice is given has an act-by date earlier than its end date.

The dataset can be retrieved with:

`git clone --depth 1 https://github.com/TheAtticusProject/cuad.git`
`cd cuad && unzip -q data.zip`

The unzip step gives CUADv1.json, about 40 MB. The check script, contractcheck.py, uses the standard library only and does not call a model. The notice parser looks for a period such as "ninety (90) days" near a cue like "at least" or "prior." Months are counted as 30 days. The key setting to change is RENEWS_ON, which the script uses as a test end date.

Run it with:

`python3 contractcheck.py CUADv1.json`

Result 1: the deadline trap is common

The script loaded 510 contracts. It found 176 contracts with a renewal or extension clause. Of those, 115, or 23 percent, say "automatic." A further 91 contracts, 18 percent of all 510, also carry a notice period to stop renewal. The notice period parsed to days in 82 of those 91 cases.

The median notice period was 90 days, with a range of 15 to 365 days. The parsed notices fell into these bands: 21 were 1 to 30 days, 37 were 31 to 90 days, 22 were 91 to 180 days, and 2 were 181 to 400 days. Using the test date RENEWS_ON as January 1, 2027, a median 90-day notice means acting by October 3, 2026, not January 1, 2027.

Nine notice periods were unreadable, and five of those were redacted with asterisks in the filing. One readable example came from CENTRACKINTERNATIONALINC_10_29_1999-EX-1, where the parser found 15 days from "fifteen (15) days." The AlixPartners talk involved a 30-day miss, and 21 of the 82 readable periods in CUAD are 30 days or less. Without the cue, the parser would risk grabbing a term length instead of a notice period.

Result 2: orphan amendments can disappear

The CUAD sample is like a filtered export: it does not contain every parent document. The script found 9 amendments and identified 3 orphans: Columbia Laboratories, Cerus, and ADMA. For those filers, no parent document appears elsewhere in the set. If an agent is asked for their terms and only sees the amendment, the answer can look complete while missing the agreement it changes.

The script also notes that title-based checks undercount amendments. More amendments can hide under plain agreement names, so relying on a word like "amendment" in the title is not enough.

Result 3: restated documents can double-count spend

The script checked for "amended and restated." Four contracts put the phrase in the first 300 characters, meaning the document itself is replacing an earlier version. Fifteen contracts use the phrase within the first 3,000 characters. Eleven of those use it later, often citing other agreements, so a keyword hit alone is not proof.

This matters because a restated agreement can carry a new total. If both the earlier version and the restated version are in the same folder, a simple sum can count the same deal twice.

Result 4: coverage checks catch silent drops

The coverage check asks whether every input came out the other end. In CUAD, empty texts were 0, the longest contract was 338,211 characters, and 81 contracts exceeded 100,000 characters. If a preprocessing stage has a timeout, the longest documents are likely to be the ones that fail first.

The fix is counting: documents in must equal rows out. The article also suggests a small companion file, mine.py, for your own pipeline. In the example, input_ids lists every file fed in, such as c1, c2, and c3, while rows lists what came out, such as c1 with spend of $120,000 and c3 with spend of 0. The missing set then reveals c2. That is the basic guard against silent drops.

The AlixPartners fix: structure the data first

The talk does not stop at the failure. The proposed fix is to structure the data before asking Claude Code to answer. There are five stages: optical character recognition, classifying each document as an agreement, amendment, or non-disclosure agreement, extracting terms, rates, and spend, mapping to spend categories, and only then running Claude Code with custom tools such as "get rate card page" and business rules built in.

The speakers say the pipeline took two weeks to build. After that, more than 40,000 contracts ran overnight, and every answer could be checked in code. That is the key difference between a fast answer and a defensible one: the model may write the summary, but the structured data layer decides what the model is allowed to see.

What readers should watch for

If you use AI agents on contract data, the useful check is not whether the answer sounds complete. It is whether the pipeline can prove that the right documents, clauses, versions, and coverage rows were included. Start with the same four questions: Are there auto-renewing contracts with notice periods that make the real deadline earlier than the end date? Are amendments linked to parent contracts? Are restated totals replacing older totals rather than adding to them? And do input counts match output counts after preprocessing?

The AlixPartners case is valuable because the errors were not exotic. A missing parent link, a termination date read instead of a renewal date, and a restated total can all look tidy to an agent. The script over 510 CUAD contracts shows why: 18 percent of the sample already contains the deadline trap, and the rest of the checks reveal similar structural risks. Claude Code looked right, but the data was not ready.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.