Can You Trust AI to Pull Details From Documents?
Bart Porcincula · Published
AI-extracted document fields need source checking. In my missing-field contract test, the small model supplied a value in 6.3% of cases where the answer was absent from the excerpt; this is not an error rate for all document answers.
Suppose you ask an AI assistant to find a renewal date in a contract. A date appears in the output, the spreadsheet cell is filled, and the job looks complete.
There are two things to check: whether that is the right date, and whether the source actually gives a renewal date at all.
My research tested the second problem. I asked a small local model to extract five kinds of field from short contract excerpts. In 315 cases, the requested answer was absent from the excerpt. The model supplied a value anyway in 20 of those cases: 6.3%.
That rate applies to missing fields in this experiment. It is not a claim that 6.3% of all AI document answers are invented.
Asking it to leave blanks did not solve the problem
I tried explicitly allowing the model to omit an answer. Its invention rate was 6.7%. I also required either a quoted source or an empty answer. That produced a 6.0% invention rate, while making the model worse at finding fields that were present.
Those small differences did not demonstrate an improvement. A simple check that the quoted words appeared in the document caught none of the 19 inventions in the quote-required condition. Words can appear in the source and still fail to support the value being claimed.
For a business owner, the practical question is whether the proposed answer is supported by the relevant part of the document. A filled cell or an attached quotation does not settle that question.
Training changed the trade-off
In a separate experiment, I trained a small model to find contract clauses and reproduce them word for word. Compared with an untrained version reading the input in the same way, it made fewer false claims and reproduced text more faithfully. It also missed more of the clauses that were there. Its overall score was essentially unchanged.
Training made the model more careful in that test. It did not make it uniformly better. If missing a renewal clause matters more than reviewing an extra false alarm, that trade-off may be unsuitable for your workflow.
What a useful review process could look like
For a first trial, choose a small set of fields and keep the original document beside the proposed answers. Allow a field to remain unresolved. Have the reviewer record both wrong values and values the model missed.
Include documents where the field is absent, documents with several competing dates, and examples with different layouts. If your work involves scans, test scans: the experiments described here do not establish the quality of reading images or handwriting.
Measure review time as well as extraction accuracy. The workflow may be useful if checking a draft record is faster than building it from scratch. It may be a poor fit if someone has to reread every page to establish whether each suggestion is sound.
AI can prepare document details for review. These results give us a reason to test that role carefully before relying on an automatically completed record.
A date can be real and still be the wrong answer
Imagine a service agreement that shows a signature date and a date for the first invoice, but gives no fixed renewal date. This is an illustrative example. An extraction assistant might select one of the visible dates and put it into a “renewal date” column. Every character could come from the document while the completed field is still unsupported.
That is why asking for a quotation is useful for review but insufficient as proof. The reviewer needs to establish what the quoted passage means for the particular field. A source match checks that words exist; it does not establish the relationship between those words and the answer.
The missing-field experiment directly tested that distinction. The quote check failed to catch inventions in the tested condition. It would be a mistake to market a source-linked answer as verified solely because the source contains the quoted words.
Decide what each field is allowed to mean
Before commissioning extraction, write a short definition for every requested field. For an invoice, does “total” mean the amount before tax, the amount after tax, or the amount still payable after a credit? For a contract, are you asking for an explicit expiration date or a date someone would need to calculate from other terms?
Separate copying a value from interpreting or calculating one. You may choose to allow an explicit total to be copied while leaving a calculated renewal date for a reviewer. If a calculation is part of the job, specify which inputs and rule support it, and show those inputs in the review output.
Also distinguish “not found in the supplied text” from “does not exist in the full document.” If only an excerpt is processed, the assistant has not examined the remaining pages. A useful output makes that boundary visible instead of treating an incomplete input as a complete search.
These definitions are business decisions. A technical person can implement them, but cannot reliably infer which meaning your staff expect from an ambiguous spreadsheet heading.
Design the review around the field
A reviewer should be able to inspect a proposed value, its source location, and its status together. For a first version, ask for separate states such as proposed, unresolved, and approved. Retain the original file and record any correction rather than overwriting the evidence with a polished answer.
Do not let a believable-looking confidence label substitute for checking whether the answer is right. If the implementer proposes using a score to decide which records skip review, ask them to demonstrate the actual error rate among the records that would pass. A score printed by a system is not, by itself, a measured probability of correctness.
Give more attention to fields whose mistakes create more work or trigger consequential action. A misspelled description and an incorrect payment amount deserve separate reporting. This article's experiments do not establish that any particular financial or contractual decision can safely be automated.
A practical extraction brief
Here is an example scope an owner could hand to an implementer:
- Document scope: one named document type, including the layouts and scan quality the business actually receives.
- Field definitions: a short description of each requested value, with examples of acceptable answers and unresolved cases.
- Evidence: the original document and a usable page or passage reference for each proposal.
- Review: staff approve each record; unsupported values remain unresolved and can be assigned for follow-up.
- Evaluation: report wrong values, missed present values, invented absent values, and unreadable inputs separately.
- Deliverable: an export of approved records plus an error log that staff can inspect.
Ask for the results on examples the implementer did not use to adjust the instructions. Include documents with no answer, several plausible answers, and values spread across pages. A demonstration containing only documents with an obvious answer tests a smaller problem than your business may have.
Measure whether checking is easier than entering
The commercial question is whether the draft makes the full task quicker at an acceptable standard. Time staff entering a sample manually, then time staff reviewing proposed records of similar difficulty. Include opening the source, resolving uncertainty, and correcting mistakes.
If the review requires rereading the entire document for every field, the extraction output may contribute little. If it reliably brings the relevant passage into view and leaves uncertain cases clearly marked, it may support a useful workflow. That remains a trial outcome to measure on your documents. The testing guide describes how to agree those conditions before accepting an implementation.
Frequently asked questions
Can I trust AI to extract invoice or contract fields without checking?
Treat extracted fields as proposals and compare them with the source before using them. A plausible value can be wrong or supplied when the document does not contain it.
What should happen when a document does not contain a requested field?
Mark the field as missing or unresolved and preserve a link to the source. Do not treat a plausible guess as a confirmed value.
How often does AI invent missing document fields?
In one small-model contract test, it supplied a value in 6.3% of cases where the answer was absent from the excerpt. That figure is specific to the test and is not an error rate for all document extraction.
Research behind this article
- Missing-field extraction study: short excerpts from a contract dataset, five fields, one small model.
- Contract-model training study: a separate experiment on clause extraction. Its results should not be combined with the missing-field experiment into one accuracy figure.
Cite this guide
Bart Porcincula. "Can You Trust AI to Pull Details From Documents?." Published 2026-11-08. https://porsync.com/ai-for-business/document-details