ResearchPractical AIProjectsOpen SourceAbout
Research/The Quote Was Real. The Contract Answer Was Still Wrong.
Local AIContract ExtractionOutput ValidationEvaluationOriginal Research

The Quote Was Real. The Contract Answer Was Still Wrong.

A literal quote check caught zero of 49 answers on contract fields labeled absent across three local deployments. It also discarded seven answers that passed the recall scorer. The same payment-allocation passage fooled all three deployments on a governing-law question.

Bart Porcincula · Published · Updated

Finding

On the same 630 field–excerpt pairs, the quote check kept all 19 labeled-absent answers from Qwen3.5-9B, all 17 from Gemma 4 12B and all 13 from Qwen3.6-27B. Requiring a quote changed overlap recall by −6.7, +1.9 and −1.0 percentage points, respectively. Strict value match fell on all three deployments.

Contract-field extraction replication on CUAD contract excerpts

The original 630 field–excerpt pairs cover five contract fields, with 315 labeled present and 315 labeled absent. Gemma 4 12B Q4_K_M and Qwen3.6-27B UD-IQ3_XXS completed 3,780 requests through llama.cpp b11753. The historical Qwen3.5-9B Q4_K_M run used b11408. The comparison is between quantized deployments on the same cases.

MeasurementValueNote
Labeled-absent answers removed by quote check0 of 49 across three deployments
Overlap-correct answers removed by quote check7 across three deployments
Gemma overlap recall, baseline to quote requirement285 → 291 of 315
Qwen 27B overlap recall, baseline to quote requirement293 → 290 of 315

Three models cited the same payment clause

I asked three local deployments to extract the governing law from a contract excerpt. Each cited a passage that mentioned the Colombian Civil Code. The passage explained how to apply a payment when a debtor owes several debts. The excerpt gave no governing-law designation. All three quotes passed a check that looked for the quoted words in the excerpt.

The filter removed seven answers that passed recall scoring

I reran the contract-field extraction study on Gemma 4 12B and Qwen3.6-27B, using the same 630 field–excerpt pairs as the original Qwen3.5-9B run. The quote arm asked for verbatim evidence and a value, with an explicit marker for an unstated field. The offline filter kept non-null answers only when the evidence appeared in the excerpt after normalization.

DeploymentAnswers on labeled-absent fields, before → afterOverlap-correct answers, before → after
Qwen3.5-9B, original19 → 19275 → 272
Gemma 4 12B17 → 17291 → 287
Qwen3.6-27B13 → 13290 → 290

Each column uses 315 cases per deployment. The 49 labeled-absent answers are outputs across runs on a shared sample. The check verified where the words came from. The payment example shows how a genuine quote can still fail to support the field being extracted.

The original recall drop was larger

Relative to the baseline, requiring a quote cut overlap recall by 6.7 percentage points on the original 9B run. The same change was +1.9 points on Gemma and −1.0 point on the 27B. Both new estimates had 95% contract-clustered intervals spanning zero. The original stricter value-match score fell on all three deployments, by 15.2, 12.7 and 18.7 points respectively. That score uses normalized substring matching and can penalize longer values even when they share the expected words.

The frozen baseline already told the model to use null when the field was unstated. Its paired schema arm used the identical wording and enforced a nullable-value schema. Both new deployments had identical headline counts in those two arms. The quote arm changed the prompt, schema and way of signaling abstention together.

The results cover these deployments and this excerpt sample. Absence labels can be noisy, and I spot-checked semantic support on individual examples. Contract-field extraction replication report

Original contract-field extraction study

The public release notes identify included summaries and configurations, excluded raw inputs and responses, and reproduction limits.

Cite this research

Bart Porcincula. "The Quote Was Real. The Contract Answer Was Still Wrong.." Published 2026-11-09; updated 2026-11-16. https://porsync.com/research/local-ai-contract-extraction-quote-check-replication

Permanent article URL