The AI Could Read the Answer. The Search Pipeline Had to Find It First
I tested a chatbot on 96 questions about a fictional knowledge base. Giving it the whole parent article after search raised accuracy from 78% to 89%; giving it the answer sections directly reached 99%.
Bart Porcincula · Published
Finding
On 96 questions over a synthetic HTML knowledge base, Claude Haiku 5.5 scored 78% with heading-based retrieval and 89% when the top three parent articles were included; the paired difference was 10.4 percentage points (95% bootstrap interval 3 to 19). Given gold sections, it scored 99%.
Synthetic knowledge-base retrieval pipeline comparison (November 2026)
96 questions over 58 fictional HTML articles, using one Haiku 5.5 configuration. Pipeline stages were frozen after a retrieval-only assessment and a 10-question pilot. Scores were checked by automated scorers after an author audit and corrections.
| Measurement | Value | Note |
|---|---|---|
| Raw HTML, fixed chunks | 58% | 96 questions |
| Cleaned text, fixed chunks | 70% | 96 questions |
| Heading chunks with path | 78% | 96 questions |
| Heading chunks plus parent articles | 89% | 96 questions; paired improvement 10.4 points, 95% interval 3 to 19 |
| Gold sections supplied | 99% | 96 questions; diagnostic, not a deployable search pipeline |
| Entire knowledge base in prompt | 94% | 48 questions, not directly comparable to the 96-question percentages |
The short version
A knowledge-base chatbot can only answer from the information its search finds. In this test, improving what search delivered mattered more than changing the model. A pipeline that retrieved heading-based chunks scored 78% on 96 questions. Expanding the top three distinct hits to their full parent articles raised that to 89%. When the correct sections were supplied directly, the same model scored 99%.
The knowledge base and its policies were fictional. This is one model and one synthetic dataset, not a promise of the same improvement on a real company's documents.
What I tested
I built 58 fictional HTML articles and 96 questions, including direct lookups, tables, qualifications, conflicting versions and questions the documents could not answer. Claude Haiku 5.5 answered using different versions of the same retrieval pipeline: search found relevant text, then the model used that text to answer.
The baseline used raw HTML and fixed-size chunks. Later versions cleaned the HTML, split by headings and added heading paths. The parent-expansion version searched the heading chunks, then supplied the full articles containing the top three distinct hits. Each stage changes the context the model receives; the comparison does not isolate every design choice independently.
Results
| Material given to the model | Accuracy | Questions |
|---|---|---|
| Raw HTML, fixed chunks | 58% | 96 |
| Cleaned text, fixed chunks | 70% | 96 |
| Heading chunks and paths | 78% | 96 |
| Heading chunks, expanded to parent articles | 89% | 96 |
| Correct sections supplied directly | 99% | 96 |
| Entire knowledge base in the prompt | 94% | 48 |
Parent expansion improved accuracy by 10.4 points; a paired bootstrap interval ran from 3 to 19 points. It helped with qualifications: the qualifier question class rose from 40% to 93%. Supplying the whole knowledge base used much more context; its apparent advantage over retrieval on the 48-question subset was not statistically clear.
Why a correct answer can be missing
Some fixed-size chunks ended mid-sentence. In several failures the model said the source did not contain enough information instead of inventing an answer. That is safer than fabrication, but it still leaves the user without an answer. A separate diagnostic that supplied the correct sections reached 99%, suggesting retrieval was a major source of errors in this particular test.
A second small test compared ways to update the index after articles changed. On 20 questions, replacing all chunks for each changed article scored 85%, versus 70% for a naive chunk update; the difference was concentrated in questions about deleted articles. That sample is too small to treat the difference as a general estimate.
Limits
The corpus was synthetic, cleaner and more consistent than a real site. The main arms used one model and one run each, with one repeat of the cleaned-text arm. The author audited about 30 answers, corrected two scorer defects and rescored the cached answers; this was not an independent audit. No reranker or managed search service was tested. The combined country and superseded-document filter was not tested on top of the parent-expansion arm.
The practical lesson is to inspect what search actually gives the model and test missing qualifications, stale documents and unanswered questions before assuming the model itself needs replacing.
Cite this research
Bart Porcincula. "The AI Could Read the Answer. The Search Pipeline Had to Find It First." Published 2026-11-13; updated 2026-11-13. https://porsync.com/research/knowledge-base-ai-retrieval-parent-article-expansion
Permanent article URL