ResearchPractical AIProjectsOpen SourceAbout
Research/A Small Model Recovered More Details When a Different Model Wrote Its Summary
Local AILong ContextSummarizationEvaluationOriginal Research

A Small Model Recovered More Details When a Different Model Wrote Its Summary

The same Qwen3.5-9B answered 17.5% of detail questions from its own short structured summary and 77.5% from Qwen3.6-27B’s. Asking the 9B to write a longer summary raised its score to 78.8%.

Bart Porcincula · Published · Updated

Finding

On 240 matched probes per condition, Qwen3.5-9B passed 42 from its own short structured summary, 186 from the 27B writer’s short summary, 189 from its own longer summary, 209 from the 27B writer’s longer summary and 211 from the full transcript.

Summary writer and answer model replication

Forty synthetic work logs and six fixed detail probes per log. Three local models produced 360 summaries. Seven tested writer and answer-model pairings across three summary conditions, plus three full-transcript baselines, yielded 5,760 scored answers. The deployments were Qwen3.5-9B Q4_K_M, Qwen3.6-27B UD-IQ3_XXS and Gemma 4 12B Q4_K_M on llama.cpp b11753.

MeasurementValueNote
9B answering from its own short / longer structured summary42 / 189 of 240
9B answering from the 27B short / longer structured summary186 / 209 of 240
9B answering from the full transcript211 of 240
Gemma answering from its own short / longer structured summary166 / 166 of 240

Hold the answer model fixed and change what it gets to read

I gave local models 40 synthetic work logs containing details such as exact identifiers, values and constraints. After replacing a log with a summary, I asked six questions that could be scored against the original. The follow-up varied who wrote the summary and who answered from it.

What Qwen3.5-9B receivedPasses / 240Pass rate
Its own short structured summary4217.5%
Qwen3.6-27B’s short structured summary18677.5%
Its own longer structured summary18978.8%
Qwen3.6-27B’s longer structured summary20987.1%
Full transcript21187.9%

The short-summary writer swap gained 60 percentage points. A paired bootstrap that resampled whole work logs gave a 95% interval of 49.6 to 69.6 points. Swapping the answer model to 27B while keeping the 9B short summaries raised the score from 42 to 53 out of 240.

A larger summary budget helped some pairings

The short structured condition capped the response at 900 tokens. The longer condition asked for 900–1,200 tokens with an emphasis on exact details and raised the cap to 1,500. The 9B writer averaged about 197 tokens in the short condition and 1,485 in the longer one. The change combines the effect of the new instruction and the higher cap.

Gemma answered 166 of 240 probes correctly from its own summaries in each condition, even as its average summary grew from about 295 to 1,473 tokens. The 9B’s score of 209 on the longer 27B summaries was two answers below its full-transcript score on this sample. That gap alone cannot establish equivalence.

Scope and records

The test used synthetic work logs, exact-detail probes and a partial cross of three quantized deployments. It also included a freeform summary condition. The Gemma run used a streamlined text-only chat template, checked against the native template on representative inputs. The original context-summary article describes the earlier self-summary comparison. Original context-summary study

The replication record includes all tested pairings, summary lengths, detail survival and stream-paired comparisons. Context-summary replication evidence

The public release notes identify included summaries and configurations, excluded raw inputs and responses, and reproduction limits.

Cite this research

Bart Porcincula. "A Small Model Recovered More Details When a Different Model Wrote Its Summary." Published 2026-11-09; updated 2026-11-16. https://porsync.com/research/local-ai-summary-writer-detail-recovery

Permanent article URL