ResearchPractical AIProjectsOpen SourceAbout
Research/What Does a Small AI Keep When It Summarises Its Own Work? Mostly the Dates.
EvaluationContext CompressionMemoryAgentsLocal AIPre-RegisteredOriginal Research

What Does a Small AI Keep When It Summarises Its Own Work? Mostly the Dates.

Part 5 of 5. When a conversation gets long, the standard fix is to have the model compress it. I planted six kinds of detail across 20 steps of work log, had the 9B compress its own context three ways, then asked for each detail back. Uncompressed it got 88%. A free summary got 3%, a structured template 20% and a copy-exactly template 33%. A files-plus-tool approach got 12%, because the model almost never opened a file.

Bart Porcincula · Published · Updated

Finding

Across 40 generated streams of 20 work-log steps (about 16K tokens), 6 planted details each and 1,440 probes, Qwen3.5-9B passed 88% with no compression, 3% after a free-form summary, 20% after a structured template, 33% after a copy-verbatim template, 12% with a file list and a read_file tool, and 27% with the file list plus a template; only the deadline date survived a template reliably (85 to 90%), and the model opened a file on 8.8% of probes.

Context compression vs pointers, 40 streams x 6 detail types x 6 arms (Oct 2026)

Each stream is 20 generated work-log steps (about 630 words, 16K tokens in total) with six planted details: a file path, a number, a prohibition, who decided something, a deadline date and an implied constraint. The context is either kept (NC), compressed once by the same 9B in one of three ways (free summary SF, structured template ST, template with copy-verbatim instruction SV), or replaced by a list of files with a read_file tool (P, at most 3 reads per question) or that list plus the template (P+ST). Each detail is then probed and checked by exact match. 1,440 probes. Qwen3.5-9B Q4_K_M, f16 cache, temperature 0, thinking off, paired bootstrap, 10,000 draws.

MeasurementValueNote
No compression: probes passed88% (83-92)100% on five detail types; the implied constraint 25%, matching part 3
Free-form summary (SF)3% (1-6)0% on path, number and decider
Structured template (ST)20% (15-26)deadline 85%, prohibition 20%, others 3-5%
Template with copy-verbatim instruction (SV)33% (28-39)+30 points over the free summary (24-36); deadline 90%, prohibition 42%
File list plus read_file tool (P)12% (8-16)75 points below no compression
File list plus template (P+ST)27% (22-33)
Share of probes where the model opened a file (P / P+ST)8.8% / 7.5%0 malformed tool calls; the tool worked when called
Detail present in the compressed text (ST / SV)path 3% / 12%; number 53% / 62%; deadline 85% / 90%
Probe passed when the detail survived (ST / SV)68% / 74%most of the loss happens at compression
Compressed size, mean (SF / ST / SV)142 / 158 / 278 tokensfrom about 16,000; under 2% of the context

The short version

Once a long conversation runs out of room, the usual fix is to have the model summarise what has happened and carry on from the summary. The risk is that the summary drops the one detail you later need. I measured how many details survive when a small model compresses its own context. All five use the same setup: the open Qwen3.5-9B model compressed to 4 bits, run through llama.cpp on one 16 GB RTX 5060 Ti, at temperature 0 (the setting that makes a model as repeatable as it gets), with its optional "thinking" mode off. Each experiment's questions, thresholds and analysis were written down before it was run.

  • Uncompressed, the model got 88%. After any compression I tried, far less. A free summary: 3%. A structured template: 20%. The same template told to copy exact strings: 33%.
  • The one thing that survives is the date. Deadline dates made it through the template 85 to 90% of the time. File paths: 3 to 12%. Names: 5 to 23%.
  • Most of the loss happens when the summary is written, not when it is used. When a detail does survive into the summary, the model uses it correctly 68 to 74% of the time.
  • Keeping the files and giving the model a tool to open them did not help, because it almost never used the tool. It opened a file on 8.8% of questions, and scored 12%.

Takeaway: do not assume a small model's summary of its own work keeps what matters. Test it by planting known details and asking for them back. If you cannot afford to lose specifics, store them outside the summary and make retrieval a step the system performs, not a choice the model has to make.

What I built

Twenty-step work logs, generated by a program, each about 630 words of plausible filler with six planted details: a file path, a number (a retry limit), a prohibition ("never restart the X service while Y is running"), who approved a plan, a deadline date, and an implied constraint. The logs are identical across arms, and every probe is checked by exact match.

ArmWhat the model has when asked
NCthe whole 16K-token log, uncompressed
SFa free-form summary the model wrote
STnotes in fixed sections (files, numbers, rules, decisions, dates, constraints, open items)
SVthe same sections, with "copy paths, numbers, names, dates and rules verbatim"
Pa list of files with one-line titles and a read_file tool (up to 3 reads per question)
P+STthe file list, the tool, and the template notes

Results

Armpathnumberprohibitiondeciderdeadlineimplied
No compression1.001.001.001.001.000.25
Free summary0.000.000.170.000.030.00
Template0.030.050.200.050.850.05
Template, verbatim0.120.170.420.230.900.15
Files + tool0.000.400.150.000.120.05
Files + template0.030.230.200.300.880.00

The compressed notes were very short: 142 tokens on average for the free summary, 158 for the template, 278 for the verbatim template, out of about 16,000. The model keeps what it judges salient (dates) and drops identifiers. Asking for verbatim copying helped by 30 points over a free summary, but 33% is still a long way below 88%.

The tool that did not get used

In the file-pointer arms the tool worked: zero malformed calls. But the model opened a file on only 8.8% of questions (7.5% when the template notes were also present). Without a read it answers from the file list, and fails. The one detail where it did better (number, 40%) is a short answer it may sometimes have guessed. I did not separate guessing from a single lucky read.

Pre-set predictions

Free-summary survival of paths and numbers under 70%: not refuted (0 and 5%). Verbatim beats free by 15 points or more: not refuted (+30). Implied constraints survive worst in every arm: refuted, since in some arms paths or names fare worse. Pointers within 10 points of no compression: refuted (75 points below). A 27B model as compressor helping the free summary more than the template: not tested.

What I would not claim

  • The model wrote very short compressions and I set no minimum length. This measures what a small model keeps by default, not what it could keep if told to write a thousand tokens. A longer-budget version was not part of this original run; the later follow-up below tests it.
  • The same 9B did the compressing and the answering. A larger compressor (the 27B) was not part of this original run, so this original comparison says nothing about whether the loss is about model size.
  • The implied-constraint column mostly shows the floor from part 3, because even uncompressed it is only 25%.
  • The pointer result depends on the model choosing to call a tool. A prompt or harness that forces a read would be a different design.
  • Synthetic, templated logs where the topic named in each question matches the log. Real logs are messier and may be harder or easier.

Follow-up published: The summary-writer comparison tests a 27B writer and longer summaries on 240 matched probes per condition. The 9B reader passed 42 probes from its own short structured summary, 186 from the 27B writer's short summary and 189 from its own longer summary. Those are separate follow-up conditions, not replacements for the original results above.

Part 5 of 5. Previous: paste the whole folder or retrieve the notes. Start of the series: one long session or a fresh start each step.

Can an AI summary serve as a reliable record of ongoing work?

A readable summary did not preserve most of the planted details in this experiment. Keep important identifiers, approved values, and decisions in explicit records and verify that the assistant reads them when needed. The measured retention rates describe this model and these generated work logs; they do not establish the memory quality of every AI product.

Cite this research

Bart Porcincula. "What Does a Small AI Keep When It Summarises Its Own Work? Mostly the Dates.." Published 2026-10-30; updated 2026-11-16. https://porsync.com/research/what-a-small-ai-keeps-when-it-summarises-its-own-context

Permanent article URL