Does a Small AI Model Stop Following a Rule as the Conversation Gets Long? Not Up to 64,000 Tokens.
Part 2 of 5. A common worry is that a model forgets standing instructions as the context grows. I stated one rule once, buried it among filler tasks up to 64,000 tokens, and then tested whether the model followed it. It followed the rule every time at every length, wherever the rule sat. The catch, in part 3, is how the rule is phrased.
Bart Porcincula · Published
Finding
Qwen3.5-9B followed a once-stated rule on 42 of 42 probes at every context length from 4K to 64K tokens, quoted it back on 42 of 42, and showed no drop when the rule sat in the middle of the context instead of the start (also 42 of 42 at every length). With no failures in 42 contexts per cell, the true failure rate is bounded at roughly 7%, not zero.
Standing-rule use across context length, 4K to 64K (Oct 2026)
One standing rule stated once in a multi-turn chat of filler tasks, then a probe task the rule governs (use) or a question asking for the rule back (retrieval). Six rule types, 7 seeds, so 42 contexts per length, with identical contexts across arms. Lengths 4K, 8K, 16K, 32K and 64K tokens. Qwen3.5-9B Q4_K_M, f16 cache, temperature 0, thinking off. Extra arms: rule placed mid-context, a sanity arm of filler tasks alone, and a no-rule control at 4K and 16K.
| Measurement | Value | Note |
|---|---|---|
| Rule followed, 4K through 64K | 42 of 42 at every length | 100% at 4K, 8K, 16K, 32K and 64K |
| Rule quoted back when asked, 4K through 64K | 42 of 42 at every length | |
| Rule placed in the middle of the context, followed | 42 of 42 at every length | no start-versus-middle effect |
| Gap between using the rule and quoting it | 0 points at every length | |
| Plausible true failure rate given zero failures in 42 | up to about 7% | rule of three: 3 / 42 |
| No-rule control: probes passed without the rule stated | 0 of 7 for five of six rules | iso_date is the exception: 6 of 7 at 4K, 5 of 7 at 16K, because the model writes ISO dates by default |
| Sanity arm (filler tasks only) correct | 83% at 4K, 95% at 8K, 100% from 16K | all nine misses are the word-reversal filler task, a capability limit, not a context effect |
The short version
The usual story about long contexts is that models lose track of what you told them near the start. I tested that for one kind of instruction: a rule stated plainly, once. All five use the same setup: the open Qwen3.5-9B model compressed to 4 bits, run through llama.cpp on one 16 GB RTX 5060 Ti, at temperature 0 (the setting that makes a model as repeatable as it gets), with its optional "thinking" mode off. Each experiment's questions, thresholds and analysis were written down before it was run.
- No cliff to 64,000 tokens. The 9B followed the rule on all 42 probes at 4K, 8K, 16K, 32K and 64K.
- It could also quote the rule back every time, so the rule was not just applied, it was retrievable.
- Placing the rule in the middle changed nothing, so the "lost in the middle" effect did not show up here.
- The test was an easy case, and I say so below. A ceiling on stated rules does not say the same for rules you only imply. That is part 3, and it is where it breaks.
Takeaway: if you tell a small model a rule outright, context length up to 64K is not the reason it fails. Look at how the rule is stated before blaming the window.
The setup
Each test builds a chat of filler tasks (small text jobs the model does one after another) up to a target length, with one standing rule stated at the start or in the middle, for example a rule about date format or name style. At the end I ask for a task the rule governs and check the output with code, no human or AI judging.
| Arm | What it tests |
|---|---|
| U (use) | does the output obey the rule |
| R (retrieval) | can the model say what the rule was |
| U-pos | the same as U with the rule in the middle |
| S (sanity) | filler tasks alone, no rule, to check the filler is doable |
| N (control) | the probe with the rule turn removed |
Six rule types, seven random seeds each, so 42 contexts per length. The contexts are identical across arms, so the comparison is paired.
Results
| Length | Use | Retrieval | Use, rule in middle | Sanity |
|---|---|---|---|---|
| 4K | 100% | 100% | 100% | 83% |
| 8K | 100% | 100% | 100% | 95% |
| 16K | 100% | 100% | 100% | 100% |
| 32K | 100% | 100% | 100% | 100% |
| 64K | 100% | 100% | 100% | 100% |
My four pre-set predictions: that retrieval would stay at or above 90% to 32K (it did); that use would fall at least 15 points below retrieval at the longest length (it did not, the gap was zero); that a rule in the middle would be used less than one at the start (it was not); and that the 9B's cliff would come sooner than a larger model's (cannot be confirmed, since the 9B showed no cliff and I did not run the larger model).
Two things that temper the result
One of the six rules barely tests anything. In the control with the rule removed, five of the six probes failed every time (0 of 7), so they really do need the rule. The sixth, ISO date format, passed 5 or 6 of 7 with no rule at all, because the model writes ISO dates by default. So the real claim is 100% on five rules the model would otherwise break, plus one that is mostly default behaviour.
The sanity arm is below 100% at the shortest lengths (83% at 4K, 95% at 8K). All nine misses are one filler task, reversing a word, where the model mis-reverses letters. That is a limit of the model, not of the context, and it does not touch the rule results.
What I would not claim
- This is the easy case: one short rule, six simple rule types, outputs code can check, a model well inside its trained window of 262,144 tokens. It does not cover several competing rules, rules that need applying inside long reasoning, or rules you only imply (part 3).
- Forty-two contexts per cell with no failures bounds the failure rate at roughly 7%. It does not show it is zero.
- I did not run the 27B model here, because its prefill speed at the card's memory edge (part 1) makes a full grid take many hours, and there was no 9B cliff for it to explain.
- One run per cell at temperature 0. Another seed set could differ in a few contexts.
Part 2 of 5. Previous: one long session or a fresh start each step. Next: the same rule, only implied.
Cite this research
Bart Porcincula. "Does a Small AI Model Stop Following a Rule as the Conversation Gets Long? Not Up to 64,000 Tokens.." Published 2026-10-30; updated 2026-10-30. https://porsync.com/research/small-model-context-cliff-does-it-follow-a-stated-rule-at-64k
Permanent article URL