A Small AI Follows a Stated Rule 100% of the Time and the Same Rule Only Implied 24% of the Time
Part 3 of 5. In part 2 a rule stated outright was followed every time. Here the same six rules are given only as a reason (for example, the importer rejects commas in amounts) instead of an order. The model followed them about a quarter of the time, at every context length, and a pre-step that asks it to list the standing rules did not close the gap.
Bart Porcincula · Published · Updated
Finding
Qwen3.5-9B followed a rule stated outright on 100% of probes and the same rule implied by a stated reason on 24% at 4K (26% at 16K and 32K), a gap of 76 points (95% interval 62 to 88) that did not grow with length; a pre-pass in which the 9B listed the standing rules scored 21% at 4K (a change of -2 points, interval includes zero), and a 27B model's list scored 33% (+10 points, interval also includes zero).
Stated vs implied rules, with and without a rules pre-pass (Oct 2026)
The six rule types from part 2, each given either as an order (stated, S) or as a reason from which the rule follows (implied, I). Two further arms add a pre-pass in which a model first lists the standing rules from the conversation (9B: I+X; 27B: I+X27), and the list is added to the context. Lengths 4K, 16K and 32K; 42 probes per arm per length; paired bootstrap, 10,000 draws. Qwen3.5-9B Q4_K_M as the worker; the 27B pre-pass used Qwen3.6-27B UD-IQ3_XXS (a different quantisation and cache type).
| Measurement | Value | Note |
|---|---|---|
| Stated rule followed (4K / 16K / 32K) | 100% / 100% / 100% | |
| Implied rule followed (4K / 16K / 32K) | 24% / 26% / 26% | 95% intervals 12-38%, 14-40%, 14-40% |
| Implied gap, stated minus implied (4K) | 76 points (62 to 88) | 74 points at 16K and at 32K |
| Implied rule, with a 9B-written rules list (4K / 16K / 32K) | 21% / 19% / 17% | gain over no list -2, -7, -10 points; intervals include zero |
| Implied rule, with a 27B-written rules list (4K / 16K / 32K) | 33% / 33% / 31% | gain +10, +7, +5 points; intervals include zero; recovers 6-13% of the gap |
| Per rule, implied: ISO date / 24-hour time / dollar format | 71% / 52% / 29% | |
| Per rule, implied: metric units / no asterisks / snake_case | 0% / 0% / 0% | never followed on any arm |
| Pre-pass lists that captured the implied rule (manual read of 30 9B lists) | about 30% | a keyword check found 16% |
| Mean time: 9B pre-pass vs the probe it precedes | 1.9 s vs 7.1 s | the 27B pre-pass averaged 24 s |
The short version
People give rules to an AI two ways: as orders ("write amounts without commas") and as reasons ("the importer rejects amounts with commas"). A person would treat the second as the first. I checked whether a small model does. All five use the same setup: the open Qwen3.5-9B model compressed to 4 bits, run through llama.cpp on one 16 GB RTX 5060 Ti, at temperature 0 (the setting that makes a model as repeatable as it gets), with its optional "thinking" mode off. Each experiment's questions, thresholds and analysis were written down before it was run.
- Stated rule: followed 100% of the time. Implied rule: followed 24% at 4K. The gap is 76 points.
- The gap does not grow with length. It is 74 points at 16K and at 32K. Length is not what breaks it.
- A "list the standing rules first" pre-step did not fix it. The 9B's own list made no measurable difference (21% at 4K, down from 24%). A 27B model's list got to 33%, a change that sits inside the noise.
- Half of the six implied rules were never followed on any arm. Metric units, no asterisks and snake_case scored 0%.
Takeaway: if a constraint matters to a small model's output, say it as an order, not as background. A model that follows an instruction perfectly may not infer the instruction from the reason behind it, and giving it a second pass to infer it did not help.
How "implied" was done
Each rule was written two ways. Stated: an order to follow the rule. Implied: a sentence giving a reason from which the rule follows, for example that a downstream tool handles these values as plain text or that an importer rejects commas in amounts, with no instruction to the model. The probe task is the same either way, and the outputs are checked by code. The stated arm reuses part 2's rows, because they use the same contexts and seeds.
Results
| Arm | 4K | 16K | 32K |
|---|---|---|---|
| Stated | 100% | 100% | 100% |
| Implied | 24% | 26% | 26% |
| Implied + 9B rules list | 21% | 19% | 17% |
| Implied + 27B rules list | 33% | 33% | 31% |
Per rule (implied, all lengths pooled): ISO date 71%, 24-hour time 52%, dollar format 29%, and 0% for metric units, no asterisks and snake_case.
My five pre-set predictions went like this: the stated and implied arms would be within 10 points at 4K (refuted, 76 apart); the gap would grow by 15 points or more by 32K (refuted, flat); the 9B's own list would recover at least half the gap at 32K (refuted, it went negative); the 27B's list would beat the 9B's (point estimates agree, but neither is distinguishable from no list at all); and the 9B pre-pass would cost less time than the probe (true: 1.9 s against 7.1 s, but the 27B pre-pass takes 24 s, longer than the probe).
Why the pre-pass failed
A keyword check says the lists name the implied rule 16% of the time. I read 30 of the 9B lists myself and found the rule captured in about 9 of them (30%): dollar format 3 of 3, ISO date 2 of 2, 24-hour time in 3 of the ones where a timetable note was present, snake_case 1 of 6, and no asterisks and metric units 0. The model mostly restates the format of the task ("reply with the number only") and skips the constraint that was only ever given as a reason. So the failure is earlier than the probe: the model does not turn a stated reason into a rule, with or without help.
The 27B's gain is nearly all dollar format: it writes the importer's constraint into the list, and the 9B then follows it. That is a real effect, but it is one rule of six.
What I would not claim
- ISO date is a weak probe (part 2: the model writes ISO dates by default in 5 to 6 of 7 seeds), so its 71% is partly default behaviour.
- The 27B pre-pass ran on a different quantisation and cache type from the 9B arms. Take its comparison as indicative.
- One pre-pass prompt, written once. A different wording could do better. This tests that prompt.
- Forty-two probes per cell, 21 per rule per arm. The per-rule numbers are coarse.
- The manual audit is one reader, 30 lists, 9B only.
Part 3 of 5. Previous: does a small model stop following a rule as the context grows. Next: paste the whole folder or retrieve the relevant notes.
How should I write company instructions for a local AI assistant?
State the required action directly, then test it on examples where the rule matters. In this controlled study, explanations that merely implied the rule were followed far less often than explicit instructions. Perfect performance on the explicit probes is not a guarantee for an entire company handbook or a different model.
Cite this research
Bart Porcincula. "A Small AI Follows a Stated Rule 100% of the Time and the Same Rule Only Implied 24% of the Time." Published 2026-10-30; updated 2026-11-08. https://porsync.com/research/ai-follows-stated-rules-but-not-implied-ones
Permanent article URL