ResearchPractical AIProjectsOpen SourceAbout
Research/Two Local Models Followed Every Written Rule and Missed Many Implied Ones
Local AILong ContextInstruction FollowingEvaluationOriginal Research

Two Local Models Followed Every Written Rule and Missed Many Implied Ones

Qwen3.6-27B and Gemma 4 12B passed all 252 explicit-rule checks across roughly 4K, 16K and 32K tokens. With the rules implied, Qwen passed 52–55% and Gemma passed 24–29%. Giving each model its own extracted rule list never raised its score.

Bart Porcincula · Published · Updated

Finding

At each of three context lengths, both models passed 42 of 42 explicit-rule probes. Qwen passed 23, 22 and 22 with implied rules and 19, 20 and 21 with a same-model extraction pass. Gemma passed 12, 10 and 11 with implied rules and 12, 8 and 10 with extraction.

Instruction-following replication of stated and implied rules

Six synthetic rule types crossed with seven seeds and three target context lengths. Two local model deployments, four scored conditions and one extraction call per case produced 1,260 calls. The run used Qwen3.6-27B UD-IQ3_XXS and Gemma 4 12B Q4_K_M on llama.cpp b11753 with one slot, 40,960-token capacity, q8_0 KV cache, temperature zero and reasoning disabled.

MeasurementValueNote
Explicit-rule passes across both models252 / 252
Qwen implied, then implied plus extraction23 / 22 / 22, then 19 / 20 / 21Each count is out of 42 at 4K, 16K and 32K, respectively.
Gemma implied, then implied plus extraction12 / 10 / 11, then 12 / 8 / 10Each count is out of 42 at 4K, 16K and 32K, respectively.

The gap was already there at 4K

I put a rule near the start of a synthetic work session, filled the middle with unrelated turns and tested the rule at the end. In one version the rule was written out. In another, the model had to infer it from a clue. The probe and its checker stayed the same. For example, the test checked whether an identifier came back in snake_case when the convention was either stated or implied.

Model and condition~4K~16K~32K
Qwen3.6-27B, stated424242
Qwen3.6-27B, implied232222
Qwen3.6-27B, implied plus extracted rules192021
Gemma 4 12B, stated424242
Gemma 4 12B, implied121011
Gemma 4 12B, implied plus extracted rules12810

Each cell is out of 42 probes. The original Qwen3.5-9B run also found a large gap at 4K. The two new deployments showed little change in their implied-rule scores from 4K to 32K.

Asking the model to list the rules added a call

The extraction pass asked the same model to list applicable rules from the implied history. I put its list directly before the probe and kept the original history available. Across all six model-and-length comparisons, the extra call produced five lower scores and one tie. The largest observed drop was four probes out of 42.

Scope and records

The replication used six synthetic rules and one quantized deployment per model. The checks score the output, and some formats can appear even with the neutral history. The results cover a same-model extraction pass. The original study records its separate cross-model rewrite test. Original stated-and-implied-rules study

The public record contains per-rule counts, paired intervals and model settings. Raw responses are not distributed in this release. Instruction-following replication evidence

The public release notes identify included summaries and configurations, excluded raw inputs and responses, and reproduction limits.

Cite this research

Bart Porcincula. "Two Local Models Followed Every Written Rule and Missed Many Implied Ones." Published 2026-11-09; updated 2026-11-16. https://porsync.com/research/local-ai-stated-vs-implied-rules-replication

Permanent article URL