ResearchPractical AIProjectsOpen SourceAbout
Research/Reasoning on Two Model and Engine Setups: Coder on Strata Was Faster, While the Thinking 9B Was More Accurate
Local AIReasoningThinking ModelsMixture of ExpertsPre-RegisteredOriginal Research

Reasoning on Two Model and Engine Setups: Coder on Strata Was Faster, While the Thinking 9B Was More Accurate

I gave a 9B model with thinking turned on and Niko1221's Strata Coder with thinking off the same queue of reasoning puzzles for six hours each on one 16 GB graphics card. The Coder on Strata delivered about 11.9 times the correct answers per hour compared with the 9B on llama.cpp. Model, engine and thinking mode all changed together. The thinking 9B was 10.4 points more accurate, with the gap coming from constraint puzzles and log diagnosis. There was no difficulty level where thinking paid back its time.

Bart Porcincula · Published

Finding

Over a fixed 6 h window each, on an RTX 5060 Ti 16 GB at concurrency 1, on the first 633 items both arms finished, Niko1221's Strata Coder IQ1_M (thinking off) delivered 1,159 correct items per hour against 97.7 for Qwen3.5-9B Q4_K_M with thinking on: Coder/9B 11.85 (95% CI 10.13 to 13.74, paired bootstrap, 10,000 draws). This comparison changes engine and thinking mode as well as model. A later test on short business tasks found the same Coder 6.8 times faster on Strata than stock llama.cpp; it does not isolate the engine effect on these reasoning tasks. The 9B was more accurate, 93.05% against 82.62% (+10.43 points, CI +7.74 to +13.11). The Coder led on correct per hour at every tier of 2, 4, 6 and 8 reasoning steps (15.0 times down to 10.7 times), so there was no crossover. The 9B's median item took 14.9 s against 2.44 s, and 6.95% of its items hit the 16,000-token limit with an empty reply, counted as wrong.

Fixed 6-hour window, Qwen3.5-9B thinking vs Niko1221's Strata Coder IQ1_M direct, 16 GB VRAM (Nov 2026)

One run per arm, one machine, one day each. RTX 5060 Ti 16 GB, 31.6 GB RAM. 9B: llama.cpp b11753, Q4_K_M, 32K context, thinking on. Coder: Strata Terminal 0.1.54, IQ1_M, thinking off through a pass-through proxy. Same shuffled queue of 12,000 synthetic reasoning items (seed 0) in four families (word problems, table lookup chains, constraint puzzles, log-chain diagnosis) at 2, 4, 6 and 8 reasoning steps. Temperature 0, concurrency 1, 300 s cap, 16,000 max tokens, exact-match scoring. Hypotheses were written before the first scored run.

MeasurementValueNote
Items finished in 6 h9B thinking 633 vs Coder 7,752
Correct items per hour (first 633 items)9B 97.7 vs Coder 1,159
Coder / 9B-think correct per hour (headline)11.8595% CI 10.13 to 13.74
Accuracy (first 633 items)9B 93.05% vs Coder 82.62%9B +10.43 points, CI +7.74 to +13.11
Coder / 9B ratio by tier 2, 4, 6, 815.0, 11.2, 10.8, 10.7the Coder won at every tier
9B accuracy lead on log chains+29.7 points98.2% vs 68.5%
9B accuracy lead on puzzles+12.3 points77.9% vs 65.6%; the 9B managed 36 correct per hour here
Word problems and table chainsaccuracy tie97.3% vs 98.6% and 99.4% vs 100%
Median seconds per item9B 14.9 vs Coder 2.44p95 235 s vs 5.6 s
9B items cut at the 16,000-token limit6.95%all empty replies, counted wrong
Control: 9B with thinking off, same items31.0% accuracysecondary, 62.1 points below thinking on

The short version

The previous study gave both models short tasks with thinking off, and the 9B never needed the bigger model. This one asks the question that study could not: when the task needs reasoning, is it better to spend a fixed time budget on a small model that thinks, or on a bigger, slower model that answers directly? Both ran the same queue of synthetic reasoning items for six hours.

  • The giant model won on throughput. Niko1221's Strata Coder, answering directly, delivered 1,159 correct answers per hour. Qwen3.5-9B with thinking on delivered 97.7. The ratio was 11.85 (95% interval 10.13 to 13.74).
  • The thinking model was more accurate. 93.05% against 82.62%, a 10.4-point lead (interval +7.7 to +13.1).
  • There was no break-even difficulty. I expected harder items to eventually favour the 9B on correct per hour. They did not: the Coder led at 2, 4, 6 and 8 reasoning steps, by 15 times at the easiest and 10.7 times at the hardest.
  • Thinking was what made the 9B work at all. The same 9B with thinking off scored 31%.

Takeaway: if you are paying for time, a fast direct model beats a slow thinking one by roughly an order of magnitude here, even though the thinking model is more accurate. If a wrong answer is costly and time is not, the picture flips.

How it was run

Same card (RTX 5060 Ti, 16 GB), one run per arm, one day each, concurrency 1, temperature 0. The queue was 12,000 synthetic items in a fixed shuffled order, generated by my own seeded code so nothing is in a training set and every answer is exact: arithmetic word problems, table lookup chains, constraint puzzles with a unique solution, and log diagnosis where the cause is hidden behind a chain of linked entries with a decoy. Each family came in four difficulty tiers set by the number of reasoning steps. Scoring was exact match on a single answer line; an unparsable reply, a length cut, a timeout or an error counted as wrong and its time still counted. The headline compared the first 633 items both arms finished, with a paired bootstrap (10,000 draws, seed 0). The hypotheses, pool and software versions were fixed before the first scored item.

What the hypotheses said

HypothesisVerdict
The Coder has more correct per hour overallSupported (11.85)
On the easiest tier the Coder wins by at least 2 timesSupported (15.0 times)
On the hardest tier the 9B is at least 10 points more accurateSupported, narrowly (+10.3 points; the interval reaches below 10)
There is a crossover tier where the 9B wins on correct per hourRefuted (the Coder won every tier)
The 9B hits the 300 s cap on at least 2% of items, and its slowest 5% are at least 3 times its median, while the Coder's are under 2 timesRefuted (the 9B's tail was heavy, 15.8 times its median, but ended at the token limit and never at the 300 s cap; the Coder's was 2.28)

Where the 9B's accuracy came from

On word problems and table chains the two models were tied (97.3% against 98.6%, and 99.4% against 100%), so thinking only cost time there. The 9B's lead came from two families: constraint puzzles (77.9% against 65.6%) and log diagnosis (98.2% against 68.5%, a 30-point gap). Puzzles were also where thinking was most expensive, at 36 correct answers per hour, which is why the throughput ratio there was 52 times.

What thinking cost

The 9B used 2,332 completion tokens per item on average, against 118 for the Coder, and its median item took 14.9 seconds against 2.44. Its slowest 5% took 235 seconds or more. Nearly 7% of its items ran into the 16,000-token limit and returned nothing, so the thinking budget itself cost it roughly 7 points of accuracy it might otherwise have had. That limit was set before the run and not tuned after.

What I am not claiming

  • One run per arm, one machine, one day each. The bootstrap covers which items were sampled, not run-to-run noise.
  • The tasks are synthetic and English-only. Tier is a count of reasoning steps, not a measured human difficulty.
  • The Coder is a heavily compressed (IQ1_M) quantisation, so this is about that quantisation on that engine, not about the model family.
  • The 9B finished only 633 items, so the per-family-per-tier breakdowns rest on 31 to 53 items and I do not draw conclusions from them.
  • One request at a time. Batched serving, which could change the picture for both models, is the next study.

Cite this research

Bart Porcincula. "Reasoning on Two Model and Engine Setups: Coder on Strata Was Faster, While the Thinking 9B Was More Accurate." Published 2026-11-02; updated 2026-11-02. https://porsync.com/research/thinking-small-model-vs-direct-giant-model-fixed-six-hour-budget-reasoning-tasks

Permanent article URL