ResearchPractical AIProjectsOpen SourceAbout
Research/The Same Coding Model Got Six Times as Many Incidents Right per Hour on a Different Engine
Local AIMixture of ExpertsThroughputReplicationOriginal Research

The Same Coding Model Got Six Times as Many Incidents Right per Hour on a Different Engine

An earlier test found a nearly sevenfold speed gap between two ways of running the same large model on one 16 GB card. I repeated the comparison on incident triage. Niko1221's Strata engine delivered 1,278 correct incidents per hour versus 208 for my original llama.cpp setup, with less than a point of observed accuracy difference. Tuning llama.cpp later narrowed the gap.

Bart Porcincula · Published

Finding

With Coder IQ1_M on the same first 762 synthetic incidents, Niko1221's Strata engine delivered 6.14 times the correct items per hour of stock llama.cpp configured with all experts on the CPU (95% paired interval 5.95 to 6.34). Accuracy was 82.68% versus 81.89%, a difference of +0.79 points (95% interval -0.26 to +1.84).

Same-model engine comparison on 762 synthetic incidents (Nov 2026)

Coder IQ1_M on llama.cpp b11753 and Niko1221's Strata engine 0.1.54 with the same prompt and item order, one request at a time and temperature zero. The matched set consists of the first 762 items, which stock completed in its three-hour window.

MeasurementValueNote
Correct per hour, stock / Niko1221's Strata engine208.0 / 1,277.6
Correct-per-hour ratio6.14×95% paired interval 5.95 to 6.34
Median time per incident12.55 s / 2.35 s
Accuracy difference+0.79 points95% interval -0.26 to +1.84

The short version

I had already seen a 6.8-fold gap on a batch of short tasks. I wanted to know whether it would hold when the work was reading a runbook and triaging an incident. I kept the model file, prompts and card the same and changed how I ran the model.

SetupAll fields rightCorrect incidents per hourMedian time
Stock llama.cpp, all experts on CPU81.89%208.012.55 s
Niko1221's Strata engine82.68%1,277.62.35 s

Stock finished 762 items in three hours. I compared those same 762 items on both sides, so a different mix of easy and hard incidents cannot explain the ratio. The observed accuracy gap was under one point, although the interval leaves room for a difference larger than that.

What the sixfold gap includes

I ran each configuration once and reused the earlier incident-triage rows for Niko1221's Strata engine. The uncertainty interval covers which incidents were sampled. Repeated runs would be needed to measure how much the setups vary over time. The faster setup also used multi-token prediction, which tries to propose several tokens ahead, while the original stock setup ran without it.

The comparison held up on a second task family against that original setup. In the tuned comparison, letting stock llama.cpp keep some experts on the GPU brought the gap down from 6.14 to 2.71 times.

Cite this research

Bart Porcincula. "The Same Coding Model Got Six Times as Many Incidents Right per Hour on a Different Engine." Published 2026-11-07; updated 2026-11-07. https://porsync.com/research/same-coder-two-engines-incident-triage-16gb

Permanent article URL