ResearchPractical AIProjectsOpen SourceAbout
Research/Same Test, Run Seven Times: How Much of a Local Coding AI's Result Is Luck?
Local AICoding ModelsEvaluationRun-to-Run NoiseOriginal Research

Same Test, Run Seven Times: How Much of a Local Coding AI's Result Is Luck?

Readers pushed back on my coding-model test: three runs is too few, and where did the time actually go? So I ran the same ten jobs seven times on two models, then 17 new jobs, then a runaway case a third time. The 20-second speed gap I had reported turned out to be inside the run-to-run spread. The surprise was not speed: the faster model failed on writing new code, which my first ten jobs never tested.

Bart Porcincula · Published

Finding

Across 7 repeats of the same 10 bug-fixing tasks on one 16 GB card, gpt-oss-20b took a median 151 s (range 144 to 244 s) and Qwen3.5-9B 190 s (160 to 208 s), so the 20 s gap reported earlier sits inside the run-to-run spread; both made 62 tool calls (median), and Qwen's median step was 1.9 s against 0.7 s for gpt-oss; gpt-oss failed 2 of 70 task-runs (a wrong Roman-numeral case and a deleted module), so "10 of 10 every time" did not hold at 7 repeats; on 17 new tasks (12 write-new-code, 5 fix) over 3 runs, Qwen passed 51 of 51 and gpt-oss 45 of 51, with every gpt-oss failure on write-new-code; and the 390-call runaway seen once with Qwen3-Coder-30B did not repeat in 10 further runs (max 8 calls). Small samples, tasks written by me.

Noise bar and task-mix follow-up for small coding models on a 16 GB card (Oct 2026)

Stage A: the earlier ten bug-fixing tasks run 7 times each on gpt-oss-20b (MXFP4) and Qwen3.5-9B (Q4_K_M), same settings as before, temperature not lowered, with per-call logging taken from the assistant's saved event stream. Stage B: 17 new tasks (12 write-new-code, 5 fix), each verified to fail as shipped and pass with a reference fix, run 3 times per model. Stage C: one task (a wrong-operator fix) on Qwen3-Coder-30B (UD-Q2_K_XL) 5 times uncapped and 5 times with a 20-call cap. llama.cpp b11753, OpenCode, 32K context, 600 s per task, one RTX 5060 Ti; a pass is the project's own tests passing with test files unmodified.

MeasurementValueNote
Stage A: wall time per 10 tasks, gpt-oss-20b, 7 runs151 s median (144 to 244)IQR 146 to 188
Stage A: wall time per 10 tasks, Qwen3.5-9B, 7 runs190 s median (160 to 208)IQR 183 to 193; median gap +38.6 s, ranges overlap
Stage A: tool calls per run (median)62 vs 62range 58 to 102 for gpt-oss, 55 to 64 for Qwen
Stage A: median time per step0.7 s vs 1.9 sgpt-oss vs Qwen, 569 and 418 steps pooled; the gap is slower steps, not more calls
Stage A: tool errors3 and 2 in 70 task-runs eachthe earlier run's 24 / 2 / 8 did not repeat; errors moved between tasks
Stage A: tasks passed per rungpt-oss 9, 10, 10, 10, 10, 10, 9; Qwen 10 in all 7gpt-oss failures: Roman numerals (MCMXCIIII for MCMXCIV) and a deleted module
Stage B: pass per run of 17 new tasksgpt-oss 16, 14, 15; Qwen 17, 17, 17all gpt-oss failures were write-new-code; its 5 fix tasks passed 3 of 3
Stage B: write-new-code pass rate (12 tasks x 3 runs)Qwen 36/36, gpt-oss 30/36ranking by pass rate favours Qwen; by time favours gpt-oss
Stage B: Qwen wall time per run556, 388, 569 smostly one task (w03_parse_duration, median 218 s, 27 calls); gpt-oss 338, 295, 327 s
Stage C: Qwen3-Coder-30B on the task that ran away once10 of 10 passed, max 8 callsthe 390-call run did not repeat; a call cap of 20 was never reached, so it was not tested

The short version

In my test of small coding models I reported that gpt-oss-20b finished ten bug-fixing jobs in 156 seconds and Qwen3.5-9B in 176, both passing all ten. Readers raised the same objections on both: three runs is too few, what is the spread, where did the time go, and do ten jobs of one kind say anything about the job you actually do? They were right. So I wrote the plan down, ran it, and report it as it came out.

  • The 20-second gap is noise. Across seven runs the two models' time ranges overlap. gpt-oss was usually faster (median 151 s against 190 s), but one gpt-oss run took 244 s, and the spread alone is 100 s.
  • Where the time goes: same number of tool calls (62 each), so not more work. Qwen's steps are slower, 1.9 seconds each against 0.7.
  • "10 of 10 every time" did not hold. gpt-oss failed one task in 2 of 7 runs. I checked the saved work: a wrong Roman-numeral rule, and a module it deleted.
  • The task mix matters more than the speed. On 17 new jobs, 12 of them writing new code, Qwen passed all 51 attempts and gpt-oss 45. Every gpt-oss miss was a new-code job. My first ten jobs were all fix-existing-code, so they never saw this.

Takeaway: the faster model failed on the kind of work the first test did not cover. A speed gap of tens of seconds is not a reason to pick a model, and a perfect score on ten jobs of one kind is not a promise about another kind.

Stage A: the same ten jobs, seven times each

The settings were unchanged from the first test, and I did not lower the temperature. Randomness is the point, and a lower temperature is not how people use the tool.

gpt-oss-20bQwen3.5-9B
Time per 10 tasks, median (range)151 s (144 to 244)190 s (160 to 208)
Tool calls, median (range)62 (58 to 102)62 (55 to 64)
Tool errors in 70 task-runs3, all one run of one task2, in two different tasks
Tasks passed per run9, 10, 10, 10, 10, 10, 910 in every run

I had written a hypothesis before the runs: spread in wall time of at least 20 seconds, so that the reported gap would be inside it. That held. Errors, which I had reported as 24, 2 and 8 across three runs of a different model, did not repeat here: they were rare and they moved from task to task, so they do not belong to a particular job.

Stage B: seventeen new jobs

I wrote 12 jobs that ask the model to write new code (a least-recently-used cache, a minimum-value stack, a duration parser, a topological sort and so on) and 5 more fix jobs. Each was checked before use: it fails as shipped and passes with a reference fix, and the tests are unchanged.

gpt-oss-20bQwen3.5-9B
Passed per run (of 17)16, 14, 1517, 17, 17
Time per run338, 295, 327 s556, 388, 569 s
Tool calls, median (range)118 (106 to 123)128 (115 to 153)

gpt-oss missed jobs it had to write from scratch: a missing import, a syntax error from bad indentation, a parser that did not reject bad input, a sort that raised a false cycle error, a wrong text-case split. All five fix jobs passed three of three, and every failure was a real code error, not a vague prompt.

Qwen's longer time is mostly one job. The duration parser took a median 218 seconds, and every other Qwen job took between 14 and 32. Reading the transcripts, it tried a regular expression and probed it with many small test commands. Remove that one job and the two models are at 306 s and 323 s. So the speed result of this stage, in a sample of three runs and one outlier job, should not be read as a ranking.

Stage C: the runaway that did not repeat

In the first test, Qwen3-Coder-30B fixed one job and then kept calling tools for 390 calls until the 600-second timeout. I ran that job ten more times, five normally and five with a limit of 20 calls. All ten passed in about 18 seconds with at most 8 calls. So it happened once in eleven runs. I will not say the model "runs away". I will say it happened once and I could not reproduce it.

The call-limit question is unanswered: the limit of 20 was never reached, so it changed nothing. A fair test needs a case that really triggers the loop.

What I would take from it

  1. Report the middle and the spread, not one run. A 20-second gap on three runs was a coin flip.
  2. Test the kind of work you do. Ten fix jobs said nothing about writing new code.
  3. Read the failures. Both gpt-oss failures were real mistakes in the saved code, which only shows up when someone opens the files.

Limits

  • Seven and three runs are small samples. I report medians and ranges, not confidence intervals.
  • I wrote the stage B and C tasks and their tests. They are mine, so they are not comparable to a published benchmark, and an author's bias in the prompts is possible.
  • The 17 tasks add up to 27 in all, so whether the ranking holds on a much larger set is only partly answered.
  • Two models in stages A and B, one in stage C, one quantisation each, one assistant, one card.
  • Stage C's second question is unanswered because the loop did not appear.

The finding, stated for citation

On one 16 GB card, the reported 20-second time gap between gpt-oss-20b and Qwen3.5-9B on ten bug-fixing tasks sits inside seven-run noise (medians 151 s and 190 s, ranges overlapping, 62 tool calls each, Qwen's steps about 2.7 times slower); gpt-oss failed 2 of 70 task-runs at seven repeats; on 17 new tasks Qwen passed 51 of 51 and gpt-oss 45 of 51, with every gpt-oss failure on write-new-code; and Qwen3-Coder-30B's one 390-call runaway did not repeat in 10 further runs.

Cite this research

Bart Porcincula. "Same Test, Run Seven Times: How Much of a Local Coding AI's Result Is Luck?." Published 2026-10-28; updated 2026-10-28. https://porsync.com/research/same-test-twice-run-to-run-noise-small-coding-models

Permanent article URL