The Same Coding Model Improved When Its Harness Changed
A separate 288-attempt follow-up put Aider ahead on three models and both Patchsmith versions near ceiling on Qwen3.6. Harness changes helped some models, without establishing a general winner.
Bart Porcincula · Published
Finding
In the separate n09-n16 cohort, Patchsmith v5b/v6/Aider passed 13/19/23 of 24 for gpt-oss, 14/15/17 for Qwen3-Coder, 24/24/23 for Qwen3.6 and 18/17/23 for Edge0. Edge0 v5b is not blind; cohorts are never pooled. Two simultaneous v6 changes and unequal budgets prevent a causal harness ranking.
Patchsmith and Aider with the same local models (November 2026)
Independent-test outcomes on ten tuning fixtures repeated five times, four historical fixes repeated three times, and eight later fixtures repeated three times per arm. Later fixtures were informed by both arms' failures. Patchsmith v5b and the gpt-oss recovery were changed after inspecting results. Cohorts remain separate; quants and serving flags differ across models. The separate n09-n16 follow-up adds 288 saved attempts; curated rows and offline analysis are public.
| Measurement | Value | Note |
|---|---|---|
| Separate n09-n16: v5b / v6 / Aider, each out of 24 | gpt-oss 13/19/23; Coder 14/15/17; Qwen3.6 24/24/23; Edge0 18/17/23 | 288 attempts; Edge0 v5b not blind; cohorts never pooled |
| Edge0 tuning: v1 / v2 / v3 / Aider / v4 / v5 | 21 / 22 / 18 / 43 / 49 / 48 of 50 | tuning set, not generalisation; raw rows correct v3 from 19 to 18 |
| Edge0 replay, same arm order | 6 / 5 / 4 / 4 / 5 / 8 of 12 | four commits, too small to rank arms |
| Edge0 later fixtures: v5b / Aider | 20/24 / 20/24 | v5b post-hoc, not blind |
| Qwen3-Coder: Patchsmith / Aider | 19/24 / 14/24 | Q3_K_M, three attempts per fixture |
| gpt-oss: Patchsmith pre / post fix; Aider | 14/24 / 15/24; 22/24 | MXFP4; post-hoc fix, Aider not rerun; 8 then 6 HTTP 500 failures |
| Qwen3.6: Patchsmith / Aider | 24/24 / 23/24 | UD-IQ3_XXS; near ceiling; one attempt is noise |
A weak coding result can have more than one cause
When a local coding model fails to repair a small program, it is tempting to blame the model. But the software around it decides which files it sees, how it edits them, when it runs tests and when it must stop. That software is the harness.
Edge0 initially struggled in Patchsmith, my tool-call loop. Aider, an established coding harness, did much better on the same tuning tasks. After changes to Patchsmith, the gap narrowed. On eight later fixtures, Patchsmith v5b passed 20/24, matching Aider's 20/24. The Patchsmith result was not blind: I changed it after seeing its first failures on those fixtures.
Repeating the comparison on three other models made the conclusion more useful. Patchsmith was not consistently the weaker harness. It did encounter a substantial serving and output-format problem with gpt-oss.
What counted as a completed repair
Each attempt had to pass independent tests. A model saying it was finished did not count. Three workloads remain separate: ten synthetic maintenance tasks repeated five times, four historical fixes repeated three times, and eight later fixtures repeated three times. I do not pool these totals or combine them with frontier results.
Patchsmith asks the model to call tools to read, edit and test files. Aider v0.86.2 used whole-file edits, source supplied up front, tests marked read-only, automatic testing and no repository map. Patchsmith allowed 30 turns and 30,000 tokens of live context; Aider's fixture runner had a 600-second process timeout. These are whole harnesses with different prompts, stopping rules and editing interfaces, not a controlled test of format alone.
Patchsmith explicitly requested temperature zero and thinking off. Aider's wrapper described temperature zero but did not explicitly set it in the command, so sampling parity is not established.
Models and serving settings
The desktop had an RTX 5060 Ti 16 GB and about 32 GB of system RAM. Quantization compresses weights and can affect quality. Every comparison across models is confounded by these different formats and serving settings.
| Model | Weight format | Serving settings |
|---|---|---|
| Edge0-120B | q3k | edge0-infer, context 32,768, port 8000 |
| Qwen3-Coder-30B-A3B-Instruct | unsloth Q3_K_M, 14.71 GB | Vulkan llama-server, context 32,768, q8_0 K/V cache, 6 CPU MoE layers |
| gpt-oss-20b | ggml-org MXFP4, 12.11 GB | Vulkan llama-server, context 32,768; no extra KV/MoE launch flags; automatic fitting, four slots and 8,192 MiB prompt RAM cache in logs |
| Qwen3.6-35B-A3B | UD-IQ3_XXS, 13.21 GB | Vulkan llama-server, context 32,768, q8_0 K/V cache, 8 CPU MoE layers; low-bit quality risk |
The non-Edge0 launcher used port 8001, flash attention on, Jinja templates and 99 requested GPU layers. It was the native llama-server executable distributed by Docker. Effective gpt-oss KV dtype and the exact server build were not pinned in the saved launch record. This is not a ranking of the original models.
First, the harness learned the tuning tasks
| Harness | Tuning passes /50 | Historical-fix replay /12 |
|---|---|---|
| Patchsmith v1 | 21 | 6 |
| Patchsmith v2 | 22 | 5 |
| Patchsmith v3 | 18 | 4 |
| Aider | 43 | 4 |
| Patchsmith v4 | 49 | 5 |
| Patchsmith v5 | 48 | 8 |
The v3 result is corrected here: the handoff said 19/50, but its saved independent-test rows and generated table give 18/50.
Early changes gave Patchsmith a file listing, more room before repeated-call stops, warnings about excessive reading and detection of edits that left a file unchanged. Version 4 supplied source, tests and initial failing test output in the prompt. Version 5 made that preload smaller and more selective, shortened old tool output as context filled, and added a last redirect before stopping a repeated call.
Those descriptions come from the code. They do not isolate which change caused improvement. The ten tasks were the tuning set. The 48/50 result is not evidence of generalisation. The four replay commits are too few to rank the arms.
The runs used uncommitted harness changes on top of the live-context-budget implementation. A source commit was created later during this write-up; it does not retrospectively pin every run-time snapshot.
Then came the later fixtures
The eight later tasks covered token buckets, semantic versions, word wrapping, decimal ledgers, business days, LRU caches, template expansion and topological sorting. They were written after looking at both arms' failures, so their design was not independent of tuning.
Patchsmith v5 first passed 17/24. After changes, v5b passed 20/24, not blind, alongside Aider's 20/24. Equal totals hid different weaknesses: both struggled with the ledger; Patchsmith also failed business-day and cache attempts, while Aider timed out twice on topological sorting.
| Model | Patchsmith passes /24 | Aider passes /24 |
|---|---|---|
| Edge0 | 20, v5b, not blind | 20 |
| Qwen3-Coder | 19 | 14 |
| gpt-oss before fix | 14 | 22 |
| gpt-oss after post-hoc fix | 15 | 22, original control not rerun |
| Qwen3.6 | 24 | 23 |
Only three attempts were made per fixture. Differences of one attempt are noise, and Qwen3.6 is near the ceiling of this small set. Nothing here establishes that Patchsmith beats Aider.
The gpt-oss failures were different
Qwen3-Coder's failed Patchsmith runs spent 16-30 turns on semantic versions or topological sorting before blocking or hitting the turn limit. Server logs showed no parser exceptions, and saved traces described unsuccessful algorithms. Aider also failed those tasks. That is consistent with task-solving limits in this setup, rather than an observed transport fault. Patchsmith did solve all three template attempts where Aider solved none; that difference warrants investigation, not a general superiority claim.
gpt-oss had another problem. Before the fix, eight Patchsmith attempts ended in HTTP 500 errors from the output parser. Three involved a finish call sent on the wrong channel; five involved malformed JSON arguments. A narrow recovery change handled the finish-call case after those failures were inspected.
The new run still had six HTTP 500 failures: five code-writing arguments cut off inside a JSON string and one malformed channel/tool message. The result moved from 14/24 to 15/24, against Aider's original 22/24. That one-run improvement is not a reliable estimate of the fix's benefit. Other models were not rerun after the recovery change.
Aider's whole-file format avoids putting edited code inside a JSON argument. The pattern is consistent with a format cost on gpt-oss, but whole-harness differences prevent saying format alone explains the gap. A gpt-oss-only whole-file fallback is an untested follow-up; the data do not warrant a general fallback.
A separate follow-up changed the comparison
The n09-n16 follow-up used eight new synthetic tasks, three repeats, three arms and four models: 288 saved attempts, with no missing or duplicate keys. These tasks were held out from the observed earlier tuning, but the task author was not independent and this was not a preregistered evaluation. Edge0 v5b is not blind. Cohorts are never pooled with n01-n08, tuning, replay or frontier results.
| Model | Patchsmith v5b /24 | Patchsmith v6 /24 | Aider /24 |
|---|---|---|---|
| gpt-oss | 13 | 19 | 23 |
| Qwen3-Coder | 14 | 15 | 17 |
| Qwen3.6 | 24 | 24 | 23 |
| Edge0 | 18 | 17 | 23 |
Version 6 raised the completion cap from 2,048 to 4,096 tokens and withdrew read-only tools after five reads until an edit or test. Because both changed together, these results cannot isolate their effects. Patchsmith retained a 30-turn limit, 30k live-context budget and 420-second request timeout; Aider had a 600-second process timeout. Sampling parity remains unpinned, and cross-model quantization and backend differences remain confounds.
gpt-oss gained six passes under v6, while remaining below Aider in this cohort. Qwen3-Coder's earlier Patchsmith advantage did not replicate. Edge0 showed no v6 gain; Qwen3.6 remained near ceiling. Three repeats on eight tasks are too narrow for a general harness ranking.
The successful Edge0 batch followed a reported reboot and retained context 32,768. Two failed 12:43 startup launches and an earlier morning failure are operational exclusions, not failed task attempts. Surviving logs do not establish that RAM caused those startup failures or independently verify the reboot.
What would settle the remaining question
The initial Edge0 deficit did not persist after harness tuning. Harness reliability depends on the model and serving setup, and changing the surrounding software can change what the same model completes.
The next comparison needs frozen source hashes, pinned serving and sampling settings, genuinely unseen tasks, counterbalanced arm order and more distinct repairs. A causal format claim requires an isolated format change. A broken OpenCode runner and smoke cohorts are excluded; they provide no extra comparison evidence.
This does not resolve the separate local routing study. Plugin token accounting, host and shadow integration, health calibration, natural recovery and held-out routing evidence remain open.
Evidence availability
The first-set report and separate n09-n16 report are public. The follow-up includes curated trial rows, configuration, portable offline analysis and original-source hashes. The release notes disclose scope and transformations.
Fixture source, reference solutions, private implementation, generated patches, prompts, transcripts and raw test output are withheld. Readers can recompute counts from the selected outcome rows, but cannot independently rescore generated code from this package.
Cite this research
Bart Porcincula. "The Same Coding Model Improved When Its Harness Changed." Published 2026-11-19; updated 2026-11-19. https://porsync.com/research/same-coding-model-different-harness
Permanent article URL