Under Load, the Speed Gap Holds but the Latency Target Does Not: The Same Giant Coder Model on Two Engines, Tested with Detonate
My last study measured two engines one request at a time. This one sends several requests at once, using the open-source Detonate load tester, and repeats the whole sweep to see what is stable. Niko1221's Strata engine generated about 3 to 4 times the tokens per second of stock llama.cpp at every load level, in both runs. Neither engine met a normal latency target beyond one user, each ran a single slot, and stock llama.cpp met it for one user in the first run but not in the second.
Bart Porcincula · Published · Updated
Finding
Same Qwen3.8-Flash-Next Coder IQ1_M file, RTX 5060 Ti 16 GB, one slot on both engines, Detonate v0.1.0 concurrency sweeps, 256-token prompts and 64-token answers, 60 s per level, two runs per engine (run 1 at 1, 2, 4 and 8 users; run 2 at 1, 2, 3, 4 and 8). Output speed: Niko1221's Strata engine (Terminal 0.1.54) 31-33 tokens per second at every level in both runs, against 7-10 for stock llama.cpp b11753 (3.1 to 3.6 times in run 1, 3.7 to 4.3 times in run 2). Strata's numbers repeated almost exactly; stock's did not. Time to first token at the 95th percentile, run 1 then run 2: Strata 0.88 / 0.85 s at 1 user, 3.0 / 3.0 s at 2, 7.2 / 6.9 s at 4, 39.8 / 30.6 s at 8; stock 1.78 / 4.12 s at 1 user, 21.7 / 12.1 s at 2, 28.8 / 27.3 s at 4, 61.1 / 57.1 s at 8. Run 2 added 3 users: Strata 5.0 s, stock 28.0 s. Against the default target (95th-percentile first token within 2 s and per-token time within 100 ms) Strata serves 1 user in both runs and fails from 2 up. Stock served 1 user in run 1 (narrowly, 1.78 s) and none in run 2. Both engines break (over 30 s) at 8 users in both runs. No errors at any level.
Same Coder IQ1_M on Niko1221's Strata engine vs stock llama.cpp under concurrent load, 16 GB VRAM, two runs (Nov 2026)
Two runs per engine, one machine. RTX 5060 Ti 16 GB, 31.6 GB RAM. Detonate v0.1.0 (commit 69be777, Apache-2.0, by AntonioGr7), closed-loop concurrency sweeps (run 1: 1, 2, 4, 8 users; run 2: 1, 2, 3, 4, 8 users), 256 input and 64 output tokens, 60 s per level, default target (p95 time to first token 2 s, p95 per-token time 100 ms, errors 1%). Stock llama.cpp b11753: --cpu-moe, 32K context, one slot, thinking off, hit directly. Niko1221's Strata engine, Terminal 0.1.54: 32K context, one slot, thinking off through a pass-through proxy. The workload is smaller than Detonate's default because the default does not finish on this model. Each engine was relaunched between runs.
| Measurement | Value | Note |
|---|---|---|
| Output tokens per second at 1 user, run 1 / run 2 | Niko1221's Strata engine 31 / 32 vs stock 10 / 8 | 3.1 times, then 3.9 times |
| Output tokens per second at 8 users, run 1 / run 2 | Niko1221's Strata engine 32 / 32 vs stock 9 / 9 | 3.6 times, then 3.7 times |
| p95 time to first token at 1 user, run 1 / run 2 | Strata 0.88 / 0.85 s vs stock 1.78 / 4.12 s | the stock figure moved by more than 2 times between runs |
| p95 time to first token at 2 / 3 / 4 / 8 users (run 2) | Strata 3.0 / 5.0 / 6.9 / 30.6 s vs stock 12.1 / 28.0 / 27.3 / 57.1 s | |
| p95 per-token time (both runs) | Strata 21-23 ms vs stock 66-95 ms | |
| Users within the target, run 1 / run 2 | Strata 1 / 1, stock 1 / 0 | Strata's limit lies between 1 and 2 users; stock's was marginal |
| Breaking point (p95 first token over 30 s) | 8 users on both engines in both runs | |
| Errors | 0.0% at every level on both engines in both runs |
The short version
My last study compared the same giant Coder model on Niko1221's Strata engine and stock llama.cpp, one request at a time. That says nothing about what happens when several people ask at once. This one uses Detonate, an open-source load tester by AntonioGr7, to send simultaneous users at each engine. I ran the whole sweep twice, relaunching both engines in between, to see which numbers repeat.
- The speed gap holds under load, and repeats. Niko1221's Strata engine produced 31-33 tokens per second at every level in both runs. Stock llama.cpp produced 7-10. That is about 3 to 4 times.
- The latency target does not hold. Against Detonate's default target (95th-percentile first token within 2 seconds, per-token time within 100 ms), Niko1221's Strata engine serves exactly one user in both runs and fails from two up.
- Stock's single passing level was fragile. In run 1 stock llama.cpp just passed at one user (1.78 s against 2.0 s). In run 2 the same setup took 4.1 s and passed for no one. I do not know why. Strata's one-user figure was 0.88 s then 0.85 s.
- Both break at eight users, in both runs. The 95th-percentile wait for a first token passes 30 seconds. Strata's figure at eight users was 39.8 s, then 30.6 s, so that one sits close to the line.
- No errors at any level. The requests queued, they did not fail.
Takeaway: a faster engine raises throughput but does not create capacity that a single slot does not have. Both engines ran one slot, so extra users simply wait in line. Whether either engine does better with more slots is not tested here.
What the numbers say
Tokens per second and p95 first-token wait, run 1 then run 2 (a dash means that level was not in run 1):
| Users | Stock tokens/s | Stock first token (p95) | Strata tokens/s | Strata first token (p95) |
|---|---|---|---|---|
| 1 | 10 / 8 | 1.78 / 4.12 s | 31 / 32 | 0.88 / 0.85 s |
| 2 | 7 / 9 | 21.7 / 12.1 s | 31 / 32 | 3.0 / 3.0 s |
| 3 | - / 7 | - / 28.0 s | - / 32 | - / 5.0 s |
| 4 | 8 / 9 | 28.8 / 27.3 s | 32 / 33 | 7.2 / 6.9 s |
| 8 | 9 / 9 | 61.1 / 57.1 s | 32 / 32 | 39.8 / 30.6 s |
The headline I wrote down before the run was "the most users each engine serves within the target, Strata over stock". That came out 1 against 1 in run 1 and 1 against 0 in run 2, so it is too coarse to carry the result. The more useful figure is throughput, and it repeated. Time per token is also much lower on Niko1221's Strata engine (21-23 ms against 66-95 ms), but the target is missed on waiting time, not on speed.
Stock's first-token waits at two to four users scattered a lot between runs (21.7 s then 12.1 s at two users), which is what you would expect when only 8 to 16 requests finish in a 60-second window. Strata finished 30 to 38 per level and its figures barely moved.
How it was run
Same card (RTX 5060 Ti, 16 GB), the same Coder file on both engines, one slot on both. Detonate v0.1.0 ran a closed-loop sweep: each user sends a request, waits for the answer, then sends the next, for 60 seconds per level. Requests were 256 input tokens and 64 output tokens. Stock llama.cpp b11753 was hit directly with the experts kept on the CPU; Niko1221's Strata engine (Terminal 0.1.54) went through a small proxy that turns thinking off, as in the previous study. The engines ran one at a time and were relaunched between run 1 and run 2. Run 2 added a three-user level to look between one and two users, but the limit still sits between one and two.
What I am not claiming
- Two runs per engine is better than one but is not a variance estimate. Stock's one-user result moved from a narrow pass to a clear fail and I did not find the cause (the first run had a warm-up check beforehand, the second did not, but I did not test that).
- The workload is not Detonate's default. The default (1,024 input and 256 output tokens, up to 512 users) does not finish on a model that generates about 10 tokens per second, so I used a smaller shape. I did not freeze and commit the plan before the scored runs, as I had intended.
- Both engines ran a single slot. Niko1221's Strata engine and llama.cpp both support other settings; I did not explore them.
- The stock arm hits the model directly and the other goes through a proxy, as in the previous study. Detonate ran on the same machine as the engine.
- Detonate's own grade (C- on both in run 1; D for stock and C- for Strata in run 2) is quoted as its output and is not used as a result; its score formula is not documented.
- Nothing here measures accuracy, and nothing says anything about any other engine or product.
Does a fast local AI test mean several employees can use it at once?
No. Neither tested engine met the study's response-time target beyond one user, and both used a single processing slot. A speed result from one request at a time does not establish multi-user capacity. Test the number of simultaneous staff requests and the wait your workflow can tolerate.
Cite this research
Bart Porcincula. "Under Load, the Speed Gap Holds but the Latency Target Does Not: The Same Giant Coder Model on Two Engines, Tested with Detonate." Published 2026-11-03; updated 2026-11-08. https://porsync.com/research/giant-coder-model-two-engines-concurrent-load-detonate-niko1221-strata-vs-stock-llama-cpp
Permanent article URL