ResearchPractical AIProjectsOpen SourceAbout

Research

Original Data

Experiments I ran on running, training and evaluating small AI models, with the actual numbers, including the results that didn't flatter the idea. Each one starts with a short plain-English summary.

Start with a question

Read the lab notes

What can I run locally?

Start with the evidence-based decision guide, then compare model fit and serving choices.

Why does my model fail?

Begin with run-to-run variation, then follow the failure-mode series and long-context experiments.

Does training help?

See where fine-tuning changed precision and recall, and how the teacher limited a smaller model.

Small-model failure modes

Controlled tests of how a small model's answers go wrong, from classification and tool use to table summaries and receipt extraction.

Local AIContract ExtractionOutput Validation

The Quote Was Real. The Contract Answer Was Still Wrong.

A literal quote check caught zero of 49 answers on contract fields labeled absent across three local deployments. It also discarded seven answers that passed the recall scorer. The same payment-allocation passage fooled all three deployments on a governing-law question.

Published 2026-11-09

Tool UseLocal AIOutput Validation

Gemma Invented Zero Search Parameters and Returned 307 Invalid Replies

Gemma 4 12B made 885 tool calls without inventing an argument key. It still failed the final-answer format in 307 of 600 attempts, compared with 72 on Qwen3.5-9B. A third local deployment stopped at the development gate after four searches used category values outside the catalogue.

Published 2026-11-09

Local AIDocument ExtractionEvaluation

Three Local Models Read the Same Receipts. Repair Fixed No Totals.

Gemma 4 12B and Qwen3.6-27B got all 42 printed totals right on their first attempt. Qwen3.5-9B repeated its one wrong total, which passed the arithmetic check. An instruction and a repair pass changed some line-item scores while exact-total counts held across all three models.

Published 2026-11-09

Local AIData SummariesEvaluation

A Numbers-Only Instruction Halved Flags on One Local Model

The original 13-to-6 drop in checker flags reproduced on Qwen3.5-9B. Gemma 4 12B went from zero to one, while Qwen3.6-27B went from six to five. A review of passing summaries found claims that misused valid source numbers.

Published 2026-11-09

Local AIData SummariesEvaluation

The Numbers Matched the Table. The AI Summary Still Got the Facts Wrong.

I tested summaries of 200 public CO2 tables with a small local model and a numeric checker. The checker rejected supported approximations while letting false comparisons and trends through. In an agent review of 50 passing summaries, 11 contained clear factual contradictions. A clean numeric-membership score measured much less than factual accuracy.

Published 2026-11-08

Local AIDocument ExtractionEvaluation

The AI's Receipt Total Added Up. It Was Still Wrong.

On 42 eligible receipt transcripts, a local model produced four arithmetic contradictions. The only wrong total passed the arithmetic check: the model had multiplied line amounts by quantities again, then added its mistaken numbers correctly. One repair fixed a line amount, but total accuracy stayed 41/42.

Published 2026-11-08

Tool UseLocal AIOutput Validation

No Invented Search Filters, but 72 Answers Failed the Format Check

I tested whether telling a local AI that a search tool's filters were its only filters would stop it inventing new ones. Across 600 attempts on 150 synthetic shopping requests, it invented none. The more visible problem was finishing: 72 replies failed the required output format, and one further reply searched the wrong category. The shortest added instruction had the fewest format failures in this run, but that pattern did not recur on Gemma 4 12B (76 versus 77 invalid finals). The original filter-error question hit a zero-error baseline.

Published 2026-11-08

Part 1 of 13EvaluationPrompt EngineeringLocal AI

Run the Same AI Prompt Twice: How Much of a "Change" Is Just Noise?

Part 1 of 6. Before you can say an edit to a prompt changed the answers, you need to know how much they change when you edit nothing. I measured that on 500 support tickets. Edits that look cosmetic, like reordering the options, changed more answers than a deliberate improvement did, and the improvement made the model slightly less accurate.

Published 2026-10-29

Part 2 of 13Prompt InjectionAI SafetyLocal AI

One Polite Sentence Redirected a Small AI Classifier on 71% of Tested Tickets

Part 2 of 6. If a model sorts text that strangers write, anyone can append a sentence and steer it. I tested six such sentences on 500 tickets and four defences. A polite sentence asking for a different team worked on 71% of tickets. Telling the model its output must be a valid label did nothing, and wrapping the ticket in markers helped on exactly one kind of attack.

Published 2026-10-29

Part 3 of 13Information ExtractionContract ReviewHallucination

When the Answer Isn't in the Contract, a Small AI Makes One Up 6% of the Time

Part 3 of 6. Pulling dates and terms out of contracts is a common AI job, and most of the time the field is simply not in the excerpt. The baseline already instructed the model to use null when a field was absent; a JSON schema did not materially change the result, and a stricter rule that required a quoted source cost accuracy without reducing the invention.

Published 2026-10-29

Part 4 of 13Structured OutputReliabilitySoftware Design

The Check That Fails Without Telling You: A Required Phrase Missed 9% of the Time

Part 4 of 6. A common way to wire an AI into code is to ask it to start its reason with a fixed phrase, then test for that phrase. It looks reliable and it is not: the model sometimes just doesn't write it, and your code concludes nothing happened. A separate yes/no field did not have the problem.

Published 2026-10-29

Part 5 of 13Structured OutputLatencySoftware Design

Put the Answer Before the Explanation: 1.8 Seconds Down to 0.56, and What It Costs

Part 5 of 6. If the model writes its explanation first, nothing is usable until it finishes. If the decision comes first, a screen or a downstream step can start right away. I tested both orders, with and without a cap on the explanation, on 500 tickets. The order roughly tripled how fast the answer appeared but barely changed total time, and the cap saved less than I predicted.

Published 2026-10-29

Part 6 of 13EvaluationAI GradingAnchoring

Does an AI Grader That Can See the Answer Just Agree With It? Not Quite. The Explanation Did.

Part 6 of 6. A second AI checking the first one's work is a common safeguard. The worry is that a grader shown the proposed answer simply agrees with it. I tested two small graders on 734 answers, half of them wrong. Showing the answer did not make them agree more. Showing the first AI's reason did, and both graders were weak either way.

Published 2026-10-29

Long sessions and memory

A five-part series: what long conversations cost, what a small model keeps from them, and when pasting everything beats retrieving.

RetrievalAI EvaluationKnowledge Bases

The AI Could Read the Answer. The Search Pipeline Had to Find It First

I tested a chatbot on 96 questions about a fictional knowledge base. Giving it the whole parent article after search raised accuracy from 78% to 89%; giving it the answer sections directly reached 99%.

Published 2026-11-13

Local AILong ContextInstruction Following

Two Local Models Followed Every Written Rule and Missed Many Implied Ones

Qwen3.6-27B and Gemma 4 12B passed all 252 explicit-rule checks across roughly 4K, 16K and 32K tokens. With the rules implied, Qwen passed 52–55% and Gemma passed 24–29%. Giving each model its own extracted rule list never raised its score.

Published 2026-11-09

Local AILong ContextSummarization

A Small Model Recovered More Details When a Different Model Wrote Its Summary

The same Qwen3.5-9B answered 17.5% of detail questions from its own short structured summary and 77.5% from Qwen3.6-27B’s. Asking the 9B to write a longer summary raised its score to 78.8%.

Published 2026-11-09

Part 1 of 8CostLocal AIContext Window

One Long AI Session or a Fresh Start Each Step? What It Costs on a Single Graphics Card

Part 1 of 5. If an AI helps with a long job, you can keep one growing conversation or start clean on every step and re-supply what matters. I timed both over 60 steps. On the 9B model, the long conversation was the cheapest, as long as the server remembers the start of the prompt. On the 27B, running at the memory edge of the card, starting fresh was cheaper.

Published 2026-10-30

Part 2 of 8EvaluationContext WindowInstruction Following

Does a Small AI Model Stop Following a Rule as the Conversation Gets Long? Not Up to 64,000 Tokens.

Part 2 of 5. A common worry is that a model forgets standing instructions as the context grows. I stated one rule once, buried it among filler tasks up to 64,000 tokens, and then tested whether the model followed it. It followed the rule every time at every length, wherever the rule sat. The catch, in part 3, is how the rule is phrased.

Published 2026-10-30

Part 3 of 8EvaluationInstruction FollowingImplied Constraints

A Small AI Follows a Stated Rule 100% of the Time and the Same Rule Only Implied 24% of the Time

Part 3 of 5. In part 2 a rule stated outright was followed every time. Here the same six rules are given only as a reason (for example, the importer rejects commas in amounts) instead of an order. The model followed them about a quarter of the time, at every context length, and a pre-step that asks it to list the standing rules did not close the gap.

Published 2026-10-30

Part 4 of 8EvaluationRetrievalRAG

Paste the Whole Folder or Retrieve the Relevant Notes? On a Small Model, Pasting Won Up to 120 Notes

Part 4 of 5. The standard advice for working over a pile of notes is retrieval: find the few relevant ones and show only those. I tested that against simply pasting the whole folder, from 10 to 120 notes (about 46,000 tokens). Pasting was more accurate at every size. Retrieval tied it on single-note questions and was 2 to 20 times faster. The entire gap was questions that need two notes.

Published 2026-10-30

Part 5 of 8EvaluationContext CompressionMemory

What Does a Small AI Keep When It Summarises Its Own Work? Mostly the Dates.

Part 5 of 5. When a conversation gets long, the standard fix is to have the model compress it. I planted six kinds of detail across 20 steps of work log, had the 9B compress its own context three ways, then asked for each detail back. Uncompressed it got 88%. A free summary got 3%, a structured template 20% and a copy-exactly template 33%. A files-plus-tool approach got 12%, because the model almost never opened a file.

Published 2026-10-30

Running models locally

Speed, cost, memory and coding models on one 16 GB graphics card.

Local AICoding AgentsEvaluation

Expert Caching Made Tokens Faster. Coding Throughput Did Not Improve Clearly

A 6,000 MiB cache improved generation speed, but passed 8/50 repair attempts versus 9/50 without it. A larger timing-only cache was faster, with no agent result.

Published 2026-11-15

Local AIGPU MemoryInference

A 30B Model Fit on My 16 GB Card. Eight Layers Changed Its Speed

On one 16 GB GPU, moving eight layers of a 30B model off the card cut short-prompt generation from 21.7 to 11.4 tokens per second while saving GPU memory. This measures speed and memory, not answer quality.

Published 2026-11-13

Local AICoding ModelsAgent Tools

Can a Small Local AI Make the Code Change? 360 Attempts Across Six Jobs

I gave two local models and Claude the same four small bugs and two small features, repeating each job 20 times per model. The 9B and Claude passed every attempt. The larger local Coder passed 119 of 120 directly. On the remaining attempt, the runner verified that it had undone the local edits, then Claude solved the job from a fresh copy.

Published 2026-11-07

Local AICoding ModelsBenchmarks

Seven Small Coding AIs, Four Benchmarks, One 16 GB Card: None Cleared the Bar

My first coding test was too easy to rank anything, so I ran the same models up a ladder of public benchmarks, hardest last, with the pass marks and a time limit written down first. The best model solved 9 of 50 real open-source bug reports. Five of seven were stopped before the end, not for being wrong but for being too slow to wait for. The result is a floor, and the reasons each model stopped matter more than the scores.

Published 2026-10-28

Local AICoding ModelsEvaluation

Same Test, Run Seven Times: How Much of a Local Coding AI's Result Is Luck?

Readers pushed back on my coding-model test: three runs is too few, and where did the time actually go? So I ran the same ten jobs seven times on two models, then 17 new jobs, then a runaway case a third time. The 20-second speed gap I had reported turned out to be inside the run-to-run spread. The surprise was not speed: the faster model failed on writing new code, which my first ten jobs never tested.

Published 2026-10-28

Local AICoding ModelsBenchmarks

Which Small Coding AI Models Are Worth Running on a 16 GB Graphics Card?

I put eight coding models on one 16 GB graphics card and gave them ten small bug fixes through the same coding assistant. Six fixed every job. The clearest differences were in runtime and tool use. One model wrote out its tool calls as text, so the assistant could never run them. I used these results to pick candidates for the harder benchmark that followed.

Published 2026-10-25

Local AICost AnalysisSpeed vs Accuracy

Running AI on Your Own Computer: What Makes It Faster, and What Quietly Changes the Answers

I tested what happens when a small AI model sorts 500 customer-support tickets on one ordinary graphics card, changing one setup choice at a time. The best setup was 2.2x faster than the default and cost a fraction of a cent per thousand tickets, against about $22 per thousand through a paid AI service. Overall accuracy barely moved. But individual answers did: the faster, more compressed setups gave a different answer on about 1 ticket in 12.

Published 2026-10-06

Big models on small hardware

What happens when a model built for a server is squeezed onto one consumer graphics card.

Local AIMixture of ExpertsThroughput

The Same Coding Model Got Six Times as Many Incidents Right per Hour on a Different Engine

An earlier test found a nearly sevenfold speed gap between two ways of running the same large model on one 16 GB card. I repeated the comparison on incident triage. Niko1221's Strata engine delivered 1,278 correct incidents per hour versus 208 for my original llama.cpp setup, with less than a point of observed accuracy difference. Tuning llama.cpp later narrowed the gap.

Published 2026-11-07

Local AIMixture of ExpertsThroughput

Tuning llama.cpp Cut a Sixfold Speed Gap to 2.7 on My 16 GB Card

My first llama.cpp setup kept all of the model's experts on the CPU. Moving some onto the GPU more than doubled its output. Niko1221's Strata engine still got 2.71 times as many incidents right per hour as the tuned setup. I also turned off its multi-token prediction and tested a small draft model in stock llama.cpp to see how much each accounted for.

Published 2026-11-07

Local AIMixture of ExpertsCoding Models

A 74B Coding Model on a 16 GB Card: It Fit With Room for a 116K-Token Prompt

I ran a community build of Nemotron Bolt on a 16 GB graphics card. It used 14.8 GB of video memory with 116,100 tokens in the prompt and generated about 50 tokens a second. Its coding and tool-call scores came within 2.5 points of the builder's reported scores. On 24 everyday questions, my smaller 9B model got five more right.

Published 2026-11-07

Local AIRetrievalIncident Triage

Giving a Local AI the Runbook Raised Incident-Triage Accuracy to 57%. A Bigger Model Reached 80%

I gave two local models the same incident reports, retrieved runbook pages and system state. The 9B got all five required fields right on 56.9% of incidents. The bigger Coder got 80.3% right. The 9B finished more correct incidents per hour across different engines, but the accuracy gap was far outside the five-point margin I set for swapping one for the other.

Published 2026-11-07

Local AIInference EnginesLoad Testing

Under Load, the Speed Gap Holds but the Latency Target Does Not: The Same Giant Coder Model on Two Engines, Tested with Detonate

My last study measured two engines one request at a time. This one sends several requests at once, using the open-source Detonate load tester, and repeats the whole sweep to see what is stable. Niko1221's Strata engine generated about 3 to 4 times the tokens per second of stock llama.cpp at every load level, in both runs. Neither engine met a normal latency target beyond one user, each ran a single slot, and stock llama.cpp met it for one user in the first run but not in the second.

Published 2026-11-03

Local AIBatch ThroughputConcurrency

Run Eight Requests at Once and a 9B Model Delivers 1.32 Times the Correct Answers per Hour, Not the 2 Times I Predicted. The Giant Coder Model Got Slower.

My earlier batch studies sent one request at a time. A batch job has nobody waiting, so it can send several at once. I tested 1, 2, 4 and 8 parallel slots for the 9B model and 1 and 2 for the giant Coder model on Niko1221's Strata engine, over the same queue of short business tasks. Batching helped the 9B by 1.32 times at eight slots, accuracy stayed within a third of a point, and the price was a 9-fold rise in the slowest 5% of response times. The Coder got slower at two slots.

Published 2026-11-03

Local AIReasoningThinking Models

Reasoning on Two Model and Engine Setups: Coder on Strata Was Faster, While the Thinking 9B Was More Accurate

I gave a 9B model with thinking turned on and Niko1221's Strata Coder with thinking off the same queue of reasoning puzzles for six hours each on one 16 GB graphics card. The Coder on Strata delivered about 11.9 times the correct answers per hour compared with the 9B on llama.cpp. Model, engine and thinking mode all changed together. The thinking 9B was 10.4 points more accurate, with the gap coming from constraint puzzles and log diagnosis. There was no difficulty level where thinking paid back its time.

Published 2026-11-02

Local AIInference EnginesMixture of Experts

Engine or Model? The Same Giant Coder Model Delivered 6.8 Times the Correct Answers per Hour on Niko1221's Strata Engine Than on Stock llama.cpp

My earlier studies compared a 9B model on stock llama.cpp with a giant Coder model on Niko1221's Strata engine, so every gap mixed the model and the engine together. This one runs the same Coder file on both engines over the same queue of short business tasks. Niko1221's Strata engine delivered about 6.8 times the correct answers per hour, with accuracy within a point. The planned 2x2 became three cells, because the 9B does not load on that engine.

Published 2026-11-02

Local AIMixture of ExpertsBatch Throughput

An Eight-Hour Batch Window on One 16 GB Card: The Small Model Delivered 1.68 Times the Correct Answers per Hour, the Giant One Was More Accurate

I gave a 9B model and Niko1221's Strata Coder the same shuffled queue of 30,000 short business tasks (intent classification, invoice extraction, log triage) for eight hours each on one 16 GB graphics card, and counted correct items per hour. The 9B finished 24,805 items and the Coder 14,022. The Coder was 4.2 points more accurate, all of it on extraction, but the 9B delivered about 1.68 times the correct answers per hour. The models used different engines; a later same-Coder test measured a 6.8-fold engine gap.

Published 2026-11-01

Local AIMixture of ExpertsCoding Models

A Giant Coding Model on a 16 GB Graphics Card: Fast Enough to Chat With, Too Slow to Finish a Coding Benchmark

Niko1221's Strata engine is an open-source runtime that squeezes a very large mixture-of-experts model onto one consumer graphics card by keeping the hot parts in video memory and the rest in system memory and on disk. I ran its coding model on a 16 GB card with 32 GB of RAM. It decoded at 42-58 tokens per second, but my coding benchmark was stopped by a rule I wrote beforehand: at 12 finished cases it projected 11.9 hours against an 8-hour budget. There is no benchmark score, and I am not reporting one.

Published 2026-10-30

Training and tuning small models

Fine-tuning and distillation: what the extra training buys and what it doesn't.

Local AIDistillationPre-Registered

Can the Apprentice Replace the Master? A Small AI Learning From a Big One

Big commercial AI models are good and cost money every time you use them. Small ones run on your own computer for the price of electricity. The tempting idea is to let the small one learn from the big one's answers and then retire the big one. I tested that on a real task, sorting customer banking questions into 77 categories, and the honest result is: the apprentice gets a lot better, but on this task it does not yet match the master. What it learns depends heavily on how good the master is.

Published 2026-10-22

Small Language ModelsFine-TuningContract Review

Teaching a Small AI to Read Contracts: It Got More Careful, Not More Thorough

I tried to train a small AI model, small enough to run on one ordinary computer, to find 41 kinds of clause in real contracts and quote them word for word. Compared fairly, with both versions reading the contract the same way, training made it far more careful (fewer false claims, almost no invented text) and slightly less thorough, and left the overall score unchanged. The bar I set in advance was not cleared, so the main comparison against a larger model never ran.

Published 2026-10-03

Small Language ModelsDistillationLoRA

Copying a Bigger AI's Judgement Into a Smaller One: The Teacher Sets the Ceiling

An experiment replacing a mid-size AI classifier with a small trained model. The small model's size, the amount of training data and the output format made no measurable difference. Changing who wrote the training answers made the largest one. And without real outcomes to check against, every score turned out to measure agreement with a judge, not correctness.

Published 2026-09-28

Routing and cascades

Splitting work between a small model and a big one, and the signals that fail to route it.

Local AICoding AgentsEvaluation

A 120B Model Fit on My 16 GB Card. Reliable Coding Was Still Unresolved

Edge0 ran interactively. A separate routing-accounting retry passed 15/15 final checks, with two frontier repairs, but showed no speed win. The router remains frontier-only.

Published 2026-11-15

Local AIQuantizationModel Cascades

Squeeze a Big AI Model Into Less Memory, or Run a Smaller One? A Pre-Registered Test on a 16 GB Card

When your graphics card is the limit, you can run a small model at full quality or a much bigger one compressed to fit. I tested eight setups on 102 real contracts, with the rules written down first. The compressed big models won clearly, and the most compressed of them (13.5 GB) did as well as the least compressed (16 GB). The best of them landed within noise of a frontier model with no escalation at all. Deciding which contracts to hand up to the frontier model mostly still did not work: of 28 setup-and-signal pairs, 1 passed, 7 were flaky and 20 failed.

Published 2026-10-18

Model CascadesCost AnalysisSmall Language Models

Can a Small AI Tell You When to Call the Big One? A Pre-Registered Test That Said No

A popular cost-saving idea: let a cheap small AI handle everything, and escalate to an expensive frontier model only when the small one seems unsure. I tested it on 102 real contracts, with the rules written down before any results existed. No confidence signal reliably beat escalating at random. Even where escalating individual answers did help, it saved no money, because the frontier model charges for reading the contract, not for answering.

Published 2026-10-12

Coding harnesses

Same model, different harness: what the scaffolding around a coding model contributes.

Method and decision guides

How to test AI claims honestly, and the rules that came out of the tests.

Pre-RegisteredEvaluationMethodology

Write Down Your Bet Before You Test the AI

Most AI results are explained after the fact: run it, look at the numbers, then decide what they mean. I do it the other way round. Before a single test runs, I write down the question, the score that counts as a pass and how much wobble I'll treat as noise, then lock the page so it can't be quietly edited. This is why that habit matters for anyone buying or building with AI, with a live example where I had to change my method halfway and wrote the change down instead of hiding it.

Published 2026-10-22

Pre-RegisteredEvaluationSafety Checks

I Broke an AI on Purpose. The Safety Check Never Said "Worse"

Before you swap a working AI for a cheaper one, you want a check that says whether the new one is safe to ship. I tested that check the way you test a smoke alarm: I damaged the new model on purpose and waited for the alarm. It never went off. The damage was real and the check noticed it, but my own rule for what counts as an alarm was stricter than the damage I had caused. This is what that means for anyone relying on a "no worse than before" check, and what I did and did not conclude.

Published 2026-10-22

Decision GuideLocal AIPre-Registered

When Should a Task Run on a Local Model? Seven Rules With the Evidence Behind Each

I ran five pre-registered experiments on small AI models over two months, most of them ending in a trade-off or a "no", plus one framework study. Read one at a time they look scattered; together they answer the question people actually ask: should this task run on my own hardware? This is the rulebook that came out of them — seven rules, each with the study that measured it, the conditions it was measured under, and where it stops applying. Six are supported by measurements, one is a method whose thresholds are conventions. A small command-line tool asks six questions and prints the rules that bear on your case, and every output says "evidence from 3 tasks on one GPU".

Published 2026-10-18

Edge AISmall Language ModelsQuantisation

Before You Build: A Way to Test Whether a Small AI Can Do the Job on Your Device

Most projects that put AI on a phone, kiosk or other small device start by training a model and end by discovering it never fit the device. This is the reverse order: a measurement-first method that answers "can a small AI model do this job on this hardware at all?" before any training, using a simple pass/fail gate based on how much quality survives compression and whether it fits the device.

Published 2026-07-05