ResearchPractical AIProjectsOpen SourceAbout
Research/When Should a Task Run on a Local Model? Seven Rules With the Evidence Behind Each
Decision GuideLocal AIPre-RegisteredEvaluationMethodology

When Should a Task Run on a Local Model? Seven Rules With the Evidence Behind Each

I ran five pre-registered experiments on small AI models over two months, most of them ending in a trade-off or a "no", plus one framework study. Read one at a time they look scattered; together they answer the question people actually ask: should this task run on my own hardware? This is the rulebook that came out of them — seven rules, each with the study that measured it, the conditions it was measured under, and where it stops applying. Six are supported by measurements, one is a method whose thresholds are conventions. A small command-line tool asks six questions and prints the rules that bear on your case, and every output says "evidence from 3 tasks on one GPU".

Bart Porcincula · Published

Finding

A pre-registered decision guide for "should this run local?" can be written as seven rules (R1-R7) whose 89 published numbers are each bound by file hash to the write-up they were published in and to the raw aggregates they were computed from: check fit by measured peak memory rather than file size (R1, supported); test quantisation retention before training, >= 0.85 acceptable and < 0.70 collapse (R2, a convention rather than a measurement); serving setup is a speed lever, not an accuracy lever, and 4-bit flipped 8% of individual answers (R3, supported); local marginal cost is $0.0032 against $22.43 per 1,000 tickets in electricity (R4, supported); fine-tuning trades recall for precision at an unchanged overall score, 0.697 vs 0.698 [-0.043, +0.061] (R5, supported); the teacher's labels set the ceiling, +10 points then +2.7 not significant after the rubric fix (R6, supported); and small-model confidence does not route whole documents, with 1 of 28 setup-and-signal pairs passing the registered test at every escalation level in the second cascade study (R7, supported). Evidence covers 3 task families on one 16 GB GPU.

The rulebook: seven decision rules for local vs frontier AI, with their audited evidence (Oct 2026)

The decision guide's evidence registry, one entry per rule R1-R7: claim, status, conditions, scope limit, and every audited number with its confidence interval, split and sample size. Status is "supported" when a pre-registered measurement backs the claim and "method" when the thresholds are conventions fixed in advance. Each number is bound by sha256 to the write-up it was published in and to the raw aggregate it was read from; two audits and a checkout re-hash check that binding. Evidence spans three task families on one 16 GB consumer graphics card.

MeasurementValueNote
Rules in the rulebook76 supported by measurements, 1 method (R2's thresholds are a convention)
Audited numbers behind the rules8935 (R1) + 3 (R2) + 6 (R3) + 3 (R4) + 12 (R5) + 10 (R6) + 20 (R7)
R1 — check fit first, by measured peak memorysupporteda 27B model at about 3 bits fits the 16 GB tier; the least-compressed file missed it
R2 — decide feasibility before trainingmethodkeep >= 0.85 of 8-bit quality after compression, < 0.70 is collapse; thresholds are fixed conventions
R3 — serving setup is a speed lever, not an accuracy leversupported2.2x (178.4 vs 80.3 tickets/min); 4-bit flipped 8% of individual answers
R4 — local marginal cost is electricitysupported$0.0032 vs $22.43 per 1,000 tickets on the same task
R5 — fine-tuning buys precision, not recallsupportedmacro-F1 0.697 vs 0.698 [-0.043, +0.061]; precision 0.783 -> 0.925, recall 0.869 -> 0.816
R6 — the teacher sets the ceilingsupported+10 points on a stronger teacher's labels; +2.7 and not significant after the rubric fix
R7 — don't route whole documents on small-model confidencesupported1 of 28 setup-and-signal pairs passed at every escalation level; 7 flaky, 20 failed
Task families the evidence covers3contract clause extraction, ticket triage, lead scoring — on one 16 GB RTX 5060 Ti

The short version

Over two months I ran five pre-registered experiments on small AI models — the kind you run on your own computer — plus one framework study. Pre-registering means writing down the question, the metric, the pass bar and what counts as noise before any result exists, then freezing the plan with a hash so it cannot be quietly moved later. Most of my studies ended as a "no" or a trade-off.

Read one at a time they look scattered. Together they answer the question people actually ask: should this task run on a local model, or not? So I wrote the rulebook: seven rules, R1 to R7. Each one carries the claim, the conditions it was measured under, the evidence (which study, which split, how many items, the confidence interval), a scope limit saying where it stops applying, and a status.

  • Six rules are supported by measurements; one is a method. R2's thresholds (keep at least 0.85 of your 8-bit quality after compression; below 0.70 is collapse) are conventions fixed in advance, not measured cliffs, and the rulebook says so rather than dressing them up as findings.
  • Every number is audited twice. Each of the 89 published numbers is bound by file hash to the verbatim string in the write-up it came from and to the raw aggregate it was computed from. A test fails if either drifts.
  • The rules are narrow on purpose. The evidence is three task families — contract clause extraction, ticket triage and lead scoring — on one 16 GB graphics card. Every output of the companion tool repeats that line.
  • There is a tool. prereg-decide asks six questions and prints the rules that bear on your case, with each rule's scope limit. It makes no model calls and prints nothing that is not already an audited number.

Takeaway: the order that worked here is: check what actually fits your measured memory (R1), test feasibility before you train (R2), and only then argue about accuracy — because at this scale the price question is already settled (R4).

The rules

#RuleStatusMeasured in
R1Check fit first: assign a model to a memory tier from its measured peak during a real run, never from file size or a vendor specsupportedSqueeze a Big AI Model Into Less Memory
R2Decide feasibility before training: quality kept after compression >= 0.85 vs 8-bit is acceptable, < 0.70 is collapse; if nothing clears inside the device's limits, fine-tuning cannot save the projectmethodBefore You Build: Testing Whether a Small AI Can Do the Job
R3Serving setup is a speed lever, not an accuracy lever — but 4-bit flipped 8% of individual answers, so use 8-bit when per-item consistency matterssupportedRunning AI on Your Own Computer: What Makes It Faster
R4Local marginal cost is about $0.003 per 1,000 tickets in electricity against about $22 through a paid API on the same task — the question is accuracy, not pricesupportedsame study
R5Fine-tuning buys precision and verbatim fidelity, not recall; it did not move the overall score against an input-matched baseline (0.697 vs 0.698)supportedTeaching a Small AI to Read Contracts
R6When distilling, the label source sets the accuracy ceiling — spend on better labels before a bigger studentsupportedCopying a Bigger AI's Judgement Into a Smaller One
R7Do not route whole documents on small-model confidence (no signal beat random at every tested level); per-answer routing finds the frontier model's wins but saves nothing when the bill is the document readsupportedCan a Small AI Tell You When to Call the Big One?

How a rule is written

Every rule has the same four parts, and the fourth is the one that usually goes missing in write-ups.

The claim is one sentence, stated so it can be checked. R7's, for example: do not route whole documents on small-model confidence.

The conditions say exactly how it was measured — which split, which sample size, which verdict rule. R7's condition is a two-tier gate borrowed from the rest of the toolkit: a routing signal only counts as working if its 95% confidence interval beats escalating at random at every tested escalation level (20%, 30% and 40%). Passing at one level is "flaky", not working. That single rule is why so much of the cascade evidence reads as a "no".

The evidence is the study, with the numbers and their intervals — not a link to a leaderboard.

The scope says where the rule stops applying, and it is mandatory. R1's scope, in full: one 16 GB card, memory tiers emulated from measured peaks rather than tested on 8 or 12 GB cards; one task; one frontier reference at one sample; thin peak traces for three of the setups; thinking mode off; at 102 contracts a two-to-three-point difference does not separate; local compute not priced. A rule without a scope column is an opinion.

How the numbers stay honest

The rulebook is not a document, it is generated from an evidence registry — one JSON entry per rule holding the claim, status, conditions, scope, and every published number with its confidence interval, split and sample size, plus the sha256 of the source file each number was read from.

  • Two audits per number. One checks the number against the verbatim string in the published write-up; the other checks it against the raw aggregates. A checkout-mode gate re-hashes the live source files and fails if a single byte changed.
  • The rulebook table is generated. The registry renders the README section; a test fails if it is edited by hand, so the prose can never drift from the data.
  • CI on every push runs the whole suite plus a credential audit (secrets, absolute paths, day-job traces, dataset licences).

89 numbers, seven rules, and a failure at any of those checks blocks the release.

The tool

prereg-decide is a reasoning aid over the registry, not an oracle. Six questions in: task type (classification or extraction), input length, available VRAM, the accuracy floor you need, whether per-item consistency matters, and volume — plus a flag if you are about to train. The rules that bear on those answers come out, each with its scope limit and a link to the study.

prereg-decide --task extraction --input-tokens 30000 --vram 16 --floor 0.75
prereg-decide --task classification --volume 50000 --json
prereg-decide                  # interactive: prompts for the six inputs

Every output carries the evidence line — evidence from 3 tasks on one GPU — because the most dangerous failure mode of a decision guide is sounding more general than its data. The tool has no model calls, no downloads, and nothing it prints that is not already an audited registry number.

The toolkit behind it (freeze-and-hash plans, noise floors beside every delta, a two-tier verdict, the registry and the CLI) is Apache-2.0 and available on request.

Limits

  • Three task families, one GPU. Contract clause extraction, ticket triage and lead scoring, all on one 16 GB RTX 5060 Ti. Rules are what those studies measured, not general laws.
  • Memory tiers are emulated from measured peaks on a 16 GB card, not tested on smaller cards, and three setups' peaks come from one or two readings.
  • One frontier model, one sample of its answers, and its price is notional (a subscription run read at API-equivalent rates). At 102 contracts a two-to-three-point difference does not separate.
  • R2's thresholds are conventions and R2 itself is marked "method". R7's single winner (contract length on the ternary arm, 1 hit out of 28) was picked after the fact and is carried as a lead, not a claim.
  • Cost figures are marginal electricity at a stated exchange rate, with no hardware amortisation in that column.
  • A rule changes only through a new pre-registered study. Two are queued: routing with a cheap frontier tier priced at registration time (R7's scope line would gain a price, not a rewrite), and a bounded-question scorer that would earn a place in the registry only if it passed.

Where this came from

The six write-ups behind the rules: the first cascade test, the second, the serving study, the fine-tuning study, the distillation study, and the feasibility framework.

The finding, stated for citation

A decision guide for running AI tasks locally can be written as seven pre-registered rules whose 89 published numbers are each auditable against both the published write-up and the raw aggregates: fit by measured peak memory rather than file size; test compression retention before training; treat serving setup as a speed lever and expect 4-bit to flip about 8% of individual answers; expect local marginal cost around $0.003 per 1,000 items against about $22 through a paid API; expect fine-tuning to trade recall for precision at an unchanged overall score; expect the label source, not the student model, to set the ceiling when distilling; and do not route whole documents on small-model confidence. The evidence spans 3 task families on one 16 GB consumer GPU, and every tool output states that scope.

Cite this research

Bart Porcincula. "When Should a Task Run on a Local Model? Seven Rules With the Evidence Behind Each." Published 2026-10-18; updated 2026-10-18. https://porsync.com/research/when-should-this-run-local-rulebook

Permanent article URL