ResearchPractical AIProjectsOpen SourceAbout
Research/I Broke an AI on Purpose. The Safety Check Never Said "Worse"
Pre-RegisteredEvaluationSafety ChecksAI Strategy

I Broke an AI on Purpose. The Safety Check Never Said "Worse"

Before you swap a working AI for a cheaper one, you want a check that says whether the new one is safe to ship. I tested that check the way you test a smoke alarm: I damaged the new model on purpose and waited for the alarm. It never went off. The damage was real and the check noticed it, but my own rule for what counts as an alarm was stricter than the damage I had caused. This is what that means for anyone relying on a "no worse than before" check, and what I did and did not conclude.

Bart Porcincula · Published

Finding

A paired non-inferiority check on 965 banking questions downgraded its verdict as a small model was deliberately damaged, from "no worse than 3 points" at 20% scrambled training answers (accuracy 0.851 to 0.834) to "no worse than 5 points only" at 40% (to 0.821), but never returned "worse", because that verdict needs proof of a loss above 5 points; the registered test of the check therefore failed at both damage levels, and the plan's rule to stop before opening the held-out test set applied.

Deliberate-damage test of a model-swap safety check (Oct 2026, validation data)

A small model that had learned from a strong teacher was retrained with a share of its training answers scrambled, then compared with the undamaged version on the same 965 validation questions by a paired non-inferiority check with tolerance margins of 3 and 5 points. The test was registered in advance with one pre-allowed escalation (from 20% to 40% scrambled), run once each.

MeasurementValueNote
Undamaged model accuracy0.851965 validation questions
20% of training answers scrambled0.834 (-1.7 points)check said: non-inferior within 3 points. The registered test needed "worse", so it failed.
40% of training answers scrambled0.821 (-3.0 points)check said: non-inferior within 5 points only (95% interval for the loss: 1.6 to 4.4 points). The registered test needed "worse", so it failed again.
What "worse" requiresa proven loss above 5 pointsthe whole interval has to sit beyond the looser margin, which a 3-point loss cannot reach
Held-out test setnever openedthe plan said the phase stops if this test fails

The short version

Say you run an AI that works and someone offers a cheaper replacement. The responsible step is a swap check: run both on the same questions and ask whether the new one is "no worse than the old, within a tolerance". It's the right idea. The question I wanted to answer is whether the check is any good, and the way to test a smoke alarm is to light a small fire.

So I lit one. I took a small AI model that had learned from a strong one, retrained it with some of its training answers deliberately scrambled, and asked the check to flag it. I wrote the pass rule first: the check must call the damaged model "worse". It didn't. Not at 20% scrambled, and not at 40%.

What happened

Training answers scrambledAccuracyWhat the check said
none (the good model)0.851n/a
20%0.834no worse than 3 points
40%0.821no worse than 5 points only

The check isn't blind. As the damage grew, its verdict slipped from "within 3 points" to "within 5 points only", and at 40% its interval said the loss was somewhere between 1.6 and 4.4 points, which is clearly not zero. What it never did is say "worse".

Why, and why it still counts as a failure

Its rule is cautious. It says "worse" only when it has shown the loss is bigger than 5 points. My damage cost 1.7 and then 3.0 points. Real, but under the line it needs for an alarm.

That is a perfectly reasonable design for a check whose job is to say "yes, safe to swap". It is a trap if you read its silence as "no problem found". "Not proven worse" and "proven fine" are different statements, and a verdict of "within 5 points" can hide a loss you would care about.

I could have lowered my pass mark afterwards so the test passed. I didn't. The plan allowed one escalation, from 20% to 40%, and I used it, wrote it down before running it, and ran it once. Both failures stand. The plan also said that if this test failed, I should stop before opening the final exam set. So I did, and that set is still sealed.

What to take from this

  • Ask what the alarm threshold is. For any "no worse than before" check, find out how large a loss it takes before it says "worse", not just how it behaves when nothing is wrong.
  • Test it with a fault you caused. If a vendor shows a swap check, ask what it did when they deliberately broke the model, and how badly it was broken.
  • Read the tier, not just the colour. "Safe within 5 points" and "safe within 3 points" are different promises. Whether 5 points is acceptable depends on what a wrong answer costs you.
  • Sample size sets the sharpness. With 965 questions, a 3-point loss can be seen but not always proven past a 5-point line. More questions sharpen the check, at a labelling cost.

What this does not show

One task, one kind of damage (scrambled training answers), one run at each level, validation data only. It does not show that the check is broken, only that my registered pass rule asked it for something it is not designed to give. Fixing the rule is a separate piece of work, and I haven't done it.

Cite this research

Bart Porcincula. "I Broke an AI on Purpose. The Safety Check Never Said "Worse"." Published 2026-10-22; updated 2026-10-22. https://porsync.com/research/i-broke-an-ai-on-purpose-the-safety-check-never-said-worse

Permanent article URL