A test that cannot fail is not evidence

How can a test pass and still be worthless?

The clearest example VenSoc has hit recently: a contact form test submitted an invalid form and asserted that an element with the alert role appeared. It passed. It had always passed.

The server action behind that form was, at the time, completely broken. The module exported a validation schema alongside the action itself, which the framework refuses at runtime. Every submission threw. The form displayed a generic error — which is an element with the alert role. The assertion was satisfied by the exact failure it existed to detect.

This is the shape of the problem. The assertion was too loose to distinguish success from failure, and looseness is invisible while everything works. Fifty other assertions passed in the same run. The only trace was a stack in the web-server log of an otherwise green suite.

The fix was to assert on strings only the working code can produce — the exact validation message from the schema, and the exact state the action returns. A crashed module cannot generate either. The test now fails when the thing it guards is broken, which is the entire job.

What is mutation testing, in practice?

Break the code on purpose and confirm the test goes red. That is the whole practice. Formal mutation testing tools automate it by generating many small mutants, but the manual version — pick the behaviour that matters, break it deliberately, run the test, restore — captures most of the value and takes minutes.

The discipline matters most for the tests you rely on most. A guard on this project asserts that no client name appears anywhere in the source. Running it after writing it produced a pass. Reassuring — and meaningless, because a guard that has never been observed to reject anything is indistinguishable from a guard with a broken regular expression.

Deliberately reintroducing each forbidden name and confirming a red build turned that pass into evidence. It also surfaced something unrelated: an earlier edit had silently deleted two constants the content module depended on. The type checker caught that only because the mutation run made the file compile again.

The heuristic that follows is simple. Any test you would cite as proof that something is safe should be one you have watched fail.

Practice

Break the guarded behaviour, run the test, expect red

Applies to

Any test or gate you would cite as proof

Manual cost

Minutes per assertion

What it detects

Loose assertions, dead guards, broken matchers

Why does AI-assisted development make this worse?

Because the characteristic failure of AI-generated code is not code that looks wrong. It is code that looks entirely right and references something that does not exist.

A model that has seen `created_at` and `updated_at` in a schema will confidently write `deleted_at` into a query. It compiles. It passes review, because it reads exactly like the surrounding code. It fails against a database that never had that column. The same pattern produces plausible API fields, plausible environment variables and plausible file paths.

Generated tests inherit the same tendency. A test written against an assumed shape asserts the assumed shape, so implementation and test agree with each other and both disagree with reality. The suite is green and self-consistent and wrong.

Two practices contain this. Verify every identifier at its source before it is used — read the migration, read the service metadata, read the config file. And make every test earn trust by failing on demand. Neither is novel; both become load-bearing when a large share of the implementation is generated faster than it can be read line by line.

Which checks belong in the build rather than in review?

Any rule that a reasonable person will skip when they are in a hurry, which is most of them.

On this project seven checks were promoted from convention to build gate, and every one caught a real defect that human review had already passed over. Colour contrast, checked across every semantic token pair on every surface in both themes, caught four accessibility failures including an inverted panel at 1.9:1. A placeholder-content gate caught fabricated metrics on their way into a production build. A crawler-visibility gate fetches every route as GPTBot and asserts load-bearing sentences are in the raw HTML. A console gate loads every route in a real browser and fails on any error. A headers gate asserts a specific relaxation cannot leak into production. A confidentiality gate caught a documentation comment that named all four entities it was written to protect. A sitemap gate caught twelve advertised routes that returned 404 and, in the same run, that every case study was missing from the sitemap entirely.

The pattern is consistent: each defect was invisible in a browser, harmless-looking in a diff, and would have been found by a client rather than by us.

A gate is worth building when the failure is silent, the rule is objective, and the cost of a miss is external. It is not worth building for matters of taste, and a badly written gate that blocks good work is worse than no gate — which is why each one has to be proven by mutation before it is trusted.

What does “done” mean if a green suite is not enough?

It means a command was run and its output shown. Not "the tests should pass" — the command, and what it printed.

The failure this prevents is specific and common: work reported as delivered where tests were written but never executed, a file was created but left empty, or an integration was mocked and never wired. Each of these produces a confident report and a system that does not work, and each is caught by the trivial discipline of pasting the output.

A completion report should also state what is not done. A report that lists three deferred items is more trustworthy than one that claims everything is finished, because the second one is almost never true and the reader knows it.

None of this is sophisticated. It is the difference between an engineering practice that can be audited and one that has to be believed, and on a system somebody else depends on, that is the whole distinction.