I can't read the code, so I test the behavior

A field note from Chad Mathews, drafted with Claude. Chad directs the work and lived these decisions; Claude does the writing; Chad edits every line and approves it before it publishes. Where an idea or a reference came from Claude, the note says so.

Here’s the constraint I actually operate under: when the AI writes code for me, I can’t read it and know whether it’s right. A senior engineer opens the file, scans the logic, and spots the off-by-one or the missing null check by reading. I don’t have that. Pretending I do would be the fastest way to ship something broken with confidence.

So I verify a different way. I can’t inspect the machine, but I can watch what it does. Every change gets tested by its behavior: enter the room from each door, confirm the player lands where intended, confirm the camera snaps to the room bounds after a respawn, confirm the enemy takes damage and drops the right flag. Not “does the code look correct” but “does the thing do the thing.”

For a game this is natural, because a game is its behavior. If it plays right, a large class of bugs simply isn’t there. My checklists are all written this way. They don’t say “review the transition handler.” They say: come into the room from every connected entrance, and confirm you spawn at the right marker each time. That’s a test I can run without reading a line.

Why this catches more than it sounds like it should

Behavioral testing has a real reputation problem. It sounds like the shallow version, the thing you do when you can’t do the “real” testing. But most bugs that reach a player are behavioral bugs. The transition that drops you in the wrong spot. The camera that doesn’t reframe. The flag that doesn’t set, so the door never opens. The attack that whiffs because it’s on the wrong collision layer. None of those need code reading to catch. They need someone to actually walk through the thing paying attention, with a written list of what “correct” looks like so attention doesn’t wander.

The discipline that makes it work is the checklist, not the play. Playing without a list, you see what you expect to see and miss the rest. A list turns “seems fine” into a set of specific claims that each have to be true: did the camera snap? did the flag set? did you land on the marker? The list is where the rigor lives.

The part I won’t pretend about

Behavioral testing cannot catch what only reading the code would.

If a function works correctly every time I play but has a security hole in how it handles input I never thought to type, I won’t find it by playing. If there’s a race condition that only shows up under load I can’t generate at my desk, playing won’t surface it. If the data model is subtly wrong in a way that corrupts state after ten thousand saves, my twenty saves look perfect. These are exactly the failures a senior engineer catches by reading and I structurally cannot.

I’ve written before that I understand that I don’t understand. This is the sharpest edge of it. My verification is real, and it’s honestly better than a lot of people’s, and it has a ceiling that a code review would not have. I’ve reduced the risk of shipping something broken. I have not eliminated it, and anyone who tells you their playtest eliminated it is selling something.

What I do about the ceiling is stay under it on purpose. I build things where behavioral failure is the dominant risk and silent-correctness failure is survivable: a game, tools I use myself. When something touches other people’s money or data, it gets more than my playtest: staged environments, security checklists, and a lot of deliberate paranoia in the places I know I can’t see. The constraint doesn’t go away. I just try never to point it at something that can’t afford it.

The name for this

Testers call this black-box testing, checking a system by its inputs and outputs without looking inside, as opposed to white-box testing, which reads the internals. I knew those terms at a basic level going in; what I didn’t have to be told was which kind I was physically capable of doing. The thing worth saying plainly is that black-box testing is a legitimate, named discipline that catches most real bugs, and the reason white-box testing exists is that black-box has a ceiling. I live below that ceiling by choice. Knowing it’s there is the difference between confidence and recklessness.