Here is how to verify AI generated code in one sentence. Run the real thing the change touches, capture what happened, and keep the check as a script someone else can rerun. Reading the diff alone is not proof. Neither is a clean compile, and neither is the agent telling you it works.
The rule behind every step below is short. Verify against the real artifact, not a proxy, a cached value or a self-report.
Match the check to the change
Different changes need different evidence. A unit test says little about a screen, and a screenshot says little about a migration. Pick the check from the kind of change.
| Kind of change | The check that proves it |
|---|---|
| Command-line tool | Run the real command. |
| Interface | Walk the changed flow in the running app. |
| Parser or migration | Replay a saved input. |
| Performance | Compare before and after profiles. |
| Storage | Read back the written value. |
If your project has no scripted way to start the app and drive it, build that first. Every check on an interface depends on it.
What counts as proof
Use this checklist on any result an agent hands you.
- Real path. The check used the path a user takes, not internal setters or test-only endpoints.
- Action and state. The evidence shows the action and the resulting state, not only the final screen.
- Side effects. Files written, rows inserted and messages sent were checked alongside what is visible.
- Mocks. A mock appears only where a production boundary already isolates the external system.
- Dry runs. If the check used a dry run or test mode, someone observed what that mode actually skips.
- Easy passes. When something passes too easily, suspect the way you observed it before you trust the result.
- Honest verdict. The result is
VERIFIED,NOT VERIFIEDorINCONCLUSIVE. Inconclusive is not a pass, and a step that could not run is named. - No weakened checks. Nobody relaxed an assertion or changed a test to fit a wrong implementation.
How sure are you
List the facts the change's safety depends on. For each one, get it as far down this list as is cheap, and say where it stopped.
- It was said. A claim with nothing behind it. Worthless on its own.
- It was pointed at. A real
file:line, or the library's own source. - It was reasoned through. The failure was walked step by step to show the bad case cannot happen.
- It was run. A script or test calls the real code and fails loudly if the claim is wrong.
- It was reproduced. The behavior was seen in the running app.
An agent summary with no evidence sits at the first step. Ask how far each claim got, push each one as far as is cheap, and record where it stopped.
Script the check
The strongest proof is a deterministic script that reruns the same comparison. A one-time look cannot be repeated by a reviewer. A script can. It turns "trust me" into "run this".
#!/bin/sh
# Replay a saved input and compare the result with a saved expected output.
# Replace ./your-command and the fixture paths with your own.
# The output and the diff stay in evidence/ for a reviewer to read.
set -u
mkdir -p evidence
if ! ./your-command fixtures/input.txt > evidence/actual.txt; then
echo "NOT VERIFIED: command failed"
exit 1
fi
status=0
diff -u fixtures/expected.txt evidence/actual.txt > evidence/diff.txt || status=$?
case "$status" in
0) echo "VERIFIED" ;;
1) echo "NOT VERIFIED"; exit 1 ;;
*) echo "INCONCLUSIVE: diff could not run"; exit 2 ;;
esacKeep the script and its output with the change. Then put the same checks where they run on every change. Our post on pre-commit hooks vs CI covers where each check belongs.
Make sure the checks can fail
Agents write tests too, and a weak test looks exactly like a strong one when it is green. Hold the tests to the same standard as the code.
- A test calls the code the way its users do and asserts against a literal expected value. If it would still pass when every imported function returned nothing, rewrite it or delete it.
- For a bug with a cheap local test path, write the smallest test that fails for the intended reason. Watch it fail, fix the bug, then rerun it.
- Prefer no new test over a bad one. A bad test mostly tests mocks, encodes implementation details or depends on timing. Use a script or a reproduction command instead.
- Give every hand-written checker a fixture that makes it fail. A check is not trustworthy until you have watched it fail on the condition it exists to catch.
- Import each real entry point the way production starts it. A green build and a green test suite do not prove the program starts.
The last two come from our engineering practices, which explain the failures behind them.
Make it routine
Verifying by hand works for one change. It does not hold up across many. Write the steps down as instructions the agent follows every time. Cover how to launch the app, how to tell it is ready, how to drive it, what evidence to capture and how to clean up. Then run those instructions once end to end. Instructions that were never executed are a draft.
The rules in this post come from the agentic engineering manual, which you can save into any repository. For the wider practice, read what agentic engineering is. To turn repeated review comments into checks that block a merge, read guardrails for AI coding agents.
- verification
- how-to
keep reading
- What is agentic engineering? A plain definitionWhat is agentic engineering? Coding agents build the software, and the codebase checks their work. A plain definition, its parts, and how to start.
- Guardrails for AI coding agents that the build enforcesGuardrails for AI coding agents hold when the build enforces them. Turn a repeated review comment into a type, a lint rule or a check the agent cannot skip.
- Pre-commit hooks vs CI: which checks belong wherePre-commit hooks vs CI, compared. When one person pushes, the hooks are the gate and CI is a second opinion. A team flips that. See which checks go where.