Stage 1: Build a Test Harness and CI Gate With a Coding Agent

This is the second playbook on the AgenticAdoptionLearningPath. It assumes you have finished AgentContainmentAndBaseline.

The work is to bring the tests and CI for one slice of your code to the point where a planted bug is caught quickly, and to do that work with a coding agent. It is your first real agentic work, chosen because it is low-stakes to ship and because it teaches the skill everything else depends on: judging agent output.

Each step below gives what to do and the reason for it. Where a step looks like ceremony, the reason is the part to read.

Why agent-written tests need more checking than agent-written code

A wrong feature usually shows itself: something visibly does not work. A wrong test shows nothing. It passes, the build is green, and the count of tests goes up. Every signal you normally use to judge progress says things are going well.

Why would an agent write a test that checks nothing? Not through carelessness. The task it is given is usually "add tests for this" or "make the tests pass", and a test that passes satisfies that task whether or not it could ever fail. The agent has no way to see the difference between a test that passes because the code is right and one that passes because it checks nothing, unless something makes that difference visible. Most of this page is about making it visible.

That produces a few recognisable kinds of test that pass without protecting anything:

Kind of weak testWhat it looks likeWhy it passes
No real assertionCalls the code and checks only that it did not throw, or that the result is not nullAlmost any behaviour satisfies it
Conditional assertionThe check sits inside an if or a loop that may never runWhen the condition is false, nothing is checked
Mirror of the implementationThe expected value is computed with the same logic as the code under testA bug in the logic appears on both sides
Mock of the subjectThe thing being tested is replaced by a stand-inThe test exercises the stand-in
Locked-in bugThe test asserts what the code does today, including behaviour that is wrongIt was written by observing the code, not the requirement
Weakened testAn existing failing test is edited, skipped or deleted so the build goes greenThe check was removed

The conditional kind is easy to miss in review. In the Wikantik codebase, polling tests of the form "if the list has items, assert on them" ran green for a long time while checking nothing, because the list was empty when the check ran. They looked like coverage and were not.

People write all six kinds too. The difference is volume and uniformity: an agent can produce two hundred tests in an afternoon, all neatly formatted, and a reviewer faced with two hundred tidy tests samples a few and approves the rest.

None of this is a reason to avoid having agents write tests. It is the reason the rest of this page is about proving the tests work.

Step 1: choose the slice and write down its boundary

Pick the module or service that will change next. A good first slice:

Write one paragraph naming the directories in scope. The agent's work in this stage stays inside them.

Why small enough to read? You are about to judge the agent's tests, and you can only judge a test if you know what the code is supposed to do. On code you have not read, you are trusting the agent's reading of it, which is the thing you are trying to check.

Why somewhere with work coming? A harness earns its keep when code changes. Tests on code nobody touches catch nothing, and the effort looks wasted when someone asks what it bought.

Why not the worst part of the system? It is tempting, because that is where tests are most needed. But the worst code is where agents and people both struggle most, and a first attempt that goes badly teaches you about that module more than about agents. Do it second.

Why write the boundary down? Agents widen scope readily. Asked to make a module testable, an agent may decide the cleanest route is to refactor its callers. A written boundary gives you something to check the diff against, and gives the agent an instruction it can follow.

Step 2: get one command that runs the slice's checks

Before adding tests, make the existing ones runnable by a single command that exits non-zero on failure and finishes in a few minutes.

Good agent tasks here:

Why does speed matter this much? An agent works in a loop of change, run, read the result. The command is run dozens of times in a session. At twenty seconds, the agent checks after every change and catches its own mistakes while they are small. At twenty minutes, it either waits, which makes every task take hours, or it batches many changes between runs, which means a failure could be caused by any of them. A slow check does not just cost time. It changes how the agent works, for the worse.

Why one command with a clear exit code? The agent decides whether it is finished from what the command reports. If "passing" means reading through pages of output for the absence of certain words, the agent will sometimes read it wrong, and so will you.

Why fix intermittent failures first? A test that fails at random makes a red build ambiguous. A person learns to re-run and move on. An agent, seeing a failure, tries to fix it: it changes working code to chase a failure that was never about its change, or it concludes the test is unreliable and weakens it. One flaky test can cost more agent effort than all the real failures combined, and it trains everyone to treat red as noise.

Step 3: pin down current behaviour with characterization tests

For code with few tests, have the agent write characterization tests: tests that record what the code does now, so that later changes that alter behaviour are noticed.

Give the agent a narrow instruction: one class or function at a time, real inputs, assertions on specific output values, no changes to production code.

Why record current behaviour instead of testing what the code should do? Because for old code, nobody is sure what it should do, and the callers depend on what it actually does. A change that makes the code more correct can still break a caller that relied on the old behaviour. The first thing you need is to know when behaviour changed at all. Whether the change was right is a separate question for a person.

Why one function at a time? A broad instruction such as "add tests for this module" gets broad, shallow tests: one per function, each checking the easy case. A narrow instruction gets the awkward inputs and the boundary cases, which is where the bugs are.

Why no changes to production code? If the agent may change the code to make it easier to test, you can no longer tell whether a test records the old behaviour or the behaviour after the agent's change. Keep the two apart: tests first, with the code frozen, then refactoring under those tests.

These tests lock in today's behaviour, bugs included. Deal with that in the open:

Why does the marking matter? An unmarked test is read, by the next person and the next agent, as a statement of intent. When someone later fixes the bug, the test fails, and the natural response is to "fix" the code back. A locked-in bug with no note is worse than no test, because it actively defends the bug.

Step 4: write new tests red first

For behaviour you are specifying, as opposed to recording, the test is written and run before the code changes, and it must fail for the expected reason. Only then is the code changed.

Make the agent show the failing run. VerificationLoopsForAgenticCoding describes enforcing this automatically.

Why insist on seeing it fail? A test that has only ever passed has shown you nothing about whether it can fail. Seeing it fail, for the right reason, is the only cheap evidence that it is connected to the behaviour. It takes seconds and it rules out four of the six weak kinds in the table above.

Why "for the expected reason"? A test can fail because of a typo, a missing import or a wrong setup. That failure says nothing about the behaviour. The failure you want is the assertion reporting that the behaviour is absent.

Why does this matter more with an agent than with a person? When test and code are written together in one pass, the agent writes the test knowing what the code does, and the test drifts toward describing the code. Written first, the test can only describe the requirement, because there is no code yet to copy from.

Step 5: prove the tests fail when the code is wrong

This is the step that separates a harness from a pile of green ticks. Do it in two ways.

By hand, on every batch of tests. Break the code on purpose and run the tests:

If the tests still pass, they do not protect that line.

By tool, on the slice. Mutation testing does the same thing automatically: it makes many small changes to the code and reports which ones no test noticed. Common tools are PIT for Java, Stryker for JavaScript, TypeScript and C#, mutmut for Python, and cargo-mutants for Rust. MutationTesting covers the method.

The list of surviving mutations is a precise task list for the agent: each one names a line and a change that nothing caught.

Why is coverage not enough? Coverage records which lines ran during the tests. It says nothing about whether anything was checked. A test that calls a function and ignores the result covers every line of it. An agent asked to raise coverage will raise it, often with exactly that kind of test, because that is the cheapest way to move the number. Breaking the code measures the thing you care about: whether a wrong change would be noticed.

Why by hand as well as by tool? The tool is thorough and slow, and its report is a list. Doing it by hand takes five minutes and shows you, directly, a tidy and plausible test passing against broken code. That experience is what changes how you read agent-written tests afterwards. The tool then covers what you would not have thought to try.

Why only on the slice? Mutation testing runs the test suite once per change it makes. Across a whole codebase that is hours or days. On one module it is minutes, which is short enough to run again after the agent fixes what it found.

Step 6: review the test diff with a fixed checklist

For every agent-written test change, check:

  1. Does each test assert a specific value or effect?
  2. Is any assertion inside a condition or loop that might not run?
  3. Is the expected value independent of the code under test?
  4. Is the real subject being exercised, not a stand-in?
  5. Were any existing tests edited, skipped or deleted? If so, why?
  6. Did any production code change? In this stage it should not have.

Why a fixed checklist and not ordinary careful review? Ordinary review looks for what seems wrong, and these tests do not seem wrong. Each item on the list corresponds to one kind of weak test from the table at the top, so the list makes you look for a specific defect instead of waiting for one to catch your eye. It also keeps review consistent on the two-hundredth test, when attention has gone.

Why does item 5 get a rule of its own? The rule is that the agent does not modify or remove an existing test without a person agreeing to that specific change. When a change breaks an existing test, there are two ways to get back to green: fix the change, or alter the test. Altering the test is nearly always less work, and an agent told to get the build passing will often take it. Sometimes that is correct, because the old test was wrong or the behaviour was meant to change. But it is a decision about what the software should do, and that belongs to a person. Without the rule, your existing tests protect you only until the first time they are inconvenient.

Step 7: make CI the gate

Why required, and not just run? A check that can be overridden will be, on the day it is most inconvenient, which is usually the day it is right. A harness that depends on someone choosing to look at it is a suggestion.

Why the same checks for both kinds of change? A lighter path for agent work becomes the path everything takes, and a heavier one gets agent work relabelled as human work. One gate also keeps the question simple: did it pass or not.

Why read the pipeline definition? Pipelines fail quietly in ways a green badge does not show. An early step fails and later steps are skipped, so the ones you rely on never ran. A release workflow builds from a tag and never runs the tests the pull request ran. A test job is filtered by path and stops matching after a directory is renamed. In each case the badge is green and the check you were counting on did nothing. The only way to know is to read what runs, in what order, and on what condition.

Exit test for stage 1: the planted bug

Ask someone who did not write the tests to introduce a realistic bug in the slice on a branch and open a pull request without saying what they changed. If you work alone, have the agent plant it in a fresh session with no access to the test files, and do not look at what it changed.

You pass when:

  1. CI fails on that pull request.
  2. It fails within a few minutes.
  3. The failure message points at the affected behaviour clearly enough to find the bug.

Repeat with two or three different bugs. If one gets through, you have found the next task for the agent.

Why a planted bug and not a coverage or mutation score? A score is a proxy, and a proxy can be met without meeting the goal. The goal is that a real mistake gets caught before it merges. The planted bug tests that directly, through the whole chain: the test exists, it runs in CI, CI blocks the merge, and the message is good enough to act on. A weak link anywhere in the chain fails the test.

Why someone who did not write the tests? Whoever wrote the tests knows what they check and will, without meaning to, plant a bug they cover. Someone else plants the bug a real change would introduce.

Why does the failure message count? Agents will be reading these failures and acting on them. A message that says only that something failed sends the agent searching, and it may "fix" the wrong thing. A message that names the behaviour sends it to the right place.

Then record what the stage cost and what it changed against your baseline, and move to AgenticTechniqueTrials.

What this stage teaches beyond the tests

By the end you will have seen, in your own code, how the agent behaves when a task is vague, when a check is slow, and when the easy way to green is the wrong one. That knowledge is specific to your codebase. No article can give it to you, and it is the main input to every trial that follows.

See Also