Stage 2: Run Measured Trials of Agentic Coding Techniques

This is the third playbook on the AgenticAdoptionLearningPath. It assumes the agent is contained, a baseline is recorded (AgentContainmentAndBaseline), and one slice of code has a harness that catches planted bugs (AgentBuiltTestHarness).

A trial is a small, time-limited test of one technique on real work, with a prediction written before it starts and a decision written when it ends. This page is the method. It is the stage you keep returning to: every new technique, and every later stage of the path, is entered through a trial.

Each part of the method below comes with its reason. The method is deliberately light, and the reasons tell you which parts you can bend and which you cannot.

Why trials instead of adopting what others recommend

Advice about agentic coding is plentiful and mostly unmeasured. Three things make it a poor guide to what will work for you.

Your own impression is not reliable either. In a 2025 randomised study by METR, experienced open-source developers took longer on tasks in their own repositories when allowed AI tools, yet believed afterwards that the tools had made them faster. The lesson is not that the tools are slow; it was one study of one group. The lesson is that the feeling of speed and the fact of speed can disagree, so the feeling cannot be the evidence.

Why would they disagree? Working with an agent replaces typing with waiting and reading. Waiting feels like nothing, and watching code appear feels like progress. The time goes into places that are easy to discount: rewriting the request, reading the result, discovering the change was subtly wrong, starting again.

A trial gives you a local, current answer that you can show to someone else.

The trial card: seven things written before you start

Write these down in one page before the trial begins.

FieldWhat to write
TechniqueOne change to how you work, stated so someone else could do it
PredictionWhat you expect to improve, by roughly how much, and why
ScopeWhich slice of code, which people, which kinds of task
PeriodStart and end date; two to three weeks is usually enough
MeasuresThe one or two baseline numbers the technique should move
Guard measuresThe numbers that must not get worse
Stop conditionsWhat would make you end the trial early

Why write it at all, for something this small? Because each field removes a specific way of fooling yourself afterwards. Without a stated technique, "what we tried" drifts to mean whatever worked. Without a period, the trial runs until the result looks good. Without measures chosen first, you pick the flattering ones later.

Why does the prediction matter most? A prediction can be wrong, and that is its value. If you expected lead time to halve and it fell by a tenth, you have learned something you would not have noticed had you simply looked at the numbers afterwards and found them "a bit better". The "why" in the prediction matters too: when a technique fails, the stated reason tells you which belief to revise.

Choosing the technique

Change one thing. Good first trials are small and close to the harness work you have just done:

Leave parallel agents and unattended work until at least two of these have been kept.

Why only one change? If you introduce written specifications, a new instruction file and parallel agents in the same fortnight and the numbers improve, you do not know which one did it. You will keep all three, including any that did nothing or did harm, and carry their cost indefinitely. One at a time is slower to start and much cheaper over a year.

Why start small? A small technique gives a clear result in a short time and costs little if it fails. It also builds the habit of running trials while the stakes are low. Large techniques have more ways to go wrong, and when one fails you want to already trust your method, so that you can believe the result.

Why hold back parallel and unattended work? Both remove you from the loop, so both depend on things you have not yet tested: that your written instructions are enough without you there to correct course, and that your gate catches what you would have caught by watching. Trialling them first tests three things at once.

Choosing measures and guard measures

Pick from the baseline you recorded in stage 0. Every trial needs at least one guard measure.

If the technique aims to improveGuard against
Lead timeEscaped defects, review load
Volume of change mergedReview time per person, CI duration
Test coverageSurviving mutations, CI duration
Review turnaroundDefects found after merge

Also record cost for the period, and one sentence from each participant on how the work felt.

Why is a guard measure mandatory? Nearly every technique can improve one number by damaging another, and the damage usually lands somewhere you were not looking. Lead time falls because review got thinner. More changes merge because each one is smaller and someone else now spends their day reviewing. Coverage rises on tests that check nothing. The guard is chosen by asking: if this technique were cheating, where would the cost show up?

Why only one or two main measures? With ten measures, something always improves by chance, and you will find it. Naming the one or two the technique is supposed to move, in advance, is what makes an improvement mean something.

Why record how it felt, when feeling is not evidence? It is not evidence of speed. It is evidence of whether the technique will last. A practice people find tedious gets dropped the moment nobody is checking, whatever its numbers were. It is better to learn that during the trial.

Setting stop conditions

Decide in advance what ends the trial early. Typical conditions:

Why decide this beforehand? Midway through, with effort already spent, the pull is to continue: one more week, the problem was a one-off. A condition written down in advance makes stopping the default. It also makes the trial easier to agree to in the first place, since everyone can see what would end it.

Running the trial on ordinary work

Why planned work and not tasks picked for the trial? You will pick tasks the technique suits. Everyone does. The result then describes how the technique performs on its best cases, and you adopt it for all cases. Ordinary planned work includes the awkward tasks, which is what the technique will actually face.

Why keep a log when you have the numbers? With this little data, the numbers say whether something changed and almost never why. The log holds the why: which tasks went wrong, in which part of the code, in what way. "Three of eleven tasks needed the agent's change thrown away, all three in the billing module" tells you to keep the technique and exclude billing. A percentage would have told you to drop it.

Why not improve the technique as you go? Adjusting is the natural thing to do and it destroys the comparison. The weeks before the change and the weeks after are now two different trials, each half the length, and neither long enough to read. Note the improvement and make it the next trial.

Why does trial work go through the normal gate? A trial that needed an exception to succeed has shown only that the technique works when the rules are relaxed. And the gate is what makes it safe to try things at all.

Reading the result honestly with small numbers

A few people over three weeks produce a few dozen changes at most. That is not enough for statistics, and you should not pretend otherwise.

The decision record: keep, adapt or drop

End every trial with a short written record. One page is enough:

  1. The trial card as written at the start.
  2. What happened: the measures and guard measures before and after, cost, and the notable entries from the log.
  3. Whether the prediction held.
  4. The decision: keep, adapt and trial again, or drop.
  5. If kept: what changes in the repository so that others get the same result, and when the numbers will be looked at again.

Why write it down, if everyone involved knows the outcome? Three reasons, for three readers.

Why record the dropped ones? They are the most persuasive entries. A list in which everything was kept looks like enthusiasm. A list that includes "tried, did not help, stopped" shows the evaluation can say no, which is what makes its yeses believable.

A trial cadence that is sustainable

Why one at a time per slice? Two trials on the same code are two changes at once, with the attribution problem described above.

Why leave a gap between trials? A kept technique changes the baseline. The next trial should be measured against how things are with that technique in routine use, not during its first, careful weeks.

Why re-run the planted-bug test? Techniques that raise output put more through the gate than it was built for. More changes mean more chances for something to slip past, and code the agent adds may not be covered by the tests you proved. The harness you verified in stage 1 was verified for the volume and the code you had then.

Exit test for stage 2

You have passed when:

  1. One technique has been through a full trial with a card written beforehand.
  2. The result was compared against the stage 0 baseline, including at least one guard measure.
  3. A keep, adapt or drop decision is written down where it will be found later.
  4. Someone outside the trial has read the record and could explain the decision.

Why the fourth item? It tests whether the record stands on its own. If a person who was not there cannot follow why the decision was made, neither will you in six months. Working alone, put the record aside for a week and read it cold.

Passing does not mean leaving. From here the path continues by running further trials on the techniques in stages 3 to 5 of the AgenticAdoptionLearningPath.

See Also