This is the third playbook on the AgenticAdoptionLearningPath. It assumes the agent is contained, a baseline is recorded (AgentContainmentAndBaseline), and one slice of code has a harness that catches planted bugs (AgentBuiltTestHarness).
A trial is a small, time-limited test of one technique on real work, with a prediction written before it starts and a decision written when it ends. This page is the method. It is the stage you keep returning to: every new technique, and every later stage of the path, is entered through a trial.
Each part of the method below comes with its reason. The method is deliberately light, and the reasons tell you which parts you can bend and which you cannot.
Advice about agentic coding is plentiful and mostly unmeasured. Three things make it a poor guide to what will work for you.
Your own impression is not reliable either. In a 2025 randomised study by METR, experienced open-source developers took longer on tasks in their own repositories when allowed AI tools, yet believed afterwards that the tools had made them faster. The lesson is not that the tools are slow; it was one study of one group. The lesson is that the feeling of speed and the fact of speed can disagree, so the feeling cannot be the evidence.
Why would they disagree? Working with an agent replaces typing with waiting and reading. Waiting feels like nothing, and watching code appear feels like progress. The time goes into places that are easy to discount: rewriting the request, reading the result, discovering the change was subtly wrong, starting again.
A trial gives you a local, current answer that you can show to someone else.
Write these down in one page before the trial begins.
| Field | What to write |
|---|---|
| Technique | One change to how you work, stated so someone else could do it |
| Prediction | What you expect to improve, by roughly how much, and why |
| Scope | Which slice of code, which people, which kinds of task |
| Period | Start and end date; two to three weeks is usually enough |
| Measures | The one or two baseline numbers the technique should move |
| Guard measures | The numbers that must not get worse |
| Stop conditions | What would make you end the trial early |
Why write it at all, for something this small? Because each field removes a specific way of fooling yourself afterwards. Without a stated technique, "what we tried" drifts to mean whatever worked. Without a period, the trial runs until the result looks good. Without measures chosen first, you pick the flattering ones later.
Why does the prediction matter most? A prediction can be wrong, and that is its value. If you expected lead time to halve and it fell by a tenth, you have learned something you would not have noticed had you simply looked at the numbers afterwards and found them "a bit better". The "why" in the prediction matters too: when a technique fails, the stated reason tells you which belief to revise.
Change one thing. Good first trials are small and close to the harness work you have just done:
Leave parallel agents and unattended work until at least two of these have been kept.
Why only one change? If you introduce written specifications, a new instruction file and parallel agents in the same fortnight and the numbers improve, you do not know which one did it. You will keep all three, including any that did nothing or did harm, and carry their cost indefinitely. One at a time is slower to start and much cheaper over a year.
Why start small? A small technique gives a clear result in a short time and costs little if it fails. It also builds the habit of running trials while the stakes are low. Large techniques have more ways to go wrong, and when one fails you want to already trust your method, so that you can believe the result.
Why hold back parallel and unattended work? Both remove you from the loop, so both depend on things you have not yet tested: that your written instructions are enough without you there to correct course, and that your gate catches what you would have caught by watching. Trialling them first tests three things at once.
Pick from the baseline you recorded in stage 0. Every trial needs at least one guard measure.
| If the technique aims to improve | Guard against |
|---|---|
| Lead time | Escaped defects, review load |
| Volume of change merged | Review time per person, CI duration |
| Test coverage | Surviving mutations, CI duration |
| Review turnaround | Defects found after merge |
Also record cost for the period, and one sentence from each participant on how the work felt.
Why is a guard measure mandatory? Nearly every technique can improve one number by damaging another, and the damage usually lands somewhere you were not looking. Lead time falls because review got thinner. More changes merge because each one is smaller and someone else now spends their day reviewing. Coverage rises on tests that check nothing. The guard is chosen by asking: if this technique were cheating, where would the cost show up?
Why only one or two main measures? With ten measures, something always improves by chance, and you will find it. Naming the one or two the technique is supposed to move, in advance, is what makes an improvement mean something.
Why record how it felt, when feeling is not evidence? It is not evidence of speed. It is evidence of whether the technique will last. A practice people find tedious gets dropped the moment nobody is checking, whatever its numbers were. It is better to learn that during the trial.
Decide in advance what ends the trial early. Typical conditions:
Why decide this beforehand? Midway through, with effort already spent, the pull is to continue: one more week, the problem was a one-off. A condition written down in advance makes stopping the default. It also makes the trial easier to agree to in the first place, since everyone can see what would end it.
Why planned work and not tasks picked for the trial? You will pick tasks the technique suits. Everyone does. The result then describes how the technique performs on its best cases, and you adopt it for all cases. Ordinary planned work includes the awkward tasks, which is what the technique will actually face.
Why keep a log when you have the numbers? With this little data, the numbers say whether something changed and almost never why. The log holds the why: which tasks went wrong, in which part of the code, in what way. "Three of eleven tasks needed the agent's change thrown away, all three in the billing module" tells you to keep the technique and exclude billing. A percentage would have told you to drop it.
Why not improve the technique as you go? Adjusting is the natural thing to do and it destroys the comparison. The weeks before the change and the weeks after are now two different trials, each half the length, and neither long enough to read. Note the improvement and make it the next trial.
Why does trial work go through the normal gate? A trial that needed an exception to succeed has shown only that the technique works when the rules are relaxed. And the gate is what makes it safe to try things at all.
A few people over three weeks produce a few dozen changes at most. That is not enough for statistics, and you should not pretend otherwise.
End every trial with a short written record. One page is enough:
Why write it down, if everyone involved knows the outcome? Three reasons, for three readers.
Why record the dropped ones? They are the most persuasive entries. A list in which everything was kept looks like enthusiasm. A list that includes "tried, did not help, stopped" shows the evaluation can say no, which is what makes its yeses believable.
Why one at a time per slice? Two trials on the same code are two changes at once, with the attribution problem described above.
Why leave a gap between trials? A kept technique changes the baseline. The next trial should be measured against how things are with that technique in routine use, not during its first, careful weeks.
Why re-run the planted-bug test? Techniques that raise output put more through the gate than it was built for. More changes mean more chances for something to slip past, and code the agent adds may not be covered by the tests you proved. The harness you verified in stage 1 was verified for the volume and the code you had then.
You have passed when:
Why the fourth item? It tests whether the record stands on its own. If a person who was not there cannot follow why the decision was made, neither will you in six months. Working alone, put the record aside for a week and read it cold.
Passing does not mean leaving. From here the path continues by running further trials on the techniques in stages 3 to 5 of the AgenticAdoptionLearningPath.