Agent Harness Engineering Introduction 12 of 12 steps written

The scaffolding around an agent loop, and what each layer of it buys

Twelve steps, four parts, three runnable tasks — six switchable layers, measured three ways.

Introduction

Your agent works in a demo and cannot be trusted in production. You have improved the prompt several times and the failures have moved rather than gone: it guesses a file path, repeats a call that taught it nothing, or announces success while wrong.

Why the prompt is not where the fix is

A working agent is a model inside a program, and almost none of that program is the model. A better prompt cannot invent a tool that does not exist, cannot name a failing case the error text withheld, and cannot stop a model that has already declared victory.

who wrote itcan you change it today
the modela vendorno
the promptyouyes, and you already have
the harnessyouyes — this volume

The third row is the subject. It is entirely yours, it is code, and unlike the model you can change it this afternoon.

What an agent harness is

An agent harness is everything around the model call: the tools it may use, the text it gets back, the checks on what it claims, the window it works in, and the record of what it did.

Six switchable layers, and this volume measures what each one buys by holding the model fixed and turning them off one at a time.

layeroffonstep
tool_surfacea shell and one overloaded file operationnarrow, typed verbs3
rich_errorsFAIL 3/6the failing cases, and what to do about them4
verifierthe model's word is acceptedthe harness checks, and refuses a false finish5
context_budgetdrop the oldest events until it fitscompact, keeping the facts that stay true6
recoveryrepeat until the budget runs outnotice the repeat, nudge once, then stop7
observabilitycounters onlythe full trace, replayable7

The honest headline, up front

On an easy task most of these layers buy nothing. They are insurance. The question is never “is this layer worth it” but what does this layer insure against, and does that happen to me — and on the tasks measured here, two layers pay on the happy path, three pay only under pressure, and one never pays in score at all and is still the reason the others could be measured.

Is this for you?

if…then
your agent announces success and is wrongstep 5 — and the answer is not a better prompt
it guesses at file paths or argument shapesstep 3, which is the largest effect measured here
it repeats a call that told it nothingstep 7
it works, and you cannot say whystep 7's second half — every finding in this volume came from a trace
it works and you trust ityou may not need this volume. Step 11 says which layers to skip

How the measurement works

One scripted, deterministic model, so a change in the outcome can only have come from the harness. Three tasks. Six layers. And three measurement modes, because the first two both under-report:

modewhat it doeswhat it hides
the ladderswitch layers on one at a timean easy task does not exercise most of them, so they look worthless
leave-one-outremove each layer from a complete harnesssame blind spot, from the other direction
stressrun each layer under a condition that exercises itnothing — this is the one that reports honestly

The third mode exists because of a result in step 8: a measurement that does not stress a layer will report that the layer is worthless, and that is a fact about the measurement.

How to read the source labels

labelmeans
conceptargued from first principles, quotes nothing from a lab
measureda lab compares two configurations and reports the difference
hands-onyou run it on your own material
Nothing here is a benchmark. One scripted model, three tasks, six cases at most each. Step 12 lists every limit, starting with the one that makes the comparison clean and also makes it narrow.
Build state: 12 of 12 steps written.