Why you still read everything your agent writes
- ai
- agents
- trust
Subscribe
New essays and episodes, sent when there is something worth reading. No noise, unsubscribe anytime.
Subscribe
New essays and episodes, sent when there is something worth reading. No noise, unsubscribe anytime.
There's an agent running inside my business that watches HubSpot for companies I move to Developing. When one appears, it researches the company, sweeps my own systems for warm paths, writes a structured profile back into the CRM, and hands me a short list of the things only I can do. Its instructions run a bit over three thousand words. And every time it runs, I read what it wrote before I act on a line of it.
That last step interests me more than the agent does. The agent exists to save me the work, and mostly it does. But the reading step never goes away, and I don't think it's just mine. Ask anyone running agents on real work. Somewhere in the loop is a human reading the output with the specific alertness of someone who doesn't quite trust what they're holding.
The standard explanation is capability. Models hallucinate, prompts underspecify, context goes missing. So we do the obvious thing and give the agent more. More instructions. More retrieved context. More tools, and more guardrails around all of it. That has been the shape of the last two years of agent engineering, and it isn't wrong exactly. Each addition helps. My profiles got better as the instructions grew.
But watch what happens to trust. It barely moves. The instruction file triples in size and the reading step stays. When the fix keeps working and the problem keeps not going away, you're probably fixing the wrong problem.
Where the failures land tells you which problem is real. My agent doesn't fail on cases the instructions cover. It fails on the cases they don't: the company that's technically in scope but obviously wrong for a reason no rule captures, the source that's citable but shouldn't be believed, the profile that follows the format and misses the point. In each case the agent does something defensible. Defensible and wrong. So I add a rule, the rulebook grows by one, and the next miss arrives from a direction the new rule doesn't face, because rules are finite and situations aren't.
You could still blame the model. Better model, fewer misses. Except the same model, in a live conversation with me, handles those exact cases well. Show it the borderline company and ask whether it feels like a fit, and it reasons about it the way I would. The capability is there. What's missing is that at the moment the judgment call arrives, I'm not in the room, and nothing I wrote stands in for me.
Look at what we hand an agent: instructions, context, tools, guardrails. All four have something in common. Every one of them is the residue of a decision I already made. An instruction is a decision, made once, written down. A guardrail is a decision about the worst case. Even the context I feed it reflects a decision about what matters. None of it is my judgment. It's a cache of my judgment: the stored outputs of decisions I've already run.
Call it cached judgment, because the cache metaphor carries the whole problem. A cache is wonderful right up until the miss. And the entire point of an agent is to handle work I haven't seen yet, which means the agent lives at the cache miss, the exact place where stored answers run out and a decision has to be made fresh. We've been scaling the cache and wondering why the misses still hurt.
The obvious fix is to write down the decision function itself. Not the outputs of my judgment but the machinery: how I trade thoroughness against speed, when a rule bends, what makes something an exception. Try it. I have. You get a page of principles that sound right and decide nothing, because you don't know your own rules. You discover what your rule is when the case arrives. Michael Polanyi put this at the centre of The Tacit Dimension in 1966: we can know more than we can tell. You don't produce a driver by teaching the theory of the car. The economist David Autor gave the idea a name in 2014, Polanyi's paradox, and used it to explain why decades of automation stalled precisely on tasks demanding flexibility and judgment. The rules exist. They just aren't tellable.
Yet judgment does transfer. Businesses move it between humans constantly, and never by documentation. Nobody hands a new hire the SOP folder on Monday and full authority on Friday. The new hire sits in the room while decisions get made, then makes small ones and gets corrected, and hears the why behind the correction. They escalate the calls that feel wrong and learn from what comes back. After a few months something has moved across that was never written anywhere, and you can tell it's moved because the escalations change: the routine questions stop and only the hard ones remain. Judgment transfers through supervised decisions, not documents. We've been onboarding agents like software and expecting them to perform like employees.
If that's the mechanism, it should predict where agents already earned trust, and it does. Coding agents got real autonomy first, and not because their prompts were better written. Code is the rare domain where the quality of a decision is cheap to check. The tests pass or they don't, and a bad change can be reviewed in minutes and rolled back in one. Tight, cheap verification does what apprenticeship does, thousands of times faster. Trust arrived where decisions were checkable, not where instructions were longest.
For everything else, teaching an agent starts to look less like prompt engineering and more like training a junior. Show it decided cases rather than rules, especially the borderline ones, with the reasoning narrated, because the hard case carries the tradeoff that the principle flattens. Teach it when not to decide: the escalation boundary is the one part of a decision policy that is easy to tell. I can't fully write down how I judge fit, but I can write down which calls are mine. The most load-bearing section of my prospect agent's instructions is a blunt list titled "the agent never does", and every recommendation in its output is labelled as a proposal for me to rule on, not a decision it has made. Then review the reasoning rather than the output, correct the tradeoff the way you'd correct a junior's, and widen the autonomy at the pace the escalations improve.
Which flips the meaning of the step I started with. I used to treat reading the agent's output as quality control, a tax paid on imperfect software. It's the opposite of a tax. The reading is where the exceptions surface and the correction happens; it's where I find out what my own rule was. It's the apprenticeship, running in the only direction it currently can. Every time I read one of those profiles, I'm doing the one part of the job I never wrote down.