Blog
Judgment Before Automation
Three products, three domains, one shape: the software explains, offers, records — and never decides. Including the time I shipped an approval that approved nothing.
I have spent this summer building three unrelated things: a design tool for campaign groups, a Mac app for photographers, and an AI advisory desk for non-profits. Different domains, different users, no shared code. Somewhere in each of them I hit the same wall, and came out the other side having made the same decision three times without noticing until last week.
That decision is what the slogan actually means, and it is more specific than "keep a human in the loop".
The poster at two in the morning
The design tool checks four things before you print: text overflow, color contrast, image resolution, safe area.
The obvious implementations are both wrong. Silently allow, and someone wheat-pastes a poster nobody can read from across a street. Hard-block, and you have told a volunteer at two in the morning that you understand their situation better than they do.
You do not. Maybe the contrast is bad because the only photograph that exists is bad, and a poster that goes up tonight beats a better one that goes up never.
So every check explains itself in plain language, beside the field that caused it, offers a fix drawn from the document's own palette — and can be set aside with a click.
The dismissal lasts the session and no longer. Storing it in the file would carry one person's decision to ignore something into everybody else's copy of it. A warning that never comes back is a warning nobody ever acts on.
The crash that stays on your machine
The Mac app needed crash reporting. The default answer is a hosted SDK — Sentry or similar, three lines of setup, reports appear in a dashboard.
Two things made that wrong. The app's entire pitch is that it does not phone home, and a hosted reporter contradicts that in the one moment the user is least able to check. And it would pull a solo project into real obligations: a privacy policy, a lawful basis, a processor agreement, for telemetry nobody asked to send.
The useful observation was that capture was never the missing piece. macOS already writes a crash log for every crash. The gap is that a non-technical user has no idea those files exist.
So: Help ▸ Report a Problem surfaces the most recent one, with app version, OS and hardware, and lets you copy it or open the raw file.
There is no submit button. There is no network call. You read the exact text first, and then you send it, or you don't.
The environment block deliberately excludes anything about your library — no project names, no file paths, not even counts. What you are looking at is all there is.
The approval that approved nothing
The third is an advisory desk that walks a non-profit through adopting AI responsibly: what the case is, what the non-AI baseline looks like, where the data boundary sits, when it gets reviewed. It ends in a document the organisation can actually act on, with approval gates the coordinator ticks off.
I ran three personas through it — a small social-support coordinator, a climate NGO operations lead, a welfare federation digitalisation lead — and wrote down what happened rather than what I hoped would happen.
One of them deliberately overrode the recommended route. The system let her, and the final report showed the override. That is right: the override is recorded, not prevented. She may well know something the rule does not.
The audit also caught something worse, and it was mine.
"Data boundary approved" was a checkbox with no data-boundary statement behind it. "Review date approved" was a checkbox with no date field. You could tick both and the document would report full approval, having recorded consent to nothing at all.
That is worse than having no gate. An empty gate manufactures evidence of diligence. A year later, when somebody asks who signed off on the data boundary, the answer is a tick against a field that never existed and everyone involved will believe the process worked.
The shape
Three products, three domains, and the same three rules each time.
Make the moment legible. Not a warning icon, not a status color. The actual problem, in the user's language, next to the thing that caused it, at the moment it matters.
Leave the decision. Every one of these can be overridden by the person in front of it, because in each case that person may hold information the system does not — the photograph that does not exist, the crash they would rather not send, the local circumstance no rule anticipated.
Record what was chosen, including the override. This is the one people skip, and it is the one that makes the other two safe. A decision nobody can reconstruct later is indistinguishable from no decision. And a record of an approval must be a record of something — or you have built the empty gate.
Why this stopped being a philosophical question
While I was writing this, OpenAI moved to court business customers with a zero-data-retention pledge, chasing accounts that had gone to competitors partly on those grounds.
Read that as a market signal rather than a product announcement. Buyers now treat where data goes and what is kept as a purchase criterion, not a footnote and vendors have started competing on it directly.
That is the same instinct these three products arrived at from the inside. Trust is not a promise in a privacy policy. It is a property of where the software decided to put things, and what it does at the moment a person has to choose.
The organizations I work with mostly do not need more automation. They need to know which decisions they are still making, be able to see them, and be able to prove afterward that they made them.
That is what Judgment Before Automation means when it costs you something: not a human rubber-stamping the output, but a human holding a decision the software was built to hand back.
If that sounds like a problem you have, that is the work I do.