Reliable For Whom?

In a college marketing class, I spent three hours standing in a shopping mall logging everyone who walked into a boba shop. Age, group size, time of day, whether they hesitated at the menu or ordered instantly.

The assignment was to recommend a strategy. Most people would have run a survey. I wanted the real customer rather than the assumed one — and after three hours, the answer was obvious from the tally sheet.

I've been doing a version of that ever since. I built learning software for older adults because mainstream interfaces assumed users who looked nothing like them. Then I spent years making AI agents reliable enough for enterprises to trust — taking generated output from under 20% success in production to over 90%.

Somewhere in the middle of that work, a different question started bothering me, and it hasn't stopped.

Reliable for whom?

Ask two people to describe a great assistant and you'll get two incompatible answers.

One wants it to just handle things. Book the flight, file the expense, send the follow-up. When it works, they say "great, it just handled it."

The other wants to be consulted. Show me what you found. Confirm before you spend anything. When that person meets the first person's assistant, they say "wait — it paid without asking me."

Same agent. Same action. One person delighted, one person's trust permanently damaged. And today both of them get the same agent, because we train on averaged preferences and ship one default behavior.

The axis that diverges

There's a distinction hiding here. Almost all the preference data being collected today measures what an agent produces: which output is better. That's outcome taste, and it converges. Ask enough people which of two summaries is cleaner and they'll largely agree.

Procedural preference is a different axis: how the work should be done. When to ask versus act, how much to verify, what to do when something is ambiguous. Unlike outcome taste, it diverges by person — which is exactly why it's what agents acting on your behalf need to know, and exactly why averaging destroys it.

I found out later that economists had already named this: procedural utility, the finding that people care how an outcome was produced and not only what it is. They wrote it in 2004. It took agents to make it measurable.

At first it looked like a sequence — correctness, then taste, then personalization. It's a tidy story, and it took me too long to see that it's wrong. It collapses two independent questions. One is what gets evaluated: the output, or the way the work was done. The other is whose preference counts: everyone's, or yours.

Cross them and you get four combinations, not three stages.

Three of those four combinations are well studied. Head-to-head comparison collects averaged outcome preference. Recommendation systems have handled individual outcome preference for twenty years. Alignment research asks when any agent should pause before acting.

The fourth is individual procedural preference — how you want the work carried out. We know least about it because it only became observable when agents started genuinely acting on our behalf, which is a development of the last two years. Personalization research has grown fast in that time, but nearly all of it aims at what a model produces rather than how a task gets done. You can't study taste in action until there are actions to watch.

Difference is the data

The instinct in machine learning is to treat variance as noise to be smoothed. Here it's the opposite. The spread between people is the signal. Todd Rose made this case about cockpits and classrooms — design for the average person and you've designed for nobody. I keep finding new rooms in that house.

And the signals already exist. Someone edits an output, overrides an action, takes control midway, redoes the work, or quietly abandons it and does the task by hand. What people do says more about how they want to be served than anything they'd type into a settings panel. Today those moments are flattened into clicks or buried in logs — the builder sees that something failed, but not whether the person wanted more control, a different sequence, or simply to be asked first.

Which is the same thing I was doing in that mall, with better instruments.

What I hope gets built

I'd like my preferences to be mine. Something that picks up how I actually work, rather than a settings panel I have to fill out in advance, and that I could take with me — so a new tool knows how to work with me on day one instead of starting from nothing. Not locked inside whichever platform happened to see me first.

I'm not alone in wanting that. A Stanford HAI workshop report this year proposed a Human Context Protocol — portable, persistent personal memory that moves across tools under the user's control, with scoped access and revocation. It's careful about what goes in the layer, and careful to separate what a person declares from what a model infers, treating inference as something to keep visible and correctable rather than to lean on.

That boundary is the interesting part. The context I'd most want carried is the kind nobody declares: not what I like, but how I want work done.

The pattern I keep returning to, across a decade and several unrelated problems: the interesting work lives in the differences between people, not in the mean.

Agents make those differences consequential for the first time. Not a smarter average — an end to averaging.

---

If you think about this too — from research, from building, or just from watching people — I'd enjoy hearing how you see it.