I Was Wrong About What Makes People Want an AI to Ask

Two weeks ago I published an argument I couldn't back up.

The argument was that AI agents are being built to be correct rather than personal: people disagree about how work should be carried out, yet products mostly ship one default way of acting.

It was a nice thesis.

I had no data.

So I launched a small pilot last weekend and let responses come in over the following week. The final dataset: 28 survey responses and 10 hour-long interviews.

Here's what came back — including the part where I was wrong.

The design

I created eight scenarios arranged into four pairs, each intended to probe a particular change in context. For each scenario I asked the same question:

How much would you want an AI assistant to do on its own?

Participants chose among five levels of autonomy — ask before every step, show me a plan first, go ahead but check at the big decision, do it and tell me after, or just handle it — plus a sixth option: I would not want an AI assistant involved in this at all.

Three people used the opt-out option, six times in total. For comparisons on the autonomy scale, I treat those responses separately rather than forcing them onto the five-point scale; opting out entirely is not quite the same thing as wanting maximum oversight.

The three manipulation pairs were:

• Flight booking: $250 refundable → $2,400 non-refundable
• File handling: clean up a messy contact list → delete files it thinks are duplicates
• Communication: reschedule a meeting with five colleagues → reply to an angry customer

The fourth was a matched control pair: research a payroll provider → research accounting software. Both were back-office financial research tasks with similar stakes and no customer- or safety-facing action. I deliberately tried to change as little as possible.

The control gave me a baseline question: how much do answers move even when I intentionally change almost nothing?

Nearly half the participants gave different answers to the two control scenarios. With a sample this small, that was a useful warning against turning every visible difference into a story.

1. Almost nobody has one "autonomy setting"

Across eight scenarios, 23 of 28 people used three or more different autonomy levels. Nobody gave the exact same response in all eight scenarios.

For one participant, the only variation was opting out entirely in two scenarios; for almost everyone else, the scored autonomy level itself changed with context. The median person's scored answers spanned 2.5 points on a five-point scale.

That does not prove there is a stable personalization function hiding underneath. But it does make one global setting look crude.

Conservative ←→ Autonomous

A person who wants an agent to just handle one task may want a plan, a checkpoint, or explicit approval on another. The useful question may not be "how autonomous is this user?" but "what changes the answer for this user?"

2. Change rates looked similar. Direction did not.

I expected the flight scenario to matter a lot. The price increased almost tenfold and the booking became non-refundable. Yet by the simplest headline measures — how many people changed levels and the median shift — it looked surprisingly close to the control.

Pair	Valid n	Changed mode	More oversight	More autonomy
Control — minimal contextual change	28	13 (46%)	3	10
Money + non-refundable	27	14 (52%)	10	4
Internal → customer-facing	26	13 (50%)	10	3
Clean contacts → delete files	27	22 (81%)	19	3

Valid n excludes opt-out responses for that pair. Direction is among participants whose scored autonomy level changed.

The file-deletion pair clearly stood apart: most people changed, and most moved toward more oversight.

The more interesting pattern was directional. In the control, people who moved mostly drifted toward more autonomy. In all three manipulation pairs, movers leaned the other way — toward more oversight.

I am deliberately not turning that into a significance claim. These are exploratory, paired responses from a small convenience sample, and the scenarios themselves are not clean causal interventions. But directionally, the manipulations did not look identical to the control.

So my summary is narrower than "two manipulations did nothing": by change rate and median shift, money and customer-facing behavior looked surprisingly close to the control. By direction, both still contained a signal worth testing properly.

3. Cost wasn't the clean boundary I expected

I had been designing around cost. My assumption was simple: as the amount of money at stake rises, people will want the agent to ask more often. A product could then use spend as a clean autonomy boundary.

The pilot did not support treating spend as the primary variable. The money pair barely exceeded the control on my headline measures, even though the people who did move tended to move toward more oversight.

By contrast, the file-deletion scenario produced the clearest shift toward caution — suggesting that reversibility, or perhaps uncertainty about the consequences of being wrong, may matter more than spend alone.

That is not the same as saying money does not matter. It means my original model was too simple.

The study demoted cost from the variable I was designing around to one candidate among several. That was probably the most useful outcome it could have produced.

4. What people say does not always line up with what they choose

I also asked participants directly: at what dollar amount would you want an agent to ask before spending?

Explicit answers ranged from $1 to $1,000, and several people refused a fixed dollar threshold altogether: "any amount," "anything above zero," or rules tied to what a normal price should be. Five people said $500. Three said $100. There was no obvious default.

Then I asked whether their threshold would change if the purchase were fully refundable. Seventeen said yes. One participant put it plainly:

Refundable is more important than total cost to me.

That sounds like a usable rule. But the flight pair — where refundability changed alongside price — showed far less movement than the file-deletion pair.

I also saw mismatches at the individual level. One participant wrote, unprompted, essentially: if it is non-reversible, ask me first; if it is reversible, go ahead. Yet that person's scenario choices did not follow the rule cleanly.

Twenty-eight hypothetical responses do not show that people cannot report their preferences. They suggest something narrower: self-report alone may be an incomplete representation of procedural preference.

5. A clue from one interview: correcting a bad model may be easier than stating a rule

One participant had already encountered a version of this problem in a real AI product. A built-in feature had inferred a profile from her history. She rejected what it produced, and her reasons were more precise than what came out of a direct preference question.

It keeps bringing this up, even though I only said it twice. It's over-indexing on those two times.

It doesn't have the context that some projects are hacky, because I never told it.

Those comments name two failure modes of behavioral inference: over-weighting rare events, and missing the context that explains an exception.

What interested me was the asymmetry. A direct question produced a partial rule. A wrong inference immediately surfaced what was missing.

That suggests a product-design hypothesis: people may be better at recognizing a bad model of themselves than producing a complete model up front. If so, rejection is not just a failure state. It is a high-value learning event.

What I think this means

The question I care about is not:

What does this person prefer?

It is:

Given this person, in this situation, what should the agent do?

The first question invites a trait — something stable you could write into a profile. The second asks for a decision, and decisions depend on circumstances.

Preference as a distribution, not a value

If procedural preference were one trait, you could store it as a number: Autonomy level = 3.

The pilot makes me suspect the useful object is closer to a function: given a person and a situation, what behavior should the agent choose?

P(preferred action | person, situation)

Thinking in distributions matters because the system also needs to represent uncertainty. "This person definitely wants me to ask here" is different from "I have no idea what this person wants here."

When confidence is high, an agent might act. When confidence is low — or when being wrong is expensive — it should ask. Interruption becomes something the system reasons about rather than something the user must configure perfectly in advance.

A settings slider is not necessarily a bad interface. It may simply be a lossy compression of something that is conditional rather than one-dimensional.

What belongs in "this situation"?

The conditioning variable is probably not one thing. Across ten interviews, several dimensions kept surfacing:

• Recoverability and reversibility. The file-deletion pair produced the strongest movement, but it also changed severity and uncertainty, so I do not treat reversibility as isolated.
• Does it leave the building? People repeatedly distinguished internal work from customer-facing work in interviews. The survey showed a directional shift toward more oversight, but not a clean separation on the headline measures.
• Am I trying to learn this, or just finish it? Several people wanted to stay involved specifically to preserve a skill, not simply to catch errors.
• Do I know this domain or tool? The same person could delegate confidently in one area and micromanage in another.
• What does checking cost me? When verification is cheap, review is easy to ask for. When reviewing costs as much as doing, the trade-off changes.
• Is this one decision, or two hundred? Per-item value and repetition change the answer; people may tolerate irreversible automation when each individual decision is trivial and review is tedious.

I am not claiming this is the right list, or a complete one. It is what surfaced from ten conversations. But it is a better starting point than a single autonomy slider.

Why this is still a hypothesis

None of this is established by 28 hypothetical responses.

The pilot supports a narrower set of observations: people's answers vary across contexts in ways a single setting would not capture; the direction of that variation differed between my control and manipulation pairs; and stated rules did not always align with scenario choices.

What I cannot answer yet is whether that variation is stable enough, and learnable enough, to predict what someone will want on a task they have not seen before.

That is the next study. It is also the test that decides whether there is a real product here or just an interesting observation.

A question the pilot cannot answer

The strongest manipulation changed contact cleanup into file deletion. My first interpretation was irreversibility.

But a non-refundable flight creates a bounded loss: you know roughly what the downside is. Deleting files can create an uncertain loss: you may not know what disappeared, how important it was, or whether you can recover it.

So perhaps the important variable is not simply reversibility. It may be whether the user can bound the consequences of being wrong.

This pilot cannot separate those explanations. The next study should.

What I'm not claiming

This was formative work, not a representative or causal study.

• 28 participants. They came largely from my own network and skew heavily technical. Directional, not representative.
• The scenarios were hypothetical. I measured what people said they would want an agent to do, not how they actually behaved while delegating to a live agent.
• Everyone was measured once. I can say responses varied across contexts. I cannot yet say an individual's pattern is stable over time.
• The matched control is not a pure error estimate. The two scenarios are similar, not identical. Their difference can reflect wording, scenario-specific preference, response variability, or even order effects. The control itself leaned toward more autonomy, which I do not yet fully understand.

Those caveats are not footnotes to the result. They define what the result is.

What's next

The next version should answer two questions: are one person's procedural preferences stable, and are they learnable?

I want to recruit 30–50 participants, work with an academic collaborator, and measure the same people more than once over a period of weeks.

Then test something harder than whether I can describe their past answers:

Can a learned model predict how a person wants an agent to behave on a task they have never seen before — better than their own written rule, and better than a population default?

If it cannot, a settings panel is enough and I'll say so. If it can, there is a different kind of personalization layer to build — one that learns how you want work carried out from what you correct, override, redo, or take over.

If you would like to participate in the next study, let me know. Especially if you are not an engineer. This first sample skewed technical, and that is one of the things I most want to fix.

And to everyone who filled out the pilot — especially the ten people who gave me an hour or more — thank you. Several of you changed what I think the problem is. That is more useful than confirming what I already believed.