Apple study finds users want AI agents that explain themselves — but don’t get in the way

As technology companies accelerate investment in autonomous AI agents capable of navigating apps and websites on behalf of users, new research from Apple suggests the interface layer — not just the model performance — may determine whether these systems gain mainstream trust.

In a recently published paper, Mapping the Design Space of User Experience for Computer Use Agents, a team of Apple researchers examined how users expect to interact with agents that can execute tasks such as booking rentals or shopping online. The findings indicate a widening gap between how AI agents are currently designed and how users believe they should behave.

The study begins with a premise: the industry has focused heavily on benchmarking AI agents for task completion, but has paid comparatively less attention to the user experience surrounding those tasks. To address this, researchers structured their work in two phases — first cataloging existing design approaches across leading agent systems, then testing user reactions through controlled interaction experiments.

In the initial phase, the team analyzed nine desktop, web, and mobile AI agents, including tools such as Claude’s Computer Use, OpenAI’s Operator, and Project Mariner. They also consulted eight UX and AI practitioners to construct a taxonomy of agent design patterns.

That framework organizes agent experience into four core categories: how users issue commands; how agents explain their actions; how users retain control; and how systems shape user expectations. Across these categories, the researchers identified 21 subcategories and 55 specific features, ranging from plan previews to interrupt mechanisms and error transparency.

The taxonomy underscores a core design tension: agents must act independently, but not opaquely.

To test how these design patterns hold up under real-world conditions, the researchers conducted a Wizard-of-Oz study involving 20 participants familiar with AI tools. Participants interacted with what they believed to be an autonomous agent via chat, assigning it either a vacation rental task or an online shopping assignment.

Behind the scenes, however, a researcher manually executed the requested actions in a separate room — simulating the agent’s behavior while preserving experimental control. Participants could interrupt execution, observe the agent’s actions in a parallel interface, and review completion messages.

During each session, the “agent” was programmed to occasionally make mistakes or encounter obstacles, such as selecting incorrect products or entering navigation loops. Afterward, participants reflected on their experience and suggested interface improvements.

The recordings and chat logs revealed consistent behavioral patterns.

Users repeatedly expressed a desire to understand what the agent was doing — but not to supervise every click. When the system exposed intermediate steps, clarified ambiguous decisions, or paused before executing consequential actions, trust increased. When it silently made assumptions, trust deteriorated quickly.

The level of desired oversight also shifted depending on context. When users were exploring unfamiliar websites, they preferred more detailed explanations and confirmation checkpoints. In contrast, for routine or low-risk tasks, they expected faster execution with minimal friction.

Consequential actions — such as making purchases, modifying payment information, or contacting others — triggered heightened expectations for control and explicit confirmation. Participants were particularly uncomfortable when the agent selected options without clearly signaling why.

Ambiguity proved to be a fault line. When faced with multiple valid choices, users preferred the system to pause and request clarification rather than proceed autonomously.

The research suggests that user trust hinges less on raw task accuracy and more on perceived alignment and transparency. Users want AI agents that communicate intent, surface uncertainty, and allow intervention — without requiring constant supervision.

The study’s implications extend beyond experimental agents. As software platforms integrate increasingly agentic features — from automated browsing to workflow execution — the interface decisions surrounding explainability and control may determine adoption rates.

The paper stops short of prescribing a single design model. Instead, it highlights variability: expectations shift based on familiarity, risk level, and task type. In practice, this may require adaptive interfaces that calibrate transparency and autonomy dynamically.