1. Where Today's Personal Agents Fail
Seven moments from the runs, one per domain. No case was resolved by every personal agent: at least four failed each one, and all seven failed the Work and Money cases, though everything they needed was in the person's own accounts or an earlier conversation.
2. Benchmark
2.1 Overview
7 domains
- Travel
- Dining
- Shopping
- Money
- Health
- Home
- Work
8 capabilities
- Memory
- Safety
- Consistency
- Personalization
- Proactivity
- Retrieval
- Communication
- Execution
Every evaluation task introduces hidden personal context that naive execution cannot resolve. Deciding facts are credited only when discovered proactively by the personal agent—either by retrieving relevant records or by raising targeted inquiries before the user points them out.
2.2 Persona
2.3 Workflow
MatrAIx Eval Infra
Control channel
Harness
Product surface
Data channel
Connection
Persona's world
Each task is an errand run end to end through the personal agent's own interface, on a fresh copy of the person's accounts reached through 52 tools over MCP or a private web portal. A language model plays the person: it answers questions and confirms actions, but never volunteers what the records already hold.
Every task and the simulated user are validated. Every single task and every long-horizon task was checked by hand end to end: the deciding records can be found, the reference solution meets every constraint, and the checks score the end state without false penalties or loopholes. The simulated user was validated for fidelity: it stays true to the person's profile and brief, answers what the personal agent asks, and never leaks a hidden constraint or volunteers what the records already hold.
3. Results
Resolved rate
4. Analysis
4.1 Efficiency and Cost
4.2 Case Studies
Proactive task. In the replay above, an unread text asked to move Priya's family dinner to Sunday and to add a guest: Instinct read it and told Priya before booking, while Grok Bot never opened it and booked Saturday for five.
Long-horizon task.
5. Citation
PersonalAgentEval is built by MatrAIx.
@misc{personalagenteval2026,
title = {PersonalAgentEval: Evaluating Personal Agents via User Simulation in Dynamic Digital Worlds},
author = {{MatrAIx}},
year = {2026},
month = oct,
url = {https://personalagenteval.com/}
}
Want your personal agent evaluated? Contact us.