PRISM: : Planning and Reasoning with Intent in Simulated Embodied Environments

When an LLM-based embodied agent fails at a household task, the culprit could be misidentified objects, forgotten sub-goals, or poor action sequencing – yet existing benchmarks report only a single success rate, making it impossible to tell which cognitive module is responsible. We present PRISM, a diagnostic benchmark that reframes this problem: rather than asking only $\mathit{\text{did}}\mathit{\text{the}}\mathit{\text{agent}}\mathit{\text{succeed?}}$, PRISM asks $\mathit{\text{which}}\mathit{\text{capability}}\mathit{\text{is}}\mathit{\text{most}}\mathit{\text{likely}}\mathit{\text{responsible}}\mathit{\text{for}}\mathit{\text{failure?}}$ Built on five photorealistic multi-room apartments (4–8 rooms each), PRISM structures 300 human-verified tasks into three capability tiers – $\mathit{\text{Basic}}\mathit{\text{Ability}}$, $\mathit{\text{Reasoning}}\mathit{\text{Ability}}$, and $\mathit{\text{Long-horizon}}\mathit{\text{Ability}}$ – that isolate perception-to-action grounding, implicit intent resolution, and sustained multi-step coordination respectively. PRISM exposes an agent-agnostic executable action API that allows arbitrary agents: LLM agents, VLM agents, symbolic planners, RL policies, and hybrid systems, to be evaluated end-to-end under the same benchmark protocol. To support deeper diagnosis, optional probes for perception, memory, and planning can be adopted, replaced, or bypassed entirely, enabling controlled component-level analysis when desired. Experiments on seven contemporary LLMs establish a clear hierarchy: explicit spatial grounding is not the dominant failure source under oracle perception, implicit intent resolution is a significant bottleneck for all model families, and long-horizon coordination exposes a stark capability cliff – lightweight models collapse to as low as 20.0% success while simultaneously consuming more tokens than their frontier counterparts, a signature of compensatory over-reasoning rather than genuine planning capability. Project page: \href{https://sj-li.com/PROJ/PRISM}{link}.