Experiences from Benchmarking Vision-Language-Action Models for Robotic Manipulation

Foundation models applied in robotics, particularly $\mathbf{\text{Vision–Language–Action}}\mathbf{\text{(VLA)}}$ models, hold great promise for achieving general-purpose manipulation. Yet, systematic real-world evaluations and cross-model comparisons remain scarce. This paper reports our $\mathbf{\text{empirical}}\mathbf{\text{experiences}}$ from benchmarking four representative VLAs – $\mathbf{\text{ACT}}$, $\mathbf{\text{OpenVLA–OFT}}$, $\mathbf{\text{RDT-1B}}$, and \boldmath{${\pi }_0$} – across four manipulation tasks conducted in both simulation and on the $\mathbf{\text{ALOHA}}\mathbf{\text{Mobile}}$ platform. We establish a $\mathbf{\text{standardized}}\mathbf{\text{evaluation}}\mathbf{\text{framework}}$ that measures performance along three key dimensions: (1) $\mathit{\text{accuracy}}\mathit{\text{and}}\mathit{\text{efficiency}}$ (success rate and time-to-success), (2) $\mathit{\text{adaptability}}$ across in-distribution, spatial out-of-distribution, and instance-plus-spatial out-of-distribution settings, and (3) $\mathit{\text{language}}\mathit{\text{instruction-following}}\mathit{\text{accuracy}}$. Through this process, we observe that \boldmath{${\pi }_0$} demonstrates superior adaptability in out-of-distribution scenarios, while $\mathbf{\text{ACT}}$ provides the highest stability in-distribution. Further analysis highlights differences in computational demands, data-scaling behavior, and recurring failure modes such as near-miss grasps, premature releases, and long-horizon state drift. These findings reveal practical trade-offs among VLA model architectures in balancing precision, generalization, and deployment cost, offering actionable insights for selecting and deploying VLAs in real-world robotic manipulation tasks.