Fast-SAM3D: 3Dfy Anything in Images but Faster

SAM3D enables scalable, open-world 3D reconstruction from complex scenes, yet its deployment is hindered by prohibitive inference latency. In this work, we conduct the $\mathbf{\text{first}}\mathbf{\text{systematic}}\mathbf{\text{investigation}}$ into its inference dynamics, revealing that generic acceleration strategies are brittle in this context. We demonstrate that these failures stem from neglecting the pipeline’s inherent multi-level $\mathbf{\text{heterogeneity}}$: the kinematic distinctiveness between shape and layout, the intrinsic sparsity of texture refinement, and the spectral variance across geometries. To address this, we present $\mathbf{\text{Fast-SAM3D}}$, a training-free framework that dynamically aligns computation with instantaneous generation complexity. Our approach integrates three heterogeneity-aware mechanisms: (1) $\mathit{\text{Modality-Aware}}\mathit{\text{Step}}\mathit{\text{Caching}}$ to decouple structural evolution from sensitive layout updates; (2) $\mathit{\text{Joint}}\mathit{\text{Spatiotemporal}}\mathit{\text{Token}}\mathit{\text{Carving}}$ to concentrate refinement on high-entropy regions; and (3) $\mathit{\text{Spectral-Aware}}\mathit{\text{Token}}\mathit{\text{Aggregation}}$ to adapt decoding resolution. Extensive experiments demonstrate that Fast-SAM3D delivers up to $\mathbf{\text{2.67}}\times$ end-to-end speedup with negligible fidelity loss, establishing a new Pareto frontier for efficient single-view 3D generation. Our code is released in https://github.com/wlfeng0509/Fast-SAM3D.