How to Guide Your Flow: Few-Step Alignment via Flow Map Reward Guidance

In generative modeling, we often wish to produce samples that maximize a user-specified reward such as aesthetic quality or alignment with human preferences, a problem known as $\mathit{\text{guidance}}$. Despite their widespread use, existing guidance methods either require expensive multi-particle, many-step schemes or rely on poorly understood approximations. We reformulate guidance as a $\mathit{\text{deterministic}}\mathit{\text{optimal}}\mathit{\text{control}}\mathit{\text{problem}}$, yielding a hierarchy of algorithms that subsumes existing approaches at the coarsest level. We show that the $\mathit{\text{flow}}\mathit{\text{map}}$, an object of significant recent interest for its role in fast inference, arises naturally in the optimal solution. Based on this observation, we propose $\mathbf{\text{Flow}}\mathbf{\text{Map}}\mathbf{\text{Reward}}\mathbf{\text{Guidance}}\mathbf{\text{(FMRG)}}$: a training-free, $\mathit{\text{single-trajectory}}$ framework that uses the flow map to both integrate and guide the flow. At text-to-image scale, FMRG matches or surpasses baselines across inverse problems and reward-guided generation with $\mathbf{\text{as}}\mathbf{\text{few}}\mathbf{\text{as}}\mathbf{\text{3}}\mathbf{\text{NFEs}}$, giving at least an order-of-magnitude speedup in comparison to prior state of the art. Code is available at https://github.com/jrrhuang/fmrg.