PushupBench: Your VLM is not good at counting pushups

Large vision-language models (VLMs) can recognize $\mathit{\text{what}}$ happens in video but fail to count $\mathit{\text{how}}\mathit{\text{many}}$ times. We introduce $\mathbf{\text{PushupBench}}$, 446 long-form clips (avg. 36.7s) for evaluating repetition counting. The best frontier model achieves 42.1% exact accuracy; open-source 4B models score $\sim$6%, matching supervised baselines. We show that accuracy alone misleads – weaker models exploit the modal count rather than reason temporally. Fine-tuning on counting with 1k samples transfers to general video understanding: MVBench (+2.15), PerceptionTest (+1.88), TVBench (+4.54), suggesting counting is a proxy for broader temporal reasoning.PushupBench incorporated in $\mathtt{\text{lmms-eval}}$ (https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/1262) and hosted on (pushupbench.com/)