RankMamba: Benchmarking Mamba's Document Ranking Performance in the Era of Transformers

Transformer structure has achieved great success in multiple applied machine learning communities, such as natural language processing (NLP), computer vision (CV) and information retrieval (IR). Transformer architecture’s core mechanism, – ,attention requires $O(n^2)$ time complexity in training and $O(n)$ time complexity in inference. Many works have been proposed to improve the attention mechanism’s scalability, such as Flash Attention and Multi-query Attention. A different line of work aims to design new mechanisms to replace attention. Recently, a notable model structure Mamba, which is based on state space models, has achieved transformer-equivalent performance in multiple sequence modeling tasks. In this work, we examine Mamba’s efficacy through the lens of a classical IR task, – ,document ranking. A reranker model takes a query and a document as input, and predicts a scalar relevance score. This task demands the language model’s ability to comprehend lengthy contextual inputs and to capture the interaction between query and document tokens. We find that $\mathbf{\text{(1)}}\mathbf{\text{Mamba}}\mathbf{\text{models}}\mathbf{\text{achieve}}\mathbf{\text{competitive}}\mathbf{\text{performance}}\mathbf{\text{compared}}\mathbf{\text{to}}\mathbf{\text{transformer-based}}\mathbf{\text{models}}\mathbf{\text{with}}\mathbf{\text{the}}\mathbf{\text{same}}\mathbf{\text{training}}\mathbf{\text{recipe;}}\mathbf{\text{(2)}}\mathbf{\text{but}}\mathbf{\text{also}}\mathbf{\text{have}}\mathbf{\text{a}}\mathbf{\text{lower}}\mathbf{\text{training}}\mathbf{\text{throughput}}\mathbf{\text{in}}\mathbf{\text{comparison}}\mathbf{\text{to}}\mathbf{\text{efficient}}\mathbf{\text{transformer}}\mathbf{\text{implementations}}\mathbf{\text{such}}\mathbf{\text{as}}\mathbf{\text{flash}}\mathbf{\text{attention.}}$ We hope this study can serve as a starting point to explore \mamba models in other classical IR tasks. Our \href{https://github.com/zhichaoxu-shufe/RankMamba}{code implementation} is made public to facilitate reproducibility. Refer to~\cite{xu-etal-2025-state} for more comprehensive experiments and results, including passage ranking.