Faster, Smaller, and Smarter: Task-Aware Expert Merging for Online MoE Inference

Sparse Mixture of Experts (SMoE) has become a preferred architecture for scaling Transformer capacity without increasing computational cost, as it activates only a small subset of experts for each input. However, deploying such an approach for $\mathit{\text{online}}\mathit{\text{inference}}$ remains challenging due to the large size of a full SMoE model and the complexity of expert routing, especially in resource-constrained edge networks. Moreover, during the online inference, task information is often unavailable, making the task-level routing error-prone. In this work, we propose a novel tree-structured adaptive neural bandit router, $\mathtt{\text{Tanbr}}$, to enable efficient and reliable online MoE inference. Instead of relying on explicit task tags, $\mathtt{\text{Tanbr}}$ estimates the task distribution over time from historical data and uses it to guide task-aware expert merging within a given pre-trained MoE. To handle the large continuous space of merging weights, $\mathtt{\text{Tanbr}}$ employs a binary tree to progressively partition the space and generate finer candidate weights. It then applies a neural bandit to learn the non-linear mapping from merging weight to model performance and decides optimal expert merging. We prove that $\mathtt{\text{Tanbr}}$ achieves a sublinear regret bound of {\small $\mathcal{O}(\sqrt{T}\log (T))$} over {\small $T$} rounds, despite operating over a continuous decision space, matching regret bounds compared to existing methods. Extensive experiments show that $\mathtt{\text{Tanbr}}$ reduces inference latency by at least {\small $45\mathrm{\%}$} and memory usage by up to {\small $25\mathrm{\%}$}, while maintaining a high accuracy compared to many state-of-the-art methods.