Multi-objective Large Language Model Alignment with Hierarchical Experts

Aligning large language models (LLMs) to simultaneously satisfy multiple objectives remains a significant challenge, especially given the diverse and often conflicting nature of human preferences. Existing alignment methods struggle to balance trade-offs effectively, often requiring costly retraining or yielding suboptimal results across the Pareto frontier of preferences. In this paper, we introduce $\mathit{\text{HoE}}$(Hierarchical Mixture-of-Experts), a $\mathit{\text{lightweight}}$, $\mathit{\text{parameter-efficient}}$, and $\mathit{\text{plug-and-play}}$ approach that eliminates the need for model training, while enabling LLMs to adapt across the entire Pareto frontier and accommodate diverse user preferences. In particular, $\mathit{\text{HoE}}$ consists of three hierarchical components: LoRA Experts, Router Experts and Preference Routing, reaching optimal Pareto frontiers and achieving a trade-off between parameter size, training cost, and performance. We evaluate $\mathit{\text{HoE}}$ across various tasks on 14 objectives and 200 different preferences among 6 benchmarks, demonstrating superior performance over 15 recent baselines. Code is available in the supplementary materials.