AgentMixer: Multi-Agent Correlated Policy Factorization

In multi-agent reinforcement learning, centralized training with decentralized execution (CTDE) methods typically assume that agents make decisions based on their local observations independently, which may not lead to a correlated joint policy with coordination. Coordination can be explicitly encouraged during training and individual policies can be trained to imitate the correlated joint policy. However, this may lead to an $\mathit{\text{asymmetric}}\mathit{\text{learning}}\mathit{\text{failure}}$ due to the observation mismatch between the joint and individual policies. Inspired by the concept of correlated equilibrium, we introduce a $\mathit{\text{strategy}}\mathit{\text{modification}}$ called AgentMixer that allows agents to correlate their policies. AgentMixer combines individual partially observable policies into a joint fully observable policy non-linearly. To enable decentralized execution, we introduce $\mathit{\text{Individual-Global-Consistency}}$ to guarantee mode consistency during joint training of the centralized and decentralized policies and prove that AgentMixer converges to an $\epsilon$-approximate Correlated Equilibrium. In the Multi-Agent MuJoCo, SMAC-v2, Matrix Game, and Predator-Prey benchmarks, AgentMixer outperforms or matches state-of-the-art methods.