Mixture of Latent Experts Using Tensor Products

In multi-task learning, the conventional approach involves training a model on multiple tasks simultaneously. However, the training signals from different tasks can interfere with one another, potentially leading to $\mathit{\text{negative}}\mathit{\text{transfer}}$. To mitigate this, we investigate if modular language models can facilitate positive transfer and systematic generalization. Specifically, we propose a novel modular language model ($\mathtt{\text{TensorPoly}}$), that balances parameter efficiency with nuanced routing methods. For $\mathit{\text{modules}}$, we reparameterize Low-Rank Adaptation ($\mathtt{\text{LoRA}}$) by employing an entangled tensor through the use of tensor product operations and name the resulting approach $\mathtt{\text{TLoRA}}$. For $\mathit{\text{routing}}\mathit{\text{function}}$, we tailor two innovative routing functions according to the granularity: $\mathtt{\text{TensorPoly-I}}$ which directs to each rank within the entangled tensor while $\mathtt{\text{TensorPoly-II}}$ offers a finer-grained routing approach targeting each order of the entangled tensor. The experimental results from the multi-task T0-benchmark demonstrate that: 1) all modular LMs surpass the corresponding dense approaches, highlighting the potential of modular language models to mitigate negative inference in multi-task learning and deliver superior outcomes. 2) $\mathtt{\text{TensorPoly-I}}$ achieves higher parameter efficiency in adaptation and outperforms other modular LMs, which shows the potential of our approach in multi-task transfer learning.