DNA: Proximal Policy Optimization with a Dual Network Architecture

This paper explores the problem of simultaneously learning a value function and policy in deep actor-critic reinforcement learning models. We find that the common practice of learning these functions jointly is sub-optimal, due to an order-of-magnitude difference in noise levels between these two tasks. Instead, we show that learning these tasks independently, but with a constrained distillation phase, significantly improves performance. Furthermore, we find that the policy gradient noise levels can be decreased by using a lower $\mathit{\text{variance}}$ return estimate. Whereas, the value learning noise level decreases with a lower $\mathit{\text{bias}}$ estimate. Together these insights inform an extension to Proximal Policy Optimization we call $\mathit{\text{Dual}}\mathit{\text{Network}}\mathit{\text{Architecture}}$ (DNA), which significantly outperforms its predecessor. DNA also exceeds the performance of the popular Rainbow DQN algorithm on four of the five environments tested, even under more difficult stochastic control settings.