One-layer transformers fail to solve the induction heads task
Clayton SanfordDaniel HsuMatus Telgarsky
所有作者
Clayton SanfordDaniel HsuMatus Telgarsky
文献摘要
A simple communication complexity argument proves that no one-layer transformer can solve the induction heads task unless its size is exponentially larger than the size sufficient for a two-layer transformer.