On Relation-Specific Neurons in Large Language Models

In large language models (LLMs), certain \emph{neurons} can store distinct pieces of knowledge learned during pretraining. While factual knowledge typically appears as a combination of \emph{relations} and \emph{entities}, it remains unclear whether some neurons focus on a relation itself – independent of any entity. We hypothesize such neurons \emph{detect} a relation in the input text and \emph{guide} generation involving such a relation. To investigate this, we study the LLama-2 family on a chosen set of relations, with a $\mathit{\text{statistics}}$-based method. Our experiments demonstrate the existence of relation-specific neurons. We measure the effect of selectively deactivating candidate neurons specific to relation $r$ on the LLM’s ability to handle (1) facts involving relation $r$ and (2) facts involving a different relation $r^{\mathrm{\prime }}\mathrm{\ne }r$. With respect to their capacity for encoding relation information, we give evidence for the following three properties of relation-specific neurons. $\mathbf{\text{(i)}}\mathbf{\text{Neuron}}\mathbf{\text{cumulativity.}}$ Multiple neurons jointly contribute to processing facts involving relation $r$, with no single neuron fully encoding a fact in $r$ on its own. $\mathbf{\text{(ii)}}\mathbf{\text{Neuron}}\mathbf{\text{versatility.}}$ Neurons can be shared across multiple closely related as well as less related relations. In addition, some relation neurons transfer across languages. $\mathbf{\text{(iii)}}\mathbf{\text{Neuron}}\mathbf{\text{interference.}}$ Deactivating neurons specific to one relation can improve LLMs’ factual recall performance for facts of other relations. We make our code and data publicly available at https://github.com/cisnlp/relation-specific-neurons.