CircuitBreak: Pruning a Sparse Safety Circuit of Neurons

White-box neuron pruning has shown that refusal in large language models (LLMs) can depend on a small, removable set of feed-forward (FFN) channels. Existing attacks either need carefully paired ``twin’’ prompts or score each neuron in isolation. Pruning also has an unavoidable capability cost, so over-removal compounds the loss. TwinBreak removes $1\mathrm{\%}$–$5\mathrm{\%}$ of parameters and NeuroStrike removes $0.3\mathrm{\%}$–$0.7\mathrm{\%}$ of neurons in targeted layers. We model safety as a small \emph{circuit} whose neurons contribute jointly rather than independently. We propose $\text{textsc}CircuitBreak$, a neuron-pruning attack that greedily searches for a sparse set of FFN neurons whose joint ablation flips a refusal readout. $\text{textsc}CircuitBreak$ re-scores every candidate after the neurons already removed and needs no twin prompts. On ten instruction-tuned models, the greedy search converges after removing $29$–$150$ neurons ($0.0066\mathrm{\%}$–$0.0388\mathrm{\%}$ of the MLP). The pruned models raise the attack success rate from $2.24\mathrm{\%}$/$7.35\mathrm{\%}$ to $87.54\mathrm{\%}$/$87.54\mathrm{\%}$ on Llama-3.2-3B-Instruct/Llama-3.1-8B-Instruct and from $8.95\mathrm{\%}$/$5.43\mathrm{\%}$ to $88.50\mathrm{\%}$/$88.18\mathrm{\%}$ on Qwen2.5-7B-Instruct/Qwen2.5-14B-Instruct, as judged by Llama Guard 3 8B on StrongREJECT.