Lipschitz Continuity in Deep Learning: A Systematic Review of Theoretical Foundations, Estimation Methods, Regularization Approaches, and Certifiable Robustness

Lipschitz continuity is a fundamental property of neural networks that characterizes their sensitivity to input perturbations. It plays a pivotal role in deep learning, governing $\mathbf{\text{robustness}}$, $\mathbf{\text{generalization}}$ and $\mathbf{\text{optimization}}\mathbf{\text{dynamics}}$. Despite its importance, research on Lipschitz continuity is scattered across various domains, lacking a unified perspective. This paper addresses this gap by providing a systematic review of Lipschitz continuity in deep learning. We explore its $\mathbf{\text{theoretical}}\mathbf{\text{foundations}}$, $\mathbf{\text{estimation}}\mathbf{\text{methods}}$, $\mathbf{\text{regularization}}\mathbf{\text{approaches}}$, and $\mathbf{\text{certifiable}}\mathbf{\text{robustness}}$. By reviewing existing research through the lens of Lipschitz continuity, this survey serves as a comprehensive reference for researchers and practitioners seeking a deeper understanding of Lipschitz continuity and its implications in deep learning.