S${}^2$NN: Sub-bit Spiking Neural Networks

Spiking Neural Networks (SNNs) offer an energy-efficient paradigm for machine intelligence, but their continued scaling poses challenges for resource-limited deployment. Despite recent advances in binary SNNs, the storage and computational demands remain substantial for large-scale networks. To further explore the compression and acceleration potential of SNNs, we propose Sub-bit Spiking Neural Networks (S${}^2$NNs) that represent weights with less than one bit. Specifically, we first establish an S${}^2$NN baseline by leveraging the clustering patterns of kernels in well-trained binary SNNs. This baseline is highly efficient but suffers from $\mathit{\text{outlier-induced}}\mathit{\text{codeword}}\mathit{\text{selection}}\mathit{\text{bias}}$ during training. To mitigate this issue, we propose an $\mathit{\text{outlier-aware}}\mathit{\text{sub-bit}}\mathit{\text{weight}}\mathit{\text{quantization}}$ (OS-Quant) method, which optimizes codeword selection by identifying and adaptively scaling outliers. Furthermore, we propose a $\mathit{\text{membrane}}\mathit{\text{potential-based}}\mathit{\text{feature}}\mathit{\text{distillation}}$ (MPFD) method, improving the performance of highly compressed S${}^2$NN via more precise guidance from a teacher model. Extensive results on vision tasks reveal that S${}^2$NN outperforms existing quantized SNNs in both performance and efficiency, making it promising for edge computing applications.