Neural Differentiation in Deep Networks : A Theoretical Framework for Expressivity and Representational Diversity

Wang, Boyuan and Jiang, Richard (2026) Neural Differentiation in Deep Networks : A Theoretical Framework for Expressivity and Representational Diversity. In: 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) :. IEEE. (In Press)

Text (NeuralDiff_CVPR_2026)
NeuralDiff_CVPR_2026.pdf - Accepted Version
Available under License Creative Commons Attribution.
Download (2MB)

Abstract

We begin by developing a mathematical framework of neural differentiation, formulated at the level of individual neurons. This framework formalizes the principle that each neuron should acquire a distinct representational role within the network, thereby avoiding redundancy and maximizing collective expressivity. Differentiation is quantified through the Neural Differentiation Index (NDI), a loss-aware measure that characterizes neuron significance from geometric, informational, and curvature-based perspectives within a unified framework. The NDI enables a rigorous characterization of how strongly a neuron diverges from its peers in both function and importance, and supports theoretical guarantees: we establish formal bounds on the error increase under NDI-guided elimination, thereby providing provable safety margins for network compression. Building on this foundation, we introduce Neural Differentiation Pruning (NDP) as a practical instantiation. NDP leverages NDI to perform adaptive, training-time neuron sparsification, followed by targeted fine-tuning, guiding networks toward compact yet highly differentiated backbones. Although the terminology draws loose intuition from biological differentiation, the framework is fully mathematical and architecture-agnostic. Experiments on modern vision benchmarks and architectures show that NDP achieves substantial structured sparsity while maintaining—or even improving—accuracy and robustness, underscoring the practical impact of the differentiation framework.