DL-Reg:A Deep Learning Regularization Technique using Linear Regression

Dialameh, Maryam and Hamzeh, Ali and Rahmani, Hossein (2020) DL-Reg:A Deep Learning Regularization Technique using Linear Regression. arXiv.

[img]
Text (2011.00368v2)
2011.00368v2.pdf - Published Version

Download (1MB)

Abstract

Regularization plays a vital role in the context of deep learning by preventing deep neural networks from the danger of overfitting. This paper proposes a novel deep learning regularization method named as DL-Reg, which carefully reduces the nonlinearity of deep networks to a certain extent by explicitly enforcing the network to behave as much linear as possible. The key idea is to add a linear constraint to the objective function of the deep neural networks, which is simply the error of a linear mapping from the inputs to the outputs of the model. More precisely, the proposed DL-Reg carefully forces the network to behave in a linear manner. This linear constraint, which is further adjusted by a regularization factor, prevents the network from the risk of overfitting. The performance of DL-Reg is evaluated by training state-of-the-art deep network models on several benchmark datasets. The experimental results show that the proposed regularization method: 1) gives major improvements over the existing regularization techniques, and 2) significantly improves the performance of deep neural networks, especially in the case of small-sized training datasets.

Item Type:
Journal Article
Journal or Publication Title:
arXiv
Subjects:
ID Code:
148865
Deposited By:
Deposited On:
10 Nov 2020 14:15
Refereed?:
No
Published?:
Published
Last Modified:
29 Nov 2020 07:06