Deep Manifold Structure Transfer for Action Recognition

Li, Ce and Zhang, Baochang and Chen, Chen and Ye, Qixiang and Han, Jungong and Guo, Guodong and JI, Rongrong (2019) Deep Manifold Structure Transfer for Action Recognition. IEEE Transactions on Image Processing, 28 (9). 4646 - 4658. ISSN 1057-7149

[thumbnail of TIP_Final_20190414]
PDF (TIP_Final_20190414)
TIP_Final_20190414.pdf - Accepted Version
Available under License Creative Commons Attribution-NonCommercial.

Download (3MB)


While intrinsic data structure in subspace provides useful information for visual recognition, it has not yet been well studied in deep feature learning for action recognition. In this paper, we introduce a new spatio-temporal manifold network (STMN) that leverages data manifold structures to regularize deep action feature learning, aiming at simultaneously minimizing the intra-class variations of learned deep features and alleviating the over-fitting problem. To this end, the manifold prior is imposed from the top layer of a convolutional neural network (CNN), and is propagated across convolutional layers during forward-backward propagation. The observed correspondence of manifold structures in the data space and feature space validates that the manifold priori can be transferred across CNN layers. STMN theoretically recasts the problem of transferring the data structure prior into the deep learning architectures as a projection over the manifold via an embedding method, which can be easily solved by an Alternating Direction Method of Multipliers and Backward Propagation (ADMM-BP) algorithm. STMN is generic in the sense that it can be plugged into various backbone architectures to learn more discriminative representation for action recognition. Extensive experimental results show that our method achieves comparable or even better performance as compared with the state-of-the-art approaches on four benchmark datasets.

Item Type:
Journal Article
Journal or Publication Title:
IEEE Transactions on Image Processing
Additional Information:
©2019 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.
Uncontrolled Keywords:
ID Code:
Deposited By:
Deposited On:
22 Jun 2019 09:12
Last Modified:
21 Sep 2023 02:37