Human Action Recognition from Various Data Modalities : A Review

Sun, Zehua and Ke, Qiuhong and Rahmani, Hossein and Bennamoun, Mohammed and Wang, Gang and Liu, Jun (2023) Human Action Recognition from Various Data Modalities : A Review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 (3): 3. pp. 3200-3225. ISSN 0162-8828

[thumbnail of 2012.11866]
Text (2012.11866)
2012.11866.pdf - Accepted Version
Available under License Creative Commons Attribution-NonCommercial.

Download (3MB)


Human Action Recognition (HAR) aims to understand human behavior and assign a label to each action. It has a wide range of applications, and therefore has been attracting increasing attention in the field of computer vision. Human actions can be represented using various data modalities, such as RGB, skeleton, depth, infrared, point cloud, event stream, audio, acceleration, radar, and WiFi signal, which encode different sources of useful yet distinct information and have various advantages depending on the application scenarios. Consequently, lots of existing works have attempted to investigate different types of approaches for HAR using various modalities. In this paper, we present a comprehensive survey of recent progress in deep learning methods for HAR based on the type of input data modality. Specifically, we review the current mainstream deep learning methods for single data modalities and multiple data modalities, including the fusion-based and the co-learning-based frameworks. We also present comparative results on several benchmark datasets for HAR, together with insightful observations and inspiring future research directions.

Item Type:
Journal Article
Journal or Publication Title:
IEEE Transactions on Pattern Analysis and Machine Intelligence
Additional Information:
©2022 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.
Uncontrolled Keywords:
?? artificial intelligencecomputational theory and mathematicssoftwareapplied mathematicscomputer vision and pattern recognition ??
ID Code:
Deposited By:
Deposited On:
22 Jun 2022 10:05
Last Modified:
25 May 2024 01:46