Foo, Lin Geng and Gong, Jia and Fan, Zhipeng and Liu, Jun (2023) System-Status-Aware Adaptive Network for Online Streaming Video Understanding. In: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) :. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition . IEEE Computer Society Press, CAN, pp. 10514-10523. ISBN 9798350301304
Full text not available from this repository.Abstract
Recent years have witnessed great progress in deep neural networks for real-time applications. However, most existing works do not explicitly consider the general case where the device's state and the available resources fluctuate over time, and none of them investigate or address the impact of varying computational resources for online video understanding tasks. This paper proposes a System-status-aware Adaptive Network (SAN) that considers the device's real-time state to provide high-quality predictions with low delay. Usage of our agent's policy improves efficiency and robustness to fluctuations of the system status. On two widely used video understanding tasks, SAN obtains state-of-the-art performance while constantly keeping processing delays low. Moreover, training such an agent on various types of hardware configurations is not easy as the labeled training data might not be available, or can be computationally prohibitive. To address this challenging problem, we propose a Meta Self-supervised Adaptation (MSA) method that adapts the agent's policy to new hardware configurations at test-time, allowing for easy deployment of the model onto other unseen hardware platforms.