Interventional Video Grounding with Dual Contrastive Learning

Nan, Guoshun and Qiao, Rui and Xiao, Yao and Liu, Jun and Leng, Sicong and Zhang, Hao and Lu, Wei (2021) Interventional Video Grounding with Dual Contrastive Learning. In: Proceedings - 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2021 :. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition . IEEE Computer Society Press, USA, pp. 2764-2774. ISBN 9781665445108

Abstract

Video grounding aims to localize a moment from an untrimmed video for a given textual query. Existing approaches focus more on the alignment of visual and language stimuli with various likelihood-based matching or regression strategies, i.e., P(Y |X). Consequently, these models may suffer from spurious correlations between the language and video features due to the selection bias of the dataset. 1) To uncover the causality behind the model and data, we first propose a novel paradigm from the perspective of the causal inference, i.e., interventional video grounding (IVG) that leverages backdoor adjustment to deconfound the selection bias based on structured causal model (SCM) and do-calculus P(Y |do(X)). Then, we present a simple yet effective method to approximate the unobserved confounder as it cannot be directly sampled from the dataset. 2) Meanwhile, we introduce a dual contrastive learning approach (DCL) to better align the text and video by maximizing the mutual information (MI) between query and video clips, and the MI between start/end frames of a target moment and the others within a video to learn more informative visual representations. Experiments on three standard benchmarks show the effectiveness of our approaches.