Experts Collaboration Learning for Continual Multi-Modal Reasoning

Xu, Li and Liu, Jun (2023) Experts Collaboration Learning for Continual Multi-Modal Reasoning. IEEE Transactions on Image Processing, 32. pp. 5087-5098. ISSN 1057-7149

Abstract

Multi-modal reasoning, which aims to capture logical and causal structures in visual content and associate them with cues from other modality inputs (e.g., texts) to perform various types of reasoning, is an important research topic in artificial intelligence (AI). Existing works for multi-modal reasoning mainly exploit offline learning, where the training samples of all types of reasoning tasks are assumed to be available at once. Here we focus on continual learning for multi-modal reasoning (i.e., continual multi-modal reasoning), where the model is required to continuously learn to solve novel types of multi-modal reasoning tasks in a lifelong fashion. Continual multi-modal reasoning is challenging since the model needs to be able to effectively learn various types of new reasoning tasks, meanwhile avoiding forgetting. Here we propose a novel brain-inspired exp erts co llaboration network (Expo), which incorporates multiple learning blocks (experts). When encountering a new task, our network dynamically assembles and updates a set of task-specific experts that are most relevant to learning the current task, by either utilizing learned experts or exploring new experts. This thus enables effective learning of new tasks, and meanwhile consolidates previously learned reasoning skills. Moreover, to automatically find optimal task-specific experts, an effective experts selection strategy is designed. Extensive experiments demonstrate the efficacy of our model for continual multi-modal reasoning.