CoGA : A Collaborative Gray-Box Adversarial Attack for Multimodal Language Models

Wu, T. and Lin, F. and Wang, G. and Liu, T. and Wang, Z. and Meng, W. and Liu, A. and Ren, K. (2026) CoGA : A Collaborative Gray-Box Adversarial Attack for Multimodal Language Models. IEEE Transactions on Information Forensics and Security, 21. pp. 870-885. ISSN 1556-6013

Text (CoGA_V2)
CoGA_V2.pdf - Accepted Version
Available under License Creative Commons Attribution.
Download (4MB)

Abstract

Multimodal language models (LMs) have shown significant potential for applications across various domains but remain vulnerable to adversarial attacks. Current research in white-box or black-box settings generally struggles with unrealistic attack assumptions and limited efficacy of targeted attacks. This paper introduces CoGA, a novel gray-box collaborative adversarial attack method for multimodal LMs. Under our gray-box settings, attackers have access only to the victim model’s input encoders. With the guidance of different modalities, we perturb the embedding representations from encoders to disrupt the semantic alignment across modalities, ultimately causing inaccurate outputs on various downstream tasks. Specifically, we integrate text embeddings into the loss calculations of the image attack and utilize image embeddings to guide the ranking of vulnerable words and the selection of final samples. Extensive experiments demonstrate that our method achieves superior attack performance across diverse models and tasks, suggesting the shared vulnerability of multimodal LMs in confronting adversarial challenges. Our work provides new insights into the security of multimodal LMs, facilitating the deployment of more robust and secure models in practical applications.