PUDD : Towards Robust Multi-modal Prototype-based Deepfake Detection

Lopez Pellicer, Alvaro and Li, Yi and Angelov, Plamen (2024) PUDD : Towards Robust Multi-modal Prototype-based Deepfake Detection. In: 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) :. IEEE, USA. ISBN 9798350365481

Text (PUDD)
CVPR_Workshop-5.pdf - Accepted Version
Available under License Creative Commons Attribution.
Download (3MB)

Abstract

Deepfake techniques generate highly realistic data, mak- ing it challenging for humans to discern between actual and artificially generated images. Recent advancements in deep learning-based deepfake detection methods, particularly with diffusion models, have shown remarkable progress. However, there is a growing demand for real-world appli- cations to detect unseen individuals, deepfake techniques, and scenarios. To address this limitation, we propose a Prototype-based Unified Framework for Deepfake Detec- tion (PUDD). PUDD offers a detection system based on similarity, comparing input data against known prototypes for video classification and identifying potential deepfakes or previously unseen classes by analyzing drops in similar- ity. Our extensive experiments reveal three key findings: (1) PUDD achieves an accuracy of 95.1% on Celeb-DF, out- performing state-of-the-art deepfake detection methods; (2) PUDD leverages image classification as the upstream task during training, demonstrating promising performance in both image classification and deepfake detection tasks dur- ing inference; (3) PUDD requires only 2.7 seconds for re- training on new data and emits 105 times less carbon com- pared to the state-of-the-art model, making it significantly more environmentally friendly.