Contrastive Training with More Data

Mander, Stephen and Piao, Scott and Rahmani, Hossein (2023) Contrastive Training with More Data. In: Eleventh International Conference on Learning Representations, 2023-05-01.

Text (66_contrastive_training_with_more)
66_contrastive_training_with_more.pdf - Published Version
Available under License Creative Commons Attribution.
Download (316kB)

Abstract

This paper proposes a new method of contrastive training over multiple data points, focusing on the scaling issue present when using in-batch negatives. Our approach compares transformer training with dual encoders versus training with multiple encoders. Our method can provide a feasible approach to improve loss modelling as encoders scale.