Contrastive Training with More Data

Mander, Stephen and Piao, Scott and Rahmani, Hossein (2023) Contrastive Training with More Data. In: Eleventh International Conference on Learning Representations, 2023-05-01. (In Press)

Text (66_contrastive_training_with_more)
66_contrastive_training_with_more.pdf - Published Version
Available under License Creative Commons Attribution.

Download (316kB)


This paper proposes a new method of contrastive training over multiple data points, focusing on the scaling issue present when using in-batch negatives. Our approach compares transformer training with dual encoders versus training with multiple encoders. Our method can provide a feasible approach to improve loss modelling as encoders scale.

Item Type:
Contribution to Conference (Paper)
Journal or Publication Title:
Eleventh International Conference on Learning Representations
Uncontrolled Keywords:
ID Code:
Deposited By:
Deposited On:
10 Aug 2023 15:15
In Press
Last Modified:
20 Sep 2023 02:39