Contrastive Training with More Data

Mander, Stephen and Piao, Scott and Rahmani, Hossein (2023) Contrastive Training with More Data. In: Eleventh International Conference on Learning Representations, 2023-05-01. (In Press)

[img]
Text (66_contrastive_training_with_more)
66_contrastive_training_with_more.pdf - Published Version
Available under License Creative Commons Attribution.

Download (316kB)

Abstract

This paper proposes a new method of contrastive training over multiple data points, focusing on the scaling issue present when using in-batch negatives. Our approach compares transformer training with dual encoders versus training with multiple encoders. Our method can provide a feasible approach to improve loss modelling as encoders scale.

Item Type:
Contribution to Conference (Paper)
Journal or Publication Title:
Eleventh International Conference on Learning Representations
Uncontrolled Keywords:
/dk/atira/pure/subjectarea/asjc/1700/1702
Subjects:
ID Code:
197517
Deposited By:
Deposited On:
10 Aug 2023 15:15
Refereed?:
Yes
Published?:
In Press
Last Modified:
20 Sep 2023 02:39