Contrastive Training with More Data

Mander, Stephen and Piao, Scott and Rahmani, Hossein (2023) Contrastive Training with More Data. In: Eleventh International Conference on Learning Representations, 2023-05-01.

[thumbnail of 66_contrastive_training_with_more]
Text (66_contrastive_training_with_more)
66_contrastive_training_with_more.pdf - Published Version
Available under License Creative Commons Attribution.

Download (316kB)

Abstract

This paper proposes a new method of contrastive training over multiple data points, focusing on the scaling issue present when using in-batch negatives. Our approach compares transformer training with dual encoders versus training with multiple encoders. Our method can provide a feasible approach to improve loss modelling as encoders scale.

Item Type:
Contribution to Conference (Paper)
Journal or Publication Title:
Eleventh International Conference on Learning Representations
Uncontrolled Keywords:
Research Output Funding/no_not_funded
Subjects:
?? no - not fundedartificial intelligence ??
ID Code:
197517
Deposited By:
Deposited On:
10 Aug 2023 15:15
Refereed?:
Yes
Published?:
Published
Last Modified:
08 Oct 2024 00:40