Mander, Stephen and Piao, Scott and Rahmani, Hossein (2023) Contrastive Training with More Data. In: Eleventh International Conference on Learning Representations, 2023-05-01. (In Press)
Text (66_contrastive_training_with_more)
66_contrastive_training_with_more.pdf - Published Version
Available under License Creative Commons Attribution.
Download (316kB)
66_contrastive_training_with_more.pdf - Published Version
Available under License Creative Commons Attribution.
Download (316kB)
Abstract
This paper proposes a new method of contrastive training over multiple data points, focusing on the scaling issue present when using in-batch negatives. Our approach compares transformer training with dual encoders versus training with multiple encoders. Our method can provide a feasible approach to improve loss modelling as encoders scale.
Item Type:
Contribution to Conference
(Paper)
Journal or Publication Title:
Eleventh International Conference on Learning Representations
Uncontrolled Keywords:
/dk/atira/pure/subjectarea/asjc/1700/1702
Subjects:
ID Code:
197517
Deposited By:
Deposited On:
10 Aug 2023 15:15
Refereed?:
Yes
Published?:
In Press
Last Modified:
20 Sep 2023 02:39