Systematic design choices for fine-tuning text-classification models : Projection space, task instructions, and label encoding

Lyu, Jian and Xue, Jingfeng and Meng, Weizhi and Han, Weijie and Chen, Junbao and Liu, Zeyang (2026) Systematic design choices for fine-tuning text-classification models : Projection space, task instructions, and label encoding. Applied Soft Computing, 187: 114341. ISSN 1568-4946

Text (ASOC-D-25-07347)
ASOC-D-25-07347.pdf - Accepted Version
Available under License Creative Commons Attribution.
Download (3MB)

Abstract

Text classification (TC) is a foundational task in natural language processing (NLP), where supervised fine-tuning (SFT) of pre-trained language models (PLMs) has become the dominant paradigm. However, systematic investigations of three key design choices within fine-tuning frameworks for TC are still lacking: (1) using a classification head projecting to the label space versus the vocabulary space, (2) augmenting input text with task instructions, and (3) integrating label text directly into training sequences. To address this gap, we introduce a principled 2 2 2 design matrix and conduct an empirical study grounded in this unified methodological framework across four core benchmark datasets (two Chinese and two English) using both encoder-only and decoder-only PLMs, and further validate our findings on multi-label and long-document benchmarks. Results indicate that classification heads projecting to the label space or the vocabulary space achieve comparable performance. Explicit task instructions, while effective in few-shot in-context learning (ICL), do not consistently improve performance in supervised fine-tuning. Notably, although decoder-only models exhibit the capability to learn from label-appended sequences, this behavior superficially resembles ICL and fundamentally arises from architectural alignment between label placement and causal next-token supervision rather than genuine reasoning over label semantics. These results provide both analytical insight into PLM supervision dynamics and actionable design guidelines for efficient TC workflows. 1 1 All source code and processed datasets are publicly available at https://github.com/JianLyu07/design_choices.