The role of Instruction-tuned Models as Annotators: Exploring Label Variation

The zero-shot learning capabilities of large language models (LLMs) make them ideal for text classification without annotation or supervised training. 

any studies have shown impressive results across multiple tasks. While tasks, data, and results differ widely, their similarities to human annotation can aid us in tackling new tasks with minimal expenses. The ultimate promise of LLMs is that their language capability lets them generalize to any text classification task. What if the answer is not to wait for one model to rule them all, but to treat their variation similar to the disagreement among human annotators? Not as individual flaws, but as specializations we can exploit.

In this talk, we will explore the potential of state-of-the-art instruction-tuned models to serve as "annotators" across various established NLP tasks, including sentiment classification, age and gender prediction, topic classification, and hate speech detection. We will answer two fundamental questions: Do we continue to require human annotators, and do the variations in human labeling also manifest in LLMs? Additionally, we will discuss the tradeoffs between speed, accuracy, cost, and bias when it comes to aggregated model labeling versus human annotation