Measuring, Understanding and Modelling Human Label Variation in Legal NLP

Human Label Variation (HLV) refers to the differences in human annotations, which reflects the plurality of human opinions. In this talk, Shanshan will present our investigations into naturally occurring HLV in legal NLP, focusing on split votes among judges in court decisions, and respondents’ different answer to political survey. These cases reveal the challenges of modeling pluralistic human values in legal NLP and emphasize the importance of responsible data curation and robust evaluation for pluralistic Human-AI alignment.