Investigating topic bias in emotion classification

Abstract

In emotion classification, texts are assigned a conceptual emotion representation such as discrete labels or dimensions of cognitive appraisal. Emotion classifiers are typically not universally applicable, but base their classification decisions on characteristic features of a specific domain. When applied to a different domain, the lack of domain-specific knowledge results in classification errors. While this behavior is typically addressed as a cross-domain or cross-corpus phenomenon, the potentially misguiding factors within one corpus have not yet been studied to the same degree. I propose an investigation of topics in emotion datasets to assess their influence on the classification decisions in emotion and appraisal classification. My contribution is threefold: First, I conduct an analysis of how topics and emotions are distributed in emotion datasets. Second, I investigate whether state-of-the-art emotion classification systems are prone to adopting the topic distribution in the training data as topic bias. Third, I evaluate debiasing methods for topic bias in the context of emotion classification. The results indicate that topic bias is introduced to emotion datasets through the applied sampling method. The topic bias within commonly used datasets in the field appears to be, except for one exception, negligible. However, if bias is present in the data, it is adopted by the resulting classifiers. In order to mitigate such bias, I investigate a naive word removal approach as well as gradient reversal, which is found to work best for bias mitigation.