Methods for mining political opinions from texts and large language models

Abstract

In democratic societies, the diversity of opinions enables individuals to express their values and engage with differing perspectives. This thesis investigates political opinions through two lenses: texts and models, examining both ideological positions and policy issue preferences. While ideological analysis is well-established, policy issue preferences represent a more nuanced, underexplored research area. Investigating political opinions from political parties is essential for understanding voter choices, policy decision-making, and the shifts in party agendas over time. In the first part of this thesis, I focus on methods for mining political opinions from party manifestos. Automating the identification of political opinions helps process large datasets, minimize annotation time, and offer timely updates on newly released information from parties. I investigate how accurately party positions can be identified from texts with minimal annotations and the level of detail achievable in this process. We also explore the extent to which party positions can be identified on a large scale across different languages and countries.Results demonstrate that the identification of party positions can be distinguished between the tasks of political scaling and positioning which have substantial differences in terms of evaluation and application. Additionally, findings indicate that improving text representations through in-domain fine-tuning significantly benefits the performance when methods depend on text similarity. And finally, party scaling across languages achieves high performance with multilingual models. Models have become my object of study with the advent of LLMs. They introduce new concerns regarding the type of biases embedded and reproduced by them. Given the importance of shedding light on political biases in LLMs, the second part of this thesis addresses the evaluation and identification of political biases in LLMs. Our research questions center on robustly evaluating LLMs for biases and identifying the political biases regarding ideology and policy issue preferences. This thesis provides definitions of political bias and political worldview, which aid in designing methods for their evaluation. Moreover, it contributes with a framework for a robust evaluation of biases in LLMs and a dataset for evaluating political opinions in LLMs. Finally, findings indicate that small parameter size models are not reliable in their answers, and that LLMs do hold consistent political worldviews in relation to some policy issues. Overall, they highlight the necessity for continued research to understand the complexities and societal implications of developing models integrating diverse political opinions into AI systems.