05 Fakultät Informatik, Elektrotechnik und Informationstechnik
Permanent URI for this collectionhttps://elib.uni-stuttgart.de/handle/11682/6
Browse
33 results
Search Results
Item Open Access Supervised semantic proximity noise and disagreement detection(2024) Choppa, TejaswiThe quality and reliability of annotated data are crucial for the development of Machine Learning models. In this work, we particularly focus on word sense annotation in context (a.k.a. Word-in-Context, WiC). WiC datasets in real-world contexts often exhibit significant disagreement. As a result, information is lost when instances are discarded during the creation of the gold label by adjudicating the annotations through majority or median judgment. Recent advancements have sought to address this issue by incorporating disagreement data through novel label aggregation methods (Uma et al., 2022). Modeling this disagreement is important because, in a real-world scenario, we often do not have clean data. We need to predict on samples where high disagreement is expected and which are inherently difficult to categorize. Predicting disagreement can help detect or filter highly complex samples. Through this thesis, we aim to build machine learning models that predict human disagreement in annotated text instances. Moreover, we focus on data with noise instances where annotators cannot confidently assign a label or the data does not fit predefined categories. We aim to measure both disagreement and noise, as they both stem from a common source: ambiguity. By modeling these aspects, we aim to design modeling approaches that predict not only the semantic proximity label but also the annotator disagreement, as well as data noisiness.Item Open Access Exploring the effects of enriched English language input on language model efficiency(2024) Zeller, TomRecent years have seen the advent of large-scale language modeling as exemplified by transformer-based models like GPT or variants of the BERT architecture. These models, which are trained on massive datasets and using compute unattainable by actors that are not of the scale of the biggest tech companies, have shown impressive feats of syntactic and semantic understanding. Naturally, interest has risen in making these models more efficient, in terms of compute as well as data requirements. Research in this area can be seen as primarily motivated by two factors: reducing the barrier for smaller actors like research institutes or end consumers to train and execute state-of-the-art models, as well as reducing the carbon footprint of these models. To achieve this goal, model compression techniques like quantization, pruning or distillation are utilized. This work aims to explore a different, less model-centric and more data-centric approach: Modifying the training and inference data, by enriching it with syntactic and semantic information. To this end, a lexical resource is created which maps English words to a form where individual characters represent values of a range of semantic and syntactic features, providing lexical information that is accessible to all model types that operate on tokens at the sub-word or character-level. Different features and methods of representation are discussed, and their effect on model performance is evaluated by pretraining a small GPT-family model and fine-tuning on downstream tasks of the SuperGLUE benchmark. Given a fixed amount of data and compute, the experiments show a performance advantage for a character-level model trained using the enriched data.Item Open Access Plug-and-play domain adaptation for neural machine translation(2023) Kadiķis, EmīlsNeural machine translation has emerged as a powerful tool, yet its performance heavily relies on training data. In a fast-changing world, dealing with out-of-domain data remains a challenge, prompting the need for adaptable translation systems. While fine-tuning is a proven effective adaptation method, it is not always feasible due to data availability, memory, and computational constraints. This thesis introduces a dynamic plug-and-play method inspired by controllable text generation to enhance machine translation across various domains without fine-tuning. This method, called Plug-and-Play Neural Machine Translation (PPNMT), uses a mono-lingual domain-specific bag-of-words to push the hidden state of the decoder through backrpopogation, making the output more in-domain. The method is tested on two types of domains: formality, gender (where the source language does not make a distinction between these aspects, but the target language does), and fine-grained technical domains (which are more based on topic inherent in the text on both the source and target sides). The method performs reasonably well for adapting the translation to different formality levels and, to a lesser extent, grammatical genders, even with an incredibly simple bag-of-words. However, it struggles with adapting the model to technical domains, and a fine-tuning baseline outperforms the proposed method in anything but very low few-shot settings in all tried domains. Despite that, the method shows some interesting behaviour, adapting to the formality on a level that goes beyond just using formal pronouns.Item Open Access Exploring retrieval-augmented language modeling for material prediction of vehicle components(2024) Wagner, FrederikJüngste Fortschritte im Bereich natural language processing (NLP), insbesondere bei großen Sprachmodellen (large language models, LLMs) wie ChatGPT, zeigen das Potenzial für ihre Anwendung bei einer Vielzahl von Aufgaben in speziellen Domänen. In der Automobilbranche könnten sie beispielsweise zur Unterstützung bei der Reparatur eines Fahrzeugs eingesetzt werden. Diese Arbeit befasst sich mit dem Problem der Vorhersage geeigneter Materialien für Fahrzeugkomponenten, wie z. B. Bremsscheiben. Es soll ermittelt werden, ob LLMs sowohl auf Allgemein- als auch auf domänenspezifisches Wissen zurückgreifen können, um genaue Vorhersagen über Komponentenmaterialien zu treffen, ohne dass eine umfangreiche Feinabstimmung (fine-tuning) erforderlich ist. Erreicht wird dies durch retrieval-augmented generation (RAG), wobei relevante Informationen aus externen Quellen abgerufen und zur Verbesserung der Modelleingabe des LLMs verwendet werden. In dieser Arbeit werden drei Ansätze verglichen: ein Standard-LLM-Modell, ein einfacher RAG-Ansatz und eine iterative RAG-Methode namens Chain-of-Verification (CoVe). In dieser Arbeit wird auch ein eigenes Annotationstool entwickelt, um eine menschliche Evaluierungsstudie zu erleichtern, da es keinen Goldstandard-Datensatz gibt. Die Ergebnisse zeigen, dass LLMs bei der Materialvorhersage gut abschneiden, und obwohl beide RAG-Ansätze die Vorhersagequalität nicht signifikant verbessern, verschlechtern sie sie auch nicht. Diese Forschungsarbeit kommt zu dem Schluss, dass LLMs mit oder ohne Retrieval-Ergänzung eine vielversprechende Lösung für die Materialvorhersage bei Fahrzeugkomponenten bieten, auch wenn es noch Herausforderungen bei der Bewertung, der Hyperparameter-Optimierung und dem Daten-Retrieval gibt.Item Open Access Gender bias in dependency parsing(2023) Go, Paul StanleyRecent high-profile advances in natural language processing (NLP) have spurred interest into identifying and rectifying socially harmful problems common in NLP systems such as gender bias. Unfortunately, many works which attempt to tackle the issue of gender bias suffer from methodological deficiencies such as the assumption of a binary and immutable concept of gender. We scrutinize one such work which found gender bias in dependency parsing and evaluate if the claims have merit. Our results were inconsistent with the gender bias findings of that paper, and further investigations through error analysis and treebank analysis revealed methodological flaws which artificially introduced differences between their female and male data sets. Mistakes made during preprocessing compromised the outcome; therefore, their results do not prove the existence of gender bias in dependency parsing. Through our findings, we suggest a different methodology for identifying and alleviating syntactic bias that is more inclusive for everyone-no matter their gender.Item Open Access Multimodal OCR post-correction on German historical documents(2023) Wu, NianhengOptical Character Recognition (OCR) post-correction is essential to digitalizing historical documents, increasing transcription accuracy, and reducing manual effort. Previous works often handle this as a text-to-text translation problem. However, the orthography of many languages, including German, has evolved across centuries, leading to many "irregular" spellings. Thus, a text-only system would face many uncertainties. Therefore, combining image features with text should be meaningful. The rise of large-scale pretrained models has brought new opportunities in this field. In this work, I will: 1) Introduce a dataset that includes historical German documents from 1783 to 1903 based on Deutsches Textarchiv with aligned golden transcription, OCR-ed textline, and their corresponding textline image; 2) Present a multimodal OCR post-correction system that combines CLIP image encoder, a pretrained image feature model, with ByT5, a byte-based language model. According to my experiments, this model outperforms the state-of-the-art text-only model.Item Open Access Enhancing a German dialect corpus with neural methods(2023) Tessadri, WolfgangWith the advent of modern chat applications, an increasing number of German dialect speakers use their dialects for written communication. The DiDi Facebook corpus (Frey et al. 2016) captures this phenomenon for South Tyrolean dialects. While the authors included a dialect/standard variety tag on the posting level, a third of these tags was undefined. By training DeBERTa and XLM-RoBERTa for dialect/standard classification we reduce these undefined instances by over 75%. We also use XLM-RoBERTa to add explicit variety labels to individual tokens. By performing a linear regression analysis of socio-linguistic variables and a label-derived dialectality metric we show that the generated labels are highly meaningful. Finally, we describe how the implemented Transformer models can be applied to gather geo-referenced dialect samples on Twitter and we discuss how this data can enrich future dialectometric research.Item Open Access Evaluating methods of improving the distribution of data across users in a corpus of tweets(2023) Milovanovic, MilanCorpora created from social network data often serve as the data source for tasks in natural language processing. Compared to other, more standardized corpora, social media corpora have idiosyncratic properties due to the fact that they consist of user-generated comments. These are, for example, the unbalanced distribution of the respective comments, a generally lower linguistic quality, and an inherently unstructured and noisy nature. Using a Twitter-generated corpus, I will investigate to what extent the unbalanced distribution of the data has an influence on two downstream tasks, relying on word embeddings. Word embeddings are a ubiquitous and frequently used concept in the field of natural language processing. The most common models are often the means to obtain semantic information about words and their usage by representing the words in an abstract word vector space. The basic idea is that semantically similar words in the mapped vector space have similar vectors. In doing so, these vectors serve as input for standard downstream tasks such as word similarity and semantic change detection. One of the most common models in current research is the use of word2vec, and more specifically, the Skip-gram architecture of this model. The Skip-gram architecture attempts to predict the surrounding words based on the current word. The data on which this architecture is trained greatly influences the resulting word vectors. In the context of this work, however, no significant improvement in the results to a fully preprocessed corpus could be found when filtering methods, widely used in the literature, without specific motivation, are used to select a subset of data according to defined criteria, neither for word similarity nor for semantic change detection. However, comparable results could be achieved with some filters, although the resulting models were trained using significantly fewer tokens as input.Item Open Access Analysis of political positioning from politician’s tweets(2023) Maurer, Maximilian MartinSocial media platforms such as Twitter have become important communication channels for politicians to interact with the electorate and communicate their stances on policy issues. In contrast to party manifestos, which lay out curated, compromised positions, the full range of positions within the ideological bounds of a party can be found on social media. This begs the question of how aligned the ideological positions of parties on social media are with their respective manifesto. To assess the alignment of social media and manifesto positions, we correlate the positions automatically retrieved from the tweets with manifesto-based positions for the German federal elections of 2017 and 2021. Additionally, we assess whether the change in positions over time is aligned between social media and manifestos. We retrieve ideological positions by aggregating distances between parties from sentence representations of their members' tweets from a corpus containing >2M individual tweets of 421 German politicians. We leverage domain-specific information by training a sentence embedding model such that representations of tweets with co-occurring hashtags are closer to each other than ones without co-occurring hashtags, following the assumption that hashtags approximate policy-related topics. Our experiments compare this political social media domain-specific model with other political domain and general domain sentence embedding models. We find high, significant correlations between the Twitter-retrieved positions and manifesto positions, especially for our domain-specific fine-tuned model. Moreover, for this model, we find overlaps in terms of how the positions change over time. These results indicate that the ideological positions of parties on Twitter correspond to the ideological positions as laid out in the manifestos to a large extent.Item Open Access An attribution method for classification tasks in Siamese models(2024) Liu, MindongExplaining the contribution of tokens on classification results in the classification task of two sentences is a challenging problem in natural language processing (NLP). This thesis studies the use of the Integrated Jacobians (IJ) in interpreting multi-class classification models with Siamese models, particularly its application in Natural Language Inference (NLI). The NLI task requires models to understand the logical relationships between two sentences, posing challenges for model interpretability. To address the fact that the original Siamese model was primarily designed for regression tasks, the thesis first expanded Siamese models for classification tasks with bilinear similarity while ensuring that the IJ methods can be utilized. It then adapts two forms of the IJ methods: exact IJ and approximate IJ, to work with newly extended Siamese models. To validate the effectiveness of the extended Siamese models using the IJ meth ods, the thesis conducted experiments on the AllNLI dataset under sentence-BERT framework. The thesis employed four different model configurations and applied both IJ methods to these models. The experimental results demonstrate that the IJ methods effectively provide explanations for us. Finally, the thesis examined the consistency between the explanations provided by the IJ methods and semantic relationships at the lexical and span levels using datasets WordNet and SpanEX. In the analysis, the IJ methods show that the models capture semantic relationships between words and spans, and there is a correlation between these relationships and the model’s predictions. This finding supports the use of the IJ methods to explain the decisions of NLP models.