05 Fakultät Informatik, Elektrotechnik und Informationstechnik

Permanent URI for this collectionhttps://elib.uni-stuttgart.de/handle/11682/6

Browse

Filters

202426

Settings

Subject: search.filters.subject.004Subject: search.filters.subject.400Start date: 2024End date: 2024

Search Results

Now showing 1 - 10 of 26
  • Thumbnail Image
    ItemOpen Access
    Cross-lingual word embeddings with multi-sense representations
    (2024) Shim, Soh-Eun
    Cross-lingual word embeddings have been found to be useful in aiding cross-lingual transfer, but work in this line of research has to date rarely addressed the monosemy constraint of static word embeddings in depth, where the collapse of multiple meanings into one form might arguably lead to subpar alignments. In this thesis, we address this gap by examining potential approaches towards the incorporation of sense information into cross-lingual alignment. We explore in specfic two variants of cross-lingual multi-sense alignment: one in which we employ the method of embedding the senses of each word as a Gaussian mixture (Athiwaratkun and Wilson, 2017), where the assumption is that multi-sense embeddings as a basis for alignment may help mitigate the meaning conflation deficiency (Camacho-Collados and Pilehvar, 2018), and in turn help improve isomorphism between vector spaces (Ruder et al., 2019). Our second method explores learning a cross-lingual multi-sense embedding space by reversing the order: we cross-lingually align uni-sense word embeddings, and attempt multi-sense enrichment as a postprocessing step by retrofitting (Pilehvar and Collier, 2016) the embedding on the Open Multilingual Wordnet (Bond et al., 2023). We observe that our model is capable of fine-grained cross-lingual semantic distinctions, where our model successfully identifies colexifications without cross-lingual supervision.
  • Thumbnail Image
    ItemOpen Access
    An attribution method for classification tasks in Siamese models
    (2024) Liu, Mindong
    Explaining the contribution of tokens on classification results in the classification task of two sentences is a challenging problem in natural language processing (NLP). This thesis studies the use of the Integrated Jacobians (IJ) in interpreting multi-class classification models with Siamese models, particularly its application in Natural Language Inference (NLI). The NLI task requires models to understand the logical relationships between two sentences, posing challenges for model interpretability. To address the fact that the original Siamese model was primarily designed for regression tasks, the thesis first expanded Siamese models for classification tasks with bilinear similarity while ensuring that the IJ methods can be utilized. It then adapts two forms of the IJ methods: exact IJ and approximate IJ, to work with newly extended Siamese models. To validate the effectiveness of the extended Siamese models using the IJ meth ods, the thesis conducted experiments on the AllNLI dataset under sentence-BERT framework. The thesis employed four different model configurations and applied both IJ methods to these models. The experimental results demonstrate that the IJ methods effectively provide explanations for us. Finally, the thesis examined the consistency between the explanations provided by the IJ methods and semantic relationships at the lexical and span levels using datasets WordNet and SpanEX. In the analysis, the IJ methods show that the models capture semantic relationships between words and spans, and there is a correlation between these relationships and the model’s predictions. This finding supports the use of the IJ methods to explain the decisions of NLP models.
  • Thumbnail Image
    ItemOpen Access
    Improving the generalisability of fake audio detection
    (2024) Lavrynovska, Viktoria
    The rapid advancements in neural speech synthesis have enabled the generation of deepfake audios that are increasingly indistinguishable from real voice recordings. Automatic fake audio detection is an emerging research area aiming to develop a reliable means of distinguishing between real and synthetic speech. Lacking generalisability of the detection models on unseen data is an issue that is commonly observed. One of the contributions of this work is the investigation of generalisability across different languages. MesoInception-4 trained on the ASVspoof19 anti-spoofing dataset builds the foundation of our detection model. The use of Mel-Frequency Cepstral Coefficients has been found to be superior to Whisper features for the cross-lingual task. Notably, our model exhibits robust performance on all evaluated languages, despite being trained exclusively on English data, and shows no evidence of language dependency or correlation with the speech quality of the language subsets. However, the findings reveal that the detection model fails to generalise well on the In-the-Wild dataset. We identify that reducing the length of audio clips and fine-tuning specific inception modules can alleviate these issues to some degree. Conversely, augmenting training data with various real-world noises from the MUSAN corpus did not significantly enhance generalisability, and the inclusion of pink noise and silence led to performance degradation on In-the-Wild data. In summary, the findings highlight the complexity of fake audio detection and underscore the importance of further research to elucidate the factors influencing performance and generalisability of detection systems.
  • Thumbnail Image
    ItemOpen Access
    RAGAR, your falsehood RADAR : RAG-augmented reasoning for political fact-checking using multimodal large language models
    (2024) Abdul Khaliq, Mohammed
    The escalating challenge of misinformation, particularly in the context of political discourse, necessitates advanced solutions for fact-checking. This thesis introduces innovative approaches to enhance the reliability and efficiency of multimodal fact-checking through the integration of large language models (LLMs) with Retrieval-augmented Generation (RAG) based advanced reasoning techniques. In the digital era, where misinformation spreads rapidly across various media, including text and images, there's a critical need for robust mechanisms capable of evaluating the veracity of political claims. This work proposes two novel methodologies, Chain of RAG (CoRAG) and Tree of RAG (ToRAG), and their hybrid implementations incorporating Chain of Thought and Chain of Verification. These approaches leverage RAG techniques utilizing multimodal LLMs with reasoning techniques. The approaches are designed to process and assess political claims by considering textual and visual information, providing a comprehensive approach to fact-checking. This thesis explores the implementation of these approaches within a multimodal fact-checking pipeline, highlighting their effectiveness in improving the accuracy of veracity predictions and the generation of explanations. By employing multimodal LLMs adept at analyzing text and images, this research advances the capability of automated systems in identifying and countering misinformation. The experimental evaluation demonstrates that the proposed RAG-augmented Reasoning (RAGAR) techniques outperform existing methods that rely on sub-question generation, offering a promising solution to the challenges of political fact-checking. This thesis contributes to the fields of computational linguistics and political science by providing an effective approach to combat fake news, thereby enhancing the integrity of political discourse in the digital age.
  • Thumbnail Image
    ItemOpen Access
    CAPTCHA mechanisms using semantic NLU tasks
    (2024) Wolkober, Marcel
    In 2019, one-fourth of all internet traffic was made of malicious bots. CAPTCHAs are a main countermeasure used as a test to detect non-human users. With the advancement of computational attackers using artificial intelligence, their success in these CAPTCHAs rises continuously. This bachelor’s thesis aims to provide new CAPTCHA mechanisms using semantic natural language understanding (NLU) tasks, which are generally considered hard to solve for advanced computational attackers. The task used for the NLU CAPTCHA challenges involves rating the semantic similarity of a word in two different contexts. A study with 275 participants was conducted to evaluate human usability. The study results show a high human difficulty for these challenges. Further, these challenges provide insufficient resistance against an advanced attacker. This leads to the conclusion that, in the current state, the semantic NLU CAPTCHA provides no benefit to existing CAPTCHAs. However, some challenge results indicate that with further adjustments, NLU tasks may still be relevant in use for CAPTCHA challenges.
  • Thumbnail Image
    ItemOpen Access
    Exploring the effects of enriched English language input on language model efficiency
    (2024) Zeller, Tom
    Recent years have seen the advent of large-scale language modeling as exemplified by transformer-based models like GPT or variants of the BERT architecture. These models, which are trained on massive datasets and using compute unattainable by actors that are not of the scale of the biggest tech companies, have shown impressive feats of syntactic and semantic understanding. Naturally, interest has risen in making these models more efficient, in terms of compute as well as data requirements. Research in this area can be seen as primarily motivated by two factors: reducing the barrier for smaller actors like research institutes or end consumers to train and execute state-of-the-art models, as well as reducing the carbon footprint of these models. To achieve this goal, model compression techniques like quantization, pruning or distillation are utilized. This work aims to explore a different, less model-centric and more data-centric approach: Modifying the training and inference data, by enriching it with syntactic and semantic information. To this end, a lexical resource is created which maps English words to a form where individual characters represent values of a range of semantic and syntactic features, providing lexical information that is accessible to all model types that operate on tokens at the sub-word or character-level. Different features and methods of representation are discussed, and their effect on model performance is evaluated by pretraining a small GPT-family model and fine-tuning on downstream tasks of the SuperGLUE benchmark. Given a fixed amount of data and compute, the experiments show a performance advantage for a character-level model trained using the enriched data.
  • Thumbnail Image
    ItemOpen Access
    Supervised semantic proximity noise and disagreement detection
    (2024) Choppa, Tejaswi
    The quality and reliability of annotated data are crucial for the development of Ma­chine Learning models. In this work, we particularly focus on word sense annotation in context (a.k.a. Word-in-Context, WiC). WiC datasets in real-world contexts of­ten exhibit significant disagreement. As a result, information is lost when instances are discarded during the creation of the gold label by adjudicating the annotations through majority or median judgment. Recent advancements have sought to ad­dress this issue by incorporating disagreement data through novel label aggregation methods (Uma et al., 2022). Modeling this disagreement is important because, in a real-world scenario, we often do not have clean data. We need to predict on samples where high disagreement is expected and which are inherently difficult to categorize. Predicting disagreement can help detect or filter highly complex samples. Through this thesis, we aim to build machine learning models that predict human disagreement in annotated text instances. Moreover, we focus on data with noise instances where annotators cannot confidently assign a label or the data does not fit predefined categories. We aim to measure both disagreement and noise, as they both stem from a common source: ambiguity. By modeling these aspects, we aim to design modeling approaches that predict not only the semantic proximity label but also the annotator disagreement, as well as data noisiness.
  • Thumbnail Image
    ItemOpen Access
    Linguistically-informed modeling of potentials for misunderstanding
    (2024) Anthonio, Talita; Roth, Michael (Dr.)
    Misunderstandings are prevalent in communication. While there is a large amount of work on misunderstandings in conversations, only little attention has been given to misunderstandings that arise from text. This is because readers and writers typically do not interact with one another. However, texts that potentially evoke different interpretations can be identified by certain linguistic phenomena, especially those related to implicitness or underspecificity. In Computational Linguistics, there is a considerable amount of work conducted on such linguistic phenomena and the computational modeling thereof. However, most of these studies do not examine when these phenomena cause misunderstandings. This is a crucial aspect, because ambiguous language does not always cause misunderstanding. In this thesis, we provide the first steps to develop a computational model that can automatically identify whether an instructional text is likely to cause misunderstandings ("potentials for misunderstanding"). To achieve this goal, we build large corpora with potentials for misunderstanding in instructional texts. We follow previous work and define misunderstandings as the existence of multiple, plausible interpretations. As these interpretations may be similar in meaning to one another, we specifically define misunderstandings as the existence of multiple plausible, but conflicting interpretations. Therefore, we find texts that potentially cause misunderstanding ("potentials for misunderstanding") by looking for passages that have several plausible interpretations that are conflicting to one another. We automatically identify such passages from revision histories of instructional texts, based on the finding that we can find potentials for misunderstanding by looking into older versions of a text, and their clarifications thereof in newer versions. We specifically look for unclarified sentences that contain implicit and underspecified language, and study their clarifications. Through several analyses and crowdsourcing studies, we demonstrate that our corpora provide valuable resources on potentials for misunderstanding, as we find that revised sentences are better than their previous ones. Furthermore, we show that the provided corpora can be used for several computational modeling purposes. The three resulting models can each be combined to identify whether a text potentially causes misunderstanding or not. More specifically, we first develop a model that can detect improvements in a text, even when they are subtle and closely dependent on the context. In an analysis, we verify that the judgements from the model on what makes a better or equally good sentence overlap with the judgements by humans. Secondly, we build a transformer-based language model that automatically resolves potentials for misunderstanding caused by implicit references. We find that modeling discourse context improves the performance of this model. In an analysis, we find that the best model is not only capable of generating the golden resolution, but also capable of generating several plausible resolutions for implicit references in instructional text. We use this finding to build a large dataset with plausible and implausible resolutions of implicit and underspecified elements. We use the resulting dataset for a third computational task, in which we train a model to automatically distinguish between plausible and implausible resolutions for implicit and underspecified elements. We show that this model and the provided dataset can be used to find passages with several, plausible clarifications. Since our definition of misunderstanding focuses on conflicting clarifications, we conduct a final study to conclude the thesis. In particular, we provide and validate a crowdsourcing set-up that allows to find the cases with conflicting, plausible, resolutions. The set-up and findings could be used in future research to directly train a model to identify passages with implicit elements that have conflicting resolutions.
  • Thumbnail Image
    ItemOpen Access
    An analysis of the domain-specific applicability of text-to-SQL systems on a linguistic database
    (2024) Ateri, Maria Vittoria
    The applicability and adaptability of text-to-SQL systems trained on reference databases to more complicated ones is an open question. This thesis attempts to provide intuitions on the challenges and limitations when applying benchmark systems to more complicated databases. For this, two exemplary systems, namely the IRNet and SmBop, both trained on the Spider dataset, are applied to the complex linguistic relational database DIRNDL. The primary aim is to analyze to what extent the systems manage to produce accurate queries and retrieve correct information when the inference is conducted on a database of greater complexity and dimensions compared to the databases contained in the Spider dataset (the main benchmark in the field). Intentionally, no re-training is performed. A comparison between the two systems is also conducted. In addition to this, the sensitivity to lexical changes and question complexity variation is part of the analysis carried out in this work. Through a qualitative evaluation, the current work provides insights into which model architecture works better for complex linguistic databases, and the limits of both systems. The main findings are that the SmBop system is superior to the IRNet one, and that SmBop is also more sensitive to lexical changes in the database schema. Nevertheless, neither system shows a satisfactory performance when the goal is the synthesis of more complex queries which are used in real-world research settings.
  • Thumbnail Image
    ItemOpen Access
    When few-shot fails : low-resource, domain-specific text classification with transformers
    (2024) Wertz, Lukas; Klinger, Roman (Prof. Dr.)
    Text classification (TC) is a foundational technique in natural language processing (NLP). The ability to automatically classify texts into predetermined categories serves a critical role in numerous applications. With recent advances in NLP owed to powerful pre-trained language models, there is also a rapidly growing interest in using TC for difficult, real-world problems in both business and industrial sectors as well as the scientific community. Using pre-trained language models, modern TC systems achieve state-of-the-art accuracy on benchmark datasets using only a handful of training examples for effective fine-tuning. However, when faced with challenging, low-resource datasets from specialised language domains, relying on small labeled datasets to train classification models is often not effective. When few-shot classification fails, we need to employ more traditional NLP techniques which increase the amount of training data in order to train accurate models. We find that existing approaches for expanding the training data are often unsuitable for deep, transformer-based classification networks. In addition, these approaches are usually tested on standard benchmark datasets, which do not properly reflect the complexity of real-world classification tasks. Consequently, there is a need for effective data augmentation or selection techniques that allow TC systems to handle complex tasks, relevant for modern industrial or business applications. Our primary goal in this work is to design effective data collection and augmentation systems for TC on low-resource datasets from technical or otherwise non-standard language domains that more closely resemble real-world applications. As a consequence, our experiments also demonstrate the limits of existing approaches and strongly motivate the need for more complex, domain-specific benchmark datasets. First, we investigate the use of generational language models for data augmentation. We find that simple language edits are smoothed out by the language model and fine-tuning on the small training data proves unstable. As such we propose a simple generation scheme, which uses specific model prompts built from the data. Second, we employ a variety of existing selection strategies for active learning. Since we find that no strategy consistently outperforms a random selection across datasets, we design an approach that combines the strategies via reinforcement learning. This allows learning which information source for data selection is most valuable and greatly improves the classification performance in early stages of the active learning process. Overall, we find that TC ist still a challenge in NLP, in particular when systems have to be designed with modern application contexts in mind. Our experiments with various baselines show, that existing augmenting techniques and AL strategies can not easily be transferred to current architectures, increasingly complex tasks or domain specific language. Consequently, the approaches presented in this work are an important step towards TC for sparse, complex datasets and real-world challenges.