05 Fakultät Informatik, Elektrotechnik und Informationstechnik
Permanent URI for this collectionhttps://elib.uni-stuttgart.de/handle/11682/6
Browse
124 results
Search Results
Item Open Access Supervised semantic proximity noise and disagreement detection(2024) Choppa, TejaswiThe quality and reliability of annotated data are crucial for the development of Machine Learning models. In this work, we particularly focus on word sense annotation in context (a.k.a. Word-in-Context, WiC). WiC datasets in real-world contexts often exhibit significant disagreement. As a result, information is lost when instances are discarded during the creation of the gold label by adjudicating the annotations through majority or median judgment. Recent advancements have sought to address this issue by incorporating disagreement data through novel label aggregation methods (Uma et al., 2022). Modeling this disagreement is important because, in a real-world scenario, we often do not have clean data. We need to predict on samples where high disagreement is expected and which are inherently difficult to categorize. Predicting disagreement can help detect or filter highly complex samples. Through this thesis, we aim to build machine learning models that predict human disagreement in annotated text instances. Moreover, we focus on data with noise instances where annotators cannot confidently assign a label or the data does not fit predefined categories. We aim to measure both disagreement and noise, as they both stem from a common source: ambiguity. By modeling these aspects, we aim to design modeling approaches that predict not only the semantic proximity label but also the annotator disagreement, as well as data noisiness.Item Open Access Event knowledge and models of logical metonymy interpretation(2014) Zarcone, Alessandra; Padó, Sebastian (Prof. Dr.)During language understanding, people do not only rely on what they read or hear, but they also exploit implicit information. For example, when they process the expression "begin the book", they understand it involves an event which is not explicitly mentioned (e.g. "begin reading the book). This thesis looks at these constructions, known as logical metonymies, which combine an event-selecting verb and entity-denoting object and involve covert events. Logical metonymies are an interesting challenge for theories of lexical semantics: they need to be reconciled with compositionality, they require the integration of context (writers typically write books, students typically read them), and they lie at the interface between lexicon and world knowledge (is the information that books are read stored in our mental lexicon or in our world knowledge?). I critically analyze previous hypotheses on logical metonymy with regard to the answer they provide to two core problems: the source problem (what events are retrieved? what type of event knowledge is assumed?) and the trigger problem (why do some constructions trigger a metonymic interpretation and others do not?). Lexicalist approaches claim that the metonymy arises from a type clash between the event-selecting verb and an entity-denoting object, and posit complex lexical items, encoding event information about artifacts (e.g. book: read), to explain the recovery of covert events. Pragmatic-based approaches argue against the idea that lexical items have an internal structure, suggesting that covert events arise from the underspecification of a logical metonymy and are inferred via non-lexical knowledge. I look with particular attention at the role of event knowledge, which lexicalist approaches place in our mental lexicon, while pragmatic-based approaches place it in our world knowledge. I propose a third hypothesis, based on thematic fit and generalized event knowledge of typical events and their participants, which have been shown to guide efficient incremental processing: I argue that contextual elements cue generalized event knowledge, which plays a key role in determining the covert event for a logical metonymy. I explore this hypothesis from an interdisciplinary perspective, employing both psycholinguistic experiments and computational models, in order to seek converging evidence and confront it with the theoretical investigation. The results from the psycholinguistic experiments and from the computational (distributional) models support the hypothesis that covert event retrieval is guided by generalized event knowledge. I also employ the computational models to analyze previous experimental results and to explore the hypothesis that thematic fit, informed by generalized event knowledge, is ultimately responsible for the trigger of the logical metonymy. I then report on more psycholinguistic evidence showing that a notion of type is indeed necessary to account for differences between metonymic and non-metonymic constructions, and that both type and thematic fit play a role in logical metonymy interpretation. Lastly, I argue for a context-sensitive model of logical metonymy interpretation that exploits an information-rich lexicon, but needs to rethink the notion of type and reconcile it with the notion of thematic fit.Item Open Access Evaluation of complex typological universals with language vectors and real-valued logics(2020) Dönicke, TillmannLanguage representations and typological universals have received increasing attention in computational linguistics over the past few years. Most approaches make use of binary language vectors from typological databases and/or focus on the correlation of only two typological variables at the same time. This thesis shows that real-valued logics can be used to evaluate even more complex formulae, as they are formulated by typologists, on continuous vectors from existing corpora. Syntactic language vectors are extracted from the Universal Dependencies treebanks (Nivre et al., 2016) and serve as the basis for the evaluation of word-order universals.Item Open Access Modeling the position and inflection of verbs in English to German machine translation(2018) Ramm, Anita; Fraser, Alexander (Prof. Dr.)Item Open Access Exploring the effects of enriched English language input on language model efficiency(2024) Zeller, TomRecent years have seen the advent of large-scale language modeling as exemplified by transformer-based models like GPT or variants of the BERT architecture. These models, which are trained on massive datasets and using compute unattainable by actors that are not of the scale of the biggest tech companies, have shown impressive feats of syntactic and semantic understanding. Naturally, interest has risen in making these models more efficient, in terms of compute as well as data requirements. Research in this area can be seen as primarily motivated by two factors: reducing the barrier for smaller actors like research institutes or end consumers to train and execute state-of-the-art models, as well as reducing the carbon footprint of these models. To achieve this goal, model compression techniques like quantization, pruning or distillation are utilized. This work aims to explore a different, less model-centric and more data-centric approach: Modifying the training and inference data, by enriching it with syntactic and semantic information. To this end, a lexical resource is created which maps English words to a form where individual characters represent values of a range of semantic and syntactic features, providing lexical information that is accessible to all model types that operate on tokens at the sub-word or character-level. Different features and methods of representation are discussed, and their effect on model performance is evaluated by pretraining a small GPT-family model and fine-tuning on downstream tasks of the SuperGLUE benchmark. Given a fixed amount of data and compute, the experiments show a performance advantage for a character-level model trained using the enriched data.Item Open Access Comparison of distributional and visual nearest neighbors(2025) Naber, SvenThis thesis investigates how semantic concepts are represented across textual and visual embedding spaces, focusing on the abstract-concrete continuum. Using 5,448 English nouns and their embeddings from both distributional language models (e.g., Word2Vec, GloVe) and vision models (e.g., ViT, DINOv2, CLIP), it compares neighborhood structure via a normalized alignment score (NAS). Results show that alignment is primarily driven by input modality rather than model architecture, with strong local overlap for concrete concepts and more diffuse agreement for abstract ones. Mean aggregation of image embeddings improves visual consistency but cannot fully bridge modality-specific limitations. The findings provide a starting point for further exploration of semantic spaces.Item Open Access More reliable retrieval augmented generation for domain-specific question-answering through domain-infused soft prompts(2025) Nassar, ZeinaTransformer-based large language models (LLMs) have revolutionized the field of artificial intelligence, enabling advancements in various applications due to their exceptional reasoning capabilities. However, these models often face challenges such as hallucination, outdated knowledge, and non-transparent reasoning, which limit their reliability in critical tasks like question answering (QA). QA systems, especially in domain-specific contexts like car manuals, require high accuracy and reliability to ensure user safety. Achieving this is complicated by the need for precise retrieval and interpretation of domain-specific information, which can be hindered by unfamiliar keywords or ambiguous questions. Open-domain QA relies on external knowledge repositories and typically follows a retriever-reader framework to locate and process evidence. In contrast, domain-specific QA often lacks sufficient gold-standard datasets, making it essential to explore techniques like retrieval-augmented generation (RAG), which combines retrieved context with the question as input to LLMs. While RAG improves grounding, hallucinations still occur when models rely on pre-trained knowledge over retrieved evidence. This thesis investigates the role of prompting techniques in improving faithfulness and reducing hallucinations in domain-specific QA. By testing domain-specific and domain-agnostic discrete prompts as well as soft prompting methods, this work aims to identify strategies for generating more accurate and grounded responses. The study addresses key research questions on the effectiveness of domain-specific information in prompts, the best way to incorporate such information, and whether dynamic soft-prompts outperform static ones in domain-specific QA scenarios. The findings aim to contribute to building more reliable and factual QA systems.Item Open Access A modular framework for coreference resolution(2012) Kobdani, Hamidreza; Schütze, Hinrich (Prof. Dr.)Coreference resolution is playing an increasingly important role in a wide range of disciplines such as theoretical, corpus and computational linguistics. It has been shown that it is beneficial in a number of natural language processing applications, including machine translation, automatic abstracting, information extraction and question answering. As a result, it has enjoyed increased interest in recent years. First, this thesis introduces a modular supervised system for coreference resolution. It is composed of separate, interchangeable components, between which there are clear well-defined logical boundaries that improve maintainability. This system has been used successfully in two international shared tasks on coreference resolution achieving good performance. The good performance of our system demonstrates the general validity of our design. In addition, a new framework for feature engineering of natural language processing will be presented that is based on a relational data model of text. It includes fast and flexible methods for implementing and extracting new features, thereby reducing the effort of creating an NLP system for a particular task. This thesis presents an instantiation and evaluation of the framework for the problem of coreference resolution in multiple languages. Competitive results were able to be obtained in a short implementation period. This demonstrates the potential power of this framework for feature engineering. An unsupervised framework will also be presented that bootstraps a complete coreference resolution system from word associations mined from a large unlabeled corpus. I will show that word associations are useful for coreference resolution - e.g., the strong association between Obama and President is an indicator of likely coreference. Association information has so far not been used in coreference resolution because it is sparse and difficult to learn from small labeled corpora. Since unlabeled text is readily available, the unsupervised approach proposed here addresses the sparseness problem. In a self-training framework, I train a decision tree on a corpus that is automatically labeled using word associations. I will show that this unsupervised system has better coreference resolution performance than other learning approaches that do not use manually labeled data.Item Open Access Fehlerbehandlung in Mensch-Maschine-Dialogen(2007) Gieselmann, Petra; Rohrer, Christian (Prof.)Seit es Computer gibt, existiert auch der Wunsch des Menschen, mit ihnen reden zu können wie mit einem anderen Menschen. Eines der berühmtesten Beispiele dafür ist sicherlich Eliza, ein Computerprogramm, das einen Psychologen simuliert, mit dem der Benutzer ein Therapiegespräch führen kann. In vielen Science-Fiction-Filmen finden sich auch immer wieder Beispiele für solche sprechenden Maschinen, wie beispielsweise HAL in 2001: Odysee im Weltraum'' oder auch der Computer auf dem Raumschiff Enterprise''. So reichen erste Dialogsysteme bereits zurück bis in die Anfänge der künstlichen Intelligenz in den fünfziger Jahren. Dennoch hatten diese Dialogsysteme bis vor wenigen Jahren noch mit so vielen Problemen zu kämpfen, dass sie kaum für einen praktischen Einsatz geeignet waren. Erst in letzter Zeit ist es durch die stetigen Verbesserungen im Bereich von Spracherkennung und Sprachverstehen und das Aufkommen von immer schnelleren und mächtigeren Rechnern möglich geworden, solche Systeme für den realen Einsatz zu bauen. Nach wie vor gibt es aber noch eine ganze Reihe ungelöster Probleme, die zum einen auf die Komplexität natürlicher Sprache und zum anderen auf den immensen Fundus an vernetztem Weltwissen und Kontextbeziehungen, über den Menschen verfügen, zurückzuführen sind. Eine der bislang größten Herausforderungen liegt darin, ein solches Dialogsystem auch für den realen Einsatz unter Alltagsbedingungen zu entwerfen. Bisher fehlt den Systemen dafür noch die nötige Fehlerrobustheit, um in Situationen, in denen das System etwas falsch verstanden hat und es zu Problemen kommt, angemessen reagieren zu können. In dieser Arbeit geht es genau um solche Fehler im Dialog, wie sie vermieden und während des laufenden Dialogs wieder behoben werden können, wenn sie nicht vorher zu vermeiden waren. Der Gegenstand dieser Arbeit ist eine datengetriebene Analyse der Fehler, die in der Mensch-Roboter-Kommunikation auftreten mit dem Ziel, diese möglichst im Vorfeld zu vermeiden. Es wird eine Fehlerklassifikation aufgestellt und es werden Methoden für die Vermeidung der verschiedenen Fehlerklassen entwickelt und evaluiert. Darüberhinaus werden auch generische Methoden zur Fehlerbehebung für die Fälle implementiert, die nicht vorher vermieden werden konnten, ebenfalls mit Hilfe datengetriebener Analysen. Damit soll es ermöglicht werden, Dialogsysteme über die Laborumgebung hinaus in realen Situationen einsetzen zu können. Dies wird am Beispiel eines Haushaltsroboters diskutiert und evaluiert. Diese Ausarbeitung gliedert sich in vier Teile: Der erste Teil beschäftigt sich mit dem Stand der Forschung in den Bereichen, die hier eine Rolle spielen. Dazu werden verschiedene Ansätze für Mensch-Maschine-Dialogsysteme beleuchtet. Im Anschluss wird die menschliche Informationsverarbeitung im Dialogbereich erläutert. Dabei geht es auch um Fehlerdialoge in zwischenmenschlichen Dialogen, die hier als Vorbild für Mensch-Roboter-Dialoge dienen. Der zweite Teil beschäftigt sich mit den durchgeführten Benutzertests und Datensammlungen und der Klassifikation von Fehlern im Dialog, die die Grundlage für die folgenden Arbeiten zur Fehlervermeidung und -behebung bilden. Zunächst erfolgt eine detaillierte Analyse von Fehlern, die bei der Mensch-Roboter-Interaktion auftreten können. Dazu werden verschiedene aufeinander aufbauende Benutzerstudien und Datensammlungen, bei denen der Roboter dem Menschen im Haushalt zur Hand geht und einfache Tätigkeiten verrichtet, durchgeführt, um eine große Menge an möglichst realistischen Daten gewinnen zu können, die nicht nur unter Laborbedingungen entstanden sind. Im dritten Teil werden verschiedene Methoden zur Fehlervermeidung und -behebung vorgestellt. Zur Fehlervermeidung werden zusätzliche Wissensquellen in den Dialogmanager integriert. Außerdem werden Mechanismen zur Anaphernresolution, Kontextmodellierung, Auflösung von Ellipsen, multimodalen Fusion und zum Umgang mit komplexen, zusammengesetzten Äußerungen entwickelt und evaluiert. Zur Fehlerbehebung werden verschiedene Strategien für effektive Klärungsfragen untersucht. Metakommunikation, wie sie in den durchgeführten Benutzertests vorkommt, wird analysiert, um eine effektivere Kommunikation gewährleisten zu können. Außerdem wird ein Mechanismus entwickelt, der es dem Roboter erlaubt, problematische Situationen zu erkennen und diese selbst durch Metakommunikation aufzulösen. Im vierten Teil werden die entwickelten Methoden anhand eines abschließenden Benutzertests evaluiert. Dabei geht es darum, das System mit allen entwickelten Mechanismen zur Fehlerbehandlung zu testen und es mit dem Basissystem zu vergleichen. Das besondere Augenmerk liegt hier auf der Übertragbarkeit der entwickelten Mechanismen auf andere Domänen und Systeme. Danach folgt das Fazit der gesamten Arbeit und eine Diskussion der zukünftigen Arbeiten im Hinblick auf mögliche Erweiterungen dieses Systems.Item Open Access Distributional measures of semantic abstraction(2022) Schulte im Walde, Sabine; Frassinelli, DiegoThis article provides an in-depth study of distributional measures for distinguishing between degrees of semantic abstraction. Abstraction is considered a “central construct in cognitive science” (Barsalou, 2003) and a “process of information reduction that allows for efficient storage and retrieval of central knowledge” (Burgoon et al., 2013). Relying on the distributional hypothesis, computational studies have successfully exploited measures of contextual co-occurrence and neighbourhood density to distinguish between conceptual semantic categorisations. So far, these studies have modeled semantic abstraction across lexical-semantic tasks such as ambiguity; diachronic meaning changes; abstractness vs. concreteness; and hypernymy. Yet, the distributional approaches target different conceptual types of semantic relatedness, and as to our knowledge not much attention has been paid to apply, compare or analyse the computational abstraction measures across conceptual tasks. The current article suggests a novel perspective that exploits variants of distributional measures to investigate semantic abstraction in English in terms of the abstract-concrete dichotomy (e.g., glory-banana) and in terms of the generality-specificity distinction (e.g., animal-fish), in order to compare the strengths and weaknesses of the measures regarding categorisations of abstraction, and to determine and investigate conceptual differences. In a series of experiments we identify reliable distributional measures for both instantiations of lexical-semantic abstraction and reach a precision higher than 0.7, but the measures clearly differ for the abstract-concrete vs. abstract-specific distinctions and for nouns vs. verbs. Overall, we identify two groups of measures, (i) frequency and word entropy when distinguishing between more and less abstract words in terms of the generality-specificity distinction, and (ii) neighbourhood density variants (especially target-context diversity) when distinguishing between more and less abstract words in terms of the abstract-concrete dichotomy. We conclude that more general words are used more often and are less surprising than more specific words, and that abstract words establish themselves empirically in semantically more diverse contexts than concrete words. Finally, our experiments once more point out that distributional models of conceptual categorisations need to take word classes and ambiguity into account: results for nouns vs. verbs differ in many respects, and ambiguity hinders fine-tuning empirical observations.