Cross-lingual word embeddings with multi-sense representations

Abstract

Cross-lingual word embeddings have been found to be useful in aiding cross-lingual transfer, but work in this line of research has to date rarely addressed the monosemy constraint of static word embeddings in depth, where the collapse of multiple meanings into one form might arguably lead to subpar alignments. In this thesis, we address this gap by examining potential approaches towards the incorporation of sense information into cross-lingual alignment. We explore in specfic two variants of cross-lingual multi-sense alignment: one in which we employ the method of embedding the senses of each word as a Gaussian mixture (Athiwaratkun and Wilson, 2017), where the assumption is that multi-sense embeddings as a basis for alignment may help mitigate the meaning conflation deficiency (Camacho-Collados and Pilehvar, 2018), and in turn help improve isomorphism between vector spaces (Ruder et al., 2019). Our second method explores learning a cross-lingual multi-sense embedding space by reversing the order: we cross-lingually align uni-sense word embeddings, and attempt multi-sense enrichment as a postprocessing step by retrofitting (Pilehvar and Collier, 2016) the embedding on the Open Multilingual Wordnet (Bond et al., 2023). We observe that our model is capable of fine-grained cross-lingual semantic distinctions, where our model successfully identifies colexifications without cross-lingual supervision.