Probabilistic modelling of morphologically rich languages
This thesis investigates how the sub-structure of words can be accounted for in probabilistic models of language. Such models play an important role in natural language processing tasks such as translation or speech recognition, but often rely on the simplistic assumption that words are opaque symbols. This assumption does not fit morphologically complex language well, where words can have rich internal structure and sub-word elements are shared across distinct word forms. Our approach is to encode basic notions of morphology into the assumptions of three different types of language models, with the intention that leveraging shared sub-word structure can improve model performance and help overcome data sparsity that arises from morphological processes. In the context of n-gram language modelling, we formulate a new Bayesian model that relies on the decomposition of compound words to attain better smoothing, and we develop a new distributed language model that learns vector representations of morphemes and leverages them to link together morphologically related words. In both cases, we show that accounting for word sub-structure improves the models' intrinsic performance and provides benefits when applied to
Botha, Jan Abraham
Natural Language Processing. ; Computational Linguistics ; Computer science (mathematics) ; language modelling ; morphology ; machine translation ; Bayesian non-parametrics ; distributed feature representations ; neural language model ; morpheme vectors
Probabilistic modelling of morphologically rich languages. This thesis investigates how the sub-structure of words can be accounted for in probabilistic models of language. Such models play an important role in natural language processing tasks such as translation or speech recognition, but often rely on the simplistic assumption that words are opaque symbols. This assumption does not fit morphologically complex language well, where words can have rich internal structure and sub-word elements are shared across distinct word forms. Our approach is to encode basic notions of morphology into the assumptions of three different types of language models, with the intention that leveraging shared sub-word structure can improve model performance and help overcome data sparsity that arises from morphological processes. In the context of n-gram language modelling, we formulate a new Bayesian model that relies on the decomposition of compound words to attain better smoothing, and we develop a new distributed language model that learns vector representations of morphemes and leverages them to link together morphologically related words. In both cases, we show that accounting for word sub-structure improves the models' intrinsic performance and provides benefits when applied to
probabilistic modelling morphologically rich language investigates sub structure word account probabilistic model language model play_important_role natural_language_processing task translation speech_recognition often rely simplistic assumption word opaque symbol assumption fit morphologically complex language well word rich internal_structure sub word element share across distinct word_form encode basic notion morphology assumption three_different type language model intention share sub word structure improve model performance help overcome data arise morphological process context language modelling formulate_new bayesian_model relies decomposition compound word attain good develop_new distribute language model learn vector representation morpheme link together morphologically related word case show account word sub structure improve model intrinsic performance provide benefit apply