Evaluating parallel corpora and translation quality for Chinese and English
Parallel bilingual corpora are important basic resources for statistical machine translation. Accurate alignment of textual elements (e.g. documents, paragraphs, sentences) in a parallel bilingual corpus is a crucial step for statistical machine translation. Rather than using sentence length, word co-occurrence, cognates, dictionaries or parts of speech, this thesis uses compression code lengths based on the Prediction by Partial Matching (PPM) compression algorithm to measure when two sentences are aligned for parallel Chinese-English corpora. PPM has been found to be an effective method as a measure of whether the information conveyed by the texts is similar at estimating the entropy of the text. Evaluation of the quality of sentence alignment is a way to measure the quality of a corpus. Evaluating parallel bilingual corpora is also an important process and usually the last step for parallel bilingual corpus creation. However, most statistics of parallel bilingual corpora are based on counts of characters, words, tokens, sentences or files. As there is a lack of advanced parallel bilingual corpus evaluation methods, this thesis adopts a new PPMbased method for parallel bilingual corpus evaluation. The method
Liu, Wei
Evaluating parallel corpora and translation quality for Chinese and English. Parallel bilingual corpora are important basic resources for statistical machine translation. Accurate alignment of textual elements (e.g. documents, paragraphs, sentences) in a parallel bilingual corpus is a crucial step for statistical machine translation. Rather than using sentence length, word co-occurrence, cognates, dictionaries or parts of speech, this thesis uses compression code lengths based on the Prediction by Partial Matching (PPM) compression algorithm to measure when two sentences are aligned for parallel Chinese-English corpora. PPM has been found to be an effective method as a measure of whether the information conveyed by the texts is similar at estimating the entropy of the text. Evaluation of the quality of sentence alignment is a way to measure the quality of a corpus. Evaluating parallel bilingual corpora is also an important process and usually the last step for parallel bilingual corpus creation. However, most statistics of parallel bilingual corpora are based on counts of characters, words, tokens, sentences or files. As there is a lack of advanced parallel bilingual corpus evaluation methods, this thesis adopts a new PPMbased method for parallel bilingual corpus evaluation. The method
evaluating parallel_corpus translation_quality chinese_english parallel bilingual corpus important basic resource statistical_machine_translation accurate alignment textual element document paragraph sentence parallel bilingual corpus crucial step statistical_machine_translation rather use sentence length word co_occurrence cognate dictionary part speech use compression code length base prediction partial matching ppm compression algorithm measure sentence align parallel chinese_english corpus ppm find effective method_measure whether information convey text similar estimate text evaluation quality sentence alignment way measure quality corpus evaluating parallel bilingual corpus also_important process usually last step parallel bilingual corpus creation however statistic parallel bilingual corpus_base count character word token sentence file lack advance parallel bilingual corpus evaluation method_adopt new method parallel bilingual corpus evaluation method