Corpus-based machine translation evaluation via automated error detection in output texts
Since the emergence of the first fully automatic machine translation (MT) systems over fifty years ago, the use of MT has increased dramatically. Consequently, the evaluation of MT systems is crucial for all stakeholders. However, the human evaluation of MT output is expensive and time-consuming, often relying on subjective quality judgements and requiring human 'reference translations' against which the output is compared. As a result, interest in more recent years has turned towards automated evaluation methods, which aim to produce scores that reflect human quality judgements. As the majority of published automated evaluation methods still require human 'reference translations' for comparison, the goal of this research is to investigate the potential of a method that requires access only to the translation. Based on detailed corpus analyses, the primary aim is to devise methods for the automated detection of particular error types in French-English MT output from competing systems and to explore correlations between automated error counts and human judgements of a translation as a whole. First, a French-English corpus designed specifically for MT evaluation was compiled. A sample of MT output from the corpus
Elliott, Debra
Corpus-based machine translation evaluation via automated error detection in output texts. Since the emergence of the first fully automatic machine translation (MT) systems over fifty years ago, the use of MT has increased dramatically. Consequently, the evaluation of MT systems is crucial for all stakeholders. However, the human evaluation of MT output is expensive and time-consuming, often relying on subjective quality judgements and requiring human 'reference translations' against which the output is compared. As a result, interest in more recent years has turned towards automated evaluation methods, which aim to produce scores that reflect human quality judgements. As the majority of published automated evaluation methods still require human 'reference translations' for comparison, the goal of this research is to investigate the potential of a method that requires access only to the translation. Based on detailed corpus analyses, the primary aim is to devise methods for the automated detection of particular error types in French-English MT output from competing systems and to explore correlations between automated error counts and human judgements of a translation as a whole. First, a French-English corpus designed specifically for MT evaluation was compiled. A sample of MT output from the corpus
corpus_based machine_translation evaluation via automate error detection output text since emergence first fully automatic machine_translation system use increase dramatically consequently evaluation system crucial stakeholder however human evaluation output expensive time consuming often rely subjective quality judgement require human reference_translation output compare interest turn_towards automate evaluation method aim produce score reflect human quality judgement majority publish evaluation method still require human reference_translation comparison goal_investigate potential method require access translation based detailed corpus_analysis primary_aim devise method automate detection particular error type french_english output compete system explore correlation automate error count human judgement translation whole first french_english corpus design specifically evaluation compile sample output corpus