Assessing MT with measures of PE effort

Abstract

Recent improvements in quality obtained by neural machine translation (NMT) have boosted its presence in the translation industry. In many domains and language combinations, translators post-edit raw MT output: they edit and correct the pre-translated text to produce the final translation. However, this process can only produce the expected results if the quality of the raw MT can be assured. MT is usually assessed with automatic metrics, as they are faster and cheaper. However, these metrics are not always good quality indicators and do not correlate to the post-editing effort. We suggest a two-step evaluation process for MT intended for post-editing. The automatic evaluations are followed by the assessment of the three dimensions of PE effort. This targeted evaluation can ensure a quality of the raw MT which does not jeopardise the final product or compromise the task of post-editors. We include a detailed description of PosEdiOn, an easy-to-use standalone tool which records PE effort, and a use case of its implementation. 18 translators post-edit texts from English into Spanish from the news domain translated with DeepL and an NMT system trained by the authors to gather PE effort metrics. We compare automatic and PE effort metrics to assess which MT system would be more suitable for post-editing.