AUTOMATIC EVALUATION OF MACHINE TRANSLATION QUALITYOF A SCIENTIFIC TEXT: FIVE YEARS LATER

Aim. The paper compares translations of Google and PROMT neural machine translation systems with translations obtained 5 years ago, when statistical machine translation and rule-based machine translation algorithms were used, respectively, as the main translation algorithms of these systems (see Bulletin of Moscow Region State University. Series: Linguistics. 2016. no. 4. pp. 174-182). Methodology. Use is made of such modern metrics for automatic quality evaluation of machine translation as BLEU, F-measure and TER to compare the quality of translations of modern neural machine translation systems using the example of Google and PROMPT online systems. Results. The evaluation of the translation quality of candidate texts generated by Google and PROMT in comparison with the reference translation using an automatic translation evaluation program reveals significant qualitative changes as compared with the results obtained 5 years ago, which indicates a dramatic improvement in the work of the above-mentioned online translation systems. Research implications. The described three methods for evaluating the quality of machine translation allow one to analyze several automatic methods for evaluating machine translation quality, i.e. methods based on string matching and n-gram models. Ways to improve the quality of machine translation are discussed. It is shown that modern systems of automatic translation quality evaluation allow errors made by machine translation systems to be identified and systematized, which will make it possible to improve the quality of translation by these systems in the future.

automatic evaluation, translation quality, machine translation, metrics, BLEU, TER, F-measure, reference translation

1. Комиссаров В. Н., Коралова А. Л. Практикум по переводу с английского языка на русский. М.: Высшая школа, 1990. 127 c.

2. Митренина О. В. Машинный перевод // Прикладная и компьютерная лингвистика / ред. Николаев И. С., Митренина О. В., Ландо Т. М. М.: УРСС, 2016. C. 156-189.

3. Улиткин И. А. Автоматическая оценка качества машинного перевода научно-технического текста // Вестник Московского государственного областного университета. Серия: Лингвистика. 2016. № 4. С. 174-182. DOI: 10.18384/2310-712X-2016-4-174-182.

4. A Study of Translation Edit Rate with Targeted Human Annotation / Snover M., Dorr B., Schwartz R., Micciulla L., Machoul J. // Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers. 2006. P. 223-231.

5. Austermьhl F. Electronic Tools for Translators. Manchester: St. Jerome, 2001. 202 p.

6. Bleu: A Method for Automatic Evaluation of Machine Translation / Papineni K., Roukos S., Ward T., Zhu W.-J. // ACL ‘02: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. 2002. P. 311-318. DOI: 10.3115/1073083.1073135.

7. Confidence Estimation for Machine Translation / Blatz J., Fitzgerald E., Foster G., Gandrabur S., Goutte C., Kulesza A., Sanchis A., Ueffing N. // COLING’ 04. Proceedings of the 20th International conference on Computational Linguistics. 2004. P. 315-321. DOI: 10.3115/1220355.1220401.

8. Gross A. Limitations of Computers as Translation Tools // Computers and Translation / ed. J. Newton. London: Routledge, 1992. P. 96-130.

9. Hutchins J. Current commercial machine translation systems and computer-based translation tools: System types and their uses // International Journal of Translation. 2005. Vol. 17 (1-2). P. 5-38.

10. Hutchins J. Machine Translation: History // Encyclopedia of Language and Linguistics / ed. K. Brown. Oxford: Elsevier, 2006. P. 375-383.

11. Language and Machines: Computers in Translation and Linguistics: a report by the Automatic Language Processing Advisory Committee, National Academy of Science / Pierce J. R., Carroll J. B., Hamp E. P., Hays D. G., et al. Washington, DC: The National Academic Press, 1966. 138 p.

12. Melamed I. D., Green R., Turian J. P. Precision and Recall of Machine Translation // NAACL-Short ‘03: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology: Companion volume of the Proceedings of HLT-NAACL 2003 (short papers). 2003. Vol. 2. P. 61-63. DOI: 10.3115/1073483.1073504.

13. Quah C. K. Translation and Technology. Basingstoke: Palgrave, 2006. 221 p.

14. Ulitkin I. A. Human Translation vs. Machine Translation: Rise of the Machines // Translation Journal. 2013. Vol. 17. № 1 [Электронный ресурс]. URL: http://translationjournal.net/journal/63mtquality.htm (дата обращения: 20.10.2021).