Evaluation of Google and Bing online translation of verb-noun collocations from English into Arabic
List of Authors
  • Awab, Su’ad , Soori, Hussein

Keyword
  • English-Arabic MT, verb-noun collocations, machine translation evaluation, evaluation metric, Arabic verb synonymy

Abstract
  • This article aims to investigate and evaluate the translation of verb-noun collocation in English into Arabic Google and Bing online translation engines. A number of sentences were used as a testing dataset to evaluate both engines. Human translations by three bilingual speakers were used as a gold standard. A simple evaluation metric was proposed to calculate the translation accuracy of verb-noun collocations. The results showed that Bing scored a verb-noun collocation value of 0.72 with a trend estimation ranging between 0.65 and 0.67. Google scored a verb-noun collocation value of 0.75 (3% higher than Bing) with a trend estimation ranging between 0.63 and 0.85. The results also showed that, in most cases, the Arabic translation output of both engines produced a one verb synonym which did not collocate with the different nouns in the testing data sentences. These results indicate that Google and Bing, so far, have not been able to resolve the verb-noun collocability problem in their Arabic output. This study and its results may shed some light on the problem and to develop new methods to improve Arabic verb noun collocability in the output translation of current machine translation engines.

Reference
  • 1. Banerjee, S., & Lavie, A. (2005, June). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization (pp. 65-72).
    2. Bing Translator (2015). Available from http://www.bing.com/translator.
    3. Breidt, E. (1993, June). Extraction of VN-collocations from text corpora: A feasibility study for German. In Proceedings of the Workshop on Very Large Corpora: Academic and Industrial Perspectives, Columbus, USA.
    4. Buckwalter, T. (2002). Buckwalter Arabic Morphological Analyzer Version 1.0, Linguistic Data Consortium (LDC) catalog number LDC2002L49 and ISBN 1-58563-257-0.
    5. Denkowski, M., & Lavie, A. (2011). METEOR 1.4 automatic machine translation evaluation system. CMU Language Technologies Institute. Retrieved from https://www.cs.cmu.edu/~alavie/METEOR/README.html#about.
    6. Google Translator (2015). Available from https://translate.google.com.
    7. Goutte, C. (2006). Automatic evaluation of machine translation quality. Presentation at the European Community, Xerox Research Centre Europe, on January, 27, 2006.
    8. Habash, N. Y. (2010). Introduction to Arabic natural language processing. Synthesis Lectures on Human Language Technologies, 3(1), 1-187.
    9. Hornby, A. S., & Cowie, A. P. (1980). Oxford Advanced Learner’s Dictionary of current English (Revised and reset of the third edition 1974). New Delhi: R. Dayal, Oxford university press.
    10. Lavie, A., Sagae, K., & Jayaraman, S. (2004). The significance of recall in automatic metrics for MT evaluation. In Machine Translation: From Real Users to Research (pp. 134-143). Springer Berlin Heidelberg.
    11. Levenshtein, V. I. (1966). Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady, 10(8), 707-710.
    12. Microsoft Translator (2015). Available from http://www.microsoft.com/en-us/translator.
    13. Oxford Advanced Learner's Dictionary (2013). Available from: http://oald8.oxfordlearnersdictionaries.com/dictionary/draw_1.
    14. Papineni, K., Roukos, S., Ward, T., & Zhu, W. J. (2002, July). BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics (pp. 311-318). Association for Computational Linguistics.
    15. Qin, Y., Wen, Q., & Wang, J. (2009, September). Automatic evaluation of translation quality using expanded N-gram co-occurrence. In Natural Language Processing and Knowledge.
    16. Engineering, 2009. NLP-KE 2009. International Conference on (pp. 1-5). IEEE.