Skip to content

BigBinnie/Benchmarking-LLM-based-Machine-Translation-on-Cultural-Awareness

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

9 Commits
 
 
 
 

Repository files navigation

Benchmarking Machine Translation with Cultural Awareness

This repository contains data and code for the paper Benchmarking Machine Translation with Cultural Awareness. If you have any questions, please reach out to the authors at [email protected].

Overview

How to Evaluate MT Systems’ Cultural Awareness?

To address this challenge, we propose

  1. CAMT (6,948 parallel sentences): a novel parallel corpus for culturally-aware machine translation

  2. CSI-Match and Pragmatic Translation Assessment: two new evaluation metrics to assess translation quality of cultural nuances, particularly for terms lacking established translations.

  3. Benchmarking both LLM-based MT and NMT systems: our results indicate that LLMs can effectively incorporate external cultural knowledge, thereby improving the pragmatic translation quality of CSIs.

CAMT Data

CAMT Corpus includes 6 language pairs, 6,983 CSIs across 18 concept categories from 235 countries. The statisics are as follows:

Pair Sent. CSIs Counts CSIs Types CSI Translations
En-Zh 778 794 601 730
En-Fr 2,073 2,213 2,213 1,130
En-Es 1,580 1,652 1,652 817
En-Hi 1,086 1,127 1,127 168
En-Ta 677 695 695 118
En-Te 754 695 695 66
Total 6,948 7,176 6,983 3,029

We performed quality checks on the En-Zh dataset and filtered out low-quality data. For the other language pairs, the data was automatically generated using our pipeline.

Pipeline for data preprocessing

  1. Entity Linking
  2. Category Classification
  3. Cultural Data Augmentation

Citation

If you use any source codes or datasets included in this repository in your work, please cite the corresponding paper. The bibtex are listed below:

@inproceedings{yao2024benchmarking,
  title={Benchmarking Machine Translation with Cultural Awareness},
  author={Yao, Binwei and Jiang, Ming and Bobinac, Tara and Yang, Diyi and Hu, Junjie},
  booktitle={Findings of the Association for Computational Linguistics: EMNLP 2024},
  pages={13078--13096},
  year={2024}
}

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published