Xunjian Yin / Research

EMNLP 2023 FindingsTranslation evaluation

Exploring Context-Aware Evaluation Metrics for Machine Translation

Xinyu Hu, Xunjian Yin, Xiaojun Wan

Peking University

Paper

Adding document context to a translation metric only at inference trades sentence-level accuracy for system-level gains. Cont-COMET trains COMET with selected context, as human annotators see it, and largely removes that trade-off.

Result
WMT22 segment-level Kendall τ with MQM judgments: Cont-COMET 0.354 vs 0.349 for COMET-21 and 0.292 when COMET-21 gets six context sentences at inference only.
New vs prior
Vernikos et al. (2022) add preceding sentences to a pretrained metric only when scoring, which performed poorly at segment level; Cont-COMET trains COMET with selected preceding and following context.
Limitation
On WMT21 News, every Cont-COMET variant's segment-level τ (0.303–0.306) stays below COMET-21's 0.311, and at WMT22 system level inference-only context scores higher (0.952 vs 0.943).
Cont-COMET's framework: context sentences are selected from the document by similarity to the current sentence, concatenated with it, and passed with the source, reference, and hypothesis through a cross-lingual encoder, pooling layer, and regression estimator to produce the evaluation score. Original Figure 1 from the paper.
Scroll sideways to read the figure. Cont-COMET's framework: context sentences are selected from the document by similarity to the current sentence, concatenated with it, and passed with the source, reference, and hypothesis through a cross-lingual encoder, pooling layer, and regression estimator to produce the evaluation score. Original Figure 1 from the paper. Full size ↗
01 / The research question

A sentence can be fluent and still be the wrong translation

Surrounding sentences can decide whom a pronoun refers to, or whether “They returned to the bank” means a riverbank or a financial institution. WMT human annotators have rated translations with document context since 2019, yet most metrics still score one sentence at a time, without the evidence the annotators had.

Adding context only at inference is not enough: a metric trained on single sentences has not learned to separate a good surrounding passage from a bad target sentence, so a prepended paragraph shifts its input and can blur the judgment. Cont-COMET selects relevant context and trains with it.

Closest prior work

Neural metrics such as COMET (Rei et al., 2020) score each sentence from its source, reference, and hypothesis, although WMT human annotation has included document context since 2019. Vernikos et al. (2022) encoded preceding sentences when computing scores and concluded that two preceding sentences work best; the paper argues that this conclusion was not tested thoroughly and notes the approach's poor segment-level correlation. Cont-COMET uses both preceding and following context, selects it by relevance, and trains the metric with it.

02 / Inside the method

Context changes the representation; the target still determines the score

Cont-COMET extends COMET-21, the 2021 release of the COMET neural metric. Source, hypothesis, and reference are each encoded with surrounding sentences, but pooling covers only the current sentence's tokens, so the score still describes the target. The hypothesis is surrounded by reference context, not by neighboring machine translations that carry their own errors; using the hypothesis's own context lowers system-level correlation (Table 3).

Section 2.2 rule · illustrative document

Which context reaches the evaluator?

Cont-COMET scores every reference sentence in the document by cosine similarity to the current sentence, discounted by α for each sentence of distance, keeps the highest-scoring ones up to a budget, and restores document order (Section 2.2). Here the current reference contains the ambiguous word “bank” (river bank or financial bank). Change the budget, the decay factor, the eligible side, and the similarities.

Sim(Ri, Rj) = cos(ri, rj) · α|i−j|

  1. −3

    Their canoe had drifted downstream.

    cos 0.92 × 0.803
    0.471Selected
  2. −2

    They spent the afternoon outdoors.

    cos 0.38 × 0.802
    0.243Not selected
  3. −1

    It had been a long day.

    cos 0.45 × 0.801
    0.360Not selected
  4. Current referenceThey returned to the bank.“bank” is ambiguous: a river bank or a financial bank. Only the context can tell the evaluator which one the translation must preserve.
  5. +1

    They pulled the canoe onto the shore.

    cos 0.88 × 0.801
    0.704Selected
  6. +2

    The trip ended before sunset.

    cos 0.31 × 0.802
    0.198Not selected

Illustrative document and cosine values (editable above); α is not reported in the paper. Adapted · Section 2.2 selection rule; selection also stops before the 512-token encoder limit, which this short document never reaches.

2 context sentences selected, in document order: −3 → current → +1. An adjacent window of the same size would use −1 instead of −3.

Illustrative: the document, its cosine similarities and every α value (the paper does not report α); the default similarities make the nearest previous sentence uninformative, so that selection and an adjacent window disagree. Adapted: the selection rule from Section 2.2. Computed: selection scores and the chosen context. As α falls, selection approaches an adjacent window of the same size; as α approaches 1, distant but similar sentences compete. No model or translation metric runs here.Section 2.2 ↗

Fine-tuning uses WMT20 Direct Assessment (DA) ratings matched to their documents, because WMT19 lacks document metadata and WMT21–22 DA scores were not available. Expert MQM (Multidimensional Quality Metrics) judgments from WMT21 and WMT22, averaged over en-de, en-ru, and zh-en, are the test sets.

  1. 01

    Select relevant context

    Rank surrounding reference sentences by semantic similarity and distance, keeping the most relevant context within the encoder’s input limit.

  2. 02

    Encode the target sentence

    Combine preceding and following context with the source, reference, and translation; pool representations over the target sentence.

  3. 03

    Train with context

    Fine-tune COMET using contextualized examples and human ratings, matching the information available during human annotation.

03 / Reading the evidence

Context-aware training, not context alone, protects sentence-level scores

The informative comparison is with COMET-21 given the same six context sentences without retraining. On WMT22 that inference-only variant ranks systems best but has the lowest segment-level correlation, while the trained metric gives up a little system-level correlation for the highest segment τ (Table 2). On WMT21 News, inference-only context lowers both levels; Cont-COMET restores system correlation, but its segment τ stays slightly below COMET-21's.

In Table 3, removing selection changes WMT22 correlations only from 0.943 to 0.942 (system) and from 0.354 to 0.353 (segment). Training without context but testing with it reproduces the inference-only pattern, and training with preceding sentences alone is worse on every reported measure. Training under the conditions of use carries the gain; selection is a small refinement.

Table 2 (selected columns). System-level Pearson correlation (Sys.Cor) and segment-level Kendall τ (Seg.Cor) with MQM human judgments, averaged over en-de, en-ru and zh-en; higher is better. “+ 3 prev & 3 next sentences” adds context to COMET-21 at inference only, without training; the WMT21.TED and system-level pairwise-accuracy columns are omitted. Source ↗

ModelWMT21.News Sys.CorWMT21.News Seg.CorWMT22 Sys.CorWMT22 Seg.Cor
COMET-210.6610.3110.9100.349
+ 3 prev & 3 next sentences0.6450.2490.9520.292
Cont-COMET: 2 context sentences selection0.6640.3060.9380.353
Cont-COMET: 4 context sentences selection0.6630.3060.9410.353
Cont-COMET: 6 context sentences selection0.6650.3030.9430.354
Cont-COMET: 8 context sentences selection0.6640.3030.9430.354

Method proposed in the paper

Scope. Evaluation covers three language pairs (en-de, en-ru, zh-en) on the WMT21 TED, WMT21 News, and WMT22 test sets. The paper notes that context-annotated training data remain scarce and that the 512-token encoder limit prevents encoding whole documents. Read the study ↗

Further reading in the paperPaper · Sections 2–5 and Tables 1–3 ↗
Read the abstract

Previous studies on machine translation evaluation mostly focused on the quality of individual sentences, while overlooking the important role of contextual information. Although WMT Metrics Shared Tasks have introduced context content into the human annotations of translation evaluation since 2019, the relevant metrics and methods still did not take advantage of the corresponding context. In this paper, we propose a context-aware machine translation evaluation metric called Cont-COMET, built upon the effective COMET framework. Our approach simultaneously considers the preceding and subsequent contexts of the sentence to be evaluated and trains our metric to be aligned with the setting during human annotation. We also introduce a content selection method to extract and utilize the most relevant information. The experiments and evaluation of Cont-COMET on the official test framework from WMT show improvements in both system-level and segment-level assessments.

Full paper ↗
Citation BibTeX 
@inproceedings{hu-etal-2023-exploring,
    title = "Exploring Context-Aware Evaluation Metrics for Machine Translation",
    author = "Hu, Xinyu  and
      Yin, Xunjian  and
      Wan, Xiaojun",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2023",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-emnlp.1021/",
    doi = "10.18653/v1/2023.findings-emnlp.1021",
    pages = "15291--15298"
}