ACL 2022Text generation · Human evaluation
How Do Seq2Seq Models Perform on End-to-End Data-to-Text Generation?
A high generation score does not explain what a model gets wrong. Fine-grained human annotation reveals how copying, pre-training, and dataset structure affect data-to-text errors.
- Result
- ToTTo human error scores: adding a copy mechanism to Transformer cuts Omission from 19.74 to 7.48 but raises Addition from 4.38 to 11.01 (lower is better).
- New vs prior
- Ferreira et al. (2019) compare pipeline and end-to-end data-to-text systems, and PolyTope (Huang et al., 2020) annotates summarization errors; this work applies MQM error annotation to five Seq2Seq models on four datasets.
- Limitation
- Each model–dataset score rests on 80 annotated outputs, and only models up to 220M parameters were tested; T5-large and BART-large were omitted for compute reasons.
Fluent text can put the right entity in the wrong role
In the paper's Table 2 example, the input gives Round Rock as the largest city of Williamson County and Georgetown as its county seat; the output calls Georgetown the largest city and omits the county seat. The sentence reads naturally, yet it contains a distorted fact and a missing one, each needing a different repair, and a single overlap score cannot say which.
Earlier human evaluations of data-to-text systems covered few samples, datasets, models, or error dimensions. Here annotators locate every error and record its type and severity, giving each model an error profile instead of one number.
Closest prior work
Ferreira et al. (2019) compared pipeline and end-to-end architectures for neural data-to-text generation, and Thomson and Reiter (2020) proposed a protocol for human evaluation of accuracy. Most model comparisons still rely on BLEU or ROUGE, which can disagree with human judgment (Novikova et al., 2017a; Reiter, 2018), and existing human evaluations cover few samples, datasets, models, or error dimensions. This study adapts the MQM-based PolyTope scheme of Huang et al. (2020), developed for summarization, to annotate five Seq2Seq models on four data-to-text datasets.
An architectural improvement can move errors between categories
19.74 → 7.48
ToTTo Omission score with copying
Adding a copy mechanism to Transformer cuts the severity-weighted Omission score but raises Addition from 4.38 to 11.01 (Table 5; lower is better).
62.88 vs 59.96
E2E BLEU, T5-small vs T5-base
BLEU ranks T5-small above T5-base, while human MQM scores reverse the order, 86.04 versus 96.36 (Tables 3–4).
Measured · Table 6
Which error types cost the most across the study
Average severity-weighted, length-normalized error score of each type over all models and datasets (Table 6, lower is better). Each score is Σ αe Le / wordcount for that type, the quantity the annotation demo in Inside the method computes for one output.
| Error type | Average error score ↓ |
|---|---|
| Omission | 5.97 |
| Inaccuracy Intrinsic | 5.25 |
| Inaccuracy Extrinsic | 2.30 |
| Addition | 1.48 |
| Duplication | 0.63 |
| Word Order | 0.56 |
| Word Form | 0.23 |
| Positive-Negative Aspect | 0.17 |
Measured · Table 6 printed averages. Shaded rows are the two error types of the Table 2 example (Omission and Inaccuracy Intrinsic); they carry the two largest average penalties. The other six types together sum to 5.37.
Copying preserves input content but also pulls in content the description does not need: on ToTTo, the copy model's Omission score falls by more than half while its Addition score more than doubles (Table 5). The paper attributes the extra additions to an auto-regressive decoder that copies long input spans and is hard to interrupt.
BLEU's overall trend agrees with human evaluation, but the paper reports a Pearson correlation of 0.61 between BLEU and the accuracy score and only 0.08 for fluency (Section 6), and rankings of close models can flip, as on E2E. Pre-training has the largest effect in Table 3: pre-trained models beat the others by a wide margin on every dataset except E2E.
Table 5 (ToTTo rows; the Positive-Negative Aspect column, 0 for every model on ToTTo, is omitted), with each model's overall ToTTo human score from Table 3. Error columns are severity-weighted, length-normalized error scores from human MQM annotation of 80 outputs per model, lower is better; the human score is 100 minus the summed error scores, up to rounding, higher is better. ptr-gen is the Transformer with a copy mechanism. On ToTTo, copying reduces Omission but raises Addition and Word Form; the rise in Addition also appears on WikiBio and WebNLG, but not on E2E. Source ↗
| Model (ToTTo) | Addition | Duplication | Extrinsic | Intrinsic | Omission | Word Form | Word Order | Human score↑ |
|---|---|---|---|---|---|---|---|---|
| Transformer | 4.38 | 2.38 | 11.03 | 17.02 | 19.74 | 0 | 0 | 45.41 |
| ptr-gen | 11.01 | 1.31 | 9.11 | 13.47 | 7.48 | 3.01 | 0 | 54.57 |
| T5-small | 0 | 0 | 0 | 5.36 | 9.19 | 0 | 0 | 85.44 |
| T5-base | 0 | 0 | 1.86 | 4.38 | 4.38 | 0 | 0.76 | 88.59 |
| BART-base | 0 | 0 | 1.79 | 2.41 | 2.66 | 0 | 2.41 | 90.71 |
Scope. All four datasets are English and differ in how much content selection they require, from fully specified WebNLG triples and partially specified E2E records to ToTTo's highlighted cells and WikiBio's unspecified infobox content (Table 1). Read the study ↗
Read the abstract
With the rapid development of deep learning, Seq2Seq paradigm has become prevalent for end-to-end data-to-text generation, and the BLEU scores have been increasing in recent years. However, it is widely recognized that there is still a gap between the quality of the texts generated by models and the texts written by human. In order to better understand the ability of Seq2Seq models, evaluate their performance and analyze the results, we choose to use Multidimensional Quality Metric(MQM) to evaluate several representative Seq2Seq models on end-to-end data-to-text generation. We annotate the outputs of five models on four datasets with eight error types and find that 1) copy mechanism is helpful for the improvement in Omission and Inaccuracy Extrinsic errors but it increases other types of errors such as Addition; 2) pre-training techniques are highly effective, and pre-training strategy and model size are very significant; 3) the structure of the dataset also influences the model’s performance greatly; 4) some specific types of errors are generally challenging for seq2seq models.
Full paper ↗Citation BibTeX
@inproceedings{yin-wan-2022-seq2seq,
title = "How Do {S}eq2{S}eq Models Perform on End-to-End Data-to-Text Generation?",
author = "Yin, Xunjian and Wan, Xiaojun",
booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics",
year = "2022",
url = "https://aclanthology.org/2022.acl-long.531/",
pages = "7701--7710"
}