Xunjian Yin / Research

LREC-COLING 2024Speech recognition · Retrieval

Contextual Modeling for Document-level ASR Error Correction

Jin Jiang1,2, Xunjian Yin1, Xiaojun Wan1, Wei Peng3, Rongjun Li3, Jingyuan Yang3, Yanquan Zhou2

1 Peking University · 2 Beijing University of Posts and Telecommunications · 3 Huawei Technologies

Paper Code

A transcript can contain clues to its own recognition errors. Retrieving context from the document and related documents helps a correction model use those clues.

Result
AISHELL-1 character error rate after correction: 3.336% with offline dataset context vs 3.528% for sentence-level BART, the strongest baseline (p < 0.05).
New vs prior
Wang et al. (2021, 2022b) add a context encoder, and doc2doc models feed many sentences, widening attention at a length and noise cost; this work retrieves context through kNN attention.
Limitation
The best results need offline access to later sentences and a datastore spanning the whole test set; a live system can use only the online setting.
The correction model is a Transformer encoder-decoder in which one decoder layer is a kNN attention layer: its queries search a datastore of (key, value) pairs, and the retrieved kNN output is mixed with the local attention output through a gate g. Original Figure 1 from the paper.
The correction model is a Transformer encoder-decoder in which one decoder layer is a kNN attention layer: its queries search a datastore of (key, value) pairs, and the retrieved kNN output is mixed with the local attention output through a gate g. Original Figure 1 from the paper. Full size ↗
01 / The research question

Correcting a transcript requires remembering what was said elsewhere

Automatic speech recognition (ASR) error correction (AEC) rewrites a recognizer's text output, and some errors cannot be fixed from one sentence. The pronouns 他 (he) and 她 (she) are both pronounced tā; in the paper's Table 1, a report names the volleyball player Zhu Ting, yet a sentence-level corrector writes 他 in the next sentence.

A longer input is the obvious fix, but attention cost grows with length and unrelated sentences add noise. The corrector instead keeps a one-sentence input and retrieves internal representations from a larger context; the recognizer itself is unchanged.

Closest prior work

Most ASR error correction (AEC) models, such as BART-based correctors (Zhao et al., 2021), work one sentence at a time. Document-level MT offers two ways to add context: a separate context encoder (sent2sent), which Wang et al. (2021, 2022b) applied to ASR correction, and multi-sentence input (doc2doc). The paper argues that both widen attention, which limits context length and adds unselected noise. kNN-MT (Khandelwal et al., 2020b) instead retrieves decoder states from a datastore built on training data; this work retrieves attention keys and values from the current document or test set.

02 / Inside the method

A memory of decoder states extends local attention

The corrector starts from BART fine-tuned for sentence-level AEC. In one decoder layer, the self-attention queries also search a datastore of attention keys and values from other processed tokens, and a gate mixes the retrieved output with local attention (Figure 1). A gate of about 0.4–0.5 works best; larger values let retrieved context override the current sentence.

Section 3.4 rules · Table 1 and Table 4 examples

Which sentences can the corrector search?

The kNN attention layer searches a datastore of attention keys and values from other sentences. Two choices decide what it holds (Section 3.4): online or offline timing, and document or dataset scope. Step through the documents and watch which sentences are searchable while sentence A3 is corrected, and whether the sentence that disambiguates it, from the paper’s Table 1 or Table 4, is among them.

Document A (followed)

  1. A1not processed yet
  2. A2cuenot processed yet
  3. A3targetnot processed yet
  4. A4not processed yet
  5. A5not processed yet

Document B

  1. B1not processed yet
  2. B2not processed yet
  3. B3not processed yet
  4. B4not processed yet
  5. B5not processed yet

Document C

  1. C1not processed yet
  2. C2not processed yet
  3. C3not processed yet
  4. C4not processed yet
  5. C5not processed yet
  • being corrected
  • searchable, same document
  • searchable, other document
  • not processed yet
  • outside this document’s datastore

Step 0 of 5: before correction startsOnline, document context · searchable: 0 sentences of document A, 0 of other documents

A3 · target
他还凭借拦网和发球分别拿到七分和一分。
BART: “He also had seven points and one point with his block and serve.”
她还凭借拦网和发球分别拿到七分和一分。
Label: “She also …”; 他 and 她 are both pronounced tā
A2 · cue
本场比赛朱婷三七次扣球得到二十一分。
Table 1: “Zhu Ting scored twenty-one points in this game …”; the paper names 朱婷 (Zhu Ting) as the context needed

Published · Table 1 sentences, BART output and label. Schematic three test documents of five sentences processed in parallel, one position per step, as in Figure 2(a); the positions of the cue and the target are reader-set and illustrative, since the paper prints the sentences but not their places in the documents.

Is the cue searchable when A3 is corrected?

Datastore modeCue in A2Cue in A4AISHELL-1 CER ↓
Online, document contextYesNonot processed yet3.412
Online, dataset contextYesNonot processed yet3.379
Offline, document contextYesYes3.407
Offline, dataset contextYesYes3.336*

Computed · Section 3.4 rules availability in this schematic. Measured · Table 2 character error rate after correction; sentence-level BART 3.528; * significant against BART at p < 0.05. Availability is necessary, not sufficient: the model still has to retrieve and use the states.

Before correction starts, the online datastore is empty. Each corrected sentence adds its attention keys and values, so later sentences can search earlier ones.

Published: the sentences, BART outputs and labels of Table 1 (他 or 她, document context) and Table 4 (管撤 or 贯彻, a sentence from another document). Schematic and illustrative: three test documents of five sentences, processed in parallel one position per step as in Figure 2(a), and the positions of the cue and the target, which the paper does not print; the cue position is reader-set. Computed: which sentences are searchable, from the Section 3.4 rules: online mode stores only sentences already corrected, offline mode starts from the finished online store, document context searches the current document and dataset context every test document. Figure 2 marks the current sentence separately, so it is not counted as context. Measured: AISHELL-1 character error rates (Table 2). The demo shows availability only; it does not run retrieval or the gate that mixes retrieved and local attention (0.4–0.5 works best, Section 5.5).Section 3.4 and Figure 2 ↗

Two choices set what the datastore holds. Document context limits retrieval to the current transcript; dataset context adds the other documents of the same split, so at test time it holds only test transcripts. Online mode adds each corrected sentence's states as it goes, so only earlier sentences are searchable; offline mode starts from the finished online store and can also use later sentences.

  1. 01

    Build a context datastore

    Store contextual representations from the current document or the wider dataset, allowing correction to access information beyond its immediate sentence.

  2. 02

    Retrieve through attention

    Add a kNN attention layer to a BART correction model, combining retrieved context with local representations.

  3. 03

    Support two timing modes

    Online correction builds the datastore incrementally; offline correction also accesses following context using the online datastore as initialization.

03 / Reading the evidence

Retrieved context helps where a longer input hurts

More context is not automatically better: a sliding window of neighboring sentences raises error rates above sentence-level BART on all four datasets, and a dual context encoder helps only slightly on three (Table 2). All four retrieval variants beat BART on every dataset, though by far less than BART's own reduction of the raw WeNet (Mandarin) and fairseq (English) output.

Offline beats online and dataset context beats document context, with one tie on LibriSpeech Other. Shuffling the documents or replacing retrieved entries with random vectors raises error rates, so the gain comes from the retrieved context itself (Table 5). The cost is inference time: with 16 neighbors, decoding takes 1.19× BART's time with a document store and 1.50× with a dataset store (Table 7).

Table 2 (selected columns). Character error rate (CER, Mandarin) and word error rate (WER, English) after correction, lower is better; RAW is the uncorrected output of the WeNet (Mandarin) or fairseq (English) recognizer (Appendix Table 10). * marks results significant at p < 0.05 against the BART baseline; the CERR/WERR reduction columns are omitted. Source ↗

MethodAISHELL-1 CER↓HKUST CER↓LIB_CLEAN WER↓LIB_OTHER WER↓
RAW4.31227.604.5539.136
BART3.52821.361.9674.318
Dual (sent2sent)3.56921.101.9584.304
Sliding k (doc2doc)3.63822.841.9954.855
Online AEC (doc)3.41221.231.8444.252
Online AEC (dataset)3.37920.681.8354.245
Offline AEC (doc)3.40720.961.8274.237
Offline AEC (dataset)3.336*20.66*1.822*4.237*

Method proposed in the paper

Scope. The authors build the four correction datasets from AISHELL-1 and HKUST (Mandarin, transcribed with WeNet) and LibriSpeech clean and other (English, transcribed with fairseq). The method is not tested beyond ASR correction; the paper names document-level translation as a next task. Read the study ↗

Further reading in the paperPaper · Sections 3–5 and Table 2 ↗Implementation ↗
Read the abstract

Contextual information, including the sentences in the same document and in other documents of the dataset, plays a crucial role in improving the accuracy of document-level ASR Error Correction (AEC), while most previous works ignore this. In this paper, we propose a context-aware method that utilizes a k-Nearest Neighbors ( kNN) approach to enhance the AEC model by retrieving a datastore containing contextual information. We conduct experiments on two English and two Chinese datasets, and the results demonstrate that our proposed model can effectively utilize contextual information to improve document-level AEC. Furthermore, the context information from the whole dataset provides even better results.

Full paper ↗
Citation BibTeX 
@inproceedings{jiang-etal-2024-contextual,
  title = {Contextual Modeling for Document-level ASR Error Correction},
  author = {Jiang, Jin and Yin, Xunjian and Wan, Xiaojun and Peng, Wei and Li, Rongjun and Yang, Jingyuan and Zhou, Yanquan},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  pages = {3855--3867},
  url = {https://aclanthology.org/2024.lrec-main.341/}
}