Figure 5(a) correction · DSGram-Eval #1
So I think we would not be alive if our ancestors did not develop sciences and technologies .
- Human sub-scores
- SC8
- EL8
- F9
- Your weights
- 8.41
- Figure 5 weights
- 8.44
- Equal weights
- 8.33
- Human overall
- 9
AAAI 2025Grammatical error correction · Evaluation
DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction in the Era of Large Language Models
Good grammatical correction must preserve meaning, avoid unnecessary edits, and read fluently. DSGram scores these dimensions separately and adapts their importance to each sentence.
- Result
- SEEDA-S sentence-level pairwise accuracy: GPT-4-weighted DSGram 0.776 vs 0.768 for SOME, the strongest baseline; equal 0.33 weights give 0.648.
- New vs prior
- SOME, following Asano et al. (2017), scores grammaticality, fluency, and meaning with three BERT models and sums them with preset weights; DSGram adds an over-editing criterion and derives sentence-specific weights through AHP.
- Limitation
- The scorer sets the ceiling: on DSGram-Eval, fine-tuned LLaMA3-8B correlates 0.419 with human overall scores, versus 0.772 for zero-shot GPT-4.
01 / The research question
A better-sounding rewrite is not always a better correction
Grammatical error correction (GEC) must fix the language while preserving what the writer meant. An LLM corrector may produce a valid paraphrase that shares few words with the reference, or rewrite too much and shift the meaning; overlap scores penalize the first, and fluency-oriented scores can reward the second. In Figure 1, BLEU cannot separate over- from under-correction, and the reference-free metric SOME misses over-correction.
DSGram scores three criteria. Semantic Coherence asks whether meaning is preserved; Edit Level asks whether the changes were necessary, rather than counting edited tokens; Fluency covers grammar and natural expression. Folding grammaticality into fluency removes a redundant vote: in SOME's annotation data the two correlate at 0.89.
Closest prior work
Reference-based GEC metrics such as M², ERRANT, and BLEU depend on gold corrections that may not cover every valid edit, and they miss the over-correction typical of LLM correctors (Fang et al. 2023). Reference-free metrics follow Asano et al. (2017): SOME scores grammaticality, fluency, and meaning preservation with three BERT models and combines them by a weighted sum. DSGram merges the largely redundant grammaticality and fluency scores, adds an Edit Level criterion for over-correction, and replaces the single weighting with sentence-specific weights that an LLM derives through AHP; IMPARA appears as a further reference-free baseline in Table 3.
03 / Reading the evidence
Dynamic weighting helps most at sentence level; the scorer sets the ceiling
0.776
SEEDA-S sentence-level accuracy
GPT-4-weighted DSGram versus 0.768 for SOME and 0.761 for IMPARA, the strongest baselines (Table 3).
0.648
Same framework, equal weights
Fixing every weight at 0.33 lowers SEEDA-S sentence accuracy from 0.776 to 0.648; at system level on SEEDA-E the gap is small, 0.927 versus 0.922 (Table 3).
Weighting can be tested apart from scoring. Applied to human sub-scores on DSGram-Eval, the authors' human-scored set, AHP weights correlate 0.8764 with human overall ratings versus 0.8544 for equal weights, a difference the paper calls significant only at a relaxed level. On SEEDA, equal weights cost far more at sentence level than at system level (Table 3).
Component scores set the ceiling. GPT-4 correlates best with human sub-scores on all three criteria, LLaMA3-70B is weakest on Edit Level, and LLaMA models fine-tuned on GPT-4-annotated data beat five-shot prompting without approaching GPT-4 (Table 4). On SEEDA-S system-level Pearson correlation, IMPARA and SOME stay ahead of DSGram (Table 3).
Table 3 (selected columns). Meta-evaluation on SEEDA, whose human judgments are sentence-based (SEEDA-S) or edit-based (SEEDA-E): Pearson r and Spearman ρ with human system rankings, and pairwise accuracy (Acc) and Kendall τ at sentence level; higher is better. DSGram rows differ only in how the weights are set; the equal-weights row is printed in the paper as Avg_DSGram (G.w), and the other two as DSGram (L.w) and DSGram (G.w). Source ↗
| Metric | System SEEDA-S r | System SEEDA-E r | System SEEDA-E ρ | Sentence SEEDA-S Acc | Sentence SEEDA-S τ |
|---|---|---|---|---|---|
| M2 | 0.658 | 0.791 | 0.764 | 0.512 | 0.200 |
| ERRANT | 0.557 | 0.697 | 0.671 | 0.498 | 0.189 |
| GLEU | 0.847 | 0.911 | 0.897 | 0.673 | 0.351 |
| SOME | 0.892 | 0.901 | 0.951 | 0.768 | 0.555 |
| IMPARA | 0.911 | 0.889 | 0.944 | 0.761 | 0.540 |
| DSGram, equal weights (0.33) | 0.797 | 0.922 | 0.930 | 0.648 | 0.296 |
| DSGram, LLaMA3-70B weights | 0.880 | 0.906 | 0.895 | 0.631 | 0.263 |
| DSGram, GPT-4 weights | 0.880 | 0.927 | 0.944 | 0.776 | 0.551 |
Method proposed in the paper
Scope. Meta-evaluation uses SEEDA and DSGram-Eval, the authors' human-scored set built from CoNLL-2014 and BEA-2019 test sentences. Only GPT-4, GPT-4o, and LLaMA-family models were tested as scorers and weight generators. Read the study ↗
Read the abstract
Evaluating the performance of Grammatical Error Correction (GEC) models has become increasingly challenging, as large language model (LLM)-based GEC systems often produce corrections that diverge from provided gold references. This discrepancy undermines the reliability of traditional reference-based evaluation metrics. In this study, we propose a novel evaluation framework for GEC models, DSGram, integrating Semantic Coherence, Edit Level, and Fluency, and utilizing a dynamic weighting mechanism. Our framework employs the Analytic Hierarchy Process (AHP) in conjunction with large language models to ascertain the relative importance of various evaluation criteria. Additionally, we develop a dataset incorporating human annotations and LLM-simulated sentences to validate our algorithms and fine-tune more cost-effective models. Experimental results indicate that our proposed approach enhances the effectiveness of GEC model evaluations.
Full paper ↗Citation BibTeX
@inproceedings{xie2025dsgram,
title = {DSGram: Dynamic Weighting Sub-Metrics for Grammatical Error Correction in the Era of Large Language Models},
author = {Xie, Jinxiang and Li, Yilin and Yin, Xunjian and Wan, Xiaojun},
booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
volume = {39},
number = {24},
pages = {25561--25569},
year = {2025},
doi = {10.1609/aaai.v39i24.34746},
url = {https://ojs.aaai.org/index.php/AAAI/article/view/34746}
}