ACL 2025 FindingsConsistent multimodal editing
MC-MKE: A Fine-Grained Multimodal Knowledge Editing Benchmark Emphasizing Modality Consistency
Correcting one multimodal answer does not ensure related answers change consistently; MC-MKE checks whether edits agree across image recognition, textual facts, and questions about images.
- Result
- On recognition edits (IE_edit) in InstructBLIP, FT(LLM) and SERAC reach 98.48% reliability but only 9.09% consistency; the most consistent method, IKE, reaches 49.05%.
- New vs prior
- MMEdit edits VQA and captioning answers and VLKEB and MIKE edit entity recognition, without testing consistency; this work covers three decomposed edit scenarios and checks that linked answers change together.
- Limitation
- Experiments cover only two 7B multimodal models (InstructBLIP, MiniGPT-v2) and four editing methods; the authors could not test larger checkpoints.
One edited answer can hide a contradiction
A multimodal model may recognize a person in an image, know which club that person plays for, and answer a club question about the image. These are three linked pieces of knowledge, and an edit can repair the requested answer while leaving a linked one unchanged. MC-MKE scores that relationship, not only the edited response.
The same new answer can have different causes. Correcting a misrecognized player changes which entity is recognized in the image; a transfer changes the player's club. Both alter the image-based club answer, but each implies a different set of answers that must change with it.
Closest prior work
Text-only editing benchmarks such as CounterFact, MQuAKE, and AToKe address language-model editing; MC-MKE builds its facts from MQuAKE. Among multimodal benchmarks, MMEdit (Cheng et al., 2024) is built from visual question answering and image captioning, while VLKEB (Huang et al., 2024) and MIKE (Li et al., 2024b) focus on entity-level recognition edits. The paper argues that none defines multimodal knowledge as recognition composed with a textual fact, so none checks whether an edit to one component carries through to the composed answer.
The reason for an edit determines its consequences
The benchmark decomposes multimodal knowledge into visual knowledge, an image–entity pair, and textual knowledge, an entity–relation–object triplet; composing them gives the image-based fact. Three edit scenarios follow: IE_edit changes recognition, SRO_edit changes a textual fact, and IRO_edit changes the image-based answer directly. Each scenario names a different linked answer to check for consistency.
Published example · Figure 2 and Eq. (2)
Which answer must change with the edit?
Choose an editing scenario and play the edit on the paper's Figure 2 example. The linked answer is recomputed from the composition (i, e) ×e=s (s, r, o) = (i, r, o), so each scenario names a different answer that must follow.
Recomputed with Eq. (2): the image-based answer (i, r, o) must change from Liverpool to Miami FC. Edit target Messi; consistency target Miami FC. The targets differ, so repeating the edit target cannot pass the consistency probe.
Published example · Figure 2 · IE_edit, recognition edit
The photograph in Figures 1 and 2 shows Messi. Before a recognition edit the model sees Mac Allister and therefore answers Liverpool; each scenario fixes a different part of that chain.
| Knowledge | Statement at this step | Role |
|---|---|---|
| Visual (i, e) | The player in the image is Messi.previously Mac Allister | Edited · reliability probe |
| Textual (s, r, o) | Messi plays for Miami FC.previously Mac Allister plays for Liverpool. | Unchanged knowledge · now selected because e = s |
| Multimodal (i, r, o) | The player in the image plays for Miami FC.previously Liverpool | Must follow · consistency probe |
Composition, Eq. (2)(image, Messi) ×e=s (Messi, plays for, Miami FC) = (image, plays for, Miami FC)
Example: Figure 2 and Section 3.3, described in text; the wrong recognition and (Mac Allister, play for, Liverpool) are from Figure 1 and Section 1. The linked answer is recomputed here from the decomposition K(i, e, s, r, o) = (i, e) ×e=s (s, r, o); no model is run. IRO_edit is read only through the textual fact because the dataset supplies a reason for every such edit (Section 3.3).
IRO_edit is ambiguous: a new image-based answer could come from a different recognized entity or from a changed fact about the same one. Because many entities could satisfy the first reading, the dataset attaches a generated reason, such as a transfer, and uses only the textual-fact reading.
Facts come from MQuAKE. An example is kept only if all three questions are answered correctly by all three filtering models, LLaVA-v1.5, InstructBLIP, and MiniGPT-v2; otherwise a failed consistency check could reflect a missing prerequisite rather than editing damage. Evaluation separates reliability on the requested change, consistency of linked answers, locality on unrelated knowledge, and generality to other images or paraphrases.
- 01
Decompose the knowledge
Represent image recognition, textual facts, and image-based factual questions as connected components rather than one undifferentiated answer.
- 02
Edit each component
Construct three editing scenarios and select facts the model can answer across all components before an edit.
- 03
Check the consequences
Evaluate consistency alongside edit success, generalization to new inputs, and preservation of unrelated knowledge.
View the paper’s figure
High reliability does not bring consistency
98.48 vs 9.09
IE_edit reliability vs consistency (%)
InstructBLIP edited with FT(LLM) or SERAC: the requested recognition edit almost always succeeds, but the image-based fact rarely follows. IKE reaches 49.05 consistency.
38.07 vs 9.09
IE_edit consistency: vision vs LLM fine-tuning (%)
On InstructBLIP, fine-tuning the vision module keeps recognition edits more consistent; for textual-fact edits the LLM module wins (MiniGPT-v2 SRO_edit: 91.24 vs 31.67).
Measured · Appendix E, Tables 9–11
Does the linked answer follow on the benchmark?
Choose a scenario, a model, and a method to compare measured reliability with consistency. Every method is plotted for the chosen model and scenario.
Measured on the full benchmark: InstructBLIP, IE_edit (Table 9)
| Method | Reliability | Consistency | Locality | Image gen. | Text gen. |
|---|---|---|---|---|---|
| FT(Vision) | 89.57 | 38.07 | 0.34 | 90.30 | 24.10 |
| FT(LLM) | 98.48 | 9.09 | 0.03 | 96.41 | 78.04 |
| MEND(Vision) | 32.39 | 18.37 | 93.15 | 23.43 | 29.73 |
| MEND(LLM) | 88.58 | 9.46 | 53.23 | 85.21 | 86.49 |
| IKE | 68.26 | 49.05 | not evaluated | not evaluated | 76.33 |
| SERAC | 98.48 | 9.09 | 87.65 | 96.41 | 68.41 |
| Method | Vision | LLM |
|---|---|---|
| FT | 38.07 | 9.09 |
| MEND | 18.37 | 9.46 |
— marks metrics the paper did not evaluate: IKE locality and image generality (no multi-image input), and image generality in SRO_edit (black-image input).
Selected method
FT(LLM) on IE_edit, InstructBLIP: reliability 98.48, consistency 9.09, locality 0.03. The edited answer appears on 98.48% of edit inputs, but the linked answer follows on only 9.09%.
Measured: Appendix E, Tables 9–11 (InstructBLIP with Vicuna-7B; MiniGPT-v2 with Llama-2-Chat-7B). Reliability is scored on the edit input, consistency on the linked probe, locality on five unrelated instances per edit, and generality on five paraphrases or five other images per edit (Section 4.2). These are benchmark averages, not model outputs for the Messi example in Method.
Editing experiments apply fine-tuning (FT), MEND (a hypernetwork that maps edits to weight changes), IKE (in-context editing), and SERAC (an edit memory with a classifier and a counterfactual model) to InstructBLIP and MiniGPT-v2, editing the vision or the language (LLM) component where applicable. High reliability does not imply consistency, especially for recognition edits.
In SRO_edit and IRO_edit the consistency target equals the edit target, so a method that outputs the new answer to every related prompt can look consistent while damaging unrelated knowledge. The authors therefore read consistency together with locality and trust it only when it holds across all three scenarios.
Scope. The study tests four editing methods on 7B multimodal checkpoints. High consistency can reflect overfitting, so it must be interpreted together with preservation of unrelated knowledge. Read the study ↗
Read the abstract
Multimodal large language models (MLLMs) are prone to non-factual or outdated knowledge issues, highlighting the importance of knowledge editing. Many benchmark has been proposed for researching multimodal knowledge editing. However, previous benchmarks focus on limited scenarios due to the lack of rigorous definition of multimodal knowledge. To better evaluate multimodal knowledge editing, we propose a decomposed definition of multimodal knowledge. Following the decomposed definition of multimodal knowledge, we introduce three scenarios and a novel requirement modality consistency. We construct MC-MKE, a fine-grained Multimodal Knowledge Editing benchmark emphasizing Modality Consistency through strict data selection. We evaluate four multimodal knowledge editing methods on MC-MKE, revealing their limitations, particularly in terms of modality consistency. Our work highlights the challenges posed by multimodal knowledge editing and motivates further research in developing effective techniques for this task.
Full paper ↗Citation BibTeX
@inproceedings{zhang-etal-2025-mc,
title = "{MC}-{MKE}: A Fine-Grained Multimodal Knowledge Editing Benchmark Emphasizing Modality Consistency",
author = "Zhang, Junzhe and Zhang, Huixuan and Yin, Xunjian and Huang, Baizhou and Zhang, Xu and Hu, Xinyu and Wan, Xiaojun",
booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
year = "2025",
url = "https://aclanthology.org/2025.findings-acl.896/",
pages = "17430--17445"
}