AAAI 2024Temporal knowledge editing

History Matters: Temporal Knowledge Editing in Large Language Model

Xunjian Yin1, Jin Jiang1, Liming Yang2, Xiaojun Wan1

1 Peking University · 2 Tsinghua University

Updating a model with today's facts can erase yesterday's answers; METO edits current and historical knowledge together so that facts remain associated with their time periods.

Result
On AToKe single edits with GPT-J, METO raises MEMIT's historical explicit-time accuracy (HES) from 2.22% to 30.31%, while current accuracy (CES) falls from 99.66% to 86.4%.
New vs prior
ROME, MEMIT, and MEND optimize only the new fact, and evaluations such as MQuAKE check related facts; this work also requires the replaced fact to survive as dated history.
Limitation
Historical recall stays incomplete: after multiple edits the best final historical score (HES*) is 21.93%, and METO lowers MEMIT's current explicit-time accuracy by 13.26 points.
Scroll sideways to read the figure. METO: the model is first queried for the fact it currently holds (its model-time knowledge), and then both that historical fact and the new facts become editing targets, optimized together with a time objective that predicts when each fact holds, using any baseline editing method. Original Figure 3 from the paper. Full size ↗

An update should not erase the past

Correcting a false fact and updating a once-true fact are different operations: when a leader changes, the old and new answers are both correct for different dates. The paper calls the second case Temporal Knowledge Editing (TKE). An editor can pass tests on the new answer, its paraphrases, and unrelated facts while erasing the earlier one, because locality tests do not check the edited fact's history.

The new benchmark, AToKe (Assessment of TempOral Knowledge Editing), asks explicit-time questions, such as who held a position in a given span, and relative-time questions, such as who held it previously. The latter require an ordered history rather than isolated date–answer pairs.

Closest prior work

Existing editors, from constrained fine-tuning and MEND to ROME and MEMIT, are built to make the model memorize the new fact, and their evaluations check edit success, paraphrases, and unrelated neighborhood facts. MQuAKE (Zhong et al., 2023) adds multi-hop questions about facts related to an edit, and Cohen et al. (2023) measure ripple effects. The paper argues that none of these checks whether the pre-edit fact is preserved as history, which is the requirement Temporal Knowledge Editing adds.

METO edits the outgoing fact and its time span

METO (Multi-Editing with Time Objective) first queries the model for the fact it currently holds, then edits that fact together with the newer ones, closing its time span and moving its wording to the past. Older facts are not replayed; the authors assume the model already treats them as history.

Released AToKe record · METO targets built from the Methodology

Two timestamped edits, one question at a time

Play the record's two edits in order. Each question form's probe and expected answer follow from the record, and the request panel shows what an existing editor and METO are asked to learn at that edit.

Edit 2 applied: Arsenal F.C., 1948–1953. Southampton F.C. is now historical. Same wording, new expected answer: CRS, HRS. New time span in the prompt: CES, CES-P, HES. Asked for the first time: HES*. The multiple-edit (AToKe-ME) case ends here.

Released record · AToKe-ME record, case_id 3 · (Billy Wrigglesworth, playsFor)

  1. 1937–1947Manchester United F.C.Historical · span closed in 1947
  2. 1947–1948Southampton F.C.Historical · span closed in 1948
  3. 1948–1953Arsenal F.C.Current · written by edit 2

What the editor is asked to learn at edit 2

Existing editor: one timestamped prompt
InputTarget
From 1948 to 1953, Billy Wrigglesworth is a player ofArsenal F.C.
Same editor with METO: every fact since the model-time fact, and its span
Fact objective inputTargetTime objective inputTarget
(Billy Wrigglesworth, playsFor, ?, 1937, 1947)Manchester United F.C.(Billy Wrigglesworth, playsFor, Manchester United F.C., ?, ?)1937 to 1947
(Billy Wrigglesworth, playsFor, ?, 1947, 1948)Southampton F.C.(Billy Wrigglesworth, playsFor, Southampton F.C., ?, ?)1947 to 1948
(Billy Wrigglesworth, playsFor, ?, 1948, 1953)Arsenal F.C.(Billy Wrigglesworth, playsFor, Arsenal F.C., ?, ?)1948 to 1953

METO's targets follow its Methodology: Ct = Cm ∪ Cm+, where Cm is the fact GPT-J held before editing (found by querying the model) and Cm+ the newer facts up to this edit; facts older than Cm are not replayed. They are written in the notation of Figure 3; the paper prints no per-record targets. Its own time-objective example edits “Donald Trump is the President of the United States from” toward “2017 to 2021”.

Probe and expected answer for each question form

Question formEdit 1 appliedAToKe-SE ends hereEdit 2 appliedAToKe-ME ends here
CESCurrent fact, explicit time
  • From 1947 to 1948, Billy Wrigglesworth is a player ofSouthampton F.C.
Not asked yet
  • From 1948 to 1953, Billy Wrigglesworth is a player ofArsenal F.C.
New time span
Not asked yet
CES-PCurrent fact, explicit time, paraphrased
  • From 1947 to 1948, Billy Wrigglesworth plays forSouthampton F.C.
Not asked yet
  • From 1948 to 1953, Billy Wrigglesworth plays forArsenal F.C.
New time span
Not asked yet
CRSCurrent fact, relative time
  • Billy Wrigglesworth's team isSouthampton F.C.
Not asked yet
  • Billy Wrigglesworth's team isArsenal F.C.
Same wording, new answer
Not asked yet
HESHistorical fact, explicit time
  • From 1937 to 1947, Billy Wrigglesworth's team wasManchester United F.C.
Not asked yet
  • From 1947 to 1948, Billy Wrigglesworth's team wasSouthampton F.C.
New time span
Not asked yet
HRSHistorical fact, relative time
  • Billy Wrigglesworth used to play forManchester United F.C.
Not asked yet
  • Billy Wrigglesworth used to play forSouthampton F.C.
Same wording, new answer
Not asked yet
HES*Every historical fact, explicit timeNot asked: HES* follows the final edit of a multiple-edit chain
  • From 1937 to 1947, Billy Wrigglesworth's team wasManchester United F.C.
  • From 1947 to 1948, Billy Wrigglesworth's team wasSouthampton F.C.
First asked
Not asked yet

Record: AToKe-ME record, case_id 3, from the authors' released dataset documentation (linked in the paper). Probes and expected answers are the record's own questions; every answer is the record's object for the probed span, so the changes are definitional, not model outputs. AToKe samples its chains from YAGO and ends each chain with one sampled counterfactual future fact, so the latest link need not be biographical.

A second objective reverses the usual query: besides supplying the object for a dated relation, the model is edited to predict the time span of a given fact. METO runs both objectives through an existing editor, constrained fine-tuning (CFT), MEND, ROME, or MEMIT, so each comparison holds the editor fixed.

AToKe has three subsets: a single edit (SE), multiple successive edits of the same fact (ME), and an extending edit (EE) that prolongs a fact's validity. Scores cover current and historical facts under explicit-time questions (CES, HES), relative-time questions (CRS, HRS), and paraphrased current questions (CES-P); HES* asks every historical question after the last edit. Baseline prompts carry timestamps too, so their failures are not due to a missing date.

  1. 01

    Evaluate a timeline

    AToKe tests single edits, repeated updates, and extensions of a fact's validity, using questions about current and historical knowledge.

  2. 02

    Edit both time periods

    METO constructs editing targets for previous facts alongside the new fact, including timestamps in the prompts.

  3. 03

    Learn when facts hold

    A time prediction objective helps the edited model associate each fact with the period in which it is valid.

METO recovers some history at a cost to current accuracy

Measured · Tables 2–3, GPT-J 6B

Which question forms does METO recover?

Choose an edit sequence, a question form, and an editor to compare measured accuracy without METO (Table 2) and with it (Table 3). All four editors stay visible for the chosen form.

Measured accuracy on HES* (Every historical fact, explicit time), GPT-J, multiple edits

Without METO (Table 2)With METO (Table 3)

  1. CFT0.01 → 0.73+0.71
  2. MEND0.25 → 21.83+21.58
  3. ROME0.26 → 15.92+15.66
  4. MEMIT0.27 → 21.93+21.66

Selected cell

MEMIT · HES* · multiple edits: 0.27% without METO → 21.93% with METO (+21.66 points). Without METO, historical probes almost never succeed; with METO, 21.93% do. Over the same edits, CES moves from 98.42% to 92.73%.

Change with METO across every question form

Percentage points as printed in Table 3, multiple edits. The selected form and editor are outlined.

Question formCFTMENDROMEMEMIT
CESCurrent fact, explicit time+0.16−1.31+1.08−5.69
CES-PCurrent fact, explicit time, paraphrased+0.02+0.45−0.57−5.31
CRSCurrent fact, relative time−0.18−1.83+5.32+7.10
HESHistorical fact, explicit time+1.61+28.25+22.78+35.72
HRSHistorical fact, relative time+1.23+28.73+17.01+25.18
HES*Every historical fact, explicit time+0.71+21.58+15.66+21.66

Measured: Table 2 (existing editors) and Table 3 (with METO), GPT-J 6B on AToKe; changes are those printed in Table 3. The multiple-edit scores average each edit of a chain; HES* is asked once after the final edit. The probes behind each form are shown for one record in Method.

Without METO, every editor nearly erases history: single-edit HES is at most 2.41%, and HES* after multiple edits at most 0.27%. With METO, HES* exceeds 15% for every editor except CFT. Retention is not free: under single edits, METO lowers paraphrased (CES-P) and relative-time (CRS) current accuracy for all four editors.

Two difficulty patterns hold with and without METO: relative-time questions are harder than explicit-time ones, likely because they require ordering facts rather than recalling dates, and repeated edits are harder than a single one.

Scope. The main experiments edit GPT-J on timestamped fact chains. Historical recall improves substantially but remains incomplete, and improvements need to be assessed alongside accuracy on current facts. Read the study ↗

Further reading in the paperPaper · temporal benchmark, METO, and Table 3 ↗
Read the abstract

The imperative task of revising or updating the knowledge stored within large language models arises from two distinct sources: intrinsic errors inherent in the model which should be corrected and outdated knowledge due to external shifts in the real world which should be updated. Prevailing efforts in model editing conflate these two distinct categories of edits arising from distinct reasons and directly modify the original knowledge in models into new knowledge. However, we argue that preserving the model's original knowledge remains pertinent. Specifically, if a model's knowledge becomes outdated due to evolving worldly dynamics, it should retain recollection of the historical knowledge while integrating the newfound knowledge. In this work, we introduce the task of Temporal Knowledge Editing (TKE) and establish a benchmark AToKe (Assessment of TempOral Knowledge Editing) to evaluate current model editing methods. We find that while existing model editing methods are effective at making models remember new knowledge, the edited model catastrophically forgets historical knowledge. To address this gap, we propose a simple and general framework termed Multi-Editing with Time Objective (METO) for enhancing existing editing models, which edits both historical and new knowledge concurrently and optimizes the model's prediction for the time of each fact. Our assessments demonstrate that while AToKe is still difficult, METO maintains the effectiveness of learning new knowledge and meanwhile substantially improves the performance of edited models on utilizing historical knowledge.

Full paper ↗
Citation BibTeX
@inproceedings{yin2024history,
  title={History matters: Temporal knowledge editing in large language model},
  author={Yin, Xunjian and Jiang, Jin and Yang, Liming and Wan, Xiaojun},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={38},
  number={17},
  pages={19413--19421},
  year={2024}
}