Context supplied with the question
Myotis lucifralis shares a roost with Myotis nattereri.
The paper prints an abbreviated context. This is its relevant co-roosting fact, paraphrased.
KnowLM Workshop @ ACL 2025Internal knowledge and context
Understanding the Interplay between Parametric and Contextual Knowledge for Large Language Models
Providing relevant context can make a model less willing to use facts it already knows; EchoQA tests when internal knowledge and external information work together or interfere.
- Result
- Llama 3.1-70B answers Unknown 62.72% of the time with complementary context, vs 23.89% with no context and 0.08% with all facts supplied.
- New vs prior
- Knowledge-conflict studies (Xie et al., 2024; Wu et al., 2024) test one relationship at a time; EchoQA compares four kinds of context in one benchmark.
- Limitation
- The instructions that restore parametric knowledge must tell the model its context is insufficient, which the authors call impractical.
01 / The research question
Knowing both facts is not the same as combining them
Retrieval often supplies only the missing part of an answer, and the model must connect it with parametric knowledge, the facts stored in its weights. This differs from deciding whether to trust a passage that contradicts memory: both sources are correct, but neither suffices alone. EchoQA tests this complementary case alongside supportive, conflicting, and irrelevant context.
A correct answer does not prove composition either: without context, a model can infer an artificial entity's properties from its name and then eliminate answer options.
Closest prior work
Earlier studies examine one relationship at a time: context that conflicts with what the model knows (Wang et al., 2023b; Xie et al., 2024) or irrelevant context (Wu et al., 2024). ConflictQA (Xie et al., 2024) places the model's own elicited evidence and conflicting evidence together in the prompt and asks it to choose; EchoQA puts only the contextual knowledge in the prompt, so the model must draw on its parametric knowledge unprompted. It also adds the complementary case, built from new entity profiles rather than the entity-description substitutions of earlier multi-hop datasets, so that the context is new to the model.
03 / Reading the evidence
Complementary context raises abstention on known facts
62.72%
Answers marked unknown
Llama 3.1-70B's unknown-answer rate rises from 23.89% without context to 62.72% with complementary scientific context under neutral instructions.
23.88%
After instruction changes
Encouraging the same model to trust its own knowledge lowers that uncertainty, although improved prompting does not eliminate the integration problem.
The pattern is not specific to one model. With complementary scientific context and the neutral instruction, the unknown-answer rate rises above its no-context level for every tested model except OpenAI o1, and at least doubles for the Llama and Qwen models. Writing all required facts into the context lowers it for every model, so the difficulty lies in drawing on memory once context is present.
In 200 random Llama 3.1-70B failures, questions it answered alone but missed with complementary context, the model relied only on the context and answer choices in 93.5% of cases. The authors hypothesize that instruction tuning, which trains answers from either context or memory, contributes to this behavior.
Table 3. Unknown-answer rate U_R (%) for complementary reasoning on ALCUNA (scientific knowledge): the share of answers marked Unknown on questions whose prerequisite fact the model answered without context. w/o K = no context; NI = Neutral Instruction with the complementary context; TYI = Trust Yourself Instruction; Gold K = all required knowledge, including the elicited parametric fact, written into the context. Lower means fewer abstentions, not higher accuracy; each model is evaluated on its own filtered example set (Table 2). NI gives the highest rate for every model except OpenAI o1 (o1-preview). Source ↗
| Model | w/o K | NI | TYI | Gold K |
|---|---|---|---|---|
| OpenAI o1 | 65.78 | 46.12 | 35.43 | 20.63 |
| GPT-4o | 36.45 | 59.90 | 26.94 | 8.08 |
| GPT-4o-mini | 46.12 | 76.13 | 68.28 | 12.52 |
| Llama 3.1-70B | 23.89 | 62.72 | 23.88 | 0.08 |
| Llama 3.1-8B | 18.12 | 58.50 | 34.35 | 12.70 |
| Qwen 2-7B | 40.28 | 81.26 | 73.62 | 28.60 |
Scope. EchoQA filters knowledge separately for each model and uses constructed context relationships. Its results describe the tested tasks, checkpoints, and instructions rather than every retrieval system. Read the study ↗
Further reading in the paperEchoQA paper · complementary knowledge, prompting conditions, and unknown-ratio analysis ↗
Research summary
Large language models (LLMs) encode vast amounts of knowledge during pre-training (parametric knowledge, or PK) and can further be enhanced by incorporating contextual knowledge (CK). Can LLMs effectively integrate their internal PK with external CK to solve complex problems? In this paper, we investigate the dynamic interaction between PK and CK, categorizing their relationships into four types: Supportive, Complementary, Conflicting, and Irrelevant. To support this investigation, we introduce EchoQA, a benchmark spanning scientific, factual, and commonsense knowledge. Our results show that LLMs tend to suppress their PK when contextual information is available, even when it is complementary or irrelevant. While tailored instructions can encourage LLMs to rely more on their PK, they still struggle to fully leverage it. These findings reveal a key vulnerability in LLMs, raising concerns about their reliability in knowledge-intensive tasks.
Full paper ↗Citation BibTeX
@inproceedings{cheng2025understanding,
title={Understanding the Interplay between Parametric and Contextual Knowledge for Large Language Models},
author={Sitao Cheng and Liangming Pan and Xunjian Yin and Xinyi Wang and William Yang Wang},
booktitle={Knowledgeable Foundation Models at ACL 2025},
year={2025},
url={https://openreview.net/forum?id=4uisAcagzw}
}