ACL 2024 · Main ConferenceEvaluating accessible knowledge
Benchmarking Knowledge Boundary for Large Language Models: A Different Perspective on Model Evaluation
A wrong answer may reflect the prompt rather than missing knowledge; searching for alternative wording reveals facts a model can answer but a fixed benchmark misses.
- Result
- On PaRaRel with LLaMA2, PGDC elicits 71.36% of facts vs 66.95% for the strongest baseline (P-few); on counterfactual CFACT it elicits 3.41% vs AutoPrompt's 88.35%.
- New vs prior
- LAMA queries facts with fixed cloze prompts, ParaRel and KAssess with a few paraphrases, and AutoPrompt optimizes trigger tokens toward a target; this work optimizes prompts under a semantic constraint.
- Limitation
- PGDC needs the known answer and white-box gradients, and human raters judge only 83.3% of LLaMA2's optimized prompts to preserve the original meaning.
A failed question does not prove ignorance
A factual question can fail because the model lacks the fact or because that wording does not elicit it. Conversely, a prompt can produce the target string without asking the original question. The paper defines a model's knowledge boundary as the facts it answers correctly under at least one expression; estimating it needs a probe that searches wordings while preserving what was asked.
If “Canberra is located in which country?” fails but “Which country contains Canberra?” succeeds, a single template underestimates the model. An instruction to print the target country, however, would show no knowledge at all. The search must go beyond a fixed template without redefining success.
Closest prior work
LAMA (Petroni et al., 2019) probes each fact with a fixed cloze prompt, and ParaRel (Elazar et al., 2021) and KAssess (Dong et al., 2023) measure consistency across a limited set of paraphrases; the paper argues that all such evaluations still rest on fixed prompts. AutoPrompt (Shin et al., 2020) optimizes discrete trigger tokens to produce a target output, which the paper finds behaves more like an adversarial attack because it also elicits counterfactual targets. PGDC instead searches for the best wording within the semantic neighborhood of the original question.
PGDC finds more true facts without forcing false ones
71.36% vs 66.95%
PaRaRel success, LLaMA2
PGDC vs the strongest baseline, P-few (few-shot prompting, counted as success if any listed paraphrase works). Zero-shot prompting with one wording reaches 34.43%.
3.41% vs 88.35%
CFACT false-target success, LLaMA2 (lower is better)
On counterfactual facts, AutoPrompt forces the false answer far more often; the paper attributes the difference to PGDC's semantic constraint.
Measured · Table 1, all four backbones · Table 3 · inputs from Tables 2 and 4
Which probes find true facts without forcing false ones?
Pick a true-fact set and a negative control. Each panel shows one backbone: how often a probe elicits the target answer of a true fact (higher is better) and of a counterfactual or fabricated one (lower is better). Select a probe to see the exact input the paper sends and to compare it across backbones.
GPT-2 (774M)PGDC prompts judged meaning-preserving: 80.5%
| Probe | PaRaRel ↑ | CFACT ↓ |
|---|---|---|
| zero | 9.27% | 1.10% |
| few | 12.61% | 1.77% |
| dis | 3.13% | 3.28% |
| P-zero | 18.18% | 3.00% |
| P-few | 20.46% | 3.92% |
| P-dis | 9.01% | 9.32% |
| PGDC | 47.68% | 2.81% |
| AutoPrompt | not reported | 92.38% |
GPT-J (6B)PGDC prompts judged meaning-preserving: 85.1%
| Probe | PaRaRel ↑ | CFACT ↓ |
|---|---|---|
| zero | 23.23% | 1.39% |
| few | 44.84% | 3.18% |
| dis | 2.40% | 2.30% |
| P-zero | 40.78% | 3.75% |
| P-few | 54.06% | 6.12% |
| P-dis | 7.25% | 6.74% |
| PGDC | 55.95% | 4.82% |
| AutoPrompt | not reported | 85.67% |
LLaMA2 (7B)PGDC prompts judged meaning-preserving: 83.3%
| Probe | PaRaRel ↑ | CFACT ↓ |
|---|---|---|
| zero | 34.43% | 1.32% |
| few | 58.23% | 4.56% |
| dis | 17.96% | 13.88% |
| P-zero | 54.78% | 3.30% |
| P-few | 66.95% | 9.31% |
| P-dis | 44.16% | 36.46% |
| PGDC | 71.36% | 3.41% |
| AutoPrompt | not reported | 88.35% |
Vicuna (7B)PGDC prompts judged meaning-preserving: 86.2%
| Probe | PaRaRel ↑ | CFACT ↓ |
|---|---|---|
| zero | 34.19% | 1.40% |
| few | 59.56% | 3.08% |
| dis | 8.40% | 4.95% |
| P-zero | 54.97% | 3.36% |
| P-few | 69.69% | 6.91% |
| P-dis | 23.06% | 14.28% |
| PGDC | 69.63% | 3.50% |
| AutoPrompt | not reported | 33.09% |
True-fact successFalse-target successBars run from 0 to 100%.
Selected probe
PGDC · PaRaRel ↑ / CFACT ↓: GPT-2 47.68% / 2.81%, GPT-J 55.95% / 4.82%, LLaMA2 71.36% / 3.41%, Vicuna 69.63% / 3.50%. Across the four backbones, false-target success equals 5–9% of true-fact success. Highest PaRaRel success of all seven probes for GPT-2, GPT-J, LLaMA2. Backbone order on PaRaRel: LLaMA2 > Vicuna > GPT-J > GPT-2.
What the probe sends: PGDC Projected gradient descent with semantic constraints
PGDC starts from the original cloze prompt and edits its embeddings; the paper's Table 2 prints three prompts it changed successfully (dataset and backbone not stated). The target answer may appear after generated tokens, not only first.
| Original prompt | PGDC prompt | Answer | Table 2 category |
|---|---|---|---|
| The associated item of source code is | The early item of source code is a simple | program | Optimal Paraphrase |
| Isatis tinctoria is a source of | Isatis tinctoria, a source of the natural dye | indigo | Reasoning and Inference |
| The host country of Australian Capital Territory is | <s> host country of Australian Capital Territory is | Australia | Format and Stop Words |
Measured: Table 1 (success rate of eliciting the target answer, four backbones), Table 3 (AutoPrompt on CFACT), and Section 4.5 (human judgment of 200 PaRaRel samples per backbone). Inputs: Table 4 (Appendix A.2) and Table 2. PaRaRel and KAssess hold true facts; CFACT and ALCUNA hold counterfactual or fabricated ones, so a sound probe should rarely succeed on them.
Baselines follow common evaluation practice: zero-shot (zero) and few-shot (few) prompting, and a discriminator format (dis) that asks whether a statement is true; the P- prefix counts a fact as known if any listed paraphrase succeeds. PGDC is highest on both true-fact datasets, PaRaRel and KAssess, for every backbone except Vicuna on PaRaRel. On the fabricated facts of CFACT (CounterFact) and ALCUNA, a sound probe should rarely elicit the false target; the discriminator baselines fail this test, which the paper links to models' preference for answering “true”.
Semantic preservation is imperfect: three annotators judging 200 PaRaRel samples find that 83.3% of LLaMA2's optimized prompts keep the original meaning, from 80.5% for GPT-2 to 86.2% for Vicuna. A representation-level constraint therefore cannot certify every rewrite.
Scope. This estimates accessible knowledge under a particular search procedure. It needs target answers and model gradients, and semantic constraints do not guarantee meaning preservation or exhaustive coverage. Read the study ↗
Read the abstract
In recent years, substantial advancements have been made in the development of large language models, achieving remarkable performance across diverse tasks. To evaluate the knowledge ability of language models, previous studies have proposed lots of benchmarks based on question-answering pairs. We argue that it is not reliable and comprehensive to evaluate language models with a fixed question or limited paraphrases as the query, since language models are sensitive to prompt. Therefore, we introduce a novel concept named knowledge boundary to encompass both prompt-agnostic and prompt-sensitive knowledge within language models. Knowledge boundary avoids prompt sensitivity in language model evaluations, rendering them more dependable and robust. To explore the knowledge boundary for a given model, we propose a projected gradient descent method with semantic constraints, a new algorithm designed to identify the optimal prompt for each piece of knowledge. Experiments demonstrate a superior performance of our algorithm in computing the knowledge boundary compared to existing methods. Furthermore, we evaluate the ability of multiple language models in several domains with knowledge boundary.
Full paper ↗Citation BibTeX
@inproceedings{yin-etal-2024-benchmarking,
title = "Benchmarking Knowledge Boundary for Large Language Models: A Different Perspective on Model Evaluation",
author = "Yin, Xunjian and Zhang, Xu and Ruan, Jie and Wan, Xiaojun",
booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics",
year = "2024",
url = "https://aclanthology.org/2024.acl-long.124/",
pages = "2270--2286"
}