Xunjian Yin / Research

NAACL 2025 FindingsDocument design for RAG

Evaluating Self-Generated Documents for Enhancing Retrieval-Augmented Generation with Large Language Models

Jiatao Li, Xinyu Hu, Xunjian Yin, Xiaojun Wan

Peking University

Paper arXiv Code

Documents generated by a model can help retrieval-augmented answers, but their usefulness depends on document style, model capacity, and how they are combined with retrieved evidence.

Result
CCU documents plus style-transformed Wikipedia average 53.6 (EM/F1, Qwen2.5-32B), vs 52.1 for GenRead Direct Mix and 45.6 for Wikipedia alone.
New vs prior
GenRead (Yu et al., 2023) generates background documents instead of retrieving them; this work asks which document types and mixing strategies help.
Limitation
Human evaluation checks only style adherence; the authors note that the factual accuracy of self-generated documents cannot be guaranteed.
The three research questions on one example question: RQ1 compares a direct answer with answering from self-generated documents, RQ2 shows four of the eight document types from the tone/granularity/structure taxonomy, and RQ3 contrasts Direct Mix with Style-Transformation Mix of Wikipedia and self-generated documents. Original Figure 1 from the paper.
Scroll sideways to read the figure. The three research questions on one example question: RQ1 compares a direct answer with answering from self-generated documents, RQ2 shows four of the eight document types from the tone/granularity/structure taxonomy, and RQ3 contrasts Direct Mix with Style-Transformation Mix of Wikipedia and self-generated documents. Original Figure 1 from the paper. Full size ↗
01 / The research question

Generated context is a representation choice

Self-generated documents (Self-Docs) are passages a model writes from its own parametric memory before answering. They draw on no external source; they externalize what the generator can already produce, which may make useful facts easier to use. If the source is generated rather than retrieved, which properties of that text decide whether it helps?

The paper treats the document type as an experimental variable. A detailed formal report and a conversational overview can share facts yet suit different tasks: fact verification needs specific evidence, while a long-form explanation needs coverage and organization.

Closest prior work

Generation-augmented methods such as GenRead (Yu et al., 2023) and recitation-augmented models (Sun et al., 2023) have a model write background passages before answering, and hybrid approaches (Zhang et al., 2023; Abdallah and Jatowt, 2024) merge generated with retrieved documents. The paper notes that this work has mostly focused on how to use such documents, for example how to handle conflicting knowledge, while overlooking how different types of generated documents affect performance. It therefore classifies Self-Docs with a Systemic Functional Linguistics taxonomy and reuses GenRead's pipeline as its generator.

02 / Inside the method

Generator size, document type, and mixing are tested separately

The experiments first separate three factors that are otherwise confounded: the size of the base model, which both writes and answers; the number of documents; and the size of the generator alone, with the answerer fixed at Qwen2.5-32B-Instruct. Documents come from GenRead's few-shot pipeline, whose examples are drawn from k-means clusters of query–document pairs.

The taxonomy, grounded in Systemic Functional Linguistics, crosses three binary choices: tone (Authoritative or Conversational), granularity (Fine- or Coarse-grained), and structure (Structured or Unstructured). Types are named by these initials: AFU is authoritative, fine-grained, and unstructured (the paper's example is a case study), CFU a personal essay, and CCU a social media post.

Finally, Direct Mix concatenates retrieved Wikipedia passages with Self-Docs, while Style-Transformation Mix first has an LLM rewrite the passages in the Self-Docs' style. Because the rewrite is itself a model step, a gain belongs to the whole pipeline, not to the retrieved content alone.

  1. 01

    Vary capacity and quantity

    Compare document generators and answering models at different sizes, then vary how many generated documents enter the context.

  2. 02

    Separate document properties

    Use a linguistics-based taxonomy of tone, detail, and structure to form eight document types for controlled comparisons.

  3. 03

    Combine information sources

    Compare direct mixing with Wikipedia passages against transforming their style before combining them with generated documents.

03 / Reading the evidence

Style-aligned mixing helps, except on long-form answers

Measured · Tables 2, 10 and 12 · examples from Figure 1 and Appendix G.2

Which document type helps which task?

Choose a tone, a granularity, and a structure to select one of the eight Self-Doc types, then choose whether the documents are used alone or mixed with Wikipedia passages. All eight types stay in the table; the second table changes one choice at a time.

Selected type

AFU Table 4 example: Case Studies

  • Authoritative (Interpersonal metafunction): a formal style that establishes authority, as in a research paper.
  • Fine-grained (Ideational metafunction): detailed information, as in a technical paper.
  • Unstructured (Textual metafunction): continuous prose with a narrative flow, as in an essay.

Question (TQA): The VS-300 was a type of what?

The VS-300 is a helicopter, specifically a prototype that marked a pivotal moment in the evolution of rotorcraft technology during the early 1940s. …
Opening of the published AFU document for this question, Figure 1 and Appendix G.2.

Scores of all eight types: Self-Docs alone (Table 2)

TypeTQAEMHotpotQAF1FEVEREMELI5F1Averagemean
AFSResearch Paper55.437.583.420.949.3
AFUCase Studies58.439.985.222.851.6
ACSNews Article56.838.783.622.050.3
ACUOpinion Pieces56.439.685.023.551.1
CFSHow-to Guides57.436.481.824.049.9
CFUPersonal Essays57.839.584.225.151.6
CCSTravel Blogs56.035.282.025.449.6
CCUSocial Media Posts56.638.783.826.451.4
WikiWikipedia passages alone52.636.366.826.745.6

EM for TQA and FEVER, F1 for HotpotQA and ELI5; Qwen2.5-32B-Instruct answers the first 500 examples of each dataset. The best Self-Doc score in each column is bold. In the mixed conditions, small figures are the printed changes from the same type alone (Table 2).

Change one choice of AFU

AdvantageTQAHotpotQAFEVERELI5Average
Authoritative over conversationalAFU vs CFU+0.62/4 pairs+0.44/4 pairs+1.04/4 pairs−2.30/4 pairs0.01/4 pairs
Fine-grained over coarse-grainedAFU vs ACU+2.03/4 pairs+0.33/4 pairs+0.22/4 pairs−0.70/4 pairs+0.53/4 pairs
Unstructured over structuredAFU vs AFS+3.03/4 pairs+2.44/4 pairs+1.84/4 pairs+1.94/4 pairs+2.34/4 pairs

Each cell is the selected type's score minus the type that differs in one choice. “n/4 pairs” counts, over all four type pairs that differ only in that choice, how often the selected choice scores higher, so a pattern can be checked beyond this one pair.

Selected type

AFU (authoritative, fine-grained, unstructured), Self-Docs alone (Table 2): TQA 58.4 (rank 1 of 8); HotpotQA 39.9 (rank 1 of 8); FEVER 85.2 (rank 1 of 8); ELI5 22.8 (rank 6 of 8); Average 51.6 (rank 1 of 8, tied with CFU). Wikipedia passages alone score higher on ELI5.

Measured: Table 2 (Self-Docs alone), Table 10 (with Wikipedia, Direct Mix), and Table 12 (with Wikipedia, Style-Transformation Mix), NAACL 2025 Findings; the page's results table repeats three rows of each mix (Table 5). Types and examples: Section 5.1 and Table 4; document openings from Figure 1 and Appendix G.2, shortened. Comparisons are computed here from the per-type rows.

Gains are task-specific. Relative to the same documents alone, Style-Transformation Mix gains most on multi-hop HotpotQA and fact-verification FEVER and leaves long-form ELI5 essentially unchanged. Among generated types, unstructured documents beat structured ones on average, authoritative fine-grained ones suit FEVER, and conversational ones suit ELI5.

Scale effects are also uneven. A larger generator helps only above a size threshold; smaller generators struggle to beat either answering without documents or Wikipedia retrieval. More documents help FEVER, while ELI5 may plateau or dip.

Scope. Main experiments use Qwen2.5-32B-Instruct and 500 examples per task across four datasets. Generated documents can contain factual errors, and effective styles vary with task and model. Read the study ↗

Further reading in the paperNAACL 2025 Findings paper · Sections 2–6; Tables 2 and 5 ↗
Read the abstract

The integration of documents generated by LLMs themselves (Self-Docs) alongside retrieved documents has emerged as a promising strategy for retrieval-augmented generation systems. However, previous research primarily focuses on optimizing the use of Self-Docs, with their inherent properties remaining underexplored. To bridge this gap, we first investigate the overall effectiveness of Self-Docs, identifying key factors that shape their contribution to RAG performance (RQ1). Building on these insights, we develop a taxonomy grounded in Systemic Functional Linguistics to compare the influence of various Self-Docs categories (RQ2) and explore strategies for combining them with external sources (RQ3). Our findings reveal which types of Self-Docs are most beneficial and offer practical guidelines for leveraging them to achieve significant improvements in knowledge-intensive question answering tasks.

Full paper ↗
Citation BibTeX 
@inproceedings{li-etal-2025-evaluating,
    title = {Evaluating Self-Generated Documents for Enhancing Retrieval-Augmented Generation with Large Language Models},
    author = {Li, Jiatao and Hu, Xinyu and Yin, Xunjian and Wan, Xiaojun},
    booktitle = {Findings of the Association for Computational Linguistics: NAACL 2025},
    year = {2025},
    url = {https://aclanthology.org/2025.findings-naacl.149/},
    doi = {10.18653/v1/2025.findings-naacl.149},
    pages = {2741--2775}
}