ICLR 2025Memory for scientific reasoning
ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoning
ChemAgent turns solved chemistry subproblems into reusable memory, helping an LLM plan, calculate, and refine new solutions with experience retained outside its model weights.
- Result
- Average accuracy over four SciBench chemistry sets with GPT-4: 57.16% for ChemAgent vs 47.66% for StructChem.
- New vs prior
- StructChem structures chemical reasoning into formula generation, step-by-step reasoning and confidence-based review; this work adds a self-updating library of solved sub-tasks that is retrieved for each new sub-task.
- Limitation
- Evaluation and refinement nearly double the cost, from about 0.012 to 0.023 million tokens per problem, excluding library construction.
Reuse the part of a solution that transfers
Two chemistry questions may share a calculation even when their full solutions differ, and retrieving a whole worked example can bury that step under irrelevant conditions. ChemAgent therefore stores smaller units, each holding a sub-task’s conditions, the sub-task and its solution, so earlier work is available where a new calculation needs it.
The library is not fixed. Solutions from development-set problems seed it, and the agent adds new experiences as it solves problems. The question is whether this growing memory of sub-tasks helps on chemistry problems it has not seen.
Closest prior work
StructChem (Ouyang et al., 2024), the main baseline, structures chemical reasoning into formula generation, step-by-step reasoning and confidence-based review, but the paper notes that it still struggles with highly complex problems. Agent self-evolution frameworks with memory, such as Agent Workflow Memory (Wang et al., 2024) and Investigate-Consolidate-Exploit (Qian et al., 2024), mainly reuse past workflows in daily tasks. ChemAgent instead decomposes solved chemistry problems into planning and execution memories, retrieves them for each sub-task, and adds newly solved sub-tasks to the library.
Memory and review both contribute, at roughly twice the token cost
57.16%
Average accuracy · GPT-4
Across four SciBench chemistry datasets, ChemAgent exceeds the 47.66% average reported for StructChem under the paper's setup.
52.12%
Without evaluation and refinement
Removing solution review lowers average accuracy from 57.16%, showing that memory and active correction contribute together.
Both components contribute: the GPT-4 average drops without evaluation and refinement, and drops further without memory, so stored examples alone do not explain the gain. Review roughly doubles the token cost per problem, not counting the one-time library construction.
The self-updating experiment on MATTER removes review, keeps only experiences judged correct and excludes a problem’s own earlier memory, so memory quality is part of the result. Figure 7 shows retrieval failing: for a gas compressed reversibly and adiabatically, where entropy does not change, the plan ignores that condition and applies ΔS = n·C_V,m·ln(T_final/T_initial), while the retrieved memories, on reaction entropy and cell potentials, do not apply. Flawed decomposition and irrelevant retrieval compound each other.
Table 1 (GPT-4 rows). Accuracy (%) on the four SciBench chemistry test sets (an answer counts as correct within 1% relative tolerance), higher is better; all rows use gpt-4-1106-preview, and Avg. is the unweighted mean of the four datasets. Few-shot + Direct reasoning and Few-shot + Python scores are taken from StructChem and SciBench, respectively, rather than rerun in this paper. The table's Llama 3.1-7b, Llama 3.1-70b, and Qwen 2.5-72b blocks are omitted; with those backbones ChemAgent raises the average over direct reasoning but trails it on ATKINS for Llama 3.1-7b (19.63 vs. 22.43) and Qwen 2.5-72b (56.07 vs. 57.01). Source ↗
| Method | CHEMMC | MATTER | ATKINS | QUAN | Avg. |
|---|---|---|---|---|---|
| Few-shot + Direct reasoning | 28.21 | 14.29 | 20.69 | 14.71 | 19.48 |
| Few-shot + Python | 38.46 | 34.69 | 57.01 | 44.12 | 43.57 |
| StructChem | 58.97 | 30.67 | 59.81 | 41.18 | 47.66 |
| ChemAgent | 74.36 | 48.98 | 61.18 | 44.12 | 57.16 |
| w/o Evaluation & Refinement | 61.54 | 44.89 | 57.94 | 44.12 | 52.12 |
| w/o Memory, Evaluation & Refinement | 58.97 | 38.78 | 57.94 | 41.18 | 49.22 |
Method proposed in the paper
Scope. Evaluation uses four SciBench textbook chemistry sets; the paper suggests, but does not test, applications such as drug discovery or materials design. Read the study ↗
Research summary
Chemical reasoning usually involves complex, multi-step processes that demand precise calculations, where even minor errors can lead to cascading failures. Furthermore, large language models (LLMs) encounter difficulties handling domain-specific formulas, executing reasoning steps accurately, and integrating code effectively when tackling chemical reasoning tasks. To address these challenges, we present ChemAgent, a novel framework designed to improve the performance of LLMs through a dynamic, self-updating library. This library is developed by decomposing chemical tasks into sub-tasks and compiling these sub-tasks into a structured collection that can be referenced for future queries. Then, when presented with a new problem, ChemAgent retrieves and refines pertinent information from the library, which we call memory, facilitating effective task decomposition and the generation of solutions. Our method designs three types of memory and a library-enhanced reasoning component, enabling LLMs to improve over time through experience. Experimental results on four chemical reasoning datasets from SciBench demonstrate that ChemAgent achieves performance gains of up to 46% (GPT-4), significantly outperforming existing methods. Our findings suggest substantial potential for future applications, including tasks such as drug discovery and materials science.
Full paper ↗Citation BibTeX
@inproceedings{tang2025chemagent,
title={ChemAgent: Self-updating Library in Large Language Models Improves Chemical Reasoning},
author={Tang, Xiangru and Hu, Tianyu and Ye, Muyang and Shao, Yanjun and Yin, Xunjian and Ouyang, Siru and Zhou, Wangchunshu and Lu, Pan and Zhang, Zhuosheng and Zhao, Yilun and Cohan, Arman and Gerstein, Mark},
booktitle={International Conference on Learning Representations},
year={2025},
url={https://openreview.net/forum?id=kuhIqeVg0e}
}