Recursive Self-Improvement Workshop, ICLR 2026Learning reward functions
Differentiable Evolutionary Reinforcement Learning
DERL learns how to combine simple reward signals: a meta-optimizer proposes rewards, trains a policy with them, and improves from that policy's validation performance.
- Result
- ALFWorld held-out-task (L2) success: 65.0% for DERL and 76.4% for DERL-pop vs 56.3% for RLVMR, all with a Qwen2.5-1.5B-Instruct policy.
- New vs prior
- Evolutionary and prompted-agent methods such as AlphaEvolve search agent or reward configurations as a black box from sparse fitness scores; this work trains the reward generator with policy gradients from validation performance.
- Limitation
- Each Meta-Optimizer update requires training several inner-loop policies, which the paper identifies as the framework’s primary computational bottleneck.
More reward signals can teach the wrong behavior
A final correct-or-incorrect reward is clear but sparse. Checks for formatting, intermediate progress or reasoning steps add feedback, but they also change what the policy is rewarded for, so the design problem is how to combine them.
In math training on GSM8K and MATH together, adding a format reward to the outcome reward raises GSM8K accuracy from 82.6% to 86.4% but lowers MATH from 58.8% to 55.9%. Averaging all primitives has the same trade-off (86.5% and 55.8%).
Closest prior work
The paper frames prior automatic agent evolution, from population-based training to prompted-agent systems such as AlphaEvolve and the Darwin Gödel Machine, as discrete black-box search that relies on sparse final fitness scores and ignores training dynamics. Meta-gradient RL (Xu et al., 2018) instead learns inside the inner optimization, which DERL avoids: it neither backpropagates through the inner-loop optimizer nor tunes continuous reward parameters. An LLM Meta-Optimizer proposes discrete symbolic reward compositions and is trained with RL on the validation performance of independently trained policies.
Generalization requires more than dense feedback
65.0%
ALFWorld · hardest split
Standard DERL reaches 65.0% task success on L2, compared with 56.3% for RLVMR using Qwen2.5-1.5B-Instruct.
43.0%
ScienceWorld · L1 split
Standard DERL improves success over RLVMR's 34.4%, supporting reward learning beyond the household-task benchmark in the same evaluation.
On ALFWorld, averaging the primitives helps on the familiar L0 split but barely changes the held-out-task L2 split, where DERL leads by a wide margin: denser feedback alone does not transfer. In math, DERL reaches 87.0% on GSM8K and 60.2% on MATH, avoiding the format reward’s trade-off.
“Differentiable” refers to the outer update, a policy-gradient estimate over generated reward programs; nothing is backpropagated through task correctness or through the inner training runs.
Table 1 (ALFWorld L1 column omitted). Task success rate (%), higher is better; inner policy Qwen2.5-1.5B-Instruct, where L0 tests seen variants, L1 unseen variants, and L2 task types held out from training. DERL-pop. initializes each inner loop from the best policy of the previous loop while keeping the total inner-loop training steps equal to DERL. GRPO baselines were run by the authors, while GiGPO and RLVMR results are taken from the RLVMR paper (Zhang et al., 2025). Source ↗
| Method | ALFWorld L0 | ALFWorld L2 | ScienceWorld L0 | ScienceWorld L1 | ScienceWorld L2 |
|---|---|---|---|---|---|
| GRPO w/ Outcome Reward | 76.6 | 29.7 | 21.1 | 13.7 | 10.9 |
| GRPO w/ Avg Reward | 88.1 | 30.5 | 37.9 | 31.3 | 18.0 |
| GiGPO | 86.7 | 48.0 | 25.8 | 15.2 | 4.7 |
| RLVMR | 89.1 | 56.3 | 46.9 | 34.4 | 26.5 |
| DERL | 91.0 | 65.0 | 47.7 | 43.0 | 30.1 |
| DERL-pop. | 91.8 | 76.4 | 98.2 | 95.3 | 31.3 |
Method proposed in the paper
Scope. Learned rewards can only combine the predefined atomic primitives; the Meta-Optimizer cannot add a check that the primitives do not provide. Read the study ↗
Research summary
The design of effective reward functions presents a central and often arduous challenge in reinforcement learning (RL), particularly when developing autonomous agents for complex reasoning tasks. While automated reward optimization approaches exist, they typically rely on derivative-free evolutionary heuristics that treat the reward function as a black box, failing to capture the causal relationship between reward structure and task performance. To bridge this gap, we propose Differentiable Evolutionary Reinforcement Learning (DERL), a bi-level framework that enables the autonomous discovery of optimal reward signals. In DERL, a Meta-Optimizer evolves a reward function (i.e., Meta-Reward) by composing structured atomic primitives, guiding the training of an inner-loop policy. Crucially, unlike previous evolution, DERL is differentiable in its meta-optimization: it treats the inner-loop validation performance as a signal to update the Meta-Optimizer via reinforcement learning. This allows DERL to approximate the "meta-gradient" of task success, progressively learning to generate denser and more actionable feedback. We validate DERL across three distinct domains: robotic agent (ALFWorld), scientific simulation (ScienceWorld), and mathematical reasoning (GSM8k, MATH). Experimental results show that DERL achieves state-of-the-art performance on ALFWorld and ScienceWorld, significantly outperforming methods relying on heuristic rewards, especially in out-of-distribution scenarios. Analysis of the evolutionary trajectory demonstrates that DERL successfully captures the intrinsic structure of tasks, enabling self-improving agent alignment without human intervention.
Full paper ↗Citation BibTeX
@misc{cheng2025differentiableevolutionaryreinforcementlearning,
title={Differentiable Evolutionary Reinforcement Learning},
author={Sitao Cheng and Tianle Li and Xuhan Huang and Xunjian Yin and Difan Zou},
year={2025},
eprint={2512.13399},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2512.13399}
}