Title: RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning

URL Source: https://arxiv.org/html/2504.20073

Markdown Content:
\reportnumber

001 \correspondingauthor Equal Contribution.

Kangrui Wang*1 Qineng Wang*1 Pingyue Zhang*1 Linjie Li*2 Zhengyuan Yang 4 Xing Jin 6 Kefan Yu 1 Minh Nhat Nguyen 7 Licheng Liu 1 Eli Gottlieb 1 Yiping Lu 1 Kyunghyun Cho 5 Jiajun Wu 3 Li Fei-Fei 3 Lijuan Wang 4 Yejin Choi 3 Manling Li 1

1 Northwestern University 2 University of Washington 3 Stanford University 4 Microsoft

5 New York University 6 University of British Columbia 7 Singapore Management University

###### Abstract

Training large language models (LLMs) as interactive agents presents unique challenges including long-horizon decision making and interacting with stochastic environment feedback. While reinforcement learning (RL) has enabled progress in static tasks, multi-turn agent RL training remains underexplored. We propose StarPO (S tate-T hinking-A ctions-R eward P olicy O ptimization), a general framework for trajectory-level agent RL, and introduce RAGEN, a modular system for training and evaluating LLM agents. Our study on four stylized environments reveals three core findings. First, our agent RL training shows a recurring mode of Echo Trap where reward variability cliffs and gradient spikes; we address this with StarPO-S, a stabilized variant with trajectory filtering, critic incorporation, and gradient stabilization. Second, we find the shaping of RL rollouts would benefit from diverse initial states, medium interaction granularity and more frequent sampling. Third, we show that without fine-grained, reasoning-aware reward signals, agent reasoning hardly emerge through multi-turn RL and they may show shallow strategies or hallucinated thoughts.

Keywords: LLM Agents, Multi-turn RL 

Website: [https://ragen-ai.github.io/](https://ragen-ai.github.io/)

Code/Environments: [https://github.com/RAGEN-AI/RAGEN](https://github.com/RAGEN-AI/RAGEN).

{CJK*}

UTF8gbsn

![Image 1: Refer to caption](https://arxiv.org/html/2504.20073v2/x1.png)

Figure 1: Previous methods focus on non-interactive tasks such as math or code generation. RAGEN implements StarPO, a general agent RL framework that supports multi-turn rollouts, trajectory-level reward assignment, and policy updates, on agent tasks requiring multi-turn stochastic interaction. 

1 Introduction
--------------

Training large language models (LLMs) to function as autonomous agents in interactive environments presents unique challenges. Unlike static tasks such as single-turn math problem solving(Shao et al., [2024a](https://arxiv.org/html/2504.20073v2#bib.bib48)) or coding(DeepSeek-AI et al., [2024](https://arxiv.org/html/2504.20073v2#bib.bib6)), agent settings require models to make sequential decisions, maintain memory across turns, and adapt to stochastic feedback from their environment. These settings—central to planning assistants, robotics, and tutoring agents—demand that models not only perform well, but also self-improve through experience.

While recent work has explored reinforcement learning (RL) for LLMs(DeepSeek-AI et al., [2025](https://arxiv.org/html/2504.20073v2#bib.bib7); OpenAI, [2024](https://arxiv.org/html/2504.20073v2#bib.bib38); Pan et al., [2025](https://arxiv.org/html/2504.20073v2#bib.bib40); Zeng et al., [2025c](https://arxiv.org/html/2504.20073v2#bib.bib70); Kumar et al., [2024](https://arxiv.org/html/2504.20073v2#bib.bib27); Gao et al., [2024](https://arxiv.org/html/2504.20073v2#bib.bib11)) using rule-based reward, it remains largely underexplored to train interactive agents that self-evolve to reason and adapt through rule-based RL. In particular, LLM agent training often exhibits training instability, complex reward signals, and limited generalization across environment changes, especially under multi-turn interaction with stochastic feedback. A key open question is: what design factors make self-evolving LLM agents learn effectively and stably?

We explore this question through a systematic study of agent learning under a general RL framework StarPO (S tate-T hinking-A ctions-R eward P olicy O ptimization). StarPO provides a unified view of multi-turn, trajectory-level agent training with flexible control over reasoning, reward assignment, and prompt-rollout structure. Built on top of StarPO, we develop RAGEN, a modular agent training and evaluation system designed to support the study of RL-based training in LLM Agents. RAGEN implements the full training loop—including rollout generation, reward assignment, and trajectory optimization—serving as a research infrastructure for systematic analysis of LLM agent training dynamics under multi-turn and stochastic environments.

Training LLM agents on real-world tasks like web browsing often depends on pretrained priors and heavy task-specific engineering. We evaluate RAGEN on four environments spanning different complexities: Bandit (single-turn, stochastic), Sokoban (multi-turn, deterministic), Frozen Lake (multi-turn, stochastic), and WebShop (multi-turn, open-domain). The first three symbolic environments are minimalistic and fully controllable, while WebShop adds real-world understanding and reasoning. Together, they enable analysis of generalization across varied decision-making challenges.

Using this setup, we analyze three key dimensions of agent learning, and summarize below findings that reveal core challenges and design principles for stable agent RL training:

1.   1.
Gradient Stability in Multi-turn RL is the Key to Stable Training. We find that multi-turn RL training often leads to a recurring instability pattern, Echo Trap, where agents overfit to locally rewarded reasoning patterns, marked by reward variability collapse, entropy drop, and gradient spikes. To mitigate this failure mode, we propose StarPO-S, a stabilized variant of our framework that improves learning robustness through variability-based trajectory filtering, critic baselining, and decoupled clipping.

2.   2.
Rollout Frequency and Diversity Shape Self-Evolution. In RL-based agent training, LLM self-generated rollout trajectories are served as core training material. We identify key rollout factors for stable agent RL training: (1) ensuring that rollouts come from diverse initial states with multiple responses per initial state, (2) implementing multiple actions each turn to improve interaction horizon within fixed turn limit, (3) maintaining a high rollout frequency to ensure online feedback reflects current policies.

3.   3.
Emerging Agent Reasoning Requires Meticulous Reward Signal. We find that simply encouraging reasoning in the action format does not guarantee reasoning behavior. Even when models are prompted to reason (e.g., with ‘<think>’ tokens) with trajectory-level optimization via StarPO, they often regress to direct action selection if reasoning offers no distinct reward advantage. We assume this is due to the simple action spaces in MDP where shallow strategies suffice. Moreover, when rewards only reflect task success, models produce hallucinated reasoning, revealing a mismatch between thoughts and environment states. These issues underscore the need for fine-grained, reasoning-aware reward design for long-horizon agent training.

![Image 2: Refer to caption](https://arxiv.org/html/2504.20073v2/x2.png)

Figure 2: The State-Thinking-Actions-Reward Policy Optimization (StarPO) framework. LLM generates reasoning-guided actions for multi-turn interactions with environments and accumulates trajectory-level rewards, normalized and used to update the LLM policy.

2 Framework
-----------

### 2.1 The MDP Formulation for Agent Training

Previous reinforcement learning (RL) for language models often assumes a single-turn setting, where the goal is to maximize the expected reward R⁢(s,a)𝑅 𝑠 𝑎 R(s,a)italic_R ( italic_s , italic_a ) over prompt-response pairs (s,a)𝑠 𝑎(s,a)( italic_s , italic_a ) sampled from a dataset 𝒟 𝒟\mathcal{D}caligraphic_D:

J step⁢(θ)=𝔼 s∼𝒟,a∼π θ(⋅|s)⁢[R⁢(s,a)].J_{\text{step}}(\theta)=\mathbb{E}_{s\sim\mathcal{D},a\sim\pi_{\theta}(\cdot|s% )}[R(s,a)].italic_J start_POSTSUBSCRIPT step end_POSTSUBSCRIPT ( italic_θ ) = blackboard_E start_POSTSUBSCRIPT italic_s ∼ caligraphic_D , italic_a ∼ italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( ⋅ | italic_s ) end_POSTSUBSCRIPT [ italic_R ( italic_s , italic_a ) ] .(1)

However, LLM-based agents must operate in interactive environments that unfold over multiple turns and exhibit stochastic feedback. To capture these dynamics, we formulate the problem as a Markov Decision Process (MDP) ℳ={S,A,P}ℳ 𝑆 𝐴 𝑃\mathcal{M}=\{S,A,P\}caligraphic_M = { italic_S , italic_A , italic_P }, where S 𝑆 S italic_S represents states (e.g., observation sequences or interaction histories), A 𝐴 A italic_A represents actions (often token sequences), and P 𝑃 P italic_P denotes the transition dynamics and reward generation process. The agent policy π θ subscript 𝜋 𝜃\pi_{\theta}italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT generates an action a t subscript 𝑎 𝑡 a_{t}italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT at each time step t 𝑡 t italic_t, conditioned on the current state s t subscript 𝑠 𝑡 s_{t}italic_s start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT and the interaction history. The environment returns a reward r t subscript 𝑟 𝑡 r_{t}italic_r start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT and a new state s t+1 subscript 𝑠 𝑡 1 s_{t+1}italic_s start_POSTSUBSCRIPT italic_t + 1 end_POSTSUBSCRIPT given the current transition dynamics:

a t∼π θ(⋅|s t,τ<t),(r t,s t+1)∼P(⋅|s t,a t),a_{t}\sim\pi_{\theta}(\cdot|s_{t},\tau_{<t}),\quad(r_{t},s_{t+1})\sim P(\cdot|% s_{t},a_{t}),italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ∼ italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( ⋅ | italic_s start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT , italic_τ start_POSTSUBSCRIPT < italic_t end_POSTSUBSCRIPT ) , ( italic_r start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT , italic_s start_POSTSUBSCRIPT italic_t + 1 end_POSTSUBSCRIPT ) ∼ italic_P ( ⋅ | italic_s start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT , italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) ,

where τ<t={s 0,a 0,r 0,…,s t−1,a t−1,r t−1}subscript 𝜏 absent 𝑡 subscript 𝑠 0 subscript 𝑎 0 subscript 𝑟 0…subscript 𝑠 𝑡 1 subscript 𝑎 𝑡 1 subscript 𝑟 𝑡 1\tau_{<t}=\{s_{0},a_{0},r_{0},...,s_{t-1},a_{t-1},r_{t-1}\}italic_τ start_POSTSUBSCRIPT < italic_t end_POSTSUBSCRIPT = { italic_s start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , italic_a start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , italic_r start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , … , italic_s start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT , italic_a start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT , italic_r start_POSTSUBSCRIPT italic_t - 1 end_POSTSUBSCRIPT } denotes the interaction history. This interactive process continues for a maximum horizon K 𝐾 K italic_K, yielding a full trajectory τ={s 0,a 0,r 0,…,s K}𝜏 subscript 𝑠 0 subscript 𝑎 0 subscript 𝑟 0…subscript 𝑠 𝐾\tau=\{s_{0},a_{0},r_{0},...,s_{K}\}italic_τ = { italic_s start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , italic_a start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , italic_r start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , … , italic_s start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT } that forms the learning material for the agent.

### 2.2 StarPO: Reinforcing Reasoning via Trajectory-Level Optimization

We introduce StarPO (State-Thinking-Action-Reward Policy Optimization), a general RL framework designed to optimize entire multi-turn interaction trajectories for LLM agents. Unlike previous methods for static tasks that treat each action independently, StarPO treats the entire trajectory—including observations, reasoning traces, actions, and feedback—as a coherent unit for rollout and model optimization. The objective is to maximize expected trajectory reward:

J StarPO⁢(θ)=𝔼 ℳ,τ∼π θ⁢[R⁢(τ)],subscript 𝐽 StarPO 𝜃 subscript 𝔼 similar-to ℳ 𝜏 subscript 𝜋 𝜃 delimited-[]𝑅 𝜏 J_{\text{StarPO}}(\theta)=\mathbb{E}_{\mathcal{M},\tau\sim\pi_{\theta}}\left[R% (\tau)\right],italic_J start_POSTSUBSCRIPT StarPO end_POSTSUBSCRIPT ( italic_θ ) = blackboard_E start_POSTSUBSCRIPT caligraphic_M , italic_τ ∼ italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT end_POSTSUBSCRIPT [ italic_R ( italic_τ ) ] ,(2)

where ℳ ℳ\mathcal{M}caligraphic_M is the MDP, τ 𝜏\tau italic_τ is a full sequence of reasoning-augmented interactions, and R⁢(τ)𝑅 𝜏 R(\tau)italic_R ( italic_τ ) denotes the cumulative reward over the entire trajectory. The policy probability π θ⁢(τ)subscript 𝜋 𝜃 𝜏\pi_{\theta}(\tau)italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_τ ) is decomposed into token-level likelihoods, making StarPO directly compatible with autoregressive LLMs. Figure[2](https://arxiv.org/html/2504.20073v2#S1.F2 "Figure 2 ‣ 1 Introduction ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning") illustrates the full StarPO process, and we break them down in detail below.

#### 2.2.2 Optimization Procedure: Learning from Reasoning-Interaction Trajectories

At each training iteration, the agent begins from an initial state s 0 subscript 𝑠 0 s_{0}italic_s start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT and generates N 𝑁 N italic_N trajectories. At each step t 𝑡 t italic_t, the agent produces a reasoning-guided structured output:

a t T=<think>⁢…⁢</think><answer>⁢a t⁢</answer>,subscript superscript 𝑎 𝑇 𝑡<think>…</think><answer>subscript 𝑎 𝑡</answer>a^{T}_{t}=\texttt{<think>}...\texttt{</think><answer>}\leavevmode\nobreak\ a_{% t}\leavevmode\nobreak\ \texttt{</answer>},italic_a start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT = <think> … </think><answer> italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT </answer> ,(3)

where a t T subscript superscript 𝑎 𝑇 𝑡 a^{T}_{t}italic_a start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT is the full action output including intermediate reasoning, and a t subscript 𝑎 𝑡 a_{t}italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT is a sequence of environment-executable sub-action. The environment then returns the next state s t+1 subscript 𝑠 𝑡 1 s_{t+1}italic_s start_POSTSUBSCRIPT italic_t + 1 end_POSTSUBSCRIPT and reward r t subscript 𝑟 𝑡 r_{t}italic_r start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT. The rollout stage produces complete trajectories τ={s 0,a 0 T,r 0,s 1,…,a K−1 T,r K−1,s K}𝜏 subscript 𝑠 0 subscript superscript 𝑎 𝑇 0 subscript 𝑟 0 subscript 𝑠 1…subscript superscript 𝑎 𝑇 𝐾 1 subscript 𝑟 𝐾 1 subscript 𝑠 𝐾\tau=\{s_{0},a^{T}_{0},r_{0},s_{1},...,a^{T}_{K-1},r_{K-1},s_{K}\}italic_τ = { italic_s start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , italic_a start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , italic_r start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT , italic_s start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , … , italic_a start_POSTSUPERSCRIPT italic_T end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_K - 1 end_POSTSUBSCRIPT , italic_r start_POSTSUBSCRIPT italic_K - 1 end_POSTSUBSCRIPT , italic_s start_POSTSUBSCRIPT italic_K end_POSTSUBSCRIPT }, where every component is LLM-generated or environment-induced and will be jointly optimized.

StarPO interleaves rollout and update steps. New rollouts can be generated on-policy using π θ subscript 𝜋 𝜃\pi_{\theta}italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT, or sampled from a replay buffer under π old subscript 𝜋 old\pi_{\text{old}}italic_π start_POSTSUBSCRIPT old end_POSTSUBSCRIPT. Each training loop consists of P 𝑃 P italic_P initial states s 0 subscript 𝑠 0 s_{0}italic_s start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT, each generating N 𝑁 N italic_N trajectories, and updates are performed with batch size E 𝐸 E italic_E for L 𝐿 L italic_L total loops. This yields S=L⋅P⋅N E 𝑆⋅𝐿 𝑃 𝑁 𝐸 S=\frac{L\cdot P\cdot N}{E}italic_S = divide start_ARG italic_L ⋅ italic_P ⋅ italic_N end_ARG start_ARG italic_E end_ARG total gradient update steps. Additional training mechanisms are discussed in §[3](https://arxiv.org/html/2504.20073v2#S3 "3 Experiment Setup ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning").

#### 2.2.3 Modular Optimization Strategies

StarPO supports a variety of policy optimization algorithms under a unified trajectory-level abstraction. For each rollout trajectory τ i={τ i,(1),…,τ i,(|τ i|)}subscript 𝜏 𝑖 subscript 𝜏 𝑖 1…subscript 𝜏 𝑖 subscript 𝜏 𝑖\tau_{i}=\{\tau_{i,(1)},\ldots,\tau_{i,(|\tau_{i}|)}\}italic_τ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT = { italic_τ start_POSTSUBSCRIPT italic_i , ( 1 ) end_POSTSUBSCRIPT , … , italic_τ start_POSTSUBSCRIPT italic_i , ( | italic_τ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | ) end_POSTSUBSCRIPT } of totally |τ i|subscript 𝜏 𝑖|\tau_{i}|| italic_τ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | tokens, we instantiate StarPO with the following optimization strategies for token-level updates:

*   •PPO(Schulman et al., [2017](https://arxiv.org/html/2504.20073v2#bib.bib45)). We use the PPO objective (More details can be found in Appendix[A](https://arxiv.org/html/2504.20073v2#A1 "Appendix A Background of Reinforcement Learning ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning")), where a critic is trained to estimate token-level value and advantages A i,t subscript 𝐴 𝑖 𝑡 A_{i,t}italic_A start_POSTSUBSCRIPT italic_i , italic_t end_POSTSUBSCRIPT:

J PPO⁢(θ)=1 G⁢∑i=1 G 1|τ i|⁢∑t=1|τ i|min⁡[π θ⁢(τ i,(t)|τ i,<t)π old⁢(τ i,(t)|τ i,<t)⋅A i,t,clip⁢(π θ⁢(τ i,(t)|τ i,<t)π old⁢(τ i,(t)|τ i,<t),1−ε,1+ε)⋅A i,t],subscript 𝐽 PPO 𝜃 1 𝐺 superscript subscript 𝑖 1 𝐺 1 subscript 𝜏 𝑖 superscript subscript 𝑡 1 subscript 𝜏 𝑖⋅subscript 𝜋 𝜃 conditional subscript 𝜏 𝑖 𝑡 subscript 𝜏 𝑖 absent 𝑡 subscript 𝜋 old conditional subscript 𝜏 𝑖 𝑡 subscript 𝜏 𝑖 absent 𝑡 subscript 𝐴 𝑖 𝑡⋅clip subscript 𝜋 𝜃 conditional subscript 𝜏 𝑖 𝑡 subscript 𝜏 𝑖 absent 𝑡 subscript 𝜋 old conditional subscript 𝜏 𝑖 𝑡 subscript 𝜏 𝑖 absent 𝑡 1 𝜀 1 𝜀 subscript 𝐴 𝑖 𝑡\small J_{\text{PPO}}(\theta)=\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|\tau_{i}|}% \sum_{t=1}^{|\tau_{i}|}\min\left[\frac{\pi_{\theta}(\tau_{i,(t)}|\tau_{i,<t})}% {\pi_{\text{old}}(\tau_{i,(t)}|\tau_{i,<t})}\cdot A_{i,t},\,\text{clip}\left(% \frac{\pi_{\theta}(\tau_{i,(t)}|\tau_{i,<t})}{\pi_{\text{old}}(\tau_{i,(t)}|% \tau_{i,<t})},1-\varepsilon,1+\varepsilon\right)\cdot A_{i,t}\right],italic_J start_POSTSUBSCRIPT PPO end_POSTSUBSCRIPT ( italic_θ ) = divide start_ARG 1 end_ARG start_ARG italic_G end_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_G end_POSTSUPERSCRIPT divide start_ARG 1 end_ARG start_ARG | italic_τ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | end_ARG ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT | italic_τ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | end_POSTSUPERSCRIPT roman_min [ divide start_ARG italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_τ start_POSTSUBSCRIPT italic_i , ( italic_t ) end_POSTSUBSCRIPT | italic_τ start_POSTSUBSCRIPT italic_i , < italic_t end_POSTSUBSCRIPT ) end_ARG start_ARG italic_π start_POSTSUBSCRIPT old end_POSTSUBSCRIPT ( italic_τ start_POSTSUBSCRIPT italic_i , ( italic_t ) end_POSTSUBSCRIPT | italic_τ start_POSTSUBSCRIPT italic_i , < italic_t end_POSTSUBSCRIPT ) end_ARG ⋅ italic_A start_POSTSUBSCRIPT italic_i , italic_t end_POSTSUBSCRIPT , clip ( divide start_ARG italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_τ start_POSTSUBSCRIPT italic_i , ( italic_t ) end_POSTSUBSCRIPT | italic_τ start_POSTSUBSCRIPT italic_i , < italic_t end_POSTSUBSCRIPT ) end_ARG start_ARG italic_π start_POSTSUBSCRIPT old end_POSTSUBSCRIPT ( italic_τ start_POSTSUBSCRIPT italic_i , ( italic_t ) end_POSTSUBSCRIPT | italic_τ start_POSTSUBSCRIPT italic_i , < italic_t end_POSTSUBSCRIPT ) end_ARG , 1 - italic_ε , 1 + italic_ε ) ⋅ italic_A start_POSTSUBSCRIPT italic_i , italic_t end_POSTSUBSCRIPT ] ,(4)

where G 𝐺 G italic_G is the number of trajectories in the batch, τ i,(t)subscript 𝜏 𝑖 𝑡\tau_{i,(t)}italic_τ start_POSTSUBSCRIPT italic_i , ( italic_t ) end_POSTSUBSCRIPT denotes the t 𝑡 t italic_t-th token in trajectory τ i subscript 𝜏 𝑖\tau_{i}italic_τ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, and τ i,<t subscript 𝜏 𝑖 absent 𝑡\tau_{i,<t}italic_τ start_POSTSUBSCRIPT italic_i , < italic_t end_POSTSUBSCRIPT is its prefix. 
*   •GRPO(Shao et al., [2024a](https://arxiv.org/html/2504.20073v2#bib.bib48)). For critic-free training leveraging GRPO, we assign a scalar reward R⁢(τ i)𝑅 subscript 𝜏 𝑖 R(\tau_{i})italic_R ( italic_τ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) to each trajectory and normalized advantage A^i,t subscript^𝐴 𝑖 𝑡\hat{A}_{i,t}over^ start_ARG italic_A end_ARG start_POSTSUBSCRIPT italic_i , italic_t end_POSTSUBSCRIPT across all tokens in τ i subscript 𝜏 𝑖\tau_{i}italic_τ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT:

A^i,t=R⁢(τ i)−mean⁢({R⁢(τ 1),…,R⁢(τ G)})std⁢({R⁢(τ 1),…,R⁢(τ G)}).subscript^𝐴 𝑖 𝑡 𝑅 subscript 𝜏 𝑖 mean 𝑅 subscript 𝜏 1…𝑅 subscript 𝜏 𝐺 std 𝑅 subscript 𝜏 1…𝑅 subscript 𝜏 𝐺\small\hat{A}_{i,t}=\frac{R(\tau_{i})-\text{mean}(\{R(\tau_{1}),\ldots,R(\tau_% {G})\})}{\text{std}(\{R(\tau_{1}),\ldots,R(\tau_{G})\})}.over^ start_ARG italic_A end_ARG start_POSTSUBSCRIPT italic_i , italic_t end_POSTSUBSCRIPT = divide start_ARG italic_R ( italic_τ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) - mean ( { italic_R ( italic_τ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) , … , italic_R ( italic_τ start_POSTSUBSCRIPT italic_G end_POSTSUBSCRIPT ) } ) end_ARG start_ARG std ( { italic_R ( italic_τ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) , … , italic_R ( italic_τ start_POSTSUBSCRIPT italic_G end_POSTSUBSCRIPT ) } ) end_ARG .(5)

The GRPO objective becomes:

J GRPO⁢(θ)=1 G⁢∑i=1 G 1|τ i|⁢∑t=1|τ i|min⁡[π θ⁢(τ i,(t)|τ i,<t)π old⁢(τ i,(t)|τ i,<t)⋅A^i,t,clip⁢(π θ⁢(τ i,(t)|τ i,<t)π old⁢(τ i,(t)|τ i,<t),1−ε,1+ε)⋅A^i,t].subscript 𝐽 GRPO 𝜃 1 𝐺 superscript subscript 𝑖 1 𝐺 1 subscript 𝜏 𝑖 superscript subscript 𝑡 1 subscript 𝜏 𝑖⋅subscript 𝜋 𝜃 conditional subscript 𝜏 𝑖 𝑡 subscript 𝜏 𝑖 absent 𝑡 subscript 𝜋 old conditional subscript 𝜏 𝑖 𝑡 subscript 𝜏 𝑖 absent 𝑡 subscript^𝐴 𝑖 𝑡⋅clip subscript 𝜋 𝜃 conditional subscript 𝜏 𝑖 𝑡 subscript 𝜏 𝑖 absent 𝑡 subscript 𝜋 old conditional subscript 𝜏 𝑖 𝑡 subscript 𝜏 𝑖 absent 𝑡 1 𝜀 1 𝜀 subscript^𝐴 𝑖 𝑡\small J_{\text{GRPO}}(\theta)=\frac{1}{G}\sum_{i=1}^{G}\frac{1}{|\tau_{i}|}% \sum_{t=1}^{|\tau_{i}|}\min\left[\frac{\pi_{\theta}(\tau_{i,(t)}|\tau_{i,<t})}% {\pi_{\text{old}}(\tau_{i,(t)}|\tau_{i,<t})}\cdot\hat{A}_{i,t},\,\text{clip}% \left(\frac{\pi_{\theta}(\tau_{i,(t)}|\tau_{i,<t})}{\pi_{\text{old}}(\tau_{i,(% t)}|\tau_{i,<t})},1-\varepsilon,1+\varepsilon\right)\cdot\hat{A}_{i,t}\right].italic_J start_POSTSUBSCRIPT GRPO end_POSTSUBSCRIPT ( italic_θ ) = divide start_ARG 1 end_ARG start_ARG italic_G end_ARG ∑ start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_G end_POSTSUPERSCRIPT divide start_ARG 1 end_ARG start_ARG | italic_τ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | end_ARG ∑ start_POSTSUBSCRIPT italic_t = 1 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT | italic_τ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | end_POSTSUPERSCRIPT roman_min [ divide start_ARG italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_τ start_POSTSUBSCRIPT italic_i , ( italic_t ) end_POSTSUBSCRIPT | italic_τ start_POSTSUBSCRIPT italic_i , < italic_t end_POSTSUBSCRIPT ) end_ARG start_ARG italic_π start_POSTSUBSCRIPT old end_POSTSUBSCRIPT ( italic_τ start_POSTSUBSCRIPT italic_i , ( italic_t ) end_POSTSUBSCRIPT | italic_τ start_POSTSUBSCRIPT italic_i , < italic_t end_POSTSUBSCRIPT ) end_ARG ⋅ over^ start_ARG italic_A end_ARG start_POSTSUBSCRIPT italic_i , italic_t end_POSTSUBSCRIPT , clip ( divide start_ARG italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_τ start_POSTSUBSCRIPT italic_i , ( italic_t ) end_POSTSUBSCRIPT | italic_τ start_POSTSUBSCRIPT italic_i , < italic_t end_POSTSUBSCRIPT ) end_ARG start_ARG italic_π start_POSTSUBSCRIPT old end_POSTSUBSCRIPT ( italic_τ start_POSTSUBSCRIPT italic_i , ( italic_t ) end_POSTSUBSCRIPT | italic_τ start_POSTSUBSCRIPT italic_i , < italic_t end_POSTSUBSCRIPT ) end_ARG , 1 - italic_ε , 1 + italic_ε ) ⋅ over^ start_ARG italic_A end_ARG start_POSTSUBSCRIPT italic_i , italic_t end_POSTSUBSCRIPT ] .(6) 

### 2.3 The RAGEN System

To implement StarPO in practice, we build RAGEN, a complete system for LLM agent training in controlled environments. RAGEN supports structured rollouts, customizable reward functions, and integration with multi-turn, stochastic environments. It serves both as the execution backend for StarPO and as a platform for studying stability, generalization, and learning dynamics in training reasoning agents. RAGEN is designed to be extensible: new environments, reward schemes, or rollout strategies can be easily plugged in, serving as a foundation for RL-based agent training.

3 Experiment Setup
------------------

### 3.1 Environments and Tasks

We evaluate LLM agents on four environments spanning symbolic and realistic decision-making: Bandit tests risk-sensitive reasoning under noisy feedback; Sokoban requires irreversible symbolic planning; Frozen Lake combines planning with probabilistic transitions; and WebShop involves natural language grounding and web environment interaction. The first three symbolic environments are deliberately minimal and fully controllable to support clean analysis, while WebShop introduces realistic task structure and language input. Environment visualizations are in Appendix[C.1](https://arxiv.org/html/2504.20073v2#A3.SS1 "C.1 Environments and Tasks ‣ Appendix C Detailed Experimental Settings ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning").

### 3.2 Training Settings

In our main experiments, we train Qwen-2.5 Instruct 0.5B models for three symbolic tasks and its 3B variant for the challenging WebShop. We also report various model performance in Appendix[D](https://arxiv.org/html/2504.20073v2#A4 "Appendix D Results on Larger Models and Various Optimization Algorithms ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning"). Models are trained with StarPO variants on H100 GPUs for 100-200 rollout–update iterations. Each batch samples P=8 𝑃 8 P{=}8 italic_P = 8 prompts, with N=16 𝑁 16 N{=}16 italic_N = 16 rollouts per prompt, up to 5 turns and 10 actions. Policy updates use GRPO or PPO with GAE (γ=1.0,λ=1.0 formulae-sequence 𝛾 1.0 𝜆 1.0\gamma{=}1.0,\lambda{=}1.0 italic_γ = 1.0 , italic_λ = 1.0), Adam optimizer, entropy bonus (β=0.001 𝛽 0.001\beta{=}0.001 italic_β = 0.001), and a response-format penalty (−0.1 0.1-0.1- 0.1). More details can be found in Appendix[C.2](https://arxiv.org/html/2504.20073v2#A3.SS2 "C.2 Training and Evaluation Settings ‣ Appendix C Detailed Experimental Settings ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning").

### 3.3 Evaluation Metrics

We evaluate on 256 fixed prompts per environment with temperature T=0.5 𝑇 0.5 T{=}0.5 italic_T = 0.5, truncating episodes after 5 turns. Metrics include: (i) success rate (task completion), (ii) rollout entropy (exploration), (iii) in-group reward variability (behavioral diversity), (iv) response length (reasoning verbosity), and (v) gradient norm (training stability). All are computed over validation instances. More details can be found in Appendix[C.3](https://arxiv.org/html/2504.20073v2#A3.SS3 "C.3 Evaluation Metrics ‣ Appendix C Detailed Experimental Settings ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning").

4 Experimental Results and Findings
-----------------------------------

### 4.1 Multi-turn Agent RL Training Introduces New Instability Pattern

We evaluate baseline StarPO across agent tasks (Figure[3](https://arxiv.org/html/2504.20073v2#S4.F3 "Figure 3 ‣ 4.1 Multi-turn Agent RL Training Introduces New Instability Pattern ‣ 4 Experimental Results and Findings ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning")). Symbolic environments like Bandit and Sokoban show early improvements but eventually collapse. PPO is more stable than GRPO in these settings, collapsing later and achieving higher performance, likely due to its critic providing smoother reward estimates. Interestingly, GRPO is more stable on Frozen Lake, likely due to the difficulty of estimating state values in this task, which may destabilize PPO (see Appendix[I](https://arxiv.org/html/2504.20073v2#A9 "Appendix I PPO Failure Mode in Frozen Lake ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning")). On WebShop, both methods succeed, likely due to strong language prior and high initial rewards reducing the need for a critic.

![Image 3: Refer to caption](https://arxiv.org/html/2504.20073v2/x3.png)

Figure 3: Baseline StarPO performance across environments. Symbolic tasks like Bandit and Sokoban lead to collapse while real-world WebShop starts high and improves fast. PPO is stronger in Bandit and Sokoban to provide more stable token-level reward signal, while GRPO is stronger in Frozen Lake whose randomness makes the state value hard to estimate, and WebShop whose strong initial performance lessens the need for a critic to stabilize gradients. 

To understand the causes of collapse, we compare early- and late-stage trajectories. In the Bandit task, early-stage trajectories exhibit diverse reasoning about symbolic meanings and expected rewards, while later-stage responses become repetitive and deterministic. This suggests that RL training may have over-amplified inherent reasoning shortcuts, reinforcing locally rewarded templates while suppressing exploration. We refer to this failure mode as an “Echo Trap”, which is similar to the findings in Shumailov et al. ([2024](https://arxiv.org/html/2504.20073v2#bib.bib52)), where the model repeatedly reuses memorized reasoning paths when trained on self-generated trajectories, leading to a collapse in diversity and long-term performance degradation. Examples are in Appendix[F](https://arxiv.org/html/2504.20073v2#A6 "Appendix F Case Study: The Emergence of Echo Trap with RL ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning").

To detect collapse, we monitor two key metrics: (1) Average Reward, where a plateau or drop signals degraded task performance, and (2) Gradient Norm, where spikes suggest unstable updates. To identify early signs, we also track rollout-level signals: (1) Reward Standard Deviation, indicating whether the policy is producing diverse outcomes or collapsing into repetitive behavior, and (2) Output Entropy, which indicates how deterministic the model’s predictions are.

![Image 4: Refer to caption](https://arxiv.org/html/2504.20073v2/x4.png)

![Image 5: Refer to caption](https://arxiv.org/html/2504.20073v2/x5.png)

Figure 4: Collapse indicators and early warning signals in multi-turn RL. Average reward and gradient norm (left-side plots) reflect collapse directly, and their plateaus and spikes confirm performance and training instability. Reward standard deviation and entropy (right-side plots) often becomes unstable before reward degrades, serving as early warning signals. 

Figure[4](https://arxiv.org/html/2504.20073v2#S4.F4 "Figure 4 ‣ 4.1 Multi-turn Agent RL Training Introduces New Instability Pattern ‣ 4 Experimental Results and Findings ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning") summarizes these dynamics across tasks and optimization methods. From the results, we draw the following conclusions regarding how model collapse emerges in multi-turn agent RL:

*   •
Reward standard deviation is an early indicator of convergence. In FrozenLake-PPO, std drops sharply at step 40, well before reward mean collapses at step 90 while performance is near-optimal. In Bandit-PPO, std bottoms out around step 70, preceding the reward peak at step 120. In Sokoban-PPO, std and mean collapse together near step 10, suggesting early saturation.

*   •
Gradient norm spikes indicate irreversible collapse. Once gradient norm spikes emerge, e.g., at step 170 (Bandit), 110 (Sokoban), and 90 (FrozenLake), even small parameter updates could induce drastic loss shifts, after which recovery becomes unlikely.

*   •
Entropy should follow a stable decay trend during effective learning. This can be seen from FrozenLake-GRPO. Rapid entropy increases or erratic changes often correlate with collapsed reasoning behavior, such as GRPO on Bandit and Sokoban.

These patterns confirm that multi-turn RL introduces unique challenges that single-turn RL methods fail to handle. In response, we introduce StarPO-S, a stabilized variant that targets sampling quality, gradient stability, and exploration regularization to avoid premature collapse.

### 4.2 StarPO-S: Stabilize Multi-turn RL with Instance Filtering and Gradient Shaping

To address the instability of multi-turn reinforcement learning, we introduce StarPO-S, a stabilized variant of StarPO that incorporates three key modifications aimed at improving training robustness and efficiency. Building on the insight that declining reward standard deviation often precedes collapse, we investigate the following question: should agents be trained more intensively on task instances where their behavior is more uncertain with higher reward variability?

We hypothesize that the most effective training samples are those where the agent exhibits outcome uncertainty—avoiding both trivial task instances and overly difficult ones. This intuition is rooted in principles of Active Learning(Settles, [2009](https://arxiv.org/html/2504.20073v2#bib.bib47)), where uncertain examples are the most informative ones models should learn from. We define trajectory-level outcome uncertainty U 𝑈 U italic_U for policy π θ subscript 𝜋 𝜃\pi_{\theta}italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT on a given agent task instance (initial state s 0 subscript 𝑠 0 s_{0}italic_s start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT in an MDP ℳ={S,A,P}ℳ 𝑆 𝐴 𝑃\mathcal{M}=\{S,A,P\}caligraphic_M = { italic_S , italic_A , italic_P }) as:

U⁢(π θ,ℳ,s 0)=Std τ∼π θ(⋅|s 0)⁢[R⁢(τ)].\text{U}(\pi_{\theta},\mathcal{M},s_{0})=\text{Std}_{\tau\sim\pi_{\theta}(% \cdot|s_{0})}\left[R(\tau)\right].U ( italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT , caligraphic_M , italic_s start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ) = Std start_POSTSUBSCRIPT italic_τ ∼ italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( ⋅ | italic_s start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ) end_POSTSUBSCRIPT [ italic_R ( italic_τ ) ] .(7)

During training, we sort prompts based on the standard deviation of reward obtained from repeated rollouts and retain only the top p 𝑝 p italic_p% highly-uncertain prompts at each training step. Figure[5](https://arxiv.org/html/2504.20073v2#S4.F5 "Figure 5 ‣ 4.2 StarPO-S: Stabilize Multi-turn RL with Instance Filtering and Gradient Shaping ‣ 4 Experimental Results and Findings ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning") shows the effect of varying p 𝑝 p italic_p in PPO and GRPO under StarPO-S. We further validate the effect of uncertainty-based filtering in Appendix[E](https://arxiv.org/html/2504.20073v2#A5 "Appendix E When Does Uncertainty-Based Filtering Help? ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning").

In PPO runs (Figure[5](https://arxiv.org/html/2504.20073v2#S4.F5 "Figure 5 ‣ 4.2 StarPO-S: Stabilize Multi-turn RL with Instance Filtering and Gradient Shaping ‣ 4 Experimental Results and Findings ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning"), left), filtering low-variability rollouts significantly delays collapse: retaining 75% of rollouts extends stability in FrozenLake from 100 to 140 steps, while 50% avoids collapse entirely. GRPO remains less stable due to its critic-free design but still benefits modestly. Filtering also improves efficiency (Figure[5](https://arxiv.org/html/2504.20073v2#S4.F5 "Figure 5 ‣ 4.2 StarPO-S: Stabilize Multi-turn RL with Instance Filtering and Gradient Shaping ‣ 4 Experimental Results and Findings ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning"), right). We adopt 25% as the default for StarPO-S. However, we note that this aggressive value may not be optimal for all scenarios. Tasks like Sokoban and FrozenLake respond well to aggressive filtering, potentially due to their relatively repetitive reasoning patterns and under-representation in pretraining, which make them tend to collapse when similar trajectories dominate the batch. We further present larger model (72B), together with frontier model such as GPT-4o and Qwen-2.5-72B, in Appendix[D](https://arxiv.org/html/2504.20073v2#A4 "Appendix D Results on Larger Models and Various Optimization Algorithms ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning"), to better contextualize our model performance.

In addition to uncertainty-based filtering, we adopt two gradient shaping techniques inspired by DAPO(Yu et al., [2025](https://arxiv.org/html/2504.20073v2#bib.bib66)) designed for single-turn RL: KL Term Removal and Clip-Higher (Asymmetric Clipping). We extend and evaluate them in the multi-turn agent setting, and find both methods boost the success rate and extend stable training phases, showing how multi-turn RL benefits from more flexible gradient shaping. Design details and performance ablation is in Appendix[D](https://arxiv.org/html/2504.20073v2#A4 "Appendix D Results on Larger Models and Various Optimization Algorithms ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning").

![Image 6: Refer to caption](https://arxiv.org/html/2504.20073v2/x6.png)

Figure 5: Effect of uncertainty-based filtering on multi-turn RL stability. Filtering out low-variability trajectories reduces collapse risk and improves success rate. On PPO variants, collapse is largely mitigated when more than half of the trajectories are filtered. Training time is reduced.

Overall Comparison. We compare StarPO-S with vanilla StarPO across three tasks in Figure[6](https://arxiv.org/html/2504.20073v2#S4.F6 "Figure 6 ‣ 4.2 StarPO-S: Stabilize Multi-turn RL with Instance Filtering and Gradient Shaping ‣ 4 Experimental Results and Findings ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning"). StarPO-S consistently delays collapse and enhances final task performance. We attribute these gains to more selective training data (via uncertainty filtering), more balanced optimization signals (via KL removal and decoupled clipping), reducing narrowed reasoning modes. In the appendix[D](https://arxiv.org/html/2504.20073v2#A4 "Appendix D Results on Larger Models and Various Optimization Algorithms ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning"), we further discuss about other variants that may stablize training and enhance performance, such as selective response mask and Bi-level Gegenal Advantage Estimation (GAE)Wang* et al. ([2025](https://arxiv.org/html/2504.20073v2#bib.bib55)).

![Image 7: Refer to caption](https://arxiv.org/html/2504.20073v2/x7.png)

Figure 6: StarPO-S improves stability and final performance across tasks. Compared to vanilla StarPO, StarPO-S reliefs collapse in all four tasks and could achieves higher success rates. 

### 4.3 Generating Useful Trajectories for RL Training

Effective RL training depends heavily on trajectory quality. We study three key rollout dimensions—task diversity, interaction granularity, and rollout frequency—by training on vanilla Sokoban and evaluating on other tasks including SokobanNewVocab, LargeSokoban, and FrozenLake Task, which we detail in Appendix[K](https://arxiv.org/html/2504.20073v2#A11 "Appendix K Generalization Evaluation Environments ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning").

Higher task diversity with response comparison improves generalization. Task diversity refers to the number of distinct prompts used for each rollout-update cycle. With a fixed batch size, it trades off against the number of responses per prompt. In our experiments (Table[1](https://arxiv.org/html/2504.20073v2#S4.T1 "Table 1 ‣ 4.3 Generating Useful Trajectories for RL Training ‣ 4 Experimental Results and Findings ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning")), we vary this trade-off and find that higher task diversity—achieved by fewer responses per prompt (e.g., 4 per prompt)—consistently yields better generalization. This only holds when each prompt includes multiple rollouts, enabling the agent to contrast different outcomes under similar conditions.

Allowing more action budgets enables planning, while overly long-horizon rollouts inject noise. We vary the number of actions allowed per turn in Table[2](https://arxiv.org/html/2504.20073v2#S4.T2 "Table 2 ‣ 4.3 Generating Useful Trajectories for RL Training ‣ 4 Experimental Results and Findings ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning"). Allowing up to 5 or 6 actions per turn yields the best performance, especially on complex environments like SokobanNewVocab and LargeSokoban. This setting provides enough room for planning while avoiding the chaos of overly long rollouts. Increasing the budget to 7 actions degrades performance, likely due to noisy transitions and diluted reward feedback.

Table 1: Effect of Task Diversity on Generalization Performance (%). Higher diversity with multiple responses yields the best performance (4 responses per prompt).

Table 2: Performance across environments under different per-turn action budgets (%). 5–6 actions per turn yields best performance, balancing well for effective multi-step planning.

Table 3: Generalization performance (%) with and without reasoning under StarPO-S. Disabling reasoning significantly reduces generalization in single-turn Bandit task, but has mixed or marginal effects in multi-turn Sokoban task.

Frequent rollout updates ensure alignment between optimization targets and current policy behavior. To investigate the effect of rollout freshness, we adopt an Online-k 𝑘 k italic_k rollout strategy, where a single set of rollouts is reused for k 𝑘 k italic_k consecutive policy updates. A smaller k 𝑘 k italic_k implies more frequent rollout collection. Notably, Online-1 corresponds to an fully online setting, with fresh rollouts collected every update iteration. As shown in Figure[7](https://arxiv.org/html/2504.20073v2#S4.F7 "Figure 7 ‣ 4.4 Reasoning Improves Generalization but Fades in Multi-Turn Settings Without Fine-Grained Rewards ‣ 4 Experimental Results and Findings ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning"), agents trained with fresher rollouts (Online-1) achieve faster convergence and better generalization across tasks compared to those with delayed updates (e.g., Online-5 or Online-10). This supports a core design principle for multi-turn RL: learning is most effective when trajectories reflect the agent’s latest behavior. Frequent rollout reduces policy-data mismatch and improves optimization stability.

### 4.4 Reasoning Improves Generalization but Fades in Multi-Turn Settings Without Fine-Grained Rewards

We examine how symbolic reasoning impacts agent generalization. While reasoning enhances performance in single-turn tasks like Bandit, it fails to grow or sustain in complex multi-turn environments like Sokoban. Below, we analyze these effects step-by-step.

Reasoning traces improve generalization in single-turn Bandit tasks. We design a controlled generalization test in symbolic Bandit environments. In the original Bandit setting, the model is trained on the [Teacher, Engineer] arm pair and evaluated on [Librarian, Trader], preserving intuitive risk-reward alignments (i.e., Engineer and Trader are high-risk, high-reward). In BanditRev, these associations are inverted, assigning counter-intuitive reward profiles to professions and making reasoning more challenging.

As shown in Table[3](https://arxiv.org/html/2504.20073v2#S4.T3 "Table 3 ‣ 4.3 Generating Useful Trajectories for RL Training ‣ 4 Experimental Results and Findings ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning"), models trained with reasoning traces generalize better in Bandit and even in the counterintuitive BanditRev, suggesting that reasoning supervision helps internalize symbolic cues beyond memorization. Despite the added difficulty in BanditRev, models with explicit reasoning consistently outperform those without, as shown in Table[3](https://arxiv.org/html/2504.20073v2#S4.T3 "Table 3 ‣ 4.3 Generating Useful Trajectories for RL Training ‣ 4 Experimental Results and Findings ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning"). This suggests that reasoning traces help the agent internalize symbolic-reward associations and generalize beyond surface-level memorization, even under semantic-reward misalignment.

In multi-turn tasks, reasoning signals fade as training progresses. In contrast to single-turn settings, we find that reasoning provides limited benefits in multi-turn environments like Sokoban and FrozenLake. Even when the output format includes explicit <think> segments, removing them (no-think variant) often yields comparable or even better performance. To understand this degradation, we analyze average response length during training (Table[4](https://arxiv.org/html/2504.20073v2#S4.T4 "Table 4 ‣ 4.4 Reasoning Improves Generalization but Fades in Multi-Turn Settings Without Fine-Grained Rewards ‣ 4 Experimental Results and Findings ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning"), Figure[14](https://arxiv.org/html/2504.20073v2#A6.F14 "Figure 14 ‣ Appendix F Case Study: The Emergence of Echo Trap with RL ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning")) and find that reasoning traces consistently shrink over time, suggesting the model is suppressing its own thought process. Interestingly, in the semantically misaligned BanditRev task—where reasoning is essential—traces remain longer, indicating that reasoning is better sustained when context is more challenging.

We presume that reasoning collapse may arise from the sparse, delayed reward structure in multi-turn tasks, which often fails to differentiate between coherent reasoning and trial-and-error success. This is supported by examples (Appendix[L](https://arxiv.org/html/2504.20073v2#A12 "Appendix L Case Study: Spurious Reward from Incorrect Reasoning ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning")) where models generate incoherent or hallucinated reasoning yet still receive high rewards. This raises an important challenge: how can we consistently reinforce useful reasoning when the reward alone may not reflect its quality? One possible approach is to decouple action correctness from reasoning quality using format-based penalties: similar to Shao et al. ([2024b](https://arxiv.org/html/2504.20073v2#bib.bib49)), we apply a small penalty to outputs lacking valid <think>–<answer> structure, encouraging structured reasoning. We believe future work may explore finer-grained reward designs such as rewarding partial correctness to reliably reinforce reasoning in long-horizon decision-making.

Table 4: Reasoning length (<think> block length) at different training steps. Token length generally declines over time, while contextually challenging problems like ReverseBandit require more reasoning than their original counterparts.

![Image 8: Refer to caption](https://arxiv.org/html/2504.20073v2/x8.png)

Figure 7: Performance under different rollout frequencies (Online-k k k italic_k). We vary the rollout reuse factor k 𝑘 k italic_k, where each batch is reused for k 𝑘 k italic_k policy updates. Lower k 𝑘 k italic_k (e.g., Online-1) means more frequent rollouts. Fresher data improves convergence by staying aligned with the current policy.

5 Related Work
--------------

Recent work has explored fine-tuning LLMs for multi-step reasoning through reinforcement learning (RL), and developing agent frameworks to structure decision-making tasks. For reasoning, techniques range from classical PPO(Schulman et al., [2017](https://arxiv.org/html/2504.20073v2#bib.bib45)) and actor-critic methods(Haarnoja et al., [2018](https://arxiv.org/html/2504.20073v2#bib.bib13)) to structured prompting via meta tokens(Goyal et al., [2024](https://arxiv.org/html/2504.20073v2#bib.bib12); Herel and Mikolov, [2024](https://arxiv.org/html/2504.20073v2#bib.bib18)). Policy variants such as RLOO(Kool et al., [2019](https://arxiv.org/html/2504.20073v2#bib.bib26)), GRPO(DeepSeek-AI et al., [2025](https://arxiv.org/html/2504.20073v2#bib.bib7)) and DAPO(Yu et al., [2025](https://arxiv.org/html/2504.20073v2#bib.bib66)) could stabilize training and improve sample efficiency. Parallel efforts like STaR(Zelikman et al., [2022](https://arxiv.org/html/2504.20073v2#bib.bib67)) and MCTS-based reasoning(Hao et al., [2023a](https://arxiv.org/html/2504.20073v2#bib.bib14)) promote step-by-step reasoning with minimal supervision.

On the agent side, systems have evolved from early reactive planning(Yao et al., [2022b](https://arxiv.org/html/2504.20073v2#bib.bib65); Xu et al., [2023](https://arxiv.org/html/2504.20073v2#bib.bib62)) to modularized decision pipelines(Liu et al., [2023](https://arxiv.org/html/2504.20073v2#bib.bib34); Wu et al., [2023](https://arxiv.org/html/2504.20073v2#bib.bib59)), multi-agent cooperation(Li et al., [2023](https://arxiv.org/html/2504.20073v2#bib.bib28); Wang et al., [2024a](https://arxiv.org/html/2504.20073v2#bib.bib56)), and embodied interaction(Lin et al., [2024b](https://arxiv.org/html/2504.20073v2#bib.bib33); Li et al., [2025](https://arxiv.org/html/2504.20073v2#bib.bib29)). Benchmarks like Sokoban(Junghanns and Schaeffer, [2001](https://arxiv.org/html/2504.20073v2#bib.bib25)), FrozenLake(Dell’Aversana, [2021](https://arxiv.org/html/2504.20073v2#bib.bib8)), and WebShop(Yao et al., [2022a](https://arxiv.org/html/2504.20073v2#bib.bib64)) provide controlled testbeds for evaluating reasoning under different dynamics. Our work builds on these advances, aiming to unify RL-based reasoning with structured agent training across symbolic and language-centric tasks. We summarize more related works in Appendix[B](https://arxiv.org/html/2504.20073v2#A2 "Appendix B Extended Related Work ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning").

6 Conclusions and Limitations
-----------------------------

We present RAGEN, a general-purpose system for training language agents with reinforcement learning in multi-turn, stochastic environments. Built upon the StarPO framework, RAGEN enables reasoning-guided trajectory optimization and reveals new challenges unique to agent training—such as gradient collapse, rollout drift, and reasoning degradation. Through extensive experiments, we identify key design principles for stabilizing training, including rollout filtering, gradient shaping, and reward-aware reasoning supervision. These insights provide a foundation for building more robust and generalizable LLM agents. Our framework offers a scalable platform for studying autonomous language agents across domains such as symbolic reasoning and web browsing. Limitations of our work include the focus on relatively small-scale tasks, the omission of established RL practices like replay buffers, and the absence of multimodal tasks—which we leave for future work.

Acknowledgements
----------------

We thank the DeepSeek team for providing the DeepSeek-R1 model and early conceptual inspirations. We are grateful to the veRL team for their infrastructure support, and to the TinyZero team for their discoveries that informed our initial exploration. We would like to appreciate insightful discussions with Han Liu, Xinyu Xing, Monica Lam, Li Erran Li, John Schulman, Akari Asai, Eiso Kant, Lu Lu, Runxin Xu, Zhihan Liu, Huajian Xin, Zijun Liu, Weiyi Liu, Weimin Wu, Yibo Wen, Jiarui Liu, Lorenzo Xiao, Ishan Mukherjee, Anabella Isaro, Haosen Sun, How-Yeh Wan, Lester Xue, Matthew Khoriaty, Haoxiang Sun, Jiajun Liu.

References
----------

*   Brockman et al. (2016) G.Brockman, V.Cheung, L.Pettersson, J.Schneider, J.Schulman, J.Tang, and W.Zaremba. Openai gym, 2016. URL [https://arxiv.org/abs/1606.01540](https://arxiv.org/abs/1606.01540). 
*   Chen et al. (2021) L.Chen, K.Lu, A.Rajeswaran, K.Lee, A.Grover, M.Laskin, P.Abbeel, A.Srinivas, and I.Mordatch. Decision transformer: Reinforcement learning via sequence modeling. _arXiv preprint arXiv:2106.01345_, 2021. 
*   Chen et al. (2023) W.Chen, Y.Su, J.Zuo, C.Yang, C.Yuan, C.Qian, C.-M. Chan, Y.Qin, Y.Lu, R.Xie, et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents. _arXiv preprint arXiv:2308.10848_, 2(4):6, 2023. 
*   Christiano et al. (2023) P.Christiano, J.Leike, T.B. Brown, M.Martic, S.Legg, and D.Amodei. Deep reinforcement learning from human preferences, 2023. URL [https://arxiv.org/abs/1706.03741](https://arxiv.org/abs/1706.03741). 
*   DeepSeek-AI (2024) DeepSeek-AI. Deepseek LLM: scaling open-source language models with longtermism. _CoRR_, abs/2401.02954, 2024. URL [https://doi.org/10.48550/arXiv.2401.02954](https://doi.org/10.48550/arXiv.2401.02954). 
*   DeepSeek-AI et al. (2024) DeepSeek-AI, Q.Zhu, D.Guo, Z.Shao, D.Yang, P.Wang, R.Xu, Y.Wu, Y.Li, H.Gao, S.Ma, W.Zeng, X.Bi, Z.Gu, H.Xu, D.Dai, K.Dong, L.Zhang, Y.Piao, Z.Gou, Z.Xie, Z.Hao, B.Wang, J.Song, D.Chen, X.Xie, K.Guan, Y.You, A.Liu, Q.Du, W.Gao, X.Lu, Q.Chen, Y.Wang, C.Deng, J.Li, C.Zhao, C.Ruan, F.Luo, and W.Liang. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence, 2024. URL [https://arxiv.org/abs/2406.11931](https://arxiv.org/abs/2406.11931). 
*   DeepSeek-AI et al. (2025) DeepSeek-AI, D.Guo, D.Yang, H.Zhang, J.Song, R.Zhang, R.Xu, Q.Zhu, S.Ma, P.Wang, X.Bi, X.Zhang, X.Yu, Y.Wu, Z.F. Wu, Z.Gou, Z.Shao, Z.Li, Z.Gao, A.Liu, B.Xue, B.Wang, B.Wu, B.Feng, C.Lu, C.Zhao, C.Deng, C.Zhang, C.Ruan, D.Dai, D.Chen, D.Ji, E.Li, F.Lin, F.Dai, F.Luo, G.Hao, G.Chen, G.Li, H.Zhang, H.Bao, H.Xu, H.Wang, H.Ding, H.Xin, H.Gao, H.Qu, H.Li, J.Guo, J.Li, J.Wang, J.Chen, J.Yuan, J.Qiu, J.Li, J.L. Cai, J.Ni, J.Liang, J.Chen, K.Dong, K.Hu, K.Gao, K.Guan, K.Huang, K.Yu, L.Wang, L.Zhang, L.Zhao, L.Wang, L.Zhang, L.Xu, L.Xia, M.Zhang, M.Zhang, M.Tang, M.Li, M.Wang, M.Li, N.Tian, P.Huang, P.Zhang, Q.Wang, Q.Chen, Q.Du, R.Ge, R.Zhang, R.Pan, R.Wang, R.J. Chen, R.L. Jin, R.Chen, S.Lu, S.Zhou, S.Chen, S.Ye, S.Wang, S.Yu, S.Zhou, S.Pan, S.S. Li, S.Zhou, S.Wu, S.Ye, T.Yun, T.Pei, T.Sun, T.Wang, W.Zeng, W.Zhao, W.Liu, W.Liang, W.Gao, W.Yu, W.Zhang, W.L. Xiao, W.An, X.Liu, X.Wang, X.Chen, X.Nie, X.Cheng, X.Liu, X.Xie, X.Liu, X.Yang, X.Li, X.Su, X.Lin, X.Q. Li, X.Jin, X.Shen, X.Chen, X.Sun, X.Wang, X.Song, X.Zhou, X.Wang, X.Shan, Y.K. Li, Y.Q. Wang, Y.X. Wei, Y.Zhang, Y.Xu, Y.Li, Y.Zhao, Y.Sun, Y.Wang, Y.Yu, Y.Zhang, Y.Shi, Y.Xiong, Y.He, Y.Piao, Y.Wang, Y.Tan, Y.Ma, Y.Liu, Y.Guo, Y.Ou, Y.Wang, Y.Gong, Y.Zou, Y.He, Y.Xiong, Y.Luo, Y.You, Y.Liu, Y.Zhou, Y.X. Zhu, Y.Xu, Y.Huang, Y.Li, Y.Zheng, Y.Zhu, Y.Ma, Y.Tang, Y.Zha, Y.Yan, Z.Z. Ren, Z.Ren, Z.Sha, Z.Fu, Z.Xu, Z.Xie, Z.Zhang, Z.Hao, Z.Ma, Z.Yan, Z.Wu, Z.Gu, Z.Zhu, Z.Liu, Z.Li, Z.Xie, Z.Song, Z.Pan, Z.Huang, Z.Xu, Z.Zhang, and Z.Zhang. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL [https://arxiv.org/abs/2501.12948](https://arxiv.org/abs/2501.12948). 
*   Dell’Aversana (2021) P.Dell’Aversana. The frozen lake problem. an example of optimization policy, 12 2021. 
*   Du et al. (2023) Y.Du, S.Li, A.Torralba, J.B. Tenenbaum, and I.Mordatch. Improving factuality and reasoning in language models through multiagent debate. _arXiv preprint arXiv:2305.14325_, 2023. 
*   Feng et al. (2025) L.Feng, Z.Xue, T.Liu, and B.An. Group-in-group policy optimization for llm agent training, 2025. URL [https://arxiv.org/abs/2505.10978](https://arxiv.org/abs/2505.10978). 
*   Gao et al. (2024) Z.Gao, W.Zhan, J.D. Chang, G.Swamy, K.Brantley, J.D. Lee, and W.Sun. Regressing the relative future: Efficient policy optimization for multi-turn rlhf, 2024. URL [https://arxiv.org/abs/2410.04612](https://arxiv.org/abs/2410.04612). 
*   Goyal et al. (2024) S.Goyal, Z.Ji, A.S. Rawat, A.K. Menon, S.Kumar, and V.Nagarajan. Think before you speak: Training language models with pause tokens, 2024. URL [https://arxiv.org/abs/2310.02226](https://arxiv.org/abs/2310.02226). 
*   Haarnoja et al. (2018) T.Haarnoja, A.Zhou, P.Abbeel, and S.Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018. URL [https://arxiv.org/abs/1801.01290](https://arxiv.org/abs/1801.01290). 
*   Hao et al. (2023a) S.Hao, Y.Gu, H.Ma, J.J. Hong, Z.Wang, D.Z. Wang, and Z.Hu. Reasoning with language model is planning with world model. _arXiv preprint arXiv:2305.14992_, 2023a. 
*   Hao et al. (2023b) S.Hao, T.Liu, Z.Wang, and Z.Hu. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. _Advances in neural information processing systems_, 36:45870–45894, 2023b. 
*   Havrilla et al. (2024) A.Havrilla, Y.Du, S.C. Raparthy, C.Nalmpantis, J.Dwivedi-Yu, M.Zhuravinskyi, E.Hambro, S.Sukhbaatar, and R.Raileanu. Teaching large language models to reason with reinforcement learning, 2024. URL [https://arxiv.org/abs/2403.04642](https://arxiv.org/abs/2403.04642). 
*   He et al. (2025) C.He, B.Zou, X.Li, J.Chen, and H.M. Junliang Xing. Enhancing llm reasoning with multi-path collaborative reactive and reflection agents, 2025. URL [https://arxiv.org/abs/2501.00430](https://arxiv.org/abs/2501.00430). 
*   Herel and Mikolov (2024) D.Herel and T.Mikolov. Thinking tokens for language modeling, 2024. URL [https://arxiv.org/abs/2405.08644](https://arxiv.org/abs/2405.08644). 
*   Hu et al. (2021) E.J. Hu, Y.Shen, P.Wallis, Z.Allen-Zhu, Y.Li, S.Wang, L.Wang, and W.Chen. Lora: Low-rank adaptation of large language models, 2021. URL [https://arxiv.org/abs/2106.09685](https://arxiv.org/abs/2106.09685). 
*   Hu et al. (2025a) J.Hu, Y.Zhang, Q.Han, D.Jiang, X.Zhang, and H.-Y. Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model, 2025a. URL [https://arxiv.org/abs/2503.24290](https://arxiv.org/abs/2503.24290). 
*   Hu et al. (2025b) L.Hu, M.Huo, Y.Zhang, H.Yu, E.P. Xing, I.Stoica, T.Rosing, H.Jin, and H.Zhang. lmgame-bench: How good are llms at playing games?, 2025b. URL [https://arxiv.org/abs/2505.15146](https://arxiv.org/abs/2505.15146). 
*   Jimenez et al. (2024) C.E. Jimenez, J.Yang, A.Wettig, S.Yao, K.Pei, O.Press, and K.Narasimhan. SWE-bench: Can language models resolve real-world github issues? _arXiv preprint arXiv:2310.06770_, 2024. 
*   Jin et al. (2025a) B.Jin, J.Yoon, P.Kargupta, S.O. Arik, and J.Han. An empirical study on reinforcement learning for reasoning-search interleaved llm agents, 2025a. URL [https://arxiv.org/abs/2505.15117](https://arxiv.org/abs/2505.15117). 
*   Jin et al. (2025b) B.Jin, H.Zeng, Z.Yue, J.Yoon, S.Arik, D.Wang, H.Zamani, and J.Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025b. URL [https://arxiv.org/abs/2503.09516](https://arxiv.org/abs/2503.09516). 
*   Junghanns and Schaeffer (2001) A.Junghanns and J.Schaeffer. Sokoban: Enhancing general single-agent search methods using domain knowledge. _Artificial Intelligence_, 129(1):219–251, 2001. ISSN 0004-3702. [https://doi.org/10.1016/S0004-3702(01)00109-6](https://arxiv.org/doi.org/https://doi.org/10.1016/S0004-3702(01)00109-6). URL [https://www.sciencedirect.com/science/article/pii/S0004370201001096](https://www.sciencedirect.com/science/article/pii/S0004370201001096). 
*   Kool et al. (2019) W.Kool, H.van Hoof, and M.Welling. Buy 4 REINFORCE samples, get a baseline for free!, 2019. URL [https://openreview.net/forum?id=r1lgTGL5DE](https://openreview.net/forum?id=r1lgTGL5DE). 
*   Kumar et al. (2024) A.Kumar, V.Zhuang, R.Agarwal, Y.Su, J.D. Co-Reyes, A.Singh, K.Baumli, S.Iqbal, C.Bishop, R.Roelofs, L.M. Zhang, K.McKinney, D.Shrivastava, C.Paduraru, G.Tucker, D.Precup, F.Behbahani, and A.Faust. Training language models to self-correct via reinforcement learning, 2024. URL [https://arxiv.org/abs/2409.12917](https://arxiv.org/abs/2409.12917). 
*   Li et al. (2023) G.Li, H.Hammoud, H.Itani, D.Khizbullin, and B.Ghanem. Camel: Communicative agents for" mind" exploration of large language model society. _Advances in Neural Information Processing Systems_, 36:51991–52008, 2023. 
*   Li et al. (2025) M.Li, S.Zhao, Q.Wang, K.Wang, Y.Zhou, S.Srivastava, C.Gokmen, T.Lee, L.E. Li, R.Zhang, W.Liu, P.Liang, L.Fei-Fei, J.Mao, and J.Wu. Embodied agent interface: Benchmarking llms for embodied decision making, 2025. URL [https://arxiv.org/abs/2410.07166](https://arxiv.org/abs/2410.07166). 
*   Lightman et al. (2023) H.Lightman, V.Kosaraju, Y.Burda, H.Edwards, B.Baker, T.Lee, J.Leike, J.Schulman, I.Sutskever, and K.Cobbe. Let’s verify step by step. _arXiv preprint arXiv:2305.20050_, 2023. 
*   Lin et al. (2024a) B.Y. Lin, Y.Fu, K.Yang, F.Brahman, S.Huang, C.Bhagavatula, P.Ammanabrolu, Y.Choi, and X.Ren. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. _Advances in Neural Information Processing Systems_, 36, 2024a. 
*   Lin et al. (2023) J.Lin, H.Zhao, A.Zhang, Y.Wu, H.Ping, and Q.Chen. Agentsims: An open-source sandbox for large language model evaluation, 2023. URL [https://arxiv.org/abs/2308.04026](https://arxiv.org/abs/2308.04026). 
*   Lin et al. (2024b) J.Lin, H.Gao, X.Feng, R.Xu, C.Wang, M.Zhang, L.Guo, and S.Xu. Advances in embodied navigation using large language models: A survey, 2024b. URL [https://arxiv.org/abs/2311.00530](https://arxiv.org/abs/2311.00530). 
*   Liu et al. (2023) B.Liu, Y.Jiang, X.Zhang, Q.Liu, S.Zhang, J.Biswas, and P.Stone. Llm+ p: Empowering large language models with optimal planning proficiency. _arXiv preprint arXiv:2304.11477_, 2023. 
*   Liu et al. (2024) Z.Liu, H.Hu, S.Zhang, H.Guo, S.Ke, B.Liu, and Z.Wang. Reason for future, act for now: A principled framework for autonomous llm agents with provable sample efficiency, 2024. URL [https://arxiv.org/abs/2309.17382](https://arxiv.org/abs/2309.17382). 
*   Liu et al. (2025) Z.Liu, C.Chen, W.Li, P.Qi, C.D. Tianyu Pang, W.S. Lee, and M.Lin. Understanding r1-zero-like training: A critical perspective, 2025. URL [https://arxiv.org/abs/2503.20783](https://arxiv.org/abs/2503.20783). 
*   Nguyen et al. (2024) M.Nguyen, A.Baker, C.Neo, A.Roush, A.Kirsch, and R.Shwartz-Ziv. Turning up the heat: Min-p sampling for creative and coherent llm outputs, 2024. URL [https://arxiv.org/abs/2407.01082](https://arxiv.org/abs/2407.01082). 
*   OpenAI (2024) OpenAI. Introducing ChatGPT o1, 2024. URL [https://openai.com/o1/](https://openai.com/o1/). Accessed: 2025-02-15. 
*   Ouyang et al. (2022) L.Ouyang, J.Wu, X.Jiang, D.Almeida, C.L. Wainwright, P.Mishkin, C.Zhang, S.Agarwal, K.Slama, A.Ray, J.Schulman, J.Hilton, F.Kelton, L.Miller, M.Simens, A.Askell, P.Welinder, P.Christiano, J.Leike, and R.Lowe. Training language models to follow instructions with human feedback, 2022. URL [https://arxiv.org/abs/2203.02155](https://arxiv.org/abs/2203.02155). 
*   Pan et al. (2025) J.Pan, J.Zhang, X.Wang, L.Yuan, H.Peng, and A.Suhr. Tinyzero. https://github.com/Jiayi-Pan/TinyZero, 2025. Accessed: 2025-01-24. 
*   Park et al. (2023) J.S. Park, J.C. O’Brien, C.J. Cai, M.R. Morris, P.Liang, and M.S. Bernstein. Generative agents: Interactive simulacra of human behavior, 2023. URL [https://arxiv.org/abs/2304.03442](https://arxiv.org/abs/2304.03442). 
*   Pfau et al. (2024) J.Pfau, W.Merrill, and S.R. Bowman. Let’s think dot by dot: Hidden computation in transformer language models, 2024. URL [https://arxiv.org/abs/2404.15758](https://arxiv.org/abs/2404.15758). 
*   Qi et al. (2025) Z.Qi, X.Liu, I.L. Iong, H.Lai, X.Sun, W.Zhao, Y.Yang, X.Yang, J.Sun, S.Yao, T.Zhang, W.Xu, J.Tang, and Y.Dong. Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning, 2025. URL [https://arxiv.org/abs/2411.02337](https://arxiv.org/abs/2411.02337). 
*   Qin et al. (2025) Y.Qin, Y.Ye, J.Fang, H.Wang, S.Liang, S.Tian, J.Zhang, J.Li, Y.Li, S.Huang, W.Zhong, K.Li, J.Yang, Y.Miao, W.Lin, L.Liu, X.Jiang, Q.Ma, J.Li, X.Xiao, K.Cai, C.Li, Y.Zheng, C.Jin, C.Li, X.Zhou, M.Wang, H.Chen, Z.Li, H.Yang, H.Liu, F.Lin, T.Peng, X.Liu, and G.Shi. Ui-tars: Pioneering automated gui interaction with native agents, 2025. URL [https://arxiv.org/abs/2501.12326](https://arxiv.org/abs/2501.12326). 
*   Schulman et al. (2017) J.Schulman, F.Wolski, P.Dhariwal, A.Radford, and O.Klimov. Proximal policy optimization algorithms, 2017. URL [https://arxiv.org/abs/1707.06347](https://arxiv.org/abs/1707.06347). 
*   Schulman et al. (2018) J.Schulman, P.Moritz, S.Levine, M.Jordan, and P.Abbeel. High-dimensional continuous control using generalized advantage estimation, 2018. URL [https://arxiv.org/abs/1506.02438](https://arxiv.org/abs/1506.02438). 
*   Settles (2009) B.Settles. Active learning literature survey. 2009. 
*   Shao et al. (2024a) Z.Shao, P.Wang, Q.Zhu, R.Xu, J.Song, X.Bi, H.Zhang, M.Zhang, Y.K. Li, Y.Wu, and D.Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024a. URL [https://arxiv.org/abs/2402.03300](https://arxiv.org/abs/2402.03300). 
*   Shao et al. (2024b) Z.Shao, P.Wang, Q.Zhu, R.Xu, J.Song, M.Zhang, Y.Li, Y.Wu, and D.Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. _arXiv preprint arXiv:2402.03300_, 2024b. 
*   Shen et al. (2024) Y.Shen, K.Song, X.Tan, D.Li, W.Lu, and Y.Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. _Advances in Neural Information Processing Systems_, 36, 2024. 
*   Shinn et al. (2024) N.Shinn, F.Cassano, A.Gopinath, K.Narasimhan, and S.Yao. Reflexion: Language agents with verbal reinforcement learning. _Advances in Neural Information Processing Systems_, 36, 2024. 
*   Shumailov et al. (2024) I.Shumailov, Z.Shumaylov, Y.Zhao, N.Papernot, R.J. Anderson, and Y.Gal. Ai models collapse when trained on recursively generated data. _Nat._, 631(8022):755–759, July 2024. URL [https://doi.org/10.1038/s41586-024-07566-y](https://doi.org/10.1038/s41586-024-07566-y). 
*   Song et al. (2024) Y.Song, D.Yin, X.Yue, J.Huang, S.Li, and B.Y. Lin. Trial and error: Exploration-based trajectory optimization for llm agents, 2024. 
*   Wang et al. (2025) C.J. Wang, D.Lee, C.Menghini, J.Mols, J.Doughty, A.Khoja, J.Lynch, S.Hendryx, S.Yue, and D.Hendrycks. Enigmaeval: A benchmark of long multimodal reasoning challenges, 2025. URL [https://arxiv.org/abs/2502.08859](https://arxiv.org/abs/2502.08859). 
*   Wang* et al. (2025) K.Wang*, P.Zhang*, Z.Wang*, Q.Wang*, Y.Gao*, L.Li*, Z.Yang, C.Wan, H.Chen, Y.Lu, and M.Li. Vagen: Training vlm agents with multi-turn reinforcement learning, 2025. URL [https://github.com/RAGEN-AI/VAGEN](https://github.com/RAGEN-AI/VAGEN). 
*   Wang et al. (2024a) Q.Wang, Z.Wang, Y.Su, H.Tong, and Y.Song. Rethinking the bounds of llm reasoning: Are multi-agent discussions the key? _arXiv preprint arXiv:2402.18272_, 2024a. 
*   Wang et al. (2024b) X.Wang, Z.Wang, J.Liu, Y.Chen, L.Yuan, H.Peng, and H.Ji. Mint: Evaluating llms in multi-turn interaction with tools and language feedback, 2024b. URL [https://arxiv.org/abs/2309.10691](https://arxiv.org/abs/2309.10691). 
*   Wei et al. (2025) Z.Wei, W.Yao, Y.Liu, W.Zhang, Q.Lu, L.Qiu, C.Yu, P.Xu, C.Zhang, B.Yin, H.Yun, and L.Li. Webagent-r1: Training web agents via end-to-end multi-turn reinforcement learning, 2025. URL [https://arxiv.org/abs/2505.16421](https://arxiv.org/abs/2505.16421). 
*   Wu et al. (2023) Q.Wu, G.Bansal, J.Zhang, Y.Wu, S.Zhang, E.Zhu, B.Li, L.Jiang, X.Zhang, and C.Wang. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. _arXiv preprint arXiv:2308.08155_, 2023. 
*   Xi et al. (2024) J.Xi, Y.He, J.Yang, Y.Dai, and J.Chai. Teaching embodied reinforcement learning agents: Informativeness and diversity of language use, 2024. URL [https://arxiv.org/abs/2410.24218](https://arxiv.org/abs/2410.24218). 
*   Xie et al. (2023) T.Xie, F.Zhou, Z.Cheng, P.Shi, L.Weng, Y.Liu, T.J. Hua, J.Zhao, Q.Liu, C.Liu, et al. Openagents: An open platform for language agents in the wild. _arXiv preprint arXiv:2310.10634_, 2023. 
*   Xu et al. (2023) B.Xu, Z.Peng, B.Lei, S.Mukherjee, Y.Liu, and D.Xu. Rewoo: Decoupling reasoning from observations for efficient augmented language models. _arXiv preprint arXiv:2305.18323_, 2023. 
*   Yang et al. (2024) A.Yang, B.Yang, B.Zhang, B.Hui, B.Zheng, B.Yu, C.Li, D.Liu, F.Huang, H.Wei, H.Lin, J.Yang, J.Tu, J.Zhang, J.Yang, J.Yang, J.Zhou, J.Lin, K.Dang, K.Lu, K.Bao, K.Yang, L.Yu, M.Li, M.Xue, P.Zhang, Q.Zhu, R.Men, R.Lin, T.Li, T.Xia, X.Ren, X.Ren, Y.Fan, Y.Su, Y.Zhang, Y.Wan, Y.Liu, Z.Cui, Z.Zhang, and Z.Qiu. Qwen2.5 technical report. _arXiv preprint arXiv:2412.15115_, 2024. 
*   Yao et al. (2022a) S.Yao, H.Chen, J.Yang, and K.Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. _Advances in Neural Information Processing Systems_, 35:20744–20757, 2022a. 
*   Yao et al. (2022b) S.Yao, J.Zhao, D.Yu, N.Du, I.Shafran, K.Narasimhan, and Y.Cao. React: Synergizing reasoning and acting in language models. _arXiv preprint arXiv:2210.03629_, 2022b. 
*   Yu et al. (2025) Q.Yu, Z.Zhang, R.Zhu, Y.Yuan, X.Zuo, Y.Yue, T.Fan, G.Liu, L.Liu, X.Liu, H.Lin, Z.Lin, B.Ma, G.Sheng, Y.Tong, C.Zhang, M.Zhang, W.Zhang, H.Zhu, J.Zhu, J.Chen, J.Chen, C.Wang, H.Yu, W.Dai, Y.Song, X.Wei, H.Zhou, J.Liu, W.-Y. Ma, Y.-Q. Zhang, L.Yan, M.Qiao, Y.Wu, and M.Wang. Dapo: An open-source llm reinforcement learning system at scale, 2025. URL [https://arxiv.org/abs/2503.14476](https://arxiv.org/abs/2503.14476). 
*   Zelikman et al. (2022) E.Zelikman, Y.Wu, J.Mu, and N.D. Goodman. Star: Bootstrapping reasoning with reasoning, 2022. URL [https://arxiv.org/abs/2203.14465](https://arxiv.org/abs/2203.14465). 
*   Zeng et al. (2025a) S.Zeng, Q.Wei, W.Brown, O.Frunza, Y.Nevmyvaka, and M.Hong. Reinforcing multi-turn reasoning in llm agents via turn-level credit assignment, 2025a. URL [https://arxiv.org/abs/2505.11821](https://arxiv.org/abs/2505.11821). 
*   Zeng et al. (2025b) W.Zeng, Y.Huang, Q.Liu, W.Liu, K.He, Z.Ma, and J.He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025b. URL [https://arxiv.org/abs/2503.18892](https://arxiv.org/abs/2503.18892). 
*   Zeng et al. (2025c) W.Zeng, Y.Huang, W.Liu, K.He, Q.Liu, Z.Ma, and J.He. 7b model and 8k examples: Emerging reasoning with reinforcement learning is both effective and efficient. [https://hkust-nlp.notion.site/simplerl-reason](https://hkust-nlp.notion.site/simplerl-reason), 2025c. Notion Blog. 
*   Zhang et al. (2025) Z.Zhang, C.Zheng, Y.Wu, B.Zhang, R.Lin, B.Yu, D.Liu, J.Zhou, and J.Lin. The lessons of developing process reward models in mathematical reasoning, 2025. URL [https://arxiv.org/abs/2501.07301](https://arxiv.org/abs/2501.07301). 
*   Zhou et al. (2024) Y.Zhou, A.Zanette, J.Pan, S.Levine, and A.Kumar. Archer: Training language model agents via hierarchical multi-turn rl, 2024. URL [https://arxiv.org/abs/2402.19446](https://arxiv.org/abs/2402.19446). 
*   Zhuang et al. (2023) Y.Zhuang, X.Chen, T.Yu, S.Mitra, V.Bursztyn, R.A. Rossi, S.Sarkhel, and C.Zhang. Toolchain*: Efficient action space navigation in large language models with a* search. _arXiv preprint arXiv:2310.13227_, 2023. 

Appendix A Background of Reinforcement Learning
-----------------------------------------------

Reinforcement learning (RL) enables foundation models to learn through interaction and reward signals. The general RL objective is:

J⁢(θ)=𝔼 s∼𝒟,a∼π θ(⋅|s)⁢[R⁢(s,a)],J(\theta)=\mathbb{E}_{s\sim\mathcal{D},a\sim\pi_{\theta}(\cdot|s)}[R(s,a)],italic_J ( italic_θ ) = blackboard_E start_POSTSUBSCRIPT italic_s ∼ caligraphic_D , italic_a ∼ italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( ⋅ | italic_s ) end_POSTSUBSCRIPT [ italic_R ( italic_s , italic_a ) ] ,(8)

where π θ subscript 𝜋 𝜃\pi_{\theta}italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT is the policy, s 𝑠 s italic_s is the input prompt, a 𝑎 a italic_a is the response, and R⁢(s,a)𝑅 𝑠 𝑎 R(s,a)italic_R ( italic_s , italic_a ) is the reward function evaluating response quality.

Common approaches use reward modeling and policy optimization for RL. Proximal Policy Optimization (PPO) (Schulman et al., [2017](https://arxiv.org/html/2504.20073v2#bib.bib45)) stabilizes training through probability ratio clipping and advantage estimation. The probability ratio is defined as:

ρ t⁢(θ)=π θ⁢(a t|s t)π θ o⁢l⁢d⁢(a t|s t)subscript 𝜌 𝑡 𝜃 subscript 𝜋 𝜃 conditional subscript 𝑎 𝑡 subscript 𝑠 𝑡 subscript 𝜋 subscript 𝜃 𝑜 𝑙 𝑑 conditional subscript 𝑎 𝑡 subscript 𝑠 𝑡\rho_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{old}}(a_{t}|s_{% t})}italic_ρ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ( italic_θ ) = divide start_ARG italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT | italic_s start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) end_ARG start_ARG italic_π start_POSTSUBSCRIPT italic_θ start_POSTSUBSCRIPT italic_o italic_l italic_d end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( italic_a start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT | italic_s start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) end_ARG(9)

The PPO objective uses this ratio with clipping:

J P⁢P⁢O⁢(θ)=𝔼 t⁢[min⁡(ρ i⁢A i,ρ i^⁢A i)−β⁢D K⁢L],subscript 𝐽 𝑃 𝑃 𝑂 𝜃 subscript 𝔼 𝑡 delimited-[]subscript 𝜌 𝑖 subscript 𝐴 𝑖^subscript 𝜌 𝑖 subscript 𝐴 𝑖 𝛽 subscript 𝐷 𝐾 𝐿 J_{PPO}(\theta)=\mathbb{E}_{t}[\min(\rho_{i}A_{i},\hat{\rho_{i}}A_{i})-\beta D% _{KL}],italic_J start_POSTSUBSCRIPT italic_P italic_P italic_O end_POSTSUBSCRIPT ( italic_θ ) = blackboard_E start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT [ roman_min ( italic_ρ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_A start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , over^ start_ARG italic_ρ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG italic_A start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) - italic_β italic_D start_POSTSUBSCRIPT italic_K italic_L end_POSTSUBSCRIPT ] ,(10)

with probability ratio ρ i=π θ⁢(o i|q)π θ o⁢l⁢d⁢(o i|q)subscript 𝜌 𝑖 subscript 𝜋 𝜃 conditional subscript 𝑜 𝑖 𝑞 subscript 𝜋 subscript 𝜃 𝑜 𝑙 𝑑 conditional subscript 𝑜 𝑖 𝑞\rho_{i}=\frac{\pi_{\theta}(o_{i}|q)}{\pi_{\theta_{old}}(o_{i}|q)}italic_ρ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT = divide start_ARG italic_π start_POSTSUBSCRIPT italic_θ end_POSTSUBSCRIPT ( italic_o start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | italic_q ) end_ARG start_ARG italic_π start_POSTSUBSCRIPT italic_θ start_POSTSUBSCRIPT italic_o italic_l italic_d end_POSTSUBSCRIPT end_POSTSUBSCRIPT ( italic_o start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT | italic_q ) end_ARG and clipped ratio ρ i^=clip⁢(ρ i,1−ε,1+ε)^subscript 𝜌 𝑖 clip subscript 𝜌 𝑖 1 𝜀 1 𝜀\hat{\rho_{i}}=\text{clip}(\rho_{i},1-\varepsilon,1+\varepsilon)over^ start_ARG italic_ρ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG = clip ( italic_ρ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , 1 - italic_ε , 1 + italic_ε ).

For advantage estimation, Generalized Advantage Estimation (GAE) (Schulman et al., [2018](https://arxiv.org/html/2504.20073v2#bib.bib46)) computes:

A t G⁢A⁢E⁢(γ,λ)=∑l=0∞(γ⁢λ)l⁢δ t+l superscript subscript 𝐴 𝑡 𝐺 𝐴 𝐸 𝛾 𝜆 superscript subscript 𝑙 0 superscript 𝛾 𝜆 𝑙 subscript 𝛿 𝑡 𝑙 A_{t}^{GAE(\gamma,\lambda)}=\sum_{l=0}^{\infty}(\gamma\lambda)^{l}\delta_{t+l}italic_A start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT start_POSTSUPERSCRIPT italic_G italic_A italic_E ( italic_γ , italic_λ ) end_POSTSUPERSCRIPT = ∑ start_POSTSUBSCRIPT italic_l = 0 end_POSTSUBSCRIPT start_POSTSUPERSCRIPT ∞ end_POSTSUPERSCRIPT ( italic_γ italic_λ ) start_POSTSUPERSCRIPT italic_l end_POSTSUPERSCRIPT italic_δ start_POSTSUBSCRIPT italic_t + italic_l end_POSTSUBSCRIPT(11)

where δ t=r t+γ⁢V⁢(s t+1)−V⁢(s t)subscript 𝛿 𝑡 subscript 𝑟 𝑡 𝛾 𝑉 subscript 𝑠 𝑡 1 𝑉 subscript 𝑠 𝑡\delta_{t}=r_{t}+\gamma V(s_{t+1})-V(s_{t})italic_δ start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT = italic_r start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT + italic_γ italic_V ( italic_s start_POSTSUBSCRIPT italic_t + 1 end_POSTSUBSCRIPT ) - italic_V ( italic_s start_POSTSUBSCRIPT italic_t end_POSTSUBSCRIPT ) is the TD error, and (γ,λ)𝛾 𝜆(\gamma,\lambda)( italic_γ , italic_λ ) control the bias-variance tradeoff.

Recently, DeepSeek-R1-Zero DeepSeek-AI ([2024](https://arxiv.org/html/2504.20073v2#bib.bib5)) implements this paradigm through Group Relative Policy Optimization (GRPO), sampling G 𝐺 G italic_G outputs {o i}subscript 𝑜 𝑖\{o_{i}\}{ italic_o start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT } [consisting of reasoning and actions] for each prompt and optimizes:

J G⁢R⁢P⁢O⁢(θ)=𝔼 q,{o i}⁢[J g⁢r⁢o⁢u⁢p⁢(θ)],subscript 𝐽 𝐺 𝑅 𝑃 𝑂 𝜃 subscript 𝔼 𝑞 subscript 𝑜 𝑖 delimited-[]subscript 𝐽 𝑔 𝑟 𝑜 𝑢 𝑝 𝜃 J_{GRPO}(\theta)=\mathbb{E}_{q,\{o_{i}\}}[J_{group}(\theta)],italic_J start_POSTSUBSCRIPT italic_G italic_R italic_P italic_O end_POSTSUBSCRIPT ( italic_θ ) = blackboard_E start_POSTSUBSCRIPT italic_q , { italic_o start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT } end_POSTSUBSCRIPT [ italic_J start_POSTSUBSCRIPT italic_g italic_r italic_o italic_u italic_p end_POSTSUBSCRIPT ( italic_θ ) ] ,(12)

where:

J g⁢r⁢o⁢u⁢p⁢(θ)=1 G⁢∑i=1 G min⁡(ρ i⁢A i,ρ i^⁢A i)−β⁢D K⁢L,subscript 𝐽 𝑔 𝑟 𝑜 𝑢 𝑝 𝜃 1 𝐺 subscript superscript 𝐺 𝑖 1 subscript 𝜌 𝑖 subscript 𝐴 𝑖^subscript 𝜌 𝑖 subscript 𝐴 𝑖 𝛽 subscript 𝐷 𝐾 𝐿 J_{group}(\theta)=\frac{1}{G}\sum^{G}_{i=1}\min(\rho_{i}A_{i},\hat{\rho_{i}}A_% {i})-\beta D_{KL},italic_J start_POSTSUBSCRIPT italic_g italic_r italic_o italic_u italic_p end_POSTSUBSCRIPT ( italic_θ ) = divide start_ARG 1 end_ARG start_ARG italic_G end_ARG ∑ start_POSTSUPERSCRIPT italic_G end_POSTSUPERSCRIPT start_POSTSUBSCRIPT italic_i = 1 end_POSTSUBSCRIPT roman_min ( italic_ρ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT italic_A start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT , over^ start_ARG italic_ρ start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT end_ARG italic_A start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT ) - italic_β italic_D start_POSTSUBSCRIPT italic_K italic_L end_POSTSUBSCRIPT ,(13)

while mostly similar to Eq. 3, the GRPO advantage is neural-model free and calculated as:

A i=r i−mean⁢({r j})std⁢({r j}).subscript 𝐴 𝑖 subscript 𝑟 𝑖 mean subscript 𝑟 𝑗 std subscript 𝑟 𝑗 A_{i}=\frac{r_{i}-\text{mean}(\{r_{j}\})}{\text{std}(\{r_{j}\})}.italic_A start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT = divide start_ARG italic_r start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT - mean ( { italic_r start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT } ) end_ARG start_ARG std ( { italic_r start_POSTSUBSCRIPT italic_j end_POSTSUBSCRIPT } ) end_ARG .(14)

Using rule-based rewards r i subscript 𝑟 𝑖 r_{i}italic_r start_POSTSUBSCRIPT italic_i end_POSTSUBSCRIPT, this pure RL approach demonstrates emergent reasoning behaviors.

Appendix B Extended Related Work
--------------------------------

Reinforcement Learning for Reasoning in LLMs. Reinforcement learning (RL) on LLMs (Christiano et al., [2023](https://arxiv.org/html/2504.20073v2#bib.bib4); Ouyang et al., [2022](https://arxiv.org/html/2504.20073v2#bib.bib39); Chen et al., [2021](https://arxiv.org/html/2504.20073v2#bib.bib2); Havrilla et al., [2024](https://arxiv.org/html/2504.20073v2#bib.bib16)) has significantly improved LLMs’ reasoning capabilities. Notable approaches include the use of Proximal Policy Optimization Algorithms (PPO) (Schulman et al., [2017](https://arxiv.org/html/2504.20073v2#bib.bib45)) which maintains training stability while enhancing performance by clipping policy updates, Group Relative Policy Optimization (GRPO) (DeepSeek-AI et al., [2025](https://arxiv.org/html/2504.20073v2#bib.bib7)) for enhancing the ability of systematic problem-solving, actor-critic methods like SAC (Haarnoja et al., [2018](https://arxiv.org/html/2504.20073v2#bib.bib13)) and ArCHer (Zhou et al., [2024](https://arxiv.org/html/2504.20073v2#bib.bib72)) that leverages an critic to promote robust exploration and stability, and meta tokens (Goyal et al., [2024](https://arxiv.org/html/2504.20073v2#bib.bib12); Herel and Mikolov, [2024](https://arxiv.org/html/2504.20073v2#bib.bib18); Pfau et al., [2024](https://arxiv.org/html/2504.20073v2#bib.bib42)) for structured thinking. Other significant developments include Process Reward Model (PRM) (Zhang et al., [2025](https://arxiv.org/html/2504.20073v2#bib.bib71); Lightman et al., [2023](https://arxiv.org/html/2504.20073v2#bib.bib30)) and Monte Carlo Tree Search (MCTS) based approaches (Hao et al., [2023a](https://arxiv.org/html/2504.20073v2#bib.bib14)) for systematic problem-solving. On the other hand, recent advances in LLM reasoning have explored techniques to enable models to generate intermediate chain-of-thought rationales. In particular, STaR (Zelikman et al., [2022](https://arxiv.org/html/2504.20073v2#bib.bib67)) iteratively leverages a small set of rationale examples along with a large dataset without rationales. Recent work like SimpleRL-Zoo (Zeng et al., [2025b](https://arxiv.org/html/2504.20073v2#bib.bib69)) DAPO (Yu et al., [2025](https://arxiv.org/html/2504.20073v2#bib.bib66)), RLOO (Kool et al., [2019](https://arxiv.org/html/2504.20073v2#bib.bib26)), Dr. GRPO (Liu et al., [2025](https://arxiv.org/html/2504.20073v2#bib.bib36)), and Open Reasoner Zero (Hu et al., [2025a](https://arxiv.org/html/2504.20073v2#bib.bib20)) all demonstrate that minimalist, reproducible RL techniques—featuring decoupled clipping, unbiased optimization, and simple reward schemes—can significantly enhance LLM reasoning performance.

Existing agent frameworks. LLM-based agent architectures have evolved from early reasoning-action frameworks (Yao et al., [2022b](https://arxiv.org/html/2504.20073v2#bib.bib65); Shinn et al., [2024](https://arxiv.org/html/2504.20073v2#bib.bib51); Xu et al., [2023](https://arxiv.org/html/2504.20073v2#bib.bib62); Lin et al., [2024a](https://arxiv.org/html/2504.20073v2#bib.bib31)) to structured approaches (Liu et al., [2024](https://arxiv.org/html/2504.20073v2#bib.bib35), [2023](https://arxiv.org/html/2504.20073v2#bib.bib34); Hao et al., [2023a](https://arxiv.org/html/2504.20073v2#bib.bib14); Zeng et al., [2025a](https://arxiv.org/html/2504.20073v2#bib.bib68)). Multi-agent systems (Du et al., [2023](https://arxiv.org/html/2504.20073v2#bib.bib9); Li et al., [2023](https://arxiv.org/html/2504.20073v2#bib.bib28); Chen et al., [2023](https://arxiv.org/html/2504.20073v2#bib.bib3); Wang et al., [2024a](https://arxiv.org/html/2504.20073v2#bib.bib56)) are designed for tasks with more complex interactions. Widely used platforms such as OpenAI Gym (Brockman et al., [2016](https://arxiv.org/html/2504.20073v2#bib.bib1)) and specialized environments including Sokoban (Junghanns and Schaeffer, [2001](https://arxiv.org/html/2504.20073v2#bib.bib25)), FrozenLake (Dell’Aversana, [2021](https://arxiv.org/html/2504.20073v2#bib.bib8)), and Webshop (Yao et al., [2022a](https://arxiv.org/html/2504.20073v2#bib.bib64)) provide diverse testbeds for evaluating these agents. Moreover, general-purpose systems (Shen et al., [2024](https://arxiv.org/html/2504.20073v2#bib.bib50); Wu et al., [2023](https://arxiv.org/html/2504.20073v2#bib.bib59); Hao et al., [2023b](https://arxiv.org/html/2504.20073v2#bib.bib15); Zhuang et al., [2023](https://arxiv.org/html/2504.20073v2#bib.bib73); Xie et al., [2023](https://arxiv.org/html/2504.20073v2#bib.bib61)) have enabled broad applications ranging from web navigation and search (Qi et al., [2025](https://arxiv.org/html/2504.20073v2#bib.bib43); Jin et al., [2025b](https://arxiv.org/html/2504.20073v2#bib.bib24); Wei et al., [2025](https://arxiv.org/html/2504.20073v2#bib.bib58); Jin et al., [2025a](https://arxiv.org/html/2504.20073v2#bib.bib23)), coding copilot (Jimenez et al., [2024](https://arxiv.org/html/2504.20073v2#bib.bib22); DeepSeek-AI et al., [2024](https://arxiv.org/html/2504.20073v2#bib.bib6); Wang et al., [2024b](https://arxiv.org/html/2504.20073v2#bib.bib57)) to GUI(Qin et al., [2025](https://arxiv.org/html/2504.20073v2#bib.bib44); Yao et al., [2022a](https://arxiv.org/html/2504.20073v2#bib.bib64)), Game(Hu et al., [2025b](https://arxiv.org/html/2504.20073v2#bib.bib21)) and embodied tasks (Lin et al., [2024b](https://arxiv.org/html/2504.20073v2#bib.bib33); Xi et al., [2024](https://arxiv.org/html/2504.20073v2#bib.bib60); Li et al., [2025](https://arxiv.org/html/2504.20073v2#bib.bib29); Feng et al., [2025](https://arxiv.org/html/2504.20073v2#bib.bib10)). Social interaction capabilities have been advanced through Generative Agents and AgentSims (Park et al., [2023](https://arxiv.org/html/2504.20073v2#bib.bib41); Lin et al., [2023](https://arxiv.org/html/2504.20073v2#bib.bib32)). Challenges persist in architectural complexity and self-correction (He et al., [2025](https://arxiv.org/html/2504.20073v2#bib.bib17)), especially for diverse, multi-step reasoning tasks (Wang et al., [2025](https://arxiv.org/html/2504.20073v2#bib.bib54); Nguyen et al., [2024](https://arxiv.org/html/2504.20073v2#bib.bib37); Song et al., [2024](https://arxiv.org/html/2504.20073v2#bib.bib53)).

Appendix C Detailed Experimental Settings
-----------------------------------------

### C.1 Environments and Tasks

![Image 9: Refer to caption](https://arxiv.org/html/2504.20073v2/x9.png)

Figure 8: Bi-Arm Bandits environment. The agent chooses between a low-risk arm (Phoenix) and a high-risk yet high-reward arm (Dragon), each linked to symbolic semantics. The agent learns to choose stable reward at early stages and reasons to pursue maximal expected reward and shift toward strategic risk-taking. 

![Image 10: Refer to caption](https://arxiv.org/html/2504.20073v2/extracted/6480928/figures/sokoban_env_example.png)

(a)Sokoban Environment. Actions are irreversible and the boxes cannot be pulled back which needs foresight and careful planning.

![Image 11: Refer to caption](https://arxiv.org/html/2504.20073v2/extracted/6480928/figures/frozenlake_env_example.png)

(b)FrozenLake Environment. Each agent move may randomly slip to a side direction, requiring the agent to adapt under uncertainty.

Figure 9: Sokoban and Frozen Lake environments. For each environment, the left shows the agent-observed text rendering; the right is a visual illustration. (a) Sokoban is a deterministic multi-turn puzzle where the agent pushes boxes onto targets. (b) Frozen Lake combines multi-turn reasoning and stochasticity where the agent needs to reach the gift to succeed. 

We construct a diverse four-environment testbed to evaluate LLM agents across key axes of decision-making complexity. Three environments—Bandit, Sokoban, and Frozen Lake—are symbolic, synthetic, and fully controllable, supporting clean analysis of RL learning from scratch. They are deliberately minimal and decoupled from real-world priors; even large models like GPT-4o perform poorly without training, highlighting the need for grounded policy learning. Complementarily, we include WebShop, a realistic multi-turn task involving natural language grounding and web navigation in a semi-structured interface. Together, the four environments enable systematic study of reasoning, training stability, and generalization in agentic LLMs across symbolic and open-domain settings.

Each environment stresses a distinct capability: Bandits tests reasoning under uncertainty, Sokoban emphasizes irreversible long-horizon planning, Frozen Lake involves stochastic transitions, and WebShop requires language understanding and goal-directed interaction.

##### Bi-Arm Bandits.

We design this environment to evaluate whether agents can form risk-sensitive hypotheses and revise them based on training. At each step, the agent must choose between two semantically symbolic options—e.g., “Dragon” vs. “Phoenix”—each linked to a fixed reward distribution (Figure[8](https://arxiv.org/html/2504.20073v2#A3.F8 "Figure 8 ‣ C.1 Environments and Tasks ‣ Appendix C Detailed Experimental Settings ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning")). The low-risk arm always returns a reward of 0.15 0.15 0.15 0.15, while the high-risk arm samples from B⁢e⁢r⁢n⁢o⁢u⁢l⁢l⁢i⁢(0.25)𝐵 𝑒 𝑟 𝑛 𝑜 𝑢 𝑙 𝑙 𝑖 0.25 Bernoulli(0.25)italic_B italic_e italic_r italic_n italic_o italic_u italic_l italic_l italic_i ( 0.25 ): higher variance, higher expected return.

Importantly, the low-risk arm wins more often per trial, even though the high-risk arm is better in expectation. This designed to test reasoning: without inductive bias, models may prefer the lo-arm due to its more frequent success, but a reasoning agent must learn to associate symbolic cues (e.g., "Dragon") with underlying reward statistics, override misleading short-term signals, and “justify” high-risk choices based on long-term expected return. We further test this by reversing the symbolic labels to probe agent’s reasoning under opposed reward systems.

Sokoban. We use the puzzle Sokoban (Figure[9](https://arxiv.org/html/2504.20073v2#A3.F9 "Figure 9 ‣ C.1 Environments and Tasks ‣ Appendix C Detailed Experimental Settings ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning")) to study multi-turn agent interaction. The agent must push a box to the goal in a grid within constrained steps. Unlike standard navigation, Sokoban is irreversible: boxes can only be pushed, not pulled back, which requires the agent to reason ahead to avoid dead-ends. The reward signal encourages efficiency and accuracy: +1 1+1+ 1 for each box on target, −1 1-1- 1 for off-target boxes, +10 10+10+ 10 upon task completion, and −0.1 0.1-0.1- 0.1 per action.

Frozen Lake. This environment (Figure[9](https://arxiv.org/html/2504.20073v2#A3.F9 "Figure 9 ‣ C.1 Environments and Tasks ‣ Appendix C Detailed Experimental Settings ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning")) combines long-horizon decision-making with stochastic transitions. The agent navigates a grid with slippery tiles; each action succeeds with probability 1/3 1 3 1/3 1 / 3 and deviates perpendicularly with probability 2/3 2 3 2/3 2 / 3. The agent should reach the goal without falling into holes. Rewards are sparse: successful trials receive a reward of +1 1+1+ 1, with all others 0 0.

WebShop. To complement the symbolic environments, we include WebShop Yao et al. ([2022a](https://arxiv.org/html/2504.20073v2#bib.bib64)), a multi-turn web-based shopping task that tests agents’ ability to ground natural language queries, navigate semi-structured interfaces, and retrieve goal-relevant information. The agent must select a product that matches a user’s request by issuing search queries, clicking links, and reading product descriptions—introducing realistic language grounding and action space challenges absent in purely symbolic settings.

### C.2 Training and Evaluation Settings

We conduct our experiments using Qwen2.5-0.5B-Instruct(Yang et al., [2024](https://arxiv.org/html/2504.20073v2#bib.bib63)), trained via the StarPO variants with a maximum of 200 rollout–update iterations on NVIDIA H100/A100 GPUs leveraging the veRL***https://github.com/volcengine/verl repository. Considering the extremely long training time of WebShop due to its long-context nature, we train on WebShop for 100 steps. Each rollout consists of K=16 𝐾 16 K=16 italic_K = 16 trajectories per environment group, based on prompt size P=8 𝑃 8 P=8 italic_P = 8 and maximum 5 interaction turns per episode. Agents are allowed up to 5 actions per turn and 10 actions per episode. The update batch size is E=32 𝐸 32 E=32 italic_E = 32, with mini-batch size 4 per GPU. Policy optimization uses GAE with (γ=1.0,λ=1.0)formulae-sequence 𝛾 1.0 𝜆 1.0(\gamma=1.0,\lambda=1.0)( italic_γ = 1.0 , italic_λ = 1.0 ) and Adam with (β 1,β 2)=(0.9,0.999)subscript 𝛽 1 subscript 𝛽 2 0.9 0.999(\beta_{1},\beta_{2})=(0.9,0.999)( italic_β start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT , italic_β start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) = ( 0.9 , 0.999 ). We use entropy regularization (β=0.001 𝛽 0.001\beta=0.001 italic_β = 0.001) For experiments with vanilla StarPO we use a KL coefficient of 0.001, using the k1 k1\rm k1 k1 estimation†††http://joschu.net/blog/kl-approx.html. without KL loss term during training, following(Yu et al., [2025](https://arxiv.org/html/2504.20073v2#bib.bib66)), and track KL post-hoc. We impose a format penalty of −0.1 0.1-0.1- 0.1 if the agent fails to output valid structured responses (e.g., missing <think> or <answer> tags), encouraging adherence to response conventions. To accelerate rollout generation, we disable enforce_eager and retain the computation graph across prefill and sampling in vLLM. We utilize Fully Sharded Data Parallel (FSDP) training strategy for multi-GPU experiments. For distributed training, we employ Ray as the multi-processing backend with XFORMERS attention implementation.

For evaluation, we choose a fixed 256 input prompts per environment and decode using temperature T=0.5 𝑇 0.5 T{=}0.5 italic_T = 0.5, sampling stochastically to better capture robustness in agent behaviors. Episode truncation occurs after 5 turns or 10 total actions.

### C.3 Evaluation Metrics

To track agent learning dynamics and detect training instabilities, we monitor the following metrics throughout training. Except for the success rate, which is evaluated on a fixed validation set, all metrics are computed over validation instances.

*   •
Average Success Rate. Measures task completion accuracy on a fixed set of validation prompts. An episode is considered successful if the agent solves the task (e.g., pulling the high-reward arm in Bandit, pushing all boxes to targets in Sokoban, reaching the goal in Frozen Lake, and a successful purchase in WebShop).

*   •
Rollout Entropy. Computes the average token-level entropy of sampled responses, capturing the exploration level and policy uncertainty. A sharp entropy drop may indicate premature policy convergence or collapse.

*   •
In-Group Reward Variance. Measures reward standard deviation across rollouts sampled from the same prompt group. High in-group variance reflects diverse behaviors and learning potential; a sudden collapse indicates reward homogenization and policy stagnation.

*   •
Total Response Length. Average number of tokens generated per rollout, measuring the verbosity and reasoning depth of the agent. Fluctuations in length may signal changes in planning style or confidence.

*   •
Gradient Norm.ℓ 2 subscript ℓ 2\ell_{2}roman_ℓ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT norm of the policy gradient vector, used as a proxy for training stability. Spikes often correlate with phase transitions in policy behavior or unstable reward signals.

These metrics provide complementary views of policy quality, update dynamics, and reasoning behavior, helping diagnose when and why agent training succeeds or fails.

Appendix D Results on Larger Models and Various Optimization Algorithms
-----------------------------------------------------------------------

We extend all our evaluation to 3B/7B/72B scale model and explore the effects of various algorithm choices like KL removal and asymmetric clipping, together with turn-aware optimization techniques including Generalized Advantage Estimation (GAE) and response masking.

##### Scaling Effects.

We extend our trained model to 3B / 7B to evaluate scaling effects of RL training. Results can be found in Figure[10](https://arxiv.org/html/2504.20073v2#A4.F10 "Figure 10 ‣ Scaling Effects. ‣ Appendix D Results on Larger Models and Various Optimization Algorithms ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning"). Note that due to the extremely long context length of WebShop could due to OOM Error within 4xH100 for 7B models, we only report 3B performance in WebShop task. On Bandit and WebShop, the larger model demonstrates significantly stronger performance compared to smaller models. However, the improvements are marginal on Sokoban and FrozenLake. We attribute this discrepancy to the nature of the environments: Sokoban and FrozenLake are symbolic, grid-based tasks with minimal overlap with pretraining data, limiting the model’s ability to leverage language priors. In contrast, Bandit and WebShop involve natural language interactions, allowing pretrained models to more effectively exploit linguistic patterns for policy learning, even in the absence of explicit environment dynamics. This can be further validated through cases in Figure[16](https://arxiv.org/html/2504.20073v2#A13.F16 "Figure 16 ‣ Appendix M Extended Case Studies ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning"),[17](https://arxiv.org/html/2504.20073v2#A13.F17 "Figure 17 ‣ Appendix M Extended Case Studies ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning"),[18](https://arxiv.org/html/2504.20073v2#A13.F18 "Figure 18 ‣ Appendix M Extended Case Studies ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning"),[19](https://arxiv.org/html/2504.20073v2#A13.F19 "Figure 19 ‣ Appendix M Extended Case Studies ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning"), where semantic-rich tasks like Bandit and WebShop presents significantly more diverse reasoning patterns and better benefit from scale.

![Image 12: Refer to caption](https://arxiv.org/html/2504.20073v2/x10.png)

Figure 10: Scaling effect across environments. The larger model outperforms smaller models on Bandit and WebShop, which benefit from language priors, but shows limited gains on symbolic, grid-based environments like Sokoban and FrozenLake.

##### Frontier Model Performance.

To contextualize our small model’s performance, we evaluate two large foundation models—GPT-4o and Qwen2.5-72B-Instruct—on SimpleSokoban and FrozenLake in a zero-shot setting. Both models are prompted with task instructions and example formats, without any fine-tuning or in-context trajectory rollouts. See Table[5](https://arxiv.org/html/2504.20073v2#A4.T5 "Table 5 ‣ Frontier Model Performance. ‣ Appendix D Results on Larger Models and Various Optimization Algorithms ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning")

Table 5: Zero-shot vs. trained performance. Our 0.5B model, trained with only 4 responses per prompt, achieves performance comparable to that of large foundation models without any fine-tuning. Response length and effectiveness values for GPT-4o / Qwen are reported per environment.

While GPT-4o and Qwen2.5-72B achieve 19–28% success rates in Sokoban and FrozenLake without any task-specific adaptation, our 0.5B model reaches 20.70% and 21.48% respectively after training from scratch. This result is notable given that our model has over 100×\times× fewer parameters. It highlights that even under strict resource constraints, careful rollout construction and policy optimization (see Section[4.3](https://arxiv.org/html/2504.20073v2#S4.SS3 "4.3 Generating Useful Trajectories for RL Training ‣ 4 Experimental Results and Findings ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning")) can match the generalization ability of significantly larger models.

##### Gradient Shaping.

We evaluate the effectiveness of KL Term Removal and Clip-Higher Yu et al. ([2025](https://arxiv.org/html/2504.20073v2#bib.bib66)), finding them useful by simply extending from single-turn static tasks to agent tasks:

*   •
KL Term Removal: We eliminate the KL divergence penalty from PPO’s objective, relying only on policy loss and entropy bonus for gradient updates. It removes the constraint to stay close to the initial model distribution and encourage the model to explore.

*   •
Clip-Higher (Asymmetric Clipping): We decouple the PPO clipping range by using a higher upper bound (ε high=0.28 subscript 𝜀 high 0.28\varepsilon_{\text{high}}=0.28 italic_ε start_POSTSUBSCRIPT high end_POSTSUBSCRIPT = 0.28) than the lower bound (ε low=0.2 subscript 𝜀 low 0.2\varepsilon_{\text{low}}=0.2 italic_ε start_POSTSUBSCRIPT low end_POSTSUBSCRIPT = 0.2). It allows the model to learn more aggressively from high-reward rollouts for more effective training.

As shown in Figure[11](https://arxiv.org/html/2504.20073v2#A4.F11 "Figure 11 ‣ Gradient Shaping. ‣ Appendix D Results on Larger Models and Various Optimization Algorithms ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning"), both methods boost the success rate and extend stable training phases, showing how multi-turn RL benefits from more flexible gradient shaping.

![Image 13: Refer to caption](https://arxiv.org/html/2504.20073v2/x11.png)

Figure 11: Effect of KL removal and asymmetric clipping on PPO stability. The two designs both improve peak performance and delay collapse in multi-turn RL. 

##### Response Masking and Bi-Level GAE.

Following the turn-aware optimization strategy proposed in Wang* et al. ([2025](https://arxiv.org/html/2504.20073v2#bib.bib55)), we evaluate the effect of response masking and bi-level GAE on a 0.5B model. As shown in Figure[12](https://arxiv.org/html/2504.20073v2#A4.F12 "Figure 12 ‣ Response Masking and Bi-Level GAE. ‣ Appendix D Results on Larger Models and Various Optimization Algorithms ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning"), both techniques contribute to improved performance in multi-turn RL tasks, which highlight the promise of turn-aware RL training algorithms for stabilizing and enhancing language agent training.

![Image 14: Refer to caption](https://arxiv.org/html/2504.20073v2/x12.png)

Figure 12: Ablation on turn-aware optimization strategies on Sokoban Task. Both response masking and bi-level GAE improve multi-turn RL performance.

Appendix E When Does Uncertainty-Based Filtering Help?
------------------------------------------------------

We hypothesize that the effectiveness of StarPO-S largely depends on the variance of rollout rewards within each environment. In settings where the task is either too easy or too difficult, the generated trajectories tend to have very low intra-group variance—meaning the model is overconfident or uniformly poor across samples. In such cases, standard StarPO may propagate misleading gradients, while StarPO-S helps by filtering out low-confidence rollouts. Conversely, in open-ended or more diverse environments (like WebShop), the rollout variance tends to be naturally high, which reduces the marginal benefit of StarPO-S filtering.

![Image 15: Refer to caption](https://arxiv.org/html/2504.20073v2/x13.png)

Figure 13: Comparison of success rate (top) and rollout variance (bottom). StarPO-S basically improve training stability on the environments having extremely easy or difficult problems like Sokoban and Frozen Lake. This would ead to instances with small rollout Stds which can be easily filtered out through StarPO-S to make training more stable. On Tasks like WebShop, the rollout Std is already consistently high, and StarPO itself already can achieve good performance. 

Figure[13](https://arxiv.org/html/2504.20073v2#A5.F13 "Figure 13 ‣ Appendix E When Does Uncertainty-Based Filtering Help? ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning") supports this intuition. The top row shows the success rates of StarPO and StarPO-S across four environments, while the bottom row shows the evolution of in_group_std and chosen_in_group_std over training. In Bandit, Sokoban, and FrozenLake, StarPO-S consistently outperforms StarPO, with the gap widening as the rollout variance drops. In WebShop, however, the high and stable variance suggests more diversity in generated responses, making the filtering effect of StarPO-S less critical—explaining the smaller performance gap.

These results indicate that StarPO-S is most beneficial when environments exhibit low rollout uncertainty, providing a simple diagnostic for when to apply it.

Appendix F Case Study: The Emergence of Echo Trap with RL
---------------------------------------------------------

Table 6: Examples of reasoning patterns in the Bandit task. Top rows show diverse reasoning from model before training; bottom rows show repetitive and collapsed reasoning after RL training.

![Image 16: Refer to caption](https://arxiv.org/html/2504.20073v2/x14.png)

Figure 14: Reasoning length over training iterations across different tasks. We track the average token count of reasoning segments (<think> blocks) during RL training. Across all environments, reasoning length declines as training progresses, with BanditRev maintaining longer traces—possibly due to greater semantic-reward conflict requiring more deliberation.

We show cases that demonstrates the Echo Trap in RL training. As shown in Table[6](https://arxiv.org/html/2504.20073v2#A6.T6 "Table 6 ‣ Appendix F Case Study: The Emergence of Echo Trap with RL ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning"), the top rows (Step 0) display varied hypotheses about Dragon and Phoenix, while the bottom rows (Step 150) converge to near-identical phrasing focused on “choosing Dragon” without justification.

Appendix G Comparing Agent RL with Supervised Fine-Tuning
---------------------------------------------------------

Apart from StarPO for RL training, we also employ Supervised Fine-tuning (SFT) as another agent training approach, evaluating it on the Sokoban and Frozen Lake task. We employ LoRA with a rank of 64 and an alpha value of 32, targeting all linear layers in the model. The SFT process uses a learning rate of 1e-4 with a training batch size of 128. We generate ground-truth trajectory data through breadth-first search (BFS), setting a maximum depth of 100 to create 1,000 training samples and 100 test samples. For SFT, we structure the multi-turn interaction as a conversational format. At each turn, the model must generate the next action from the ground-truth trajectory, encapsulating its response within <answer></answer> tags to maintain format consistency.

We analyze the comparative performance of SFT against our stable RL baseline StarPO-S. SFT achieves 74.6% and 23% performance on Sokoban and Frozen Lake, respectively, Compared to the 20.3% and 21.8% performance with StarPO-S. The results indicate that SFT demonstrates superior performance to RL approaches. We draw conclusions from the results that although rule-based RL show promise for agent tasks, there is still a need to build more scalable and effective agent RL algorithms to achieve human-comparable performance with solely model self-evolution.

Appendix H Efficient Training with Low-Rank Adaptation (LoRA)
-------------------------------------------------------------

##### Motivation.

While the main body of the paper reports results obtained by full-parameter fine–tuning, in practice such a setting may be prohibitive when scaling to larger models or longer-horizon tasks. We therefore implement a parameter-efficient variant of RAGEN based on Low-Rank Adaptation(Hu et al., [2021](https://arxiv.org/html/2504.20073v2#bib.bib19)).‡‡‡We set rank r=64 𝑟 64 r{=}64 italic_r = 64, α=64 𝛼 64\alpha{=}64 italic_α = 64, and inject adapters into all linear projections of the transformer blocks. We also increased learning rate by 10×10\times 10 × for both actor and critic.

Performance parity. Despite updating only a fraction of the model parameters, LoRA reaches a validation success rate comparable to that achieved by full fine-tuning of the entire network for the SimpleSokoban task, achieving approximately a 0.2%percent 0.2 0.2\%0.2 % success rate on the validation set.

Resource savings. We compare the hardware footprint of LoRA with full fine-tuning. Across an 80-minute training horizon we measure:

*   •
GPU memory. LoRA stabilizes at ≈𝟐𝟑%absent percent 23\mathbf{\approx 23\%}≈ bold_23 % of device memory versus ≈𝟒𝟖%absent percent 48\mathbf{\approx 48\%}≈ bold_48 % for full updates, cutting the peak allocation by >50 %.

*   •
GPU utilization. Average GPU utilization drops from ∼34%similar-to absent percent 34\sim\!34\%∼ 34 % to ∼14%similar-to absent percent 14\sim\!14\%∼ 14 %.

*   •
Power consumption. Mean power draw decreases from ∼22%similar-to absent percent 22\sim\!22\%∼ 22 % to ∼12%similar-to absent percent 12\sim\!12\%∼ 12 %, a ≈45%absent percent 45\approx 45\%≈ 45 % reduction.

Take-aways. Parameter-efficient fine-tuning provides a practically viable alternative for RAGEN: it attains comparable policy quality while more than halving memory, compute, and power demands. Consequently, future work that scales StarPO to larger backbones or longer contexts can adopt LoRA (or other adapter-based methods) as the default optimization strategy without re-engineering the training loop.

Appendix I PPO Failure Mode in Frozen Lake
------------------------------------------

Among the three evaluated environments, we observe an interesting divergence on Frozen Lake: PPO tends to collapse earlier or converge less stably than GRPO. This contrasts with the general trend where PPO demonstrates better performance, prompting further analysis.

One possible explanation lies in the environment’s long-horizon stochasticity. In Frozen Lake, agent actions always lead to highly non-deterministic transitions, and intermediate states can appear similar while leading to very different outcomes. This makes value estimation challenging. As PPO relies on a learned value function, instability in critic learning may amplify optimization noise and contribute to early collapse. GRPO, by contrast, does not rely on explicit value learning. Its reward-weighted update procedure may be more tolerant to uncertainty in these settings, leading to comparatively more stable training on Frozen Lake—even if it remains less effective in other tasks. Overall, we summarize environments with high stochasticity may pose greater challenges for value-based methods, and that critic-free approaches can serve as a useful baseline in such cases.

Appendix J Prompt Templates
---------------------------

### J.1 Bi-Arm Bandit Environment Prompts

The Bi-Arm bandit environment implements a classic reinforcement learning problem where an agent must balance exploration and exploitation. We present the prompt templates below.

### J.2 Sokoban Environment Prompts

The Sokoban environment presents a classic puzzle game where an agent must push boxes to target locations. The following sections detail the prompt structure used to interface with language models.

The environment uses a grid-based representation with specific symbols for different elements:

The instruction template only consists of available actions and restrictions:

### J.3 FrozenLake Environment Prompts

The FrozenLake environment implements a grid-world navigation task where an agent must traverse a slippery frozen surface to reach a goal. Below we detail the prompt structure used for this environment.

The environment uses a grid-based representation with specific symbols for different elements:

The instruction template only consists of available actions and restrictions:

Figure 15: A rollout with spurious reasoning. The final outcome is successful, but the reasoning traces across turns are inconsistent and sometimes factually incorrect. This reflects a common failure mode where the model optimizes for final rewards while bypassing coherent reasoning, resulting in noisy and potentially misleading supervision during RL training.

Appendix K Generalization Evaluation Environments
-------------------------------------------------

To evaluate generalization beyond the training distribution, we design two new test environments besides the three training environments that vary along different axes:

*   •
SokobanDifferentGridVocab modifies the visual vocabulary used to represent the grid. Instead of using the standard symbols (#, _, O, X, etc.), it maps grid cells to a new vocabulary such as W, G, C, etc. This tests whether the model generalizes across symbol variations while retaining underlying spatial semantics.

*   •
LargerSokoban increases the grid size from 6×6 6 6 6\times 6 6 × 6 to 8×8 8 8 8\times 8 8 × 8 and the number of boxes from 1 to 2, introducing greater spatial complexity and longer-horizon planning demands. This setting evaluates whether the policy trained on small puzzles can scale up to more complex configurations.

These environments are not seen during training and serve to probe the agent’s generalization capability under symbol shift, size scaling, and environment shift, respectively.

Appendix L Case Study: Spurious Reward from Incorrect Reasoning
---------------------------------------------------------------

While evaluating reasoning behavior in Sokoban, we observe that the model may occasionally receive non-negative or even high rewards despite exhibiting flawed or misleading reasoning traces. Figure[15](https://arxiv.org/html/2504.20073v2#A10.F15 "Figure 15 ‣ J.3 FrozenLake Environment Prompts ‣ Appendix J Prompt Templates ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning") presents a 3-turn rollout where the model successfully pushes the box onto the target, yet its intermediate decisions reflect incorrect assumptions about the game dynamics.

In Turn 1 and 2, the agent provides plausible but ultimately incoherent plans, such as "pushing the target" or “moving toward the box on the left,” which are either redundant or directionally incorrect. Despite these errors, the final action sequence still manages to reach the goal. Such cases increase reward signal noise, making it harder for RL training to distinguish between genuinely useful plans and coincidentally effective ones.

This highlights a key challenge in multi-turn RL with reasoning agents: outcome-based reward alone may not sufficiently penalize poor reasoning traces, especially in environments with sparse or delayed feedback.

Appendix M Extended Case Studies
--------------------------------

To better understand how reasoning quality evolves across model scales and environments, we present representative rollout cases across six settings: Bandit (Figure[16](https://arxiv.org/html/2504.20073v2#A13.F16 "Figure 16 ‣ Appendix M Extended Case Studies ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning")), Sokoban (Figure[17](https://arxiv.org/html/2504.20073v2#A13.F17 "Figure 17 ‣ Appendix M Extended Case Studies ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning")), FrozenLake (Figure[18](https://arxiv.org/html/2504.20073v2#A13.F18 "Figure 18 ‣ Appendix M Extended Case Studies ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning")), and WebShop (Figure[19](https://arxiv.org/html/2504.20073v2#A13.F19 "Figure 19 ‣ Appendix M Extended Case Studies ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning")), each at 0.5B and 7B model scales. We observe that larger models tend to generate longer and more coherent reasoning chains, especially in semantic-rich decision tasks like Bandit and WebShop. However, in grid-based environments like Sokoban and stochastic ones like FrozenLake, both small and large models struggle with planning and alignment, often resorting to brittle heuristics or spurious correlations. These cases align with the experiments in Section[D](https://arxiv.org/html/2504.20073v2#A4 "Appendix D Results on Larger Models and Various Optimization Algorithms ‣ RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning") that Sokoban and Frozen Lake does not have a significant performance gain from larger model scales unlike Bandit and WebShop, illustrating how reasoning quality interacts with environment structure, and highlight the challenges of stabilizing reward-grounded reasoning in stochastic or under-specified settings.

Figure 16: Reasoning-based arm selection in Bandit tasks across model scales. In both cases, the model must infer reward tendencies of symbolic arms (Dragon vs. Phoenix) based on prior knowledge. The 0.5B model offers a short justification rooted in symbolic association. The 7B model generates a more elaborate reasoning chain comparing stability and variance, reflecting its stronger prior knowledge and interpretive capacity. Both ultimately select Dragon, but through different levels of reasoning depth.

Figure 17: Sokoban rollouts across model scales. At 0.5B, the model performs minimal reasoning and often issues locally valid but suboptimal actions. The 7B model demonstrates more structured planning and symbolic alignment across turns, though still exhibits inefficiencies and heuristic moves in long-horizon settings.

Figure 18: FrozenLake rollouts across model scales. At 0.5B, the agent repeats a fixed plan regardless of outcome, suggesting limited adaptation or planning. The 7B agent receives high reward despite issuing a suboptimal command, due to stochastic transitions—highlighting the difficulty of credit assignment in such environments and the risk of reinforcing spurious patterns.

Figure 19: WebShop rollouts illustrate the impact of model scale on long-context decision making. At 0.5B, the agent becomes stuck in a loop, repeatedly selecting irrelevant options despite rich contextual information— indicating difficulty in long-horizon memory and goal tracking. In contrast, the 3B model executes a successful multi-step reasoning chain: narrowing search queries, navigating product options, selecting attributes, and finalizing purchase. This highlights the importance of scale for compositional planning in realistic, open-domain environments.