# Reasoning Cache: Continual Improvement Over Long Horizons via Short-Horizon RL Ian Wu1, Yuxiao Qu1, Amrith Setlur1 and Aviral Kumar1 1Carnegie Mellon University **Figure 1:** *Left: Illustration of the RC algorithm.* RC decoding replaces standard autoregressive decoding at both train and test time. During RC decoding, the LLM generates a reasoning trace, summarizes it, discards the original trace, and conditions subsequent reasoning on this summary. This design decouples the effective reasoning horizon from the length of any single reasoning trace, thus maintaining tractable rollout lengths for outcome-reward RL while also enabling extrapolation at test time. *Right: Performance on HMMT 2025 (November) vs. reasoning token budget.* Our RC-trained model RCT-4B (blue, trained from Qwen3-4B-Instruct-2507 at 16k train budget) extrapolates to outperform both the base model with RC decoding (green) and the specialized Qwen3-4B-Thinking-2507 reasoning model (evaluated at 256k test tokens). **Abstract:** Large Language Models (LLMs) that can continually improve beyond their training budgets are able to solve increasingly difficult problems by adapting at test time, a property we refer to as *extrapolation*. However, standard reinforcement learning (RL) operates over fixed problem distributions and training budgets, which limits extrapolation amidst distribution shift at test time. To address this, we introduce **RC**, an iterative decoding algorithm that replaces autoregressive decoding during both training and inference. RC exploits an *asymmetry* between the response generation and summarization capabilities of LLMs to construct reasoning chains that consistently improve across iterations. Models trained to use RC can extrapolate and improve over reasoning horizons more than an order of magnitude longer than those seen during training. Empirically, training a 4B model with RC using a 16k-token training budget improves performance on HMMT 2025 from 40% to nearly 70% when evaluated with 0.5m test-time tokens, outperforming both comparably sized models and many larger reasoning LLMs. We also show that models trained with RC can more effectively leverage existing scaffolds to further scale test-time performance, due to the improved summary-conditioned generation abilities learned through training. ## 1. Introduction Large language models (LLMs) exhibit the ability to solve complex problems by generating long reasoning traces at test time. As LLMs become more capable, we naturally expect them to be able to solve harder tasks by reasoning for longer, even without external supervision. This expectation mirrors human cognition: humans continually improve their reasoning by revisiting earlier conclusions and reallocating effort to discover new information in new ways over the course of problem solving. We would like our models to behave similarly, such that additional reasoning at test time translates to improved performance over horizons much longer than any single reasoning step. This underlies the notion of *in-context exploration* [33], which learns an implicit algorithm or “procedure” for allocating test-time compute such that spending more computation systematically improves outcomes. If learned robustly, in-contextexploration should enable models to continually improve over long horizons at test time, perhaps over hours or even days, and across millions of tokens, all without modifying the model weights. However, current LLM training paradigms limit the forms of in-context exploration that can be learned. Supervised fine-tuning (SFT) teaches models to imitate the content of reasoning traces, rather than learning the algorithm that generates them [4, 32]. This inhibits the learning of systematic reasoning procedures and thus the ability to perform effective in-context exploration. While reinforcement learning (RL) incentivizes models to learn reasoning procedures rather than mere imitation [38, 47], RL is typically applied over a fixed prompt distribution and a bounded training rollout length. As a result, models are optimized to exploit this finite budget, rather than to *extrapolate* beyond it. When such models encounter complex problems that require more reasoning to solve, two failure modes emerge. First, they may prematurely terminate within their training budgets and fail to make progress by reasoning for longer. Second, when models continue beyond this budget, *distribution shift* may occur as generation proceeds from conditional distributions that differ substantially from those seen during training; reasoning traces in this regime are often repetitive and verbose [19, 33]. This raises a natural question: ***how can we train models to extrapolate their reasoning far beyond their training configurations?*** To answer this question, we introduce **Reasoning Cache** (RC), an iterative decoding algorithm that replaces standard autoregressive decoding during both training and inference. In RC, the model generates a reasoning trace, summarizes it (the “cache”) and discards the original trace over multiple turns, with subsequent generation conditioning only on the previous summary rather than the full reasoning trace. For training, we introduce an RL approach that teaches the model to exploit RC by improving its summary-conditioned reasoning capabilities. Our approach combines typical on-policy RL with off-policy learning via a *replay buffer* that enables reuse of cached summaries, allowing the model to train over long effective horizons without generating prohibitively long training rollouts. The design of the decoding process used by RC is motivated by two key observations. First, iterative decoding provides explicit control over test-time compute: we can scale the number of reasoning iterations while keeping each step within the training distribution, mitigating distribution shift as effective reasoning length grows. Second, making such iteration effective requires consistent progress across steps. RC achieves this by exploiting a *summarization-generation asymmetry*: models are often better at summarizing prior reasoning and reasoning from summaries than at producing correct solutions from scratch. Our RL objective explicitly amplifies this asymmetry, enabling larger improvements across successive iterations. Empirically, models trained with RC exhibit strong and consistent extrapolation ability. While RC decoding alone is already effective when the base model can reliably follow instructions and reason from summaries, RC-trained models extrapolate substantially further. On the mathematical reasoning benchmarks HMMT 2025 (November) and IMO-AnswerBench, our RC-trained RCT-4B model substantially outperforms both Qwen3-4B-Instruct-2507 (the base model) and Qwen3-4B-Thinking-2507 (a much stronger reasoning model) through extrapolation. For example, accuracy on HMMT 2025 improves from 40% to 70% as the test budget is scaled from 16k to 512k tokens (Figure 1, right), while on IMO-AnswerBench [20], performance rises from 41% at 16k tokens to 58% at 256k tokens, surpassing larger models such as Qwen3-235B, Qwen3-30B-A3B-Instruct-2507 and Nemotron-3-Nano-30B-A3B using standard autoregressive decoding, despite training only at a 16k budget. Notably, despite having been trained solely on mathematical reasoning data, RCT-4B also achieves substantially higher performance than the base model on the FrontierScience [24] scientific reasoning benchmark when extrapolated to 256k tokens, suggesting that RC induces transferable algorithmic behavior rather than domain-specific knowledge. Finally, we show that RCT-4B is consistently better at exploiting a variety of test-time scaffolds than boththe base model and a base model post-trained with standard RL, with and without RC decoding. This indicates that optimizing for summary-conditioned generation yields models that are not only better at leveraging RC decoding, but are also generally more effective at using external context to guide reasoning. ## 2. Related Work **Long-horizon reasoning by scaling test-time compute.** Increasing the length of reasoning traces is a fundamental approach to scaling test-time compute [9, 42], as longer traces allow for more extensive in-context exploration that can substantially improve performance [7, 33, 47]. Although prior work finds that on-policy RL is effective at teaching LLMs to generate longer autoregressive responses [9], scaling in this way suffers from a fundamental limitation: models cannot reliably extrapolate much beyond the reasoning lengths seen during RL training [19, 33]. Our work therefore aims to develop methods that enable extrapolation, rather than merely scaling reasoning lengths by increasing training budgets. **Test-time extrapolation of reasoning.** Prior work attempts to enable extrapolation mainly through one of two approaches. The first modifies training via carefully designed datasets and curricula to encourage in-context exploration [2, 19, 33]. Although this has been shown to enable extrapolation to around 3–4 $\times$ the training budget, performance typically saturates beyond this. The second approach modifies the RL reward structure to implicitly encourage extrapolation (e.g., via dense rewards that credit intermediate reasoning segments [29]) but retains pure autoregressive decoding at inference time. In both cases, in-context exploration behaviors are implicitly learned through free-form autoregressive generation and thus remain coupled to length budgets in the training setup. When test-time conditional distributions fall outside the training support, as is seen when test lengths greatly exceed train lengths, these approaches do not improve further and instead result in verbose and repetitive behavior [19, 33]. **Iterative decoding for scaling test-time compute.** Prior work explores prompting LLMs to iteratively transform their own outputs to scale test-time compute, ranging from simple self-correction [12, 13] and self-refinement [21, 37] to more complex scaffolds that combine iterative and parallel compute [35, 40]. Others consider training models to apply transformations more effectively rather than relying on prompting alone [14, 26]. Such transformations may occur over multiple time steps: Venkatraman et al. [40], for example, train models to iteratively aggregate parallel reasoning traces, while Madaan et al. [22] consider distilling parallel reasoning traces into summaries that are conditioned on for final answer generation. In contrast to these works, RC scales test-time compute sequentially through extrapolation rather than in parallel. This is reflected in our training procedure, which directly optimizes for multi-step reasoning that can extend to very long horizons through the use of a summary replay buffer. Furthermore, we find that RC training enables models to better exploit other scaffolds by teaching them to more effectively leverage external guidance, thus making our approach complementary to many existing works. **Memory in multi-turn interaction.** RC summaries can be viewed as compressed memory states that are updated as the policy acts over iterations. Prior work primarily consider using similar memory states to store external context (e.g. retrieved web pages, user responses etc.) that is dynamically recalled in later steps, often as part of multi-turn question-answering or conversation systems [16, 49]. Our method instead uses memory states to store abstractions of self-generated reasoning traces for solving reasoning problems, an approach that has been explored using prompting-based approaches [10, 39, 43]. Unlike these works, we focus on training the model to better utilize these memory states for downstream reasoning, which we show yields significant improvements over prompting-only methods.### 3. Preliminaries and Notation Consider a policy $\pi_\theta(\cdot|\mathbf{x})$ that generates tokens autoregressively conditioned on $\mathbf{x}$ . At test time, the policy is given a token budget and allocates this to reason. Our interest is test-time performance as a function of the test token budget, particularly with large test-time budgets. **Standard RL training for LLM reasoning.** Let $\mathcal{D}_{\text{train}}$ denote a training distribution of prompt-answer pairs $(\mathbf{x}, \mathbf{y})$ . On-policy reinforcement learning (RL) optimizes the expected reward of the policy: $$\max_{\pi_\theta} \mathbb{E}_{\mathbf{x}, \mathbf{y} \sim \mathcal{D}_{\text{train}}} \left[ \mathbb{E}_{\mathbf{z} \sim \pi_\theta(\cdot|\mathbf{x})} [r(\mathbf{y}, \mathbf{z})] \right], \quad \text{s.t. } |\mathbf{z}| \leq H_{\text{train}}. \quad (\text{Training objective}) \quad (3.1)$$ Here, $\mathbf{z}$ denotes an on-policy rollout autoregressively sampled from $\pi_\theta$ . The rollout encodes a reasoning trace and is generated within a fixed training budget $H_{\text{train}}$ . The reward function $r(\mathbf{y}, \mathbf{z})$ evaluates the correctness of the rollout, typically by extracting the final answer from $\mathbf{z}$ and comparing it against the ground-truth label $\mathbf{y}$ . To solve this optimization problem, we can use outcome-reward policy-gradient methods: one common choice is GRPO [34] (see Appendix H), which we use throughout this work. **Test-time extrapolation of LLM reasoning.** Equation 3.1 optimizes performance only over the empirical distribution of training prompts $\mathcal{D}_{\text{train}}$ , and only under a fixed $H_{\text{train}}$ . At test time, we may want to ensure that our trained model attains high accuracy on a different prompt distribution and under a larger budget: $$\text{TestPerf}(\pi_\theta) \stackrel{\text{def}}{=} \mathbb{E}_{\mathbf{x}, \mathbf{y} \sim \mathcal{D}_{\text{test}}} \left[ \mathbb{E}_{\mathbf{z} \sim \pi_\theta(\cdot|\mathbf{x})} [r(\mathbf{y}, \mathbf{z})] \right], \quad \text{s.t. } |\mathbf{z}| \leq H_{\text{test}} \quad (\text{Test-time objective}) \quad (3.2)$$ where $H_{\text{test}}$ is the test budget; in general, the training and test distributions differ (i.e. $p_{\text{train}}(\mathbf{x}) \neq p_{\text{test}}(\mathbf{x})$ and $H_{\text{test}} \gg H_{\text{train}}$ ). When a model trained to optimize Equation 3.1 can leverage a larger test budget to achieve $\text{TestPerf}(\pi_\theta)|_{H_{\text{test}}} > \text{TestPerf}(\pi_\theta)|_{H_{\text{train}}}$ , we say that it can *extrapolate*. ### 4. Problem Statement **Does optimizing performance at $H_{\text{train}}$ (Equation 3.1) also optimize extrapolation at $H_{\text{test}}$ (Equation 3.2)?** Unfortunately, the answer is **no**. During training, the model receives positive reward only for rollouts that terminate within $H_{\text{train}}$ tokens. This implicitly penalizes longer rollouts and encourages “premature” termination near $H_{\text{train}}$ at test time. Moreover, when the model does continue beyond $H_{\text{train}}$ at test time, it must effectively operate on the sorts of conditional distributions it was never trained on. While this form of *distribution shift* is not problematic if the model has learned a true “operator” [29] that enables the chaining of behaviors to solve problems, it is unclear whether RL can learn such operators from a finite, fixed prompt set. We instead circumvent this challenge by altering the decoding algorithm such that the model never encounters significant distribution shift even when reasoning at long horizons. **Why do we need extrapolation?** Can we simply increase $H_{\text{train}}$ to match $H_{\text{test}}$ during RL training? Doing so would obviate extrapolation altogether. However, there are two main problems with this approach. First, any new test distribution we encounter may contain harder problems requiring $H_{\text{test}} \gg H_{\text{train}}$ to solve, and so it would be better to train models that can adapt to larger test budgets on-the-fly (i.e., training models to “continually adapt”). Second, memory and compute costs, as well as the effectiveness of RL training, scale aggressively with response length, making long-horizon on-policy RL prohibitively expensive and challenging. These challenges indicate that we cannot just scale $H_{\text{train}}$ , and should instead train models to extrapolate as more compute is provided.``` graph TD Problem[Problem] --> Reasoning[Reasoning] Reasoning --> Summary[Summary] Summary --> SummaryConditioned[Summary-Conditioned Reasoning] ``` **Problem** Mark writes the squares of several distinct positive integers (in base \$10\$) on a blackboard. Given that each nonzero digit appears exactly once on the blackboard, compute the smallest possible sum of the numbers on the blackboard. **Reasoning** We are given a problem: Mark writes the squares of several distinct positive integers (in base 10) on a blackboard. Given... **Summary** In this solution, the goal is to find the smallest possible sum of distinct perfect squares such that the digits in all the squares together contain each nonzero digit (1 through 9) exactly once, with no digit repeated and no zeros counting toward the nonzero digit constraint. After extensive trials of various combinations of small squares, the only valid configuration found that satisfies all conditions is: the squares of 1, 3, 5, 6, and 28 (i.e., $1^2 = 1$ , $3^2 = 9$ , $5^2 = 25$ , $6^2 = 36$ , and $28^2 = 784$ )... **Summary-Conditioned Reasoning** We are to **find the smallest possible sum** of **distinct perfect squares** such that when we write down **all digits** of those squares, we get **each nonzero digit from 1 through 9 exactly once**, and zero digits are ignored (do not count). Our task is to **minimize the sum** of the squares such that their digits (ignoring 0s) together use each digit 1–9 exactly once. **Let's first analyze and verify the previous attempt. They proposed: $1^2 = 1 \rightarrow \text{digit: 1}$ ...** **Figure 2: Illustrative example of RC’s output.** The model generates reasoning given an input problem, before conditioning on the reasoning to generate a summary between 1-2 paragraphs long. The model then conditions on the summary to generate new reasoning. As we show later, RC training improves the ability of the model to perform *summary-conditioned reasoning*, which enables the model to continually improve over long horizons. See Appendix M for a full example of an RC output. ## 5. Enabling Extrapolation with Reasoning Cache Our goal is to develop a method that trains models under a fixed token budget and a finite prompt set, while still allowing them to extrapolate beyond the training horizon. To achieve this, we replace autoregressive decoding with an iterative decoding algorithm $\text{Alg}(\pi_\theta; \mathbf{x})$ during training and inference. This algorithm, Alg, leverages the structure of reasoning along with asymmetries present in LLMs to support long-horizon reasoning at test time while remaining amenable to training under a much smaller $H_{\text{train}}$ . We begin by concretizing the key desiderata that Alg should satisfy. **Choosing an effective decoding algorithm.** An effective choice of Alg must satisfy two desiderata. *First*, it should define an iterative procedure in which the number of iterations monotonically controls test-time compute, while each iteration operates on conditional distributions that remain close to those encountered during training. An Alg satisfying this desideratum avoids the main limitations of autoregressive decoding in standard RL: (1) by enabling longer responses through increasing the iteration limit, it mitigates premature termination biases induced by fixed-length RL training; and (2) by restricting autoregressive generation within each iteration to at most $H_{\text{train}}$ tokens, it reduces train–test distribution shift even when the effective reasoning horizon is much larger. *Second*, the algorithm should retain expressivity comparable to autoregressive decoding, allowing each iteration to refine or extend the current rollout and explore new directions. An Alg that satisfies this property can make consistent progress across many iterations, enabling continual extrapolation over long horizons. ### 5.1. RC: A Multi-Turn Decoding Algorithm We now introduce a decoding algorithm that satisfies these desiderata. Our algorithm, which we call **Reasoning Cache** (RC), is an iterative decoding approach that alternates between response generation and summarization. Being an iterative decoding algorithm, RC naturally fulfills our first desideratum:**Figure 3:** *Left: Accuracy vs. test-time token budget.* RC decoding improves performance as token budget $H_{\text{test}}$ is increased far beyond $H_{\text{train}} = 16\text{k}$ . *Middle: Total tokens used vs. test-time token budget.* Total reasoning tokens used by RC increases linearly as we increase the reasoning token budget. Shaded regions indicate the 5th-95th percentile; lines indicate the mean. *Right: Accuracy vs. token budget for iterative decoding methods.* RC is a more effective method for enabling extrapolation than self-verification and self-refinement, highlighting the benefits of exploiting the summarization-generation gap. we can increase test-time compute by increasing the number of summarization-generation turns, while also avoiding significant shifts in the conditional distributions encountered at each turn by only ever autoregressively generating at bounded lengths $H_{\text{train}} \ll H_{\text{test}}$ . To satisfy our second desideratum, RC relies on two properties of LLMs. First, reasoning traces contain *redundant* tokens: many tokens encode steps that are useful for local progress but need not be retained verbatim to guide future actions. This allows us to discard a significant portion of tokens (e.g. via summarization) so long as key information is retained. Second, as we consistently find in our experiments, base LLMs often exhibit *summarization-generation asymmetry*, in that producing a correct response conditioned on a summary of a previous attempt is easier than generating a correct response from scratch; this asymmetry arises from the instruction-following abilities of LLMs, which allows them to use summaries of prior generations to guide subsequent reasoning. RC exploits this by periodically compressing responses into a cache and conditioning subsequent generation on it, allowing the model to refine, extend, or restart reasoning across iterations as needed. See Figure 2 for an example of RC’s outputs. Let $x$ denote the prompt and let $t \in \mathbb{N}$ index the decoding turn. RC maintains: **(1)** a reasoning trace $z_R^{(t)}$ and **(2)** a summary $z_S^{(t)}$ , with $z_S^{(0)}$ initialized to the empty string. At each turn, $z_R^{(t)}$ is generated under a fixed token budget $H_R$ , while $z_S^{(t)}$ is generated under $H_S \ll H_R$ . Decoding proceeds by alternately prompting the base model with two distinct system instructions $\mathcal{I}_R$ and $\mathcal{I}_S$ (see Appendix N): $$z_R^{(t)} \sim \pi_\theta \left( \cdot \mid \mathcal{I}_R, x, z_S^{(t-1)} \right), \quad (5.1)$$ $$z_S^{(t)} \sim \pi_\theta \left( \cdot \mid \mathcal{I}_S, x, z_R^{(t)}, z_S^{(t-1)} \right). \quad (5.2)$$ $\mathcal{I}_R$ instructs the model to generate reasoning conditioned on the current cache, while $\mathcal{I}_S$ instructs the model to compress the current reasoning trace and previous summary into an updated summary that encodes high-level information about the strategies employed and conclusions reached in previous turns. After $T$ turns, the final output is given by $z := z_R^{(T)}$ . We denote this iterative process (see Figure 1) as: $$(z_R^{(1)}, z_S^{(1)}, \dots, z_S^{(T-1)}, z_R^{(T)}) \sim \text{Alg}(\pi_\theta; x). \quad (5.3)$$ **Extrapolation with RC.** Because each step is allocated a fixed budget $H_R + H_S$ , the total effective budget under RC is $T \times (H_R \times H_S)$ . Since $H_R \gg H_S$ , we drop $H_S$ and approximate the budget as $T \times H_R \stackrel{\text{def}}{=} H_{\text{test}}$ . If performance improves in the regime $T' \times H_R \gg H_{\text{train}}$ , we say that RC enables extrapolation.**Figure 4:** *Left:* Accuracy at different levels of summary detail. All accuracies measured at $H_{\text{test}} = 192k$ ; red dotted line indicates base model performance without RC. Performance degrades when summary detail is either too high or too low. *Middle:* Relative accuracy improvement (over standard autoregressive decoding) on HMMT 2025. Replacing Qwen3-4B-Instruct-2507 with Qwen3-4B-Thinking-2507 for summarization (Think, Inst) reduces gains. Using Qwen3-4B-Thinking-2507 for both tasks (Think, Think) further reduces gains. *Right:* Accuracy vs. test-time token budget with various $H_R$ . Reducing $H_R$ from 16k to 8k leaves RC performance unchanged, whereas decreasing it further to 4k negatively impacts performance. ## 5.2. Experimental Evaluation **Experimental setup.** We now validate whether LLMs possess the ability to utilize RC without additional training. We evaluate RC decoding with Qwen3-4B-Instruct-2507 and Qwen3-30B-A3B-Instruct-2507 [30], two hybrid LLMs capable of both complex reasoning and instruction-following (see Appendix D for similar results from another model family). Using these LLMs as our base models, we run RC decoding for $T = 12$ turns with $H_S = 2048$ and $H_R = 16k$ , giving us a total budget of $H_{\text{test}} = 192k$ . This is far larger than both models’ $H_{\text{train}}$ , which we estimate to be about 16k (see Appendix C for evidence justifying this). We use the November version of HMMT 2025 as our evaluation dataset (this competition was conducted after the base models were released), and generate 16 RC outputs per problem. **Finding 1: RC enables extrapolation.** We plot how accuracy evolves with the token budget $H_{\text{test}}$ in Figure 3 (left). We find that RC extrapolates reasoning far beyond $H_{\text{train}} = 16k$ : 4B model accuracy increases by 17% as the test token budget is scaled from 16k to 192k, while 30B model accuracy increases by 12%. We also plot how actual token usage varies with reasoning token budget in Figure 3 (middle). We find that the cumulative number of tokens used scales linearly with the provided budget, which indicates that the model utilizes additional test time compute as provided and does not substantially shorten its responses at later iterations. Overall, our findings demonstrate that RC satisfies both desiderata outlined in Section 5 and thus enables effective extrapolation. **Finding 2: Summary-based abstractions are key to effective extrapolation.** We examine the role of summarization–generation asymmetry by experimenting with iterative decoding methods that do not utilize summarization. Concretely, we remove the summary step and instead condition each iteration directly on the full response from the previous iteration $z_R^{(t)}$ , and prompt Qwen3-4B-Instruct-2507 ( $H_R = 16k$ ) to either verify-then-correct (self-verify) or self-refine its solution (details in Appendix J). Figure 3 (right) shows that RC consistently outperforms these baselines across all $H_{\text{test}}$ values, demonstrating that summary-conditioned generation provides benefits over other iterative methods. We attribute this to two factors: first, conditioning on summaries keeps context lengths bounded and in-distribution, whereas iterating on raw responses exceeds $H_{\text{train}}$ and induces distribution shift. Second, summarization serves to remove redundant “distractor” tokens [11, 17] that obfuscate key findings and other important information, thereby yielding clearer guidance for subsequent reasoning. **Finding 3: Summary detail level matters.** Next, we study how much information summaries should retain. We vary the prompt $\mathcal{I}_S$ to produce summaries of differing detail, ranging from answer-only tomultiple paragraphs (Figure 21), and compare these to our default approach and to full-trace conditioning (self-refinement). Figure 4 (left) reports the accuracy of Qwen3-4B-Instruct-2507 at $H_{\text{test}} = 192\text{k}$ across various choices of $\mathcal{I}_S$ . Performance degrades with very short summaries, improves as more detail is added, and peaks with $\geq 2$ -paragraph summaries; omitting summarization entirely degrades performance. #### Finding 4: Base models must be good instruction-followers **for RC to be effective.** We replace Qwen3-4B-Instruct-2507 with the specialist reasoning model Qwen3-4B-Thinking-2507, which excels at reasoning but possesses weaker instruction-following abilities and thus a less clear summarization-generation asymmetry. We evaluate using the reasoning model only for summary-conditioned generation (“Think, Inst”) and for both summary and summary-conditioned generation (“Think, Think”) ( $H_R = 64\text{k}$ ). Figure 4 (middle) shows that “Think, Inst” and “Think, Think” both achieve smaller relative gains; qualitative inspection reveals that the reasoning model sometimes ignores summaries during generation and omits key details during summarization. However, note that RC still provides positive performance gains in all configurations, suggesting that sufficient asymmetry is present for training to potentially be effective. Indeed, LM-Provers et al. [18] demonstrate that RC training substantially improves the proof-writing capabilities of Qwen3-4B-Thinking-2507, confirming that sufficient summarization-generation asymmetry exists even in specialist reasoning models for the method to be effective. **Finding 5: Reducing $H_R$ by too much degrades performance.** By default, we set $H_R = H_{\text{train}} = 16\text{k}$ , and since LLMs rarely generate traces longer than $H_{\text{train}}$ , we only consider decreasing it. Figure 4 (right) shows that reducing $H_R$ from 16k to 8k has minimal impact despite the fraction of incomplete traces increasing from 0% to 20% (see Figure 5). Reducing $H_R$ further to 4k causes nearly 50% of traces to terminate early, this time substantially degrading performance. This implies that while $H_R$ can be decreased, it must still be large enough for redundancy to emerge. When $H_R$ is set too small, the resulting summaries capture only shallow progress that provides insufficient signal for continuation, which encourages the model to restart reasoning from scratch instead. ### 5.3. Analysis of Summary-Conditioned Generations We analyze the content of summary-conditioned generations produced by RC and find that they commonly exhibit three high-level strategies: (1) *verification*, where the model generates reasoning to explicitly verify intermediate or final results stated in the summary; (2) *exploration*, where the model deliberately pursues a different strategy from that used in the summary; and (3) *refinement*, where the model acknowledges the summary and repeats the same strategy without attempting verification or exploring alternatives. To quantify the prevalence of these behaviors, we extract summaries and their subsequent reasoning traces and pass them to an LLM-based annotator (Figure 22), which assigns each sample to one of the three categories above, as well as a *none* category when no references to the **Figure 5: Termination rates of Qwen3-4B-Instruct-2507 with RC as a function of length.** Measured on HMMT 2025 and across all $T = 12$ turns. Virtually all reasoning traces terminate within 16k tokens, suggesting that the model is not trained to reason beyond this. We consider a trace to have terminated after it generates boxed{}. **Figure 6: Percentage of RC reasoning traces that employ various reasoning strategies.** Verification of previous reasoning is the most common strategy, followed by exploration of new strategies.**Figure 7: Illustration of RC rollout generation for training with a replay buffer $\mathcal{B}$ .** We sample an input and a summary from $\mathcal{B}$ , and use this as the starting point to run $T_{\text{train}}$ steps of RC decoding. New summaries generated through this process are stored in $\mathcal{B}$ , replacing older summaries corresponding to the same input problem. We then sample from the $T_{\text{train}}$ newly generated summaries, and condition on these to generate training rollouts that are optimized via Equation 6.1. summary are made and reasoning restarts from scratch. Figure 6 shows that the model relies heavily on summaries to guide subsequent generations, with very few samples classified as *none*. The most common strategy is *verification*, although a substantial fraction of samples also exhibit *exploration* and *refinement*. **Key Takeaways: Inference-only experiments with RC** - • RC enables extrapolation even without RC-specific training. It exploits the *summarization-generation asymmetry* and is thus most effective when paired with instruction-following models. ## 6. Training Models to Extrapolate with RC Decoding Having established the design of Alg, we now describe our method for training models to use it. Our analysis in Section 5.2 shows that RC extrapolation depends on the model’s ability to iteratively reason from and improve upon summaries of past iterations. Accordingly, our training objective is to strengthen *summary-conditioned generation*: given a problem and a summary, we should train our model to generate improved reasoning that is more likely to yield a correct answer. The iterative structure of RC makes this objective amenable to standard outcome-reward RL as we can run RC for multiple iterations and apply gradient updates independently at each turn. By setting $H_R$ to a “typical” value for the base model, such that most responses end with a final answer, we can assign outcome-based rewards at each step and avoid credit assignment across iterations. While this approach is indeed myopic, we note that the iterative nature of RC decoding ensures that this does not reinforce the resulting bias of premature termination. Formally, we run RC for $T_{\text{train}}$ turns for each problem $x$ in a training batch. We collect the summaries generated from this $\mathbf{z}_S := (z_S^{(1)}, \dots, z_S^{(T_{\text{train}})})$ and uniformly sample $N_{\text{summ}} \leq T_{\text{train}}$ unique summaries per problem. We then generate $K$ reasoning traces conditioned on each sampled summary, assign outcome rewards and compute advantages over these samples: $$\max_{\pi_\theta} \mathbb{E}_{\mathbf{x}, \mathbf{y} \sim \mathcal{D}_{\text{train}}, t \sim U[1, T_{\text{train}}]} \left[ \mathbb{E}_{\mathbf{z}' \sim \pi_\theta(\cdot | \mathbf{x}, z_S^{(t)})} [r(\mathbf{y}, \mathbf{z}')] \right],$$ $$\text{where } (z_S^{(1)}, \dots, z_S^{(T_{\text{train}})}) \sim \text{Alg}(\pi_\theta; \mathbf{x}), \quad \text{s.t. } |\mathbf{z}'| \leq H_R. \quad (6.1)$$ This design is effective because, under the problem structure targeted by RC, optimizing each iteration locally is aligned with optimizing the full reasoning trajectory. By training each step $t$ to produce a correctanswer, we implicitly encourage the model to generate reasoning traces and summaries that contain information, such as intermediate results or partial analyses, that is useful for subsequent steps. Because training rollouts are conditioned on these summaries, the model is also explicitly trained to exploit this accumulated information when generating later responses, increasing the likelihood of future correctness. Under the assumption that successive summarization and summary-conditioned generation steps make roughly monotonic progress toward a solution, optimizing per-step correctness is sufficient to improve trajectory-level performance, despite only explicitly optimizing individual steps. We discuss limitations of this assumption and evaluate alternative training approaches in Appendix E. **Training with a summary replay buffer (off-policy reinforcement learning).** The iterative decoding structure of RC naturally enables learning from off-policy summaries because summaries serve as conditioning inputs rather than optimization targets. Learning from off-policy summaries provides two benefits. First, it enables training on later reasoning turns without the need for us to generate long on-policy decoded trajectories, which is useful because summaries from later turns may qualitatively differ from earlier ones. Second, it increases the coverage of summaries the model encounters during training, which in turn increases the robustness of the policy to test time shifts in the summary distribution. In fact, improving state coverage via a replay buffer is one of the fundamental principles in off-policy RL [6], which using past summaries enables us to implement. We therefore incorporate a *summary replay buffer* into training; see Figure 7. During the first training epoch, we follow the same on-policy procedure to optimize Equation 6.1, but also store all generated summaries, and their corresponding problems, in the replay buffer $\mathcal{B}$ . From the second epoch onward, we sample problems and summaries from $\mathcal{B}$ and condition RC rollouts on them instead of generating fresh summaries, thereby extending the maximum effective training horizon by $T_{\text{train}}$ steps per epoch. ## 7. Experimental Evaluation: Training with RC The goal of our experiments is to demonstrate the effectiveness of training with RC to improve extrapolation. To this end, we evaluate our approach on several benchmarks, compare it with related methods, and also conduct several ablation experiments to isolate the effects of our different design choices. **Training details.** We post-train a Qwen3-4B-Instruct-2507 model to utilize RC and refer to the trained model as RCT-4B. We set $K = 8$ , $N_{\text{summ}} = 2$ and $T_{\text{train}} = 3$ . We conduct training in two stages: in Stage I, we train without the summary replay buffer, focusing on optimizing early turns, including the initial turn (without any summary context). Training problems for Stage I are subsampled from the AceReason-Math dataset [3], resulting in a dataset of about 5.7k problems. For Stage II, we enable the summary replay buffer, and construct a new training set by injecting a small number of difficult problems from DAPO [46] into our Stage I dataset as part of our training curriculum. See Appendix I for details. **Benchmarks and evaluation protocols.** We evaluate RCT-4B on three math reasoning benchmarks: AIME 2025, HMMT 2025 (November version), and IMO-AnswerBench [20], as well as one scientific reasoning benchmark, FrontierScience (Olympiad) [24], which contains expert-written problems in physics, chemistry, and biology. Since our training data exclusively consists of mathematical reasoning problems, FrontierScience serves to assess whether learned extrapolation behavior generalizes to an unseen domain. We evaluate AIME and HMMT using math-verify1 for equivalence checking, and follow the official LLM judge-based evaluation protocols for IMO-AnswerBench2 and FrontierScience. We 1 2In a previous version of this paper, we reported IMO-AnswerBench results evaluated using math-verify. We subsequently**Figure 8:** Accuracy on various reasoning benchmarks as a function of token budget for Qwen3-4B-Instruct-2507 and RCT-4B with RC decoding. RCT-4B improves performance across all four reasoning benchmarks as reasoning token budget $H_{\text{test}}$ is increased beyond $H_{\text{train}}$ . This improvement is larger than that attained by using the base model. selected the latter three benchmarks for their low contamination risk: all three were released after our training datasets and our base model, with HMMT 2025 (Nov) and FrontierScience consisting entirely of new problems and IMO-AnswerBench consisting of extensively rewritten past Olympiad problems. **Baselines and comparisons.** We compare against three categories of approaches. The first consists of **autoregressive decoding** methods using open-source 4B models. These include Qwen3-4B-Instruct-2507 (our base model), Qwen3-4B-Thinking-2507, Polaris-4B [2], a Qwen3-4B-based model trained for extrapolation by expanding the output context using YaRN [25], and a version of Qwen3-4B-Instruct-2507 trained with standard GRPO at $H_{\text{train}} = 32\text{k}$ , the maximum output length we could reliably use for RL due to practical constraints. The second category of comparisons consists of other **iterative decoding** approaches that condition directly on raw past reasoning rather than on summaries. We evaluate base and trained (see Appendix J) versions of two such methods: self-refinement and self-verification, selecting $H_R$ and $H_{\text{test}}$ to be the same as for our RC experiments. We also evaluate two other iterative decoding approaches inspired by recent work [1, 23] in Appendix K. The third category of comparisons consists of approaches that do not train with RC but still use RC at inference time. We compare RCT-4B against inference-only use of RC by the base model (as in Section 5.2) and by the base model post-trained with standard RL. This comparison isolates the contribution of training with RC, rather than applying it solely at test time. See Appendix F for a discussion on the computational efficiency of RC. ## 7.1. Benchmark Results Our results are shown in Figure 8 and Table 1. Across all benchmarks and token budgets, RCT-4B outperforms the base model using RC, with the performance gap widening as the token budget increases. This indicates that training enables more effective extrapolation rather than merely improving short-horizon performance. Notably, the model also improves on FrontierScience despite being trained exclusively on mathematics problems, suggesting that RC training develops domain-general extrapolation capabilities. **Finding 1: RC compares favorably against other strong 4B-sized models.** We compare RCT-4B against strong 4B reasoning-specialized models that utilize autoregressive decoding. While these models are explicitly trained to exploit large token budgets, RCT-4B outperforms all autoregressive approaches on the three benchmarks with lowest contamination risk: HMMT 2025 (released in Nov '25), IMO-AnswerBench (released in Nov '25), and FrontierScience (released in Dec '25). In fact, we find that RCT-4B even achieves competitive results against much larger reasoning models: see Figure 11. Notably, while standard RL discovered that this approach underestimates accuracy, as `math-verify` fails to parse certain valid answer formats on this benchmark, leading to correct responses being marked as incorrect. We therefore report results using the official LLM judge-based evaluation protocol instead, replacing Gemini-2.5-Pro with Gemini-3-Flash.
AIME 2025 HMMT 2025 (Nov) IMO-AnswerBench FrontierScience
Qwen3-4B-Instruct-2507 [16k] 46.0 39.8 40.9 23.3
Qwen3-4B-Instruct-2507 (RL, 32k) [32k] 54.8 48.3 42.1 21.5
Polaris-4B [2] [90k] 79.4 60.2 47.7 23.6
Qwen3-4B-Thinking-2507 [81k] 81.3 62.5 53.8 25.7
Self-Refine (base) 53.8 50.0 45.8 27.8
Self-Verify (base) 48.9 46.7 43.9 29.7
Self-Refine (trained) 60.4 61.3 52.2 33.5
Self-Verify (trained) 61.2 62.1 52.3 31.9
Qwen3-4B-Instruct-2507 + RC 59.4 56.7 46.3 29.5
Qwen3-4B-Instruct-2507 (RL, 32k) + RC 66.0 60.2 53.4 33.5
RCT-4B + RC (Ours) 74.9 66.3 58.0 34.1
**Table 1: Evaluation results.** The top section of the table reports results of various reasoning models run with standard autoregressive generation; $H_{\text{test}}$ given in square brackets. The middle section reports various iterative decoding baselines, run for 12 (AIME, HMMT) or 16 (IMO-AnswerBench, FrontierScience) turns (equivalent to a 192k or 256k reasoning token budget) with $H_R = 16\text{k}$ . The final section reports results for RC-based results, which were run with the same token budgets as the iterative decoding baselines. RCT-4B outperforms all baseline methods on 3 out of 4 benchmarks. training improves upon the base model on the mathematics benchmarks, this standard RL model (1) remains substantially weaker than specialized reasoning models, and (2) achieves no gains on the out-of-domain FrontierScience benchmark despite training on the same data as RCT-4B. This demonstrates that RC training develops more generalizable problem-solving strategies that standard RL does not. ### Finding 2: RC training yields better iterative reasoning than other iterative training methods. RCT-4B substantially outperforms all iterative decoding methods we compare against. Training models to perform self-verification or self-refinement using an approach analogous to RC training (Appendix J) leads to large improvements over their respective base models. However, these baselines remain significantly weaker than RCT-4B on mathematical reasoning tasks, highlighting the benefit of exploiting the summarization-generation asymmetry during training. Interestingly, nearly all iterative decoding methods we evaluate, including untrained variants, outperform autoregressive decoding on FrontierScience. This suggests that iterative decoding may generalize better to out-of-domain input problems compared to standard autoregressive generation for the base models we consider in our experiments. ## 7.2. Evaluating RC on Hard Problems Our results thus far show that extrapolation with RCT-4B yields consistent improvements on standard benchmarks. However, these gains could have arisen either from solving harder problems with additional test-time compute or from sharpening performance on problems that are already partially solvable within $H_R$ . To distinguish between these effects, we evaluate models on a set of adversarially curated problems from Omni-MATH [8], following the protocol of Qu et al. [27]. These problems are selected such that the base Qwen3-4B-Instruct-2507 model fails to produce any correct solution across $N = 256$ independent attempts, even when given sufficient reasoning budget. We consider such problems to be difficult because they are unlikely to be solvable simply by scaling parallel compute [5, 41, 44]. **Evaluation results.** We evaluate both the base model and RCT-4B on this dataset using RC decoding and report pass@ $k$ in Figure 9. While both models improve with increased reasoning token budgets, the gains for RCT-4B are substantially larger. At a budget of 256k tokens, the base model achieves a pass@16 of 20%, whereas RCT-4B reaches nearly 35%. Moreover, the performance gap between the trained and**Figure 9:** Pass@ $k$ accuracy vs. token budget on a hard subset of problems sampled from [27] for Qwen3-4B-Instruct-2507 and RCT-4B. The left panel shows base model performance, the middle panel shows trained model performance, and the right panel shows the performance gap between the two. RCT-4B achieves substantially higher pass@ $k$ rates than the base model across all values of $k$ , with the performance gap increasing as reasoning token budget grows. base models widens as the token budget increases, indicating that training improves the model’s ability to utilize RC for in-context exploration. Overall, these results demonstrate that RCT-4B can solve difficult problems that the base model cannot by effectively extrapolating its reasoning at test time. For additional results directly comparing RC against parallel compute methods (majority vote), see Appendix B. ### 7.3. Ablation Studies **The effect of the summary replay buffer.** We ablate our training procedure by comparing performance on AIME 2025 after (i) Stage I training only, (ii) Stage II training without using the summary replay buffer, and (iii) Stage II training with the replay buffer enabled; see Figure 10 (left). Stage I training alone yields substantial gains over the base model, while Stage II training provides additional improvements. These gains are modest without the replay buffer but significantly larger when it is used, particularly at higher reasoning budgets. At 16k tokens, Stage II with replay buffer improves accuracy by 2.7% over Stage I, while at 192k tokens this gap grows to 9.4%. In contrast, Stage II without replay buffer yields only a ~4% improvement at 192k tokens, thus demonstrating the effectiveness of our replay buffer. **The effect of the number of training turns.** Next, we vary the number of training turns $T_{\text{train}}$ during Stage I, evaluating $T_{\text{train}} \in \{2, 3, 4\}$ ; see Figure 10 (middle). All settings improve extrapolation, with $T_{\text{train}} = 3$ performing best, followed by $T_{\text{train}} = 2$ . We believe this arises from a trade-off in gradient signal allocation: when $T_{\text{train}}$ is too large, the model receives insufficient signal on early turns (including the initial turn), while the converse is true when $T_{\text{train}}$ is too low. These results imply that we must train the model evenly across all early turns in order for good summary-conditioned reasoning to emerge. **The importance of training with RC.** Finally, we isolate the effect of RC training by comparing against a model trained with standard outcome-reward RL (GRPO). We train Qwen3-4B-Instruct-2507 with RL for the same number of steps and following the same two-stage training curriculum while increasing the training budget $H_{\text{train}}$ from 16k to 32k tokens (the maximum output length we could reliably use for RL due to practical constraints). We then conduct evaluation using RC decoding; see Figure 10 (right) and Table 1 (bottom) for our results. While RL training yields modest improvements over the base model, it falls far short of training with RC. This demonstrates that effective extrapolation via RC decoding does not emerge from standard RL alone and must be trained through RC’s structured, multi-turn objective.**Figure 10: Left: Ablation study on stagewise training configurations.** Performance improves with Stage II training, with the summary replay buffer providing additional gains that increase with reasoning budget. **Middle: Ablation study on $T_{\text{train}}$ values during Stage I.** $T_{\text{train}} = 3$ yields optimal performance across budgets by balancing early-turn training with summary-conditioned reasoning exposure. **Right: Comparison of training methods.** RC-specific training substantially outperforms standard GRPO training, demonstrating that extrapolation requires explicit training beyond simply improving reasoning capabilities. ## 7.4. Incorporating RC into test-time scaffolds Our experiments thus far show that RC training enables more effective extrapolation by improving summary-conditioned generation. We can view this as a specific instance of *self-guided reasoning*: the model conditions on self-generated abstractions [28, 45] (in our case, summaries of prior reasoning) to guide downstream reasoning. This raises a natural question: does RC training teach a generalizable skill for using self-generated guidance, or are improvements narrowly tied to reasoning from summaries? To answer this, we evaluate how RC training transfers to other self-guidance settings by incorporating RCT-4B and RC decoding into two test-time scaffolds: Recursive Self-Aggregation (RSA) [40] and the DeepseekMath (DSM) Agent [35]. RSA generates parallel reasoning traces and iteratively aggregates them, while the DSM Agent iteratively performs self-verification and self-refinement over an initial pool of solutions; see Appendix G. In both cases, the model is used not only to generate reasoning traces from scratch, but also to reason conditioned on self-generated context produced via the scaffold. **Evaluation results.** We report results on HMMT 2025 (Nov) in Table 2, which shows that RCT-4B leverages both scaffolds far more effectively than the base or RL-trained Qwen3-4B-Instruct-2507 models, even without RC decoding. This suggests that RC training imparts a generalizable capability: reasoning effectively from self-generated abstractions beyond summaries, including aggregated past traces (RSA) and self-generated feedback (DSM Agent). Replacing standard autoregressive decoding with RC for all solution-generation steps within these scaffolds yields additional gains, which we attribute to improved reasoning accuracy throughout the scaffolded process. Overall, these results indicate that RC training develops abstraction-conditioned reasoning abilities, highlighting an important direction for future work.
RSA DSM Agent
Base 66.3 57.5
RL 64.8 61.3
RCT-4B (no RC) 70.2 65.8
RCT-4B + RC (Ours) 75.4 74.6
**Table 2: RCT-4B yields additional gains when incorporated into test-time scaffolds.** These gains are higher than with the base or standard RL-trained model. Using RC decoding within the scaffold yields further gains. ### Key Takeaways: Training with RC improves extrapolation across domains - • RCT-4B outperforms iterative decoding and autoregressive decoding baselines. - • RC enables models to solve difficult problems that cannot be solved by scaling parallel compute. - • RC training yields models that are better at summary- and abstraction- conditioned reasoning in general, and can therefore effectively leverage scaffolds to further scale test-time compute.## 8. Conclusion and Perspectives on Future Work In this work, we demonstrate how LLMs can be trained to continually improve their reasoning across long horizons. Our method, RC, replaces autoregressive decoding with an iterative decoding algorithm that alternates between summarization and summary-conditioned reasoning, and trains models via outcome-reward RL to leverage this algorithm more effectively. We show that using RC allows us to overcome a fundamental limitation of autoregressive generation and standard RL: the inability to extrapolate reasoning beyond training rollout lengths. Empirically, we demonstrate that our RC-trained model achieves substantial performance gains on challenging mathematical and scientific benchmarks by extrapolating reasoning to much longer horizons than it was trained for. Furthermore, we show that extrapolation via RC enables the model to solve difficult problems that it cannot solve within its training budget, and that the trained model can leverage existing test-time scaffolds to further scale inference compute. Ultimately, we believe that RC represents an important step toward training models that can engage in the systematic, long-horizon reasoning required to solve the world’s most difficult problems. **Future work.** Future work should address the limitations of RC (see Appendix E). We identify three main directions. First, improving the training objective beyond myopic rewards: our current approach assigns rewards based on individual trace correctness, which discourages multi-turn strategies in which early turns perform exploratory, low-reward procedures that are only exploited in later turns. As we discussed in Section 5.1, we do not expect this to be a problem when each iteration of summarization still makes useful progress, but training with a non-myopic extension of RC could potentially yield significant improvements on harder problems. Second, explicitly training for summary generation: our current approach only trains the model for summary-conditioned generation. Our experiments in Section 5.2, however, show that altering summaries can significantly affect the performance of RC, suggesting that directly optimizing summary generation could also be beneficial. This direction is also closely related to prior work on training models to produce useful abstractions for guiding reasoning [28, 45] or in training aggregation mechanisms [40], with both lines of work demonstrating tangible performance gains. Third, adapting RC to tasks without final-answer based rewards: while our current work focuses on settings with closed-form, verifiable outcomes, these tasks represent only a minority of the reasoning challenges we ultimately care about. Broadening RC to operate in open-ended domains, such as proof generation, is therefore an exciting and potentially impactful direction for future research. ## Acknowledgements We would like to thank Matthew Yang, Haoran Li, Lewis Tunstall, Jasper Dekoninck, and the other members of the CMU AIRe lab for discussions and feedback. We would also like to thank the CMU FLAME center for providing the compute resources on the Orchard cluster that supported almost all of our big experiments. Additionally, we are grateful for the Delta AI cluster at NCSA and the TRC program of Google Cloud for additional computational resources. This work was supported by the Office of Naval Research under ONR N0014-24-2206 and a Schmidt Sciences AI2050 Early Career Fellowship. AS is supported by a JP Morgan AI PhD fellowship and YQ is supported by an Amazon AI PhD fellowship. ## References - [1] Milad Aghajohari, Kamran Chitsaz, Amirhossein Kazemnejad, Sarath Chandar, Alessandro Sordoni, Aaron Courville, and Siva Reddy. The markovian thinker: Architecture-agnostic linear scaling ofreasoning, 2025. URL . [2] Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Zhong, Jingjing Xu, Xipeng Qiu, Mingxuan Wang, and Lingpeng Kong. Polaris: A post-training recipe for scaling reinforcement learning on advanced reasoning models, 2025. URL . [3] Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acereason-nemotron: Advancing math and code reasoning through reinforcement learning. *arXiv preprint arXiv:2505.16400*, 2025. [4] Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training, 2025. URL . [5] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL . [6] Justin Fu, Aviral Kumar, Matthew Soh, and Sergey Levine. Diagnosing bottlenecks in deep Q-learning algorithms. *arXiv preprint arXiv:1902.10250*, 2019. [7] Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D. Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars, 2025. URL . [8] Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni-math: A universal olympiad level mathematic benchmark for large language models, 2024. URL . [9] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Honghui Ding, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jingchang Chen, Jingyang Yuan, Jinhao Tu, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaichao You, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingxu Zhou, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Tao Yun, Tian Pei, Tianyu Sun, T. Wang, Wangding Zeng, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, XinyiZhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. *Nature*, 645 (8081):633–638, September 2025. ISSN 1476-4687. doi: 10.1038/s41586-025-09422-z. URL . - [10] Matthew Ho, Chen Si, Zhaoxiang Feng, Fangxu Yu, Yichi Yang, Zhijian Liu, Zhiting Hu, and Lianhui Qin. Arcmemo: Abstract reasoning composition with lifelong llm memory, 2025. URL . - [11] Kelly Hong, Anton Troynikov, and Jeff Huber. Context rot: How increasing input tokens impacts llm performance. Technical report, Chroma Research, July 2025. URL . - [12] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet, 2024. URL . - [13] Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks, 2023. URL . - [14] Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, and Aleksandra Faust. Training language models to self-correct via reinforcement learning, 2024. URL . - [15] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In *Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles*, 2023. - [16] Yucheng Li, Bo Dong, Chenghua Lin, and Frank Guerín. Compressing context to enhance inference efficiency of large language models, 2023. URL . - [17] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts, 2023. URL . - [18] LM-Provers, Yuxiao Qu, Amrith Setlur, Jasper Dekoninck, Edward Beeching, Jia Li, Ian Wu, Lewis Tunstall, and Aviral Kumar. Qed-nano: Teaching a tiny model to prove hard theorems. , 2026. Blog post.- [19] Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl, 2025. Notion Blog. - [20] Thang Luong, Dawsen Hwang, Hoang H. Nguyen, Golnaz Ghiasi, Yuri Chervonyi, Insuk Seo, Junsu Kim, Garrett Bingham, Jonathan Lee, Swaroop Mishra, Alex Zhai, Clara Huiyi Hu, Henryk Michalewski, Jimin Kim, Jeonghyun Ahn, Junhwi Bae, Xingyou Song, Trieu H. Trinh, Quoc V. Le, and Junehyuk Jung. Towards robust mathematical reasoning, 2025. URL . - [21] Aman Madaan, Niket Tandon, Prakash Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. *arXiv preprint arXiv:2303.17651*, 2023. - [22] Lovish Madaan, Aniket Didolkar, Suchin Gururangan, John Quan, Ruan Silva, Ruslan Salakhutdinov, Manzil Zaheer, Sanjeev Arora, and Anirudh Goyal. Rethinking thinking tokens: Llms as improvement operators, 2025. URL . - [23] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, 2025. URL . - [24] OpenAI. Evaluating ai’s ability to perform scientific research tasks (frontierscience). OpenAI Research Publication, December 2025. URL . Accessed: 2026-01-20. - [25] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models, 2023. URL . - [26] Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve, 2024. URL . - [27] Yuxiao Qu, Amrith Setlur, Virginia Smith, Ruslan Salakhutdinov, and Aviral Kumar. Learning to reason on hard problems with privileged on-policy exploration. In *The 5th Workshop on Mathematical Reasoning and AI at NeurIPS 2025*, 2025. URL . - [28] Yuxiao Qu, Anikait Singh, Yoonho Lee, Amrith Setlur, Ruslan Salakhutdinov, Chelsea Finn, and Aviral Kumar. Rlad: Training llms to discover abstractions for solving reasoning problems, 2025. URL . - [29] Yuxiao Qu, Matthew Y. R. Yang, Amrith Setlur, Lewis Tunstall, Edward Emanuel Beeching, Ruslan Salakhutdinov, and Aviral Kumar. Optimizing test-time compute via meta reinforcement fine-tuning, 2025. URL . - [30] Qwen Team. Qwen3 technical report, 2025. URL . - [31] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. *CoRR*, abs/1707.06347, 2017. URL . - [32] Amrith Setlur, Nived Rajaraman, Sergey Levine, and Aviral Kumar. Scaling test-time compute without verification or rl is suboptimal. *arXiv preprint arXiv:2502.12118*, 2025.- [33] Amrith Setlur, Matthew Y. R. Yang, Charlie Snell, Jeremy Greer, Ian Wu, Virginia Smith, Max Simchowitz, and Aviral Kumar. e3: Learning to explore enables extrapolation of test-time compute for llms, 2025. URL . - [34] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL . - [35] Zhihong Shao, Yuxiang Luo, Chengda Lu, Z. Z. Ren, Jiewen Hu, Tian Ye, Zhibin Gou, Shirong Ma, and Xiaokang Zhang. Deepseekmath-v2: Towards self-verifiable mathematical reasoning, 2025. URL . - [36] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. *arXiv preprint arXiv: 2409.19256*, 2024. - [37] Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. *arXiv preprint arXiv:2303.11366*, 2023. - [38] Yiyou Sun, Yuhan Cao, Pohao Huang, Haoyue Bai, Hannaneh Hajishirzi, Nouha Dziri, and Dawn Song. Rl grokking recipe: How does rl unlock and transfer new algorithms in llms?, 2025. URL . - [39] Mirac Suzgun, Mert Yuksekgonul, Federico Bianchi, Dan Jurafsky, and James Zou. Dynamic cheatsheet: Test-time learning with adaptive memory, 2025. URL . - [40] Siddarth Venkatraman, Vineet Jain, Sarthak Mittal, Vedant Shah, Johan Obando-Ceron, Yoshua Bengio, Brian R. Bartoldson, Bhavya Kaikkhura, Guillaume Lajoie, Glen Berseth, Nikolay Malkin, and Moksh Jain. Recursive self-aggregation unlocks deep thinking in large language models, 2025. URL . - [41] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023. URL . - [42] Colin Wei, Jason Lee, Qiang Liu, and Tengyu Ma. Regularization matters: Generalization and optimization of neural nets vs their induced kernel. 2019. - [43] Tianxin Wei, Noveen Sachdeva, Benjamin Coleman, Zhankui He, Yuanchen Bei, Xuying Ning, Mengting Ai, Yunzhe Li, Jingrui He, Ed H. Chi, Chi Wang, Shuo Chen, Fernando Pereira, Wang-Cheng Kang, and Derek Zhiyuan Cheng. Evo-memory: Benchmarking llm agent test-time learning with self-evolving memory, 2025. URL . - [44] Ian Wu, Patrick Fernandes, Amanda Bertsch, Seungone Kim, Sina Pakazad, and Graham Neubig. Better instruction-following through minimum bayes risk, 2025. URL . - [45] Matthew Y. R. Yang, Hao Bai, Ian Wu, Gene Yang, Amrith Setlur, and Aviral Kumar. Int: Self-proposed interventions enable credit assignment in llm reasoning, 2026. URL .- [46] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source llm reinforcement learning system at scale, 2025. URL . - [47] Charlie Zhang, Graham Neubig, and Xiang Yue. On the interplay of pre-training, mid-training, and rl on reasoning language models, 2025. URL . - [48] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023. URL . - [49] Zijian Zhou, Ao Qu, Zhaoxuan Wu, Sunghwan Kim, Alok Prakash, Daniela Rus, Jinhua Zhao, Bryan Kian Hsiang Low, and Paul Pu Liang. Meml: Learning to synergize memory and reasoning for efficient long-horizon agents, 2025. URL .# Appendices ## A. Additional Results vs. Larger Reasoning Models **Figure 11:** Comparison of RCT-4B and a selection of other reasoning models on IMO-AnswerBench. Combining RC training and decoding enables our 4B model to outcompete many larger and newer models. We set inference hyperparameters ( $t, p, H_{\text{test}}$ ) based on the recommended values provided on each model’s Hugging Face page. ## B. Comparison with Majority Voting **Figure 12:** Accuracy and Maj@16 against sequential reasoning token budget for Qwen3-4B-Instruct-2507 and RCT-4B on IMO-AnswerBench. While majority voting can be used to improve RC, we find that utilizing compute to increase “depth” through RC is more effective than increasing “breadth” by taking majority vote over more parallel samples. In Figure 12, we plot accuracy and Maj@16 against sequential reasoning token budgets using RC decoding (true reasoning token budget for Maj@16 is the sequential reasoning token budget multiplied by 16). While majority voting can be used to improve RC, we find that utilizing compute to increase “depth” through RC is more effective than increasing “breadth” by taking majority vote over more parallel samples for our tested value of $k$ : in other words, Maj@16 performance at 16k sequential reasoning token budget**Figure 13: Performance of gpt-oss models with RC decoding (no training) on IMO-AnswerBench.** We use $H_R = 64k$ to adjust for the models’ larger output lengths, and set reasoning effort to “high” for generation and “medium” for summarization. Both models benefit from extrapolation through RC. We evaluate answer correctness using Gemini-3-Flash as an LLM judge. (giving a true reasoning token budget of 256k) is significantly worse than accuracy at 256k tokens with RC. This applies both for the base Qwen3 model and for RCT-4B. ### C. Motivating our Choices of $H_R$ In this section, we motivate the choices of $H_R$ (autoregressive decoding maximum token budget) we use throughout this work. For RC decoding, $H_R$ determines the length of individual reasoning traces within each turn. As discussed in Section 5.2, we generally choose $H_R$ to be $H_{\text{train}}$ . Unfortunately, the exact value of $H_{\text{train}}$ is typically not made public, so we must estimate it through the termination length statistics of the model: if the model generally terminates its reasoning within some length $L$ , then we can reasonably say that $L \approx H_{\text{train}}$ . Regardless, the general idea is that increasing the reasoning token budget beyond $L$ will not yield any gains in performance with autoregressive decoding, as the model will simply not generate anything longer than this. For our base Qwen3-4B-Instruct-2507 model, we set $H_R = 16k$ , which yields a very high termination rate of 99.17% (see Table 3. Our trained RCT-4B model also attains a very high termination rate of 98.75%, indicating that our training has not resulted in increased repetitiveness or undue verbosity. For our autoregressive decoding baselines, we choose the maximum token budget based on the values recommended on each model’s Hugging Face model card. As we see from Table 3, reasoning traces generated at these lengths do indeed overwhelmingly terminate successfully. A note on input context windows: many models have a stated maximum context windows that are very large. For example, Qwen3-4B-Instruct-2507 has a stated maximum context window of 262,144 tokens. However, we note that they almost never generate reasoning traces of lengths greater than 16k tokens: see Table 3. This is likely because the models were post-trained to generate outputs of up to 16k tokens in length: the 262,144 context window is only utilized for processing long-context *inputs*. ### D. RC with gpt-oss Please see results showing the efficacy of our approach on this model family in Figure 13.
Model H_R Termination Rate (%)
Qwen3-4B-Instruct-2507 16k 99.17
Qwen3-4B-Thinking-2507 81k 100.00
Polaris-4B 90k 99.79
Qwen3-4B-Instruct-2507 + Std. RL @32k 32k 99.17
RCT-4B 16k 98.75
Table 3: Termination rates for different models on HMMT 2025. We determine termination by the presence of the boxed{} pattern in the model output. ## E. Limitations of RC and Directions for Future Work While RC training yields strong empirical results, our method has several limitations that we outline in this section. We hope this discussion provides useful directions for future work. **RC training does not optimize summary generation.** Our training focuses exclusively on summary-conditioned reasoning, based on the observation that this is the primary performance bottleneck while summary generation is inherently easier for pre-trained and instruction-tuned models. We validated this assumption through preliminary experiments where we assigned rewards to summary generation based on the proportion of subsequent reasoning traces (conditioned on the summary) that produced correct answers. More formally, we optimized the following objective: $$\max_{\pi_{\theta}} \mathbb{E}_{\substack{x, y \sim \mathcal{D}_{\text{train}} \\ t \sim U[1, T_{\text{train}}]}} \left[ \mathbb{E}_{z_S^{(t)} \sim \pi_{\theta}(\cdot | \mathcal{I}_S, x, z_R^{(t)}, z_S^{(t-1)})} \left[ \frac{1}{K} \sum_{k=1}^K r(y, z_{R,k}^{(t+1)}) \right] \right],$$ where $z_{R,k}^{(t+1)} \sim \pi_{\theta}(\cdot | x, z_S^{(t)})$ for $k = 1, \dots, K$ . (E.1) We tested this both in isolation and in combination with our usual summary-conditioned generation objective, using the same hyperparameters as in Section 6 for Stage I training. As illustrated in Figure 14, optimizing for summary generation only (“Summarization Trn.”) hurts the efficacy of training, such that the resulting model is no better than the base model. Optimizing for both summary generation and summary-conditioned generation (“Both Trn.”) improves performance relative to the base model but hurts performance relative to the model trained only for summary-conditioned generation (“Reasoning Trn.”). These findings motivate the design of our main approach. We attribute these results to difficulties in credit assignment for summary generation. Even when the model generates faithful, informative summaries, it receives zero reward if subsequent reasoning fails to solve the problem, which may occur simply because the problem is too difficult to solve in a single turn and not because the summary is poor. This misalignment between summary quality and reward signal makes it difficult to effectively train summarization, although we posit that doing so effectively could further improve RC performance. Addressing this likely requires alternative reward assignment schemes for summary generation, which we leave to future work. **RC training uses myopic rewards.** RC is trained by optimizing Equation 6.1 using outcome-based rewards assigned independently at each iteration. That is, the reasoning trace generated at turn $t$ receives reward based solely on its own correctness, without explicit credit assignment to future turns $t + 1, t + 2, \dots$ Figure 14: Training for summarization generation using objective E.1 hurts performance. Training this way either in isolation or in combination with the usual summary-conditioned reasoning objective negatively impacts performance.As discussed in Section 6, this design relies on the observation that training each step to produce a correct answer implicitly encourages the model to generate reasoning and summaries that contain information useful for subsequent iterations. When each summarization and summary-conditioned generation step makes monotonic progress toward a correct solution, such myopic rewards are well aligned with trajectory-level success. In this regime, summaries serve as sufficient representations of past reasoning, and improving per-step correctness also improves the quality of information available for future steps. Indeed, under the assumption that the model can reliably extract maximal useful information from a summary when needed, optimizing per-step correctness is sufficient to optimize the long-horizon objective over multiple iterations. Moreover, by conditioning training rollouts on these summaries, the model is explicitly trained to exploit accumulated information, further increasing the likelihood of success at subsequent steps. As a result, we can optimize correctness over the full reasoning trajectory despite only explicitly optimizing individual steps. The main limitation of our approach is that the model is not incentivized to generate reasoning that is suboptimal for the current step but valuable later. For example, the model may benefit from exploring alternative solution strategies or collecting auxiliary information that only becomes useful later in the trajectory. Learning such far-sighted reasoning behaviors may be particularly important for very difficult problems requiring extensive in-context exploration. However, designing reward schemes that effectively encourage such multi-step contributions remains challenging, and we leave this direction to future work. **Summarization-generation asymmetry is not present in all LLMs.** Our analysis in Section 5.2 reveals that RC yields the most benefits when the underlying model possesses strong summarization-generation asymmetry, and that instruction-following models generally possess this asymmetry whereas highly specialized reasoning models do not. This limits the kinds of models we can apply RC to. We propose several potential solutions to this problem. The first involves warmstarting the reasoning model for summarization and summary-conditioned generation, perhaps through distillation or SFT. This approach, however, may potentially alter the reasoning behavior of the model in a detrimental way. The second solution involves using a separate model to perform summarization generation, which we previously identified as a particularly difficult task for specialized reasoning models. This approach, however, would then require us to maintain two separate models, which could pose certain practical challenges. **RC does not improve performance on all classes of reasoning problems.** While our experiments show that RC decoding and training improves model performance across mathematical and scientific reasoning benchmarks, we posit that not all classes of problem classes benefit from RC. One class of such problems are search-heavy problems, where the model must iterate through a large number of possible outcomes and select the optimal choice. The main issue here is that the *redundancy* property no longer applies as strongly as before, as many tokens generated may be important as they document the search process and keep track of what has been tried. Summarizing such traces risks discarding important information that may reduce test-time performance on the search task. **When is RC helpful?** To understand the classes of problems for which RC is most effective, it is useful to conceptualize the solution space as a graph, where nodes represent conclusions, intermediate results, or other salient states, and edges represent logical transitions between them. RC is particularly well-suited to problems whose solution graphs are *clique-like*: nodes tend to form loosely connected clusters that can be summarized compactly, with relatively sparse connections between clusters. Mathematical and scientific reasoning often exhibit this structure, as progress can be decomposed into distinct conceptual advances that admit concise summaries.In such settings, RC enables effective reasoning by allowing each iteration to explore different regions of the graph, while maintaining long-term progress by summarizing and tracking information within individual clusters. In contrast, for search-heavy problems this clique structure is largely absent, and effective reasoning requires tracking a large number of individual nodes encountered during exploration. In these cases, summaries must enumerate prior states rather than abstract them, causing limited-length summaries to quickly become overwhelmed and reducing the effectiveness of iterative summarization. In addition to scientific and mathematical reasoning, RC may also be helpful on tasks where actions yield environment feedback that is noisy and can benefit from summarization (e.g. coding with interpreter feedback). In this case, summaries may be used to keep track of environment state, as has been explored in related work [49]. Unlike these works, we focus primarily on creating abstractions of reasoning rather than environment states, but these ideas are closely related and can likely be combined. ## F. Discussions on Computational Efficiency ### F.1. Inference We begin by analyzing the computational efficiency of RC decoding compared to standard long-context autoregressive generation. We examine how RC scales with reasoning length and whether extrapolating via RC is more efficient than training models to natively handle larger token budgets through autoregressive decoding. **Notation and definitions.** Let $C$ be the input problem length, and let $N$ be the maximum generation length for standard autoregressive decoding. Under RC decoding, the model proceeds for $T$ turns, generating at each turn a reasoning statement of length $\leq H_R$ followed by a summary of length $\leq H_S$ , with $H_R \gg H_S$ . Our analysis focuses on decoder-only transformers with KV-cached decoding, where for long contexts, attention computation dominates and scales linearly with current context length. **Standard Long-Context Generation.** In standard autoregressive decoding, the model generates $N$ tokens in a single trajectory, with the context growing from length $C$ to $C + N$ . With KV caching, the incremental cost of generating the $i$ -th token scales linearly with the context length $C + i$ . Summing over all tokens, the total attention-dominated inference cost (IC) scales as $$IC_{\text{standard}} \propto \sum_{i=1}^N (C + i) = NC + \frac{1}{2}N(N + 1) = \Theta(N(C + N)) \quad (\text{F.1})$$ **RC Inference.** In RC, each reasoning step is conditioned only on the original prompt and the current summary, rather than the full previous chain-of-thought. As a result, the effective context length within each turn is bounded by approximately $C + H_S + H_R$ , and **does not grow across turns**. Across $T$ turns, the total inference compute is therefore $$IC_{\text{RC}} \propto T \cdot H_R (C + H_S + H_R) \quad (\text{F.2})$$ **Figure 15: Plot of decoding throughput against reasoning token budget.** RC decoding throughput remains constant as reasoning token budget, whereas throughput for standard autoregressive decoding decreases.**Inference Speedup.** For a fixed effective reasoning budget $N = TH_R$ , the inference speedup of RC is approximately $$\text{Speedup} = \frac{\text{IC}_{\text{standard}}}{\text{IC}_{\text{RC}}} \approx \frac{C + TH_R}{C + H_S + H_R} \approx T. \quad (\text{F.3})$$ as $TH_R \gg C$ and $H_R \gg H_S$ . Therefore, to reach $N = TH_R$ effective reasoning tokens, RC can be $\approx T$ times cheaper than autoregressive decoding in attention-dominated regimes. **Empirical study.** We conduct experiments to study the computational efficiency of RC. We run RC decoding using Qwen3-4B-Instruct-2507 and standard autoregressive decoding using Qwen3-4B-Thinking-2507 at different reasoning token budgets, logging throughput on HMMT 2025 (30 prompts) with 8 parallel rollouts. For the RC runs, we experiment with $H_R \in \{8\text{k}, 16\text{k}\}$ . We plot decoding throughput against reasoning token budget in Figure 15. This plot demonstrates that autoregressive decoding throughput rapidly decreases as token budgets increase, whereas RC decoding throughput remains constant. This is expected because RC maintains bounded context length across turns even as the effective reasoning horizon grows. RC therefore proves substantially more efficient than autoregressive decoding despite our use of a highly optimized inference engine for the latter and a naive implementation for the former: see Appendix L for implementation and hardware details. ## F.2. Training **Standard long-context RL baseline.** In both standard RL training, we perform on-policy RL (e.g., GRPO) with batch size (problems per step) = $B$ and samples per problem = $K$ (GRPO group size). Then each step generates roughly $B \cdot K \cdot N$ tokens, where $N$ is the sequence length. Since the attention cost scales with the growing context length, the forward generation compute scales as: $$\text{GenCompute}_{\text{standard}} \propto B \cdot K \cdot N(C + N). \quad (\text{F.4})$$ Including backward and optimizer computation introduces a constant multiplicative factor $\gamma$ , yielding $$\text{TrainCompute}_{\text{long}} \approx \gamma \cdot B \cdot K \cdot N(C + N). \quad (\text{F.5})$$ When $N$ is large, this scales quadratically with the rollout horizon. **RC Training.** RC training separates trajectory construction from policy optimization. Each training step consists of: (1) summary-trajectory generation: The model runs RC for $T_{\text{train}}$ turns to produce a sequence of summaries; (2) policy optimization: from this trajectory, $N_{\text{summ}}$ summaries are sampled, and for each summary, $K$ reasoning rollouts of length at most $H_R$ are generated and optimized via GRPO. The total forward generation compute per training step scales as $$\text{GenCompute}_{\text{RC}} \propto B \cdot (T_{\text{train}} + KN_{\text{summ}}) \cdot H_R(C + H_S + H_R). \quad (\text{F.6})$$ Including backward and optimizer cost yields $$\text{TrainCompute}_{\text{RC}} \approx \gamma \cdot B \cdot (T_{\text{train}} + KN_{\text{summ}}) \cdot H_R(C + H_S + H_R). \quad (\text{F.7})$$ Crucially, all optimized rollouts remain bounded by length $H_R$ , regardless of the total effective reasoning horizon supported at inference time.**Training-Time Scaling Comparison.** To reach an effective horizon $N = T_{\text{target}} H_R$ , standard long-context RL training incurs compute scaling approximately as $$\text{TrainCompute}_{\text{standard}} \propto B \cdot K \cdot T_{\text{target}}^2 H_R^2 \quad (\text{F.8})$$ while RC training scales as $$\text{TrainCompute}_{\text{RC}} \propto B \cdot (T_{\text{train}} + K N_{\text{summ}}) \cdot H_R^2 \quad (\text{F.9})$$ Thus, the relative cost satisfies $$\frac{\text{TrainCompute}_{\text{RC}}}{\text{TrainCompute}_{\text{standard}}} \approx \frac{T_{\text{train}} K N_{\text{summ}}}{K \cdot T_{\text{target}}^2}. \quad (\text{F.10})$$ This highlights a key advantage of RC: naively increasing rollout length leads to quadratic growth in training cost, whereas RC decouples the optimized rollout length from the effective reasoning horizon. By using summaries and replay, RC enables training policies that generalize to very long reasoning horizons without incurring prohibitive quadratic costs during optimization. ### F.3. Inference KV-Cache Memory The KV cache memory footprint for autoregressive decoding scales linearly with the context length: $$\text{Memory}_{\text{standard}} \propto C + N, \quad (\text{F.11})$$ while for RC, it is bounded by the maximum within-turn context length: $$\text{Memory}_{\text{RC}} \propto C + H_S + H_R, \quad (\text{F.12})$$ which is independent of $T$ . Putting these together, RC requires $\sim T \times$ lower KV memory at the same effective reasoning horizon: $$\frac{\text{Memory}_{\text{standard}}}{\text{Memory}_{\text{RC}}} \approx \frac{C + TH_R}{C + H_S + H_R} \approx T.$$ ## G. Details for Test-Time Scaffold Experiments ### G.1. Recursive Self-Aggregation In Section 7.4, we also experiment with incorporating RC into RSA [40], a scaffold that iteratively refines solutions through sampling and aggregation. In its original form, the algorithm begins by sampling $M$ solutions from scratch (conditioned only on the problem). Then, in each subsequent iteration, the algorithm creates $M$ new solutions by randomly sampling $k$ candidates from the current pool of solutions (with replacement) and prompting the model to aggregate them into a single improved solution. Over the $T_{\text{RSA}}$ successive loops, solutions compound recursively: aggregated outputs become inputs for the next round, progressively eliminating errors and reinforcing correct solutions while maintaining a constant population of $M$ solutions. We incorporate RC into RSA by replacing (1) the initial solution generation step and (2) subsequent refinement steps with RC decoding. We begin the refinement step by treating the aggregated solution as a summary that we condition on for the first step of RC refinement. For our experiments in Section 7.4, we use $k = 2$ , $M = 8$ , and $T_{\text{RSA}} = 10$ , and for the experiment incorporating RC decoding, we set the number of RC steps as $T = 8$ .## G.2. DeepseekMath Agent We also experiment with a test-time scaffold we call DeepseekMath Agent (DSM Agent). This is adapted from the scaffold used in Shao et al. [35] to improve the ability of LLMs to generate proofs for mathematical reasoning problems. At a high level, the DSM Agent implements a Generate-Verify-Refine loop that uses self-verification to iteratively improve solutions. It begins by generating an initial pool (of size $n_g$ ) of candidate solutions, and then verifies each solution using $n_v$ self-verification attempts per solution (assigning scores of 0.0 for major errors, 0.5 for minor issues, 1.0 for correct), with the final verification score determined by averaging over the $n_v$ scores. In each of the subsequent refinement iterations, the algorithm selects the highest-scoring solutions and refines them using feedback from their lowest-scoring verifications. Refined solutions are added to the growing pool and re-verified, with this process repeating until either (1) a perfect score is achieved, or (2) the maximum $T_{\text{DSM}}$ iterations are reached. At the end, the algorithm returns the highest-scoring solution as the final answer. We incorporate RC into DSM Agent by replacing the initial solution generation step with RC decoding, with the aim of improving the quality of the initial pool of candidates. For our experiments in Section 7.4, we use $n_g = 8$ , $n_v = 4$ , and $T_{\text{DSM}} = 6$ , and for the experiment incorporating RC decoding, we set the number of RC steps as $T = 8$ . ## H. Overview of GRPO GRPO optimizes the following objective: $$\mathcal{J}(\theta) = \mathbb{E}_{\mathbf{x}, \mathbf{y} \sim \mathcal{D}_{\text{train}}} \mathbb{E}_{\mathbf{z}_i \sim \pi_\theta(\cdot | \mathbf{x})} \left[ \frac{1}{K} \sum_{i=1}^K \min \left[ \frac{\pi_\theta(\mathbf{z}_i | \mathbf{x})}{\pi_{\text{old}}(\mathbf{z}_i | \mathbf{x})} A_i, \text{clip} \left( \frac{\pi_\theta(\mathbf{z}_i | \mathbf{x})}{\pi_{\text{old}}(\mathbf{z}_i | \mathbf{x})}, 1 - \epsilon, 1 + \epsilon \right) A_i \right] \right]. \quad (\text{H.1})$$ Here, $\mathbf{z}_i$ denotes the $i$ th of $K$ independently sampled rollouts (which taken together form a “group”), and $A_i$ denotes the GRPO advantage, which is computed directly from the rewards as $A_i = \frac{r_i - \text{mean}(r)}{\text{std}(r)}$ , with the mean and standard deviation calculated over group rewards. Some intuitions behind GRPO: - • For a fixed input $\mathbf{x}$ , GRPO assigns advantages to each rollout $\mathbf{z}_i$ relative to the other $K$ samples in the group, so updates depend on whether $\mathbf{z}_i$ is better or worse than its peers rather than on absolute reward values. In the case of RC training, the $K$ parallel rollouts are sampled under the same prompt and summary combination, so we assign higher advantages to summary-conditioned reasoning traces that are better able to leverage the summary to attain the correct answer. - • Normalizing advantages by the group mean and standard deviation stabilizes gradients and makes updates invariant to the overall reward scale across different inputs. - • The clipped ratio $\pi_\theta(\mathbf{z}_i | \mathbf{x})/\pi_{\text{old}}(\mathbf{z}_i | \mathbf{x})$ retains PPO’s [31] trust-region approach, preventing any single high-advantage $\mathbf{z}_i$ from applying overly large updates. ## I. Dataset Construction Details We construct our training datasets by following some of the principles outlined in An et al. [2]. Specifically, we sample problems in a way that ensures our dataset maintains reasonable difficulty given our model.We begin by sampling problems from the AceReason-Math [3] dataset ( $\sim 50k$ problems) and solving them with Qwen3-4B-Instruct-2507, setting $K = 64$ . We then evaluate these solutions and assign a reward score to each problem based on the average number of correct solutions our model generates. These scores are used for weighted sampling: we discard all samples that attain a score of 0.7 or greater, and downsample problems with other reward scores to attain the “J-shaped” reward curve described in Figure 2 of An et al. [2]. This procedure yields a dataset of around 5.7k samples, which we take as our Stage I training set. After Stage I training, our model improves and so we rebalance our training dataset such that it maintains the “J-shaped” reward curve. We reannotate our Stage I dataset with the Stage I model (with standard autoregressive decoding) and once again remove samples that attain reward scores of 0.7 or greater. We then inject $\sim 500$ difficult (zero-reward) problems from the DAPO [46] dataset (as determined via annotation with the base model) and ensure that our Stage II dataset contains challenging problems. ## J. Iterative Decoding Baseline Details In this section, we describe in detail the self-verification and self-refinement iterative decoding baselines that we compare RC against. The purpose of these baselines is to help us separate out the impact of our summarize-generate routine from the impact of using iterative decoding. As such, these baseline methods do not utilize the summarization-generation asymmetry, and instead act directly on the reasoning trace generated by the model, as is common in iterative decoding methods and test-time scaffolds [14, 26, 35]. ### J.1. Inference Concretely, let $x$ denote the input prompt and let $t \in \mathbb{N}$ index the decoding turn. Unlike RC, these baseline methods maintain only a reasoning trace $z_R^{(t)}$ at each turn, with no separate summarization step. At each turn, the reasoning trace $z_R^{(t)}$ is generated under a fixed token budget $H_R$ (we use the same $H_R = 16k$ as in our RC experiments). For **self-refinement**, decoding proceeds by alternately generating reasoning traces and prompting the model to refine them. At each turn $t$ , we sample: $$z_R^{(t)} \sim \pi_\theta(\cdot \mid \mathcal{I}_{\text{refine}}, x, z_R^{(t-1)}), \quad (\text{J.1})$$ where $\mathcal{I}_{\text{refine}}$ instructs the model to improve upon its previous reasoning trace, and $z_R^{(0)}$ is initialized as the empty string. For **self-verification**, the model is prompted to first verify its previous attempt before generating a correction. At each turn $t$ , we sample: $$z_R^{(t)} \sim \pi_\theta(\cdot \mid \mathcal{I}_{\text{verify}}, x, z_R^{(t-1)}), \quad (\text{J.2})$$ where $\mathcal{I}_{\text{verify}}$ instructs the model to verify whether its previous reasoning is correct and, if not, to provide a corrected solution. See Figures 19 and 20 for $\mathcal{I}_{\text{refine}}$ and $\mathcal{I}_{\text{verify}}$ . After $T$ decoding turns, the final output is given by $z := z_R^{(T)}$ for both methods. The key distinction from RC is that these baselines condition on the full previous reasoning trace $z_R^{(t-1)}$ rather than a compressed summary. As such, the model must conditionally generate from sequences up to $2H_R$ in length.## J.2. Training Training follows a similar scheme to RC training, except that we generate rollouts using our baseline iterative decoding methods instead of RC decoding. The idea here is to assess whether utilizing the summarization-generation gap enables us to achieve better performance through training, or whether our iterative training strategy on its own is sufficient to attain significant gains. More formally, at any given point in training, we run the iterative decoding algorithm for $T_{\text{train}}$ turns for each problem $x$ in a training batch. We collect the reasoning traces generated from these rollouts $z_R := (z_R^{(1)}, \dots, z_R^{(T_{\text{train}})})$ and then uniformly sample $N_{\text{trace}} \leq T_{\text{train}}$ unique traces per problem. We then generate $K$ reasoning traces conditioned on each sampled trace. We assign rewards based on correctness and compute advantages over these $K$ samples. Formally, the objective can be written as: $$\max_{\pi_\theta} \mathbb{E}_{x, y \sim \mathcal{D}_{\text{train}}, t \sim U[1, T_{\text{train}}]} \left[ \mathbb{E}_{z' \sim \pi_\theta(\cdot | x, z_R^{(t)})} [r(y, z')] \right], \quad |z'| \leq H_R$$ where $(z_R^{(1)}, \dots, z_R^{(T_{\text{train}})}) \sim \text{IterativeDecoding}(\pi_\theta; x).$ (J.3) We adopt the same training hyperparameters as for RC training in Section 6. ## K. Additional Iterative Decoding Comparisons We evaluate two additional prompting-only iterative decoding baselines beyond the self-refinement and self-verification methods described in Appendix J. Rather than conditioning directly on the full reasoning trace, these baselines modify the trace before conditioning on them for subsequent generation. ### K.1. Budget Forcing
AIIME 2025 HMMT 2025 (Nov) FrontierScience
Qwen3-4B-Instruct-2507 [16k] 46.0 39.8 23.3
Qwen3-4B-Instruct-2507 + RC 59.4 56.7 29.5
RCT-4B + RC (Ours) 74.9 66.3 34.1
Budget Forcing [23] 56.7 46.7 20.1
Table 4: Comparison of budget forcing with Qwen3-4B-Instruct-2507 against the autoregressive baseline and against RC. Budget forcing yields modest gains on mathematical reasoning benchmarks, but degrades performance on FrontierScience vs. the autoregressive baseline. RC-based methods outperform budget forcing on all benchmarks. The first baseline we evaluate is a budget forcing approach, inspired by Muennighoff et al. [23]. In this approach, we repeatedly append "Wait, let me continue thinking" after termination to elicit additional reasoning. We use $H_R = 16\text{k}$ , and report our results in Table 4. Table 4 shows that budget forcing yields modest gains on mathematical reasoning benchmarks relative to autoregressive decoding but degrades FrontierScience performance. Both RC-based methods substantially outperform budget forcing across all benchmarks. Analyzing output traces reveals two failure modes. First, budget forcing rarely changes the final answer after the first few turns, with continuations quickly reducing. We hypothesize that simply appending "Wait . . ." is insufficient encouragement for the model to engage in meaningful in-context exploration, unlike conditioning on entire summaries of past reasoning.