Awesome-Long-Chain-of-Thought-Reasoning

Awesome-Long-Chain-of-Thought-Reasoning

[English Tutorial] | [中文教程] | [Arxiv]

🔥 News

• 2025.07: 🎉🎉🎉 We have updated the number of reviewed papers to over 1000. Additionally, we have added bilingual supports and updated our repository more friendly for Long-CoT beginner.
• 2025.04: 🎉🎉🎉 We have updated the number of reviewed papers to over 900. Additionally, we have enhanced the presentation with more engaging teaser figure.
• 2025.03: 🎉🎉🎉 We have published a survey paper titled “Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models”. Please feel free to cite or open pull requests for your awesome studies.

🌟 Introduction

Welcome to the repository associated with our survey paper, “Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models”. This repository contains resources and updates related to our ongoing Long CoT research. For a detailed introduction, please refer to our survey paper.

Recent advancements in reasoning with large language models (RLLMs), such as OpenAI-O1 and DeepSeek-R1, have demonstrated their impressive capabilities in complex domains like mathematics and coding. A central factor in their success lies in the application of long chain-of-thought (Long CoT) characteristics, which enhance reasoning abilities and enable the solution of intricate problems.

However, despite these developments, a comprehensive survey on Long CoT is still lacking, limiting our understanding of its distinctions from traditional short chain-of-thought (Short CoT) and complicating ongoing debates on issues like “overthinking” and “test-time scaling.” This survey seeks to fill this gap by offering a unified perspective on Long CoT.

1. We first distinguish Long CoT from Short CoT and introduce a novel taxonomy to categorize current reasoning paradigms.
2. Next, we explore the key characteristics of Long CoT: deep reasoning, extensive exploration, and feasible reflection, which enable models to handle more complex tasks and produce more efficient, coherent outcomes compared to the shallower Short CoT.
3. We then investigate key phenomena such as the emergence of Long CoT with these characteristics, including overthinking, and test-time scaling, offering insights into how these processes manifest in practice.
4. Finally, we identify significant research gaps and highlight promising future directions, including the integration of multi-modal reasoning, efficiency improvements, and enhanced knowledge frameworks.

By providing a structured overview, this survey aims to inspire future research and further the development of logical reasoning in artificial intelligence.

🕹️ Content

0. How to Learn & About Us

We aim to help newcomers quickly establish domain knowledge, so our design concept is as follows: briefly introduce the main technologies involved in reasoning large models and Long CoT, allowing everyone to understand which problems different technologies can address, so that when they wish to delve deeper into the field in the future, they will have a clear starting point.

We are a team of beginners in reasoning large models, and we hope that through our own learning experiences, we can offer some assistance to future learners, accelerating the popularization and application of reasoning large models. We welcome more friends to join our project, and we are also open to friendship and academic collaboration. For any inquiries, please feel free to contact us via email at [email protected].

Daily Knowledge Resources

• Social Media:
  • Recommended WeChat Public Accounts: JIQIZHIXIN, Paper Weekly, MLNLP…
  • Recommended Twitter Accounts: AK, elvis, Philipp Schmid, …
• Cutting-edge Courses: CS336
• Community Sharing: MLNLP, JIQIZHIXIN, BAAI, NICE Academic

1. Classical Reasoning Model

• OpenAI-o1 / o3 / o4: The earliest reasoning large language models exploring Long CoT, developed by OpenAI’s first-tier models.
• Gemini: First-tier reasoning large language models developed by Google.
• Deepseek-r1: The first open-source reasoning large language model with Long CoT.
• QwQ: The first open-source large-scale reasoning large language model with Long CoT.
• Qwen3: The most commonly used open-source reasoning large language models for Long CoT developed by Alibaba.
• Seed-Thinking-v1.5: ByteDance’s open-source reasoning model for Long CoT.
• Kimi-k1.5: The earliest multimodal reasoning model for Long CoT developed by Moonshot.
• MiniMax-m1: The open-source reasoning model for Long CoT developed by MiniMax.

2. Introduction to Long-CoT Capabilities

In this chapter, we will provide the most representative technologies for each capability, along with the latest developments. A detailed list of papers can be found in the complete list.

2.1 Deep Reasoning

The core of deep reasoning ability lies in the need for sufficient logical depth to manage a large number of reasoning nodes. Without this capability, the performance of reasoning large language models (RLLMs) significantly degrades. Current methods for enhancing deep reasoning can be categorized into two main approaches: Deep Reasoning Format and Deep Reasoning Learning.

2.1.1 Deep Reasoning Format

Since reasoning models heavily depend on the format of reasoning, they tend to achieve the deepest reasoning paths in the forms they excel at. As a result, some works have begun exploring better reasoning formats for deeper reasoning.

Natural Language Deep Reasoning

• Core Idea: Aims to express deep reasoning through natural language formats.
• Representative Works:
  • Natural Program: ensures more structured and rigorous logical analysis.
  • Code I/O: restructures code-based reasoning patterns into natural language forms, further unleashing the reasoning potential of RLLMs.

---

Structured Language Deep Reasoning

• Core Idea: Aims to enhance deep reasoning through programmatic or symbolic language formats. Current research primarily focuses on using code to improve mathematical reasoning capabilities.
• Representative Works:
  • Program-of-Thought: enables models to think using code language, thereby enhancing their reasoning capabilities.
  • DeepSeek-Prover: converts natural language questions into formal statements, filters out low-quality statements, and generates proofs to create synthetic data, enhancing LLM’s theorem proving ability.
  • RBF: demonstrates why structured language is more effective than natural language in scenarios that require strong planning.

---

Latent Space Deep Reasoning

• Core Idea: Enhances LLM reasoning ability through continuous latent space operations.
• Representative Works:
  1. Token-driven: Early studies introduced implicit “planning tokens” or “thinking tokens” to guide the reasoning process in latent space.
     • Coconut (Chain of Continuous Thought): further expands this method by maintaining multiple parallel reasoning paths, enhancing complexity while ensuring efficiency.
     • Heima: performs efficient reasoning through latent hidden spaces, innovatively compressing the entire Long CoT process into a single token, resulting in significant computational resource savings.
  2. Vector-driven: Inserts an additional vector to guide the reasoning process in latent space.
     • LTMs: innovatively abstracts each layer of the LLM into “thinking blocks” and introduces the concept of a “thinking vector” for each layer. Through iterative deep computation in latent space, the model dynamically scales the computational load during testing.
  3. Manager-driven: Proposes a continuous manager mechanism to manage the latent space states.
     • Recurrent Block: uses iterative control over trained “recurrent blocks” as recursive “thinking blocks” to integrate deeper model layers during reasoning, enhancing performance without the need for specialized training data.
     • Implicit Thought Transformer (ITT): leverages raw Transformer layers as recursive “thinking blocks,” using adaptive token routing to select key tokens and residual thinking connections to control reasoning depth, thereby achieving efficient processing of key tokens.
• Relevant Repositories:
  • Awesome-Latent-CoT: provides an overview of various thought chain representations in latent space, capturing complex non-linguistic thoughts that cannot be expressed by language alone.

---

2.1.2 Deep Reasoning Learning

The deficiency of deep reasoning abilities in RLLMs can significantly reduce model performance. As a result, the academic focus has shifted towards enhancing reasoning capabilities through training. Supervised fine-tuning (SFT), as a memory process, can stabilize model output, while reinforcement learning (RL) facilitates generalization and self-learning.

---

Deep Reasoning Imitation

• Core Idea: By imitating advanced reasoning systems, deep reasoning in RLLMs can be effectively achieved, enabling models to learn complex reasoning patterns and generalize across tasks.
• Representative Works:
  1. Imitation from Human
     • GSM8K/GPT-Verifier: Introduces early imitative learning based on human-annotated deep reasoning samples.
     • ALT: Enhances deep reasoning in RLLMs by generating a large-scale dataset of human-annotated logical templates.
  2. Imitation from Advanced RLLMs
     • AceMath: Uses few-shot prompting to distill Long CoT samples from advanced LLMs, improving performance through multi-stage quality-guided SFT.
     • DART-Math: Effectively distills difficulty-dependent deep reasoning samples through rejection sampling in the synthesis stage.
     • OpenThoughts / OpenCodeReasoning / NaturalThoughts: Extends this paradigm to mathematics, code, and general scenarios.
  3. Imitation from Scaling-augmented RLLMs
     • Bansal et al.: Find that expanding the sampling scale and length improves data quality.
     • Qwen-Math / PromptCoT: Further combine large-scale sampling with reward model sample selection to generate Olympic-level difficulty deep reasoning samples.
     • FastMCTS: Utilizes Monte Carlo Tree Search (MCTS) to identify optimal deep reasoning paths.
• Latest Developments:
  • Journey P2: Knowledge distilled from advanced RLLM APIs such as o1, R1 significantly boosts small LLMs’ performance, with supervised fine-tuning methods surpassing teacher models in complex mathematical reasoning tasks.
  • s1 / LIMO: A small number of high-quality samples is sufficient to activate deep reasoning abilities in base LLMs.

---

Deep Reasoning Self-Learning

• Core Idea: Although simple imitation yields excellent performance, current models still heavily rely on human annotations or outputs from advanced models during imitation and distillation. To break through this limitation, research focuses on self-learning techniques to achieve more advanced reasoning capabilities.
• Representative Works:
  1. Self-Learning from Direct Sampling
     • STaR: Uses in-context learning (ICL) to sample deep reasoning results and treats the correctness of the final answer as an implicit reward for self-learning.
     • Reinforced Self-Training (ReST): Proposes the “Grow-Improve” paradigm, where the self-generated reasoning process is rewarded, enhancing with offline reinforcement learning.
     • ReST$^{EM}$: Generates rewards and iteratively optimizes LLMs to achieve peak performance on validation sets, significantly improving robustness.
     • TOPS: Finds that self-learning with deep reasoning samples at an appropriate reasoning depth is the most efficient.
  2. Self-Learning from Tree Search
     • PGTS: Uses policy-guided tree search, combining reinforcement learning with structured tree exploration.
     • ReST-MCTS*: Optimizes MCTS behavior through progressive trajectory extraction and curriculum preference learning, significantly improving LLMs’ reasoning ability.
• Latest Developments: Introduce an error-correction adaptive mechanism by training a verifier or using entropy to filter and optimize the reward process, thus enhancing the quality of self-learning.
  • UnCert-CoT: Dynamically schedules thought chains based on entropy-aware uncertainty, activating multi-path reasoning only under high-entropy situations, significantly improving code generation accuracy and efficiency.
  • Wang et al.: Analyzes the impact of verifiable reward-based reinforcement learning on large language models’ reasoning capabilities from the token entropy perspective, where high-entropy “branching” tokens dominate adjustments to multi-path reasoning strategies. Policy gradient optimization is applied exclusively to these high-entropy tokens.
  • CoT-Valve: Dynamically adjusts to reduce reasoning path length based on task difficulty, thus reducing computational overhead.

---

2.2 Feasible Reflection

2.2.1 Feedback

Feedback mechanisms provide multi-granularity evaluation signals for Long CoT, ranging from Overall Feedback, which evaluates the final outcome, to Process Feedback, which supervises individual steps of the reasoning process, and Hybrid Feedback, which combines both types. These mechanisms not only support reward modeling and path optimization but also lay the foundation for subsequent self-correction, serving as a crucial bridge to move RLLMs from static generation to dynamic evaluation.

---

Overall Feedback

• Core Idea: Overall feedback evaluates the complete reasoning process and the final result from a global perspective, commonly used to guide large language models in improving reasoning quality during reinforcement learning or self-optimization. Feedback forms include numerical rewards, rule checks, and natural language evaluations.
• Representative Works:
  1. Outcome Reward Models(ORM): Provide numeric reward signals to optimize output quality, suitable for tasks where accuracy is hard to assess directly.
     • Gen-Verifier: introduces the first generation verification framework based on reasoning accuracy;
     • Critic-RM: combines natural language criticism and reward prediction, significantly optimizing feedback quality;
     • Self-Rewarding LMs (SRLMs): introduce consistency mechanisms, achieving self-supervised rewards without human annotation.
  2. Rule Extraction: Uses in-task rules to verify and correct answers, enhancing the stability of feedback.
     • STaR / ReST: show that rule-based feedback based on final answers outperforms ORM in mathematical tasks;
     • OpenCodeInterpreter / AceCoder: generate program-level feedback using automated test cases in coding tasks.
  3. RLLMs Feedback (LLM-as-a-Judge): The model self-critiques and evaluates in natural language, enhancing reflection and error-correction capabilities.
     • EvalPlanner: distinguishes feedback between planning and reasoning;
     • RoT: combines reverse reasoning and reflection to assist models in discovering knowledge gaps;
     • AutoRace: provides task-specific evaluation criteria to improve feedback relevance.
• Relevant Repositories:
  • RewardBenchFor system evaluation of ORM methods.

Process Feedback

• Core Idea: Process feedback evaluates each step in the reasoning chain progressively, often in conjunction with reinforcement learning or tree search, guiding the model to fine-tune without relying on human annotations. The feedback sources primarily include process reward models and language models driven by natural language.
• Representative Works:
  1. Process Feedback from Process Reward Models (PRMs): Use automatically constructed or minimally annotated data to train stepwise reward functions, the mainstream approach in Long CoT.
     • PRM800K: Pioneers using human-annotated stepwise supervision to enhance reward stability;
     • Math-Shepherd: Automatically generates stepwise feedback using tree search to enhance PRM generalization;
     • Full-Step-DPO: Rewards the entire reasoning chain, encouraging holistic optimization;
     • AdaptiveStep: Dynamically segments reasoning steps based on confidence, enabling token-level fine-grained feedback.
  2. Process Feedback from RLLMs: Leverages the model’s own generation of natural language feedback to simulate reward signals, improving the flexibility and scalability of process supervision.
     • React / Reflexion: Generate language feedback after each action, enhancing decision-making rationality;
     • Step-DPO: Introduces a self-validation mechanism to construct positive-negative contrastive samples, optimizing the training process;
     • CACE: Proposes causal impact metrics between reasoning steps to make the entire chain more interpretable;
     • ORPS: Automatically optimizes reasoning strategies with program execution feedback, reducing human reliance.
• Relevant Repositories:
  • ProcessBench: Evaluates stepwise reasoning and reward model performance;
  • PRMBench: Focuses on comparative analysis of PRM methods in mathematical tasks.

Hybrid Feedback

• Core Idea: Hybrid feedback mechanisms combine the strengths of both overall and process feedback, assessing final outputs while focusing on intermediate reasoning steps. This unified multi-granularity evaluation system enhances the overall reasoning quality and error-correction capabilities of language models.
• Representative Works:
  • Consensus Filtering: Combines Monte Carlo estimation with LLM-as-Judge to integrate overall and stepwise feedback, enhancing reasoning consistency and accuracy;
  • Step-KTO: Merges PRM and ORM binary feedback mechanisms, emphasizing reflection-driven error correction, guiding the model to form more coherent Long CoT structures.

2.2.2 Refinement

The Refinement mechanism focuses on self-correction capabilities based on feedback information, serving as a key step in achieving closed-loop optimization in Long CoT. Through Prompt-based Refinement, spontaneous reflection is achieved; SFT-based Refinement facilitates imitation learning; and RL-based Refinement strengthens self-correction strategies. As a result, the model gradually develops the ability of “self-diagnosis—self-updating,” making the reasoning chain more robust and controllable.

---

Prompt-based Refinement

• Core Idea: By guiding the model to generate initial responses via prompts, and allowing for self-feedback and multi-round corrections in subsequent rounds, this method improves reasoning accuracy, reduces hallucinations, and supports stronger automated reflection capabilities.
• Representative Works:
  • ReAct / Reflexion: A typical implementation of the multi-round reflection and self-correction mechanism;
  • Self-Backtracking / Refiner / BackMath: Supports the model in autonomously backtracking and modifying during the reasoning process, streamlining decision paths;
  • MCTSr / ReST-MCTS: Combines tree search with confidence updates to enable multi-round dynamic reflection;
  • LLM2/ ReARTeR: Promotes the automatic evolution and stable convergence of refinement strategies in Long CoT tasks.

SFT-based Refinement

• Core Idea: By utilizing high-quality reflection data for supervised fine-tuning, the model imitates the self-correction behaviors of more advanced models, enhancing its step-by-step error correction and reflective capabilities. This is suitable for small model capability transfer and fine-grained training.
• Representative Works:
  • rStar: Enhances small model self-improvement capabilities through self-play methods;
  • Math-Minos: Trains the model using step-by-step rationale labels for fine-grained reasoning;
  • Journey Learning: Combines MCTS backtracking to generate supervision signals;
  • MM-Verify: Expands the refinement mechanism to multimodal image-text reasoning.

RL-based Refinement

• Core Idea: Through reinforcement learning mechanisms, the model is guided to self-reflect and correct during testing or reasoning processes, emphasizing self-refinement capabilities under reward guidance, thus reducing dependence on manual supervision.
• Representative Works:
  • SCoRe: Enhances the model’s self-refinement ability during testing through self-generated correction trajectories and regularization;
  • DeepSeek-R1: Uses result-level reinforcement learning to activate the model’s natural feedback and “aha” moments for corrections;
  • S$^2$R: Combines process-level reinforcement learning to achieve dynamic refinement in reasoning;
  • ReVISE: Introduces an internal verifier to decide when to trigger RL-guided reflective behaviors.

2.3 Extensive Exploration

Extensive Exploration enables reasoning large language models (RLLMs) to explore multiple reasoning paths more deeply and comprehensively when dealing with complex problems, thereby improving problem-solving accuracy and robustness. From the perspective of exploration types, extensive exploration techniques can be divided into three categories: Exploration Scaling, Internal Exploration, and External Exploration.

2.3.1 Exploration Scaling

Exploration Scaling aims to enhance the model’s ability to solve more complex problems by increasing the number or length of reasoning paths. This approach is typically suitable when the reasoning task is more complex, and a single reasoning path may not effectively lead to the correct answer.

---

Sequential Scaling

• Core Idea: By extending the reasoning chain of a single path, the model gradually deepens its thinking, thereby improving its understanding and handling of complex problems. This is especially applicable to tasks with Long CoT that require multi-step reasoning to draw conclusions, such as mathematical proofs, logical deductions, and multi-step planning.
• Representative Works:
  • OpenAI-o1 / Deepseek-R1: Extends the reasoning chain to provide detailed multi-step reasoning processes, effectively improving the ability to solve complex problems in mathematics, coding, and other areas.
  • ITT(Inner Thinking Transformer:): redefines the layer computation in Transformer as “thinking steps,” dynamically allocating computation resources to deeper reasoning on key tokens without increasing the total number of model parameters.

---

Parallel Scaling

• Core Idea: By generating multiple reasoning paths in parallel and combining, voting, or verifying their results, the model can effectively avoid the issue of a single path getting trapped in local optima or errors, thus improving robustness and accuracy in situations with high ambiguity, multiple possible solutions, or unclear outcomes.
• Representative Works:
  • Self-Consistency: Proposes generating multiple reasoning paths and selecting the most frequent answer from the results, effectively improving the stability and accuracy of the final answer.
  • ECM(Electronic Circuit Model): Borrowing the concepts of parallel and series circuits in electronics, combines reasoning paths in parallel or series, considering various possibilities and improving decision quality.

---

2.3.2 Internal Exploration

Internal Exploration primarily refers to large reasoning models (RLLMs) actively exploring and optimizing reasoning paths through their internal mechanisms (usually reinforcement learning strategies and reward mechanisms), allowing for more efficient and deeper solutions to complex reasoning problems. This method enables the model to autonomously adjust its reasoning strategy, reducing reliance on external guiding data.

---

RL Strategies

• Core Idea: Leverage reinforcement learning (RL) algorithms to guide models in actively learning and exploring diverse reasoning paths. This approach overcomes the limitations of overly uniform patterns in reasoning processes, enhancing model performance in tasks that involve high uncertainty or heavily depend on autonomous decision-making.
• Representative Works:
  • PPO(Proximal Policy Optimization): A classic RL algorithm that efficiently optimizes a model’s internal decision-making mechanism through a policy-gradient-based approach, suitable for path exploration and optimization in complex environments.
  • DivPO(Diverse Preference Optimization): Encourages models to explore a greater variety of reasoning paths to maintain decision diversity, preventing convergence to local optima.
  • GRPO(Guided Reward Policy Optimization): Designs a guided reward mechanism that enables models to more effectively explore within complex logical reasoning spaces.

---

Reward Strategies

• Core Idea: Directly guide models to explore and optimize effective reasoning paths through carefully designed reward functions, which are particularly useful in scenarios where there are explicit optimization goals or specific reasoning bottlenecks to address.
• Representative Works:
  • Deepseek-R1: Proposes a specially designed reward function to incentivize models to optimize intermediate reasoning steps, aiding the model in building a high-quality internal reasoning process.
  • ReST-MCTS*: Combines Monte Carlo Tree Search (MCTS) with reward strategies, guiding the tree search algorithm through process rewards for more accurate exploration of effective reasoning paths, improving overall reasoning quality.

---

2.3.3 External Exploration

External exploration refers to the assistance of external tools, human knowledge, or other models in guiding the model to more effectively explore diverse reasoning paths and improve its ability to solve complex problems. This approach is often used in scenarios where fine-grained guidance or external knowledge is essential for effective problem-solving. External exploration can be subdivided into two types: Human-driven Exploration and Model-driven Exploration.

---

Human-driven Exploration

• Core Idea: Utilizes human intuition, experience, or feedback to guide the model in selecting and adjusting reasoning paths, especially in situations where the model’s autonomous exploration ability is limited or the reasoning task is complex and requires decomposition into multiple sub-tasks.
• Representative Works:
  • Least-to-Most: Breaks down complex problems into simpler subproblems, solving each and using prior answers as inputs for subsequent steps, ultimately synthesizing a solution for the overall problem. This method was proposed to address the “difficulty generalization” bottleneck in traditional Chain-of-Thought approaches.
  • ToT (Tree-of-Thought): Expands the traditional “left-to-right” token generation reasoning process into a “tree structure exploration,” where each node represents a thought unit. This supports multi-path attempts, backtracking, forward reasoning, and self-evaluation within the reasoning process.

---

Model-driven Exploration

• Core Idea: Uses auxiliary models or algorithms to automatically guide the current model’s reasoning process, reducing the need for human intervention and allowing for the efficient search and optimization of numerous complex reasoning paths, thus improving automation and overall efficiency.
• Representative Works:
  • PPO-MCTS: Integrates MCTS (Monte Carlo Tree Search) with PPO-based training to enhance reasoning. The key is to retain the value network obtained during PPO training and use it during the reasoning phase to guide the MCTS in selecting more desirable output sequences, thereby improving the quality and consistency of the generated text.
  • MindStar: Reformulates complex reasoning problems (particularly mathematical ones) as search problems, where structured searches are performed over different reasoning paths to select the optimal one.
  • rStar-Math: Develops a strong mathematical reasoning system through MCTS + small model reward mechanisms + self-evolution processes, enabling small models to outperform o1-preview in mathematical capabilities.

---

3. Key Phenomena and Related Principles

3.1 Reasoning Emergence Phenomenon

Long CoT abilities naturally emerge after training, demonstrated by the model’s ability to generate multi-step, coherent reasoning processes by internalizing logical structures and contextual examples from pretraining data, even in the absence of direct supervision. Related studies have described this phenomenon as follows:

• Wang et al. found that a small number of high-quality contextual examples can effectively guide RL-based language models (RLLMs) to generate clear, logically consistent reasoning chains, indicating that the model has internalized basic reasoning patterns during pretraining.
• Madaan et al. demonstrated that even without specific problem entities, the model can still generate reasonable reasoning chains by retaining only the logical structure information, showcasing its inductive and transfer abilities regarding structural information.
• Stechly et al. pointed out that by adjusting decoding strategies or constructing specialized prompts, latent CoT abilities within the model can be explicitly activated, resulting in multi-step reasoning in complex tasks.
• Guo et al. showed that rule-based RL strategies can directly induce models to form coherent reasoning chains during pretraining, significantly improving performance in multi-step tasks.

3.2 Reasoning Boundary Phenomenon

Large language models exhibit clear performance boundaries in Long CoT: when the depth or complexity of reasoning exceeds a certain threshold, model performance significantly degrades, sometimes even resulting in logical collapse. This phenomenon suggests that current models have a “reasoning boundary,” which is the upper limit of reasoning complexity that can be supported by their parameter space and computational resources. Existing research has systematically explored this phenomenon from both theoretical modeling and empirical analysis:

• Chen et al. formally introduced the “reasoning boundary” concept, experimentally quantifying the performance critical points of models under different task complexities, and indicating that accuracy sharply declines when the reasoning task exceeds the model’s capacity.
• Bi et al. observed that performance deteriorates drastically when the model attempts to mimic overly complex CoT examples in code generation tasks, indicating that beyond a certain complexity, Long CoT examples become counterproductive.
• Feng et al. proposed a mathematical model showing that models with fixed parameter sizes cannot perform numerical calculations exceeding a certain complexity, revealing a hard limit in accuracy.
• Zhou et al. constructed the GSM-Infinite dataset and demonstrated through experiments that the upper limits of reasoning abilities vary significantly across different tasks, further emphasizing that reasoning boundaries are related to task structure.

3.3 Overthinking Phenomenon

In Long CoT, extending the reasoning chain does not always lead to performance improvement. Studies have shown that once the reasoning length exceeds the model’s capacity, accuracy decreases, a phenomenon known as “overthinking,” which reflects the non-linear marginal benefits of reasoning and error accumulation in the process.

• Chen et al. found that when the number of reasoning steps exceeds the model’s boundary, reasoning accuracy significantly drops, indicating that there is an optimal depth range for reasoning.
• Wolf et al. emphasized that the fundamental reason for performance degradation is the amplification of errors in intermediate reasoning steps, which affects the final judgment.
• Xie et al. experimentally showed that reasoning length does not have a monotonic relationship with accuracy, challenging the intuition that “longer CoT leads to better reasoning.”
• Wu et al. established a mathematical model defining the “optimal reasoning length” interval under different models and task conditions, suggesting that performance reverses once the length exceeds the optimal range.
• Chen et al. introduced the “reasoning chain Ohm’s law,” analogizing the non-linear relationship between reasoning length and performance to information flow resistance in the model.

3.4 Inference Test-Time Scaling Phenomenon

The inference test-time scaling phenomenon refers to the increase in reasoning performance by extending the computational process (e.g., reasoning chain length or sample number) during inference. This phenomenon reveals the “dynamic amplification” potential of the model, but also comes with a trade-off between exploration depth and computational cost.

• Brown et al. observed that by repeating multiple rounds of inference attempts, even if the initial attempt fails, the correct answer can be found within a certain number of trials, introducing the “language monkey” phenomenon.
• o1 demonstrated that simply increasing the reasoning chain length improves accuracy, particularly in complex mathematical tasks.
• Jin et al. pointed out that while increasing the reasoning chain length initially leads to performance improvement, beyond a certain threshold, performance deteriorates, resulting in a typical nonlinear growth curve.
• Wu et al. found that there is a logarithmic relationship between the number of inference samples and the lower bound of error, suggesting an asymptotic relationship between computational complexity (FLOPs) and inference performance.
• Chen et al. established the theoretical upper-bound of parallel inference, indicating that no matter how the sample size is increased, the model’s verification performance cannot exceed its internal reasoning ceiling.

3.5 PRM and ORM Selction Phenomenon

In reinforcement learning optimization, Long CoT tasks involve supervision of the model generation process. Researchers distinguish between two main strategies: Process Reward Model (PRM), which focuses on the reasoning process itself, and Outcome Reward Model (ORM), which only concerns whether the final output is correct. The two strategies differ significantly in terms of generalization ability, learning stability, and supervision cost.

• Lampinen et al. validated the causal relationship between intermediate steps and final answers in qualitative experiments, providing theoretical support for the rationale behind process supervision.
• Jia et al. theoretically proved that under sufficiently diverse data, ORM is not harder to optimize than PRM, with the two only differing by polynomial factors in terms of sample complexity.
• Guo et al. showed that rule-based PRM reinforcement learning significantly improves the model’s Long CoT capabilities in complex tasks but also faces the risk of reward hacking.
• Tan emphasized the importance of reward distribution in intermediate reasoning steps for complex reasoning paths, which ORM cannot provide.
• Jiang et al. pointed out that PRM is more costly in terms of data collection, as it requires labeling each reasoning step, limiting its large-scale application.

3.6 Aha Moment Phenomenon

The Aha Moment refers to the sudden integration of information during the reasoning process, leading to a key turning point in judgment, resembling human reflection and self-correction. This phenomenon highlights the model’s dynamic cognitive adjustment abilities, but its occurrence depends on the collaboration between external stimuli and internal mechanisms.

• Guo et al. first triggered the Aha Moment behavior under unsupervised conditions through rule-based rewards, with models reflecting on intermediate reasoning and self-correcting.
• Xie et al. further demonstrated through experiments that this behavior can be replicated across multiple models, verifying that it is not an偶然 event but rather an inducible strategy.
• Zhou et al. extended the Aha Moment phenomenon to multimodal tasks, showing that it is not specific to text-based tasks but reflects the model’s broader cognitive abilities.
• Liu et al. pointed out that in certain reinforcement learning frameworks (e.g., R1-Zero), Aha behavior may not genuinely exist, with lengthening the generation more likely a result of reward optimization rather than actual reflection.
• Yang et al. found that the Aha behavior often involves human-like language enhancement and dynamic uncertainty regulation, with the model more inclined to use expressions like “I think” under high-pressure tasks, reflecting its coping mechanisms for task stress.

4. Algorithms

4.1 Supervised Fine-Tuning (SFT)

In advancing large models to possess powerful Long CoT reasoning abilities, Supervised Fine-Tuning (SFT) plays a crucial role, bridging pre-training with more advanced alignment methods such as Reinforcement Learning from Human Feedback (RLHF). The core goal of SFT is to teach models how to follow instructions and initially master the ability to generate structured, step-by-step reasoning chains, thus laying the foundation for more complex reasoning tasks.

• In the context of deep reasoning, SFT is especially critical. Although the lack of sufficient reasoning depth in RLLMs significantly reduces performance, SFT stabilizes the model’s output format through a memorization process, allowing it to learn reasoning from human-labeled or distilled data. In contrast to reinforcement learning (RL), which focuses more on generalization and self-learning, SFT plays a vital role in deep reasoning imitation. It allows RLLMs to learn complex reasoning patterns by mimicking high-quality reasoning examples generated by humans, advanced RLLMs, or enhanced RLLMs, and generalizing them to new tasks. SFT not only significantly improves the model’s reasoning performance but, in some cases, enables even a small number of high-quality samples to activate the underlying LLM’s deep reasoning capabilities, allowing it to predict events outside the model’s knowledge base. This makes SFT one of the key technologies for enhancing reasoning levels and generalization abilities in RLLMs.

• For feasible reflection, SFT primarily focuses on optimization-based imitation (Refinement Imitation). In reflection-based LLM reasoning, SFT is a key mechanism for enabling self-optimization and error correction in the model. Through SFT, the model can directly learn the error-correction processes of advanced LLMs, significantly enhancing its reflective abilities, such as performing self-play reasoning, iterative feedback error correction, and even justifying and reflecting on the reasoning process through incremental natural language feedback. Additionally, SFT can integrate visual and textual reasoning in multimodal scenarios, improving the model’s critical thinking and self-correction abilities. SFT enhances the reasoning accuracy of LLMs through iterative feedback and self-correction strategies, which is especially beneficial for smaller models.

4.1.1 Core Technology

SFT consists of two core concepts: Instruction Tuning and Parameter-Efficient Fine-Tuning, PEFT。

Instruction Tuning

• Core Idea: By fine-tuning the model on a large number of instructions covering various tasks, the model’s zero-shot generalization ability on unseen tasks can be significantly enhanced. This enables the model to learn the skill of “following instructions.”
• Representative Works:
  • Finetuned Language Models Are Zero-Shot Learners (FLAN): Google’s pioneering work demonstrating that multi-task instruction fine-tuning unlocks zero-shot capabilities in LLMs for unseen tasks.
  • Instruction Tuning for Large Language Models: A Survey: A comprehensive survey systematically introducing methods, datasets, challenges, and future directions in instruction tuning.

参数高效微调(PEFT)

• Core Idea: Given the high cost of full fine-tuning (Full Fine-tuning) for LLMs, PEFT methods have emerged. These methods achieve near-full fine-tuning effects by updating only a small subset of the model’s parameters, greatly reducing hardware requirements.
• Representative Works:
  • LoRA: Low-Rank Adaptation of Large Language Models: A revolutionary LoRA technique proposed to efficiently fine-tune models by injecting low-rank adaptation matrices, currently one of the most widely used PEFT methods.
  • QLoRA: Efficient Finetuning of Quantized LLMs: A further optimization of LoRA, combining 4-bit quantization, double-weight quantization, and page optimizers, making it possible to fine-tune massive models on a single consumer-grade GPU.
  • Adapter Tuning: Inserts small neural network modules (adapters) between layers of the Transformer, updating only the parameters of these adapters during training.
  • Prompt Tuning / P-Tuning: Instead of modifying the model’s weights, it learns one or more trainable virtual tokens (soft prompts) at the input end, guiding the model to perform downstream tasks more effectively.

Technical Comparison

Technology Type	Core Idea	Advantages	Disadvantages
Full Fine-tuning	Update all model weights.	Highest performance ceiling, can fully adapt to new data.	Extremely high training cost (memory, time), prone to catastrophic forgetting, requires storing the entire model.
Parameter-Efficient Fine-tuning (PEFT)	Freeze most of the original parameters, only update a small set of additional parameters or specific subsets.	Very low training cost, fast, resistant to forgetting, small fine-tuning products (Adapters), easy to deploy.	Performance may be slightly inferior to full fine-tuning, and its adaptation to extremely complex tasks might be limited.

4.1.2 Learning Resources

Resource Name	Speaker/Author	Features	Link
Let’s build GPT: from scratch	Andrej Karpathy	A hands-on guide to building GPT from scratch, deeply understanding the fundamentals of Transformer and training processes; a prerequisite for understanding SFT.	YouTube
Hugging Face SFT Course	Hugging Face	Official SFT series tutorial, using the Hugging Face TRL codebase for SFT code practice.	Course Link
Hugging Face SFT Trainer Doc	Hugging Face	Advanced documentation for Hugging Face SFTTrainer.	Documentation Link
Hugging Face PEFT Course	Hugging Face	Official PEFT series tutorial, explaining the theory and code practices of various efficient fine-tuning techniques like LoRA.	Course Link
LLMs-from-scratch	Sebastian Raschka	Tutorial code for the official book, “Build a Large Language Model (From Scratch).”	Course Link

4.1.3 Development Frameworks

Framework	Features	Main Use Case	Resource Link
Hugging Face TRL	Official Hugging Face library, integrating various training methods like SFT, RLHF, DPO, seamlessly connecting with ecosystems (transformers, peft, accelerate).	Provides the standardized SFT trainer SFTTrainer, simplifying the training process.	GitHub
LLaMA-Factory	One-stop LLM fine-tuning platform with a Web UI, enabling users with no coding experience to easily perform SFT, PEFT, and model evaluation.	Highly user-friendly, supports massive models and datasets, suitable for beginners and quick validation.	GitHub

4.1.4 Best Practices and Common Pitfalls

1. Data Quality is Far More Important than Quantity:
   • Core Principle: It is better to use 1,000 high-quality, diverse data points than 10,000 low-quality, homogeneous ones. Low-quality data can teach the model incorrect patterns.
   • Format Consistency: Ensure that all training data follows a unified dialogue template (e.g.,ChatML), which is crucial for training the model to recognize roles and dialogue boundaries.
2. Choose the Right Fine-Tuning Strategy:
   • For most applications with limited resources, QLoRA should be prioritized as it strikes the best balance between efficiency and effectiveness.
   • If seeking optimal performance and with sufficient resources, full fine-tuning may be considered, though care should be taken to avoid the risk of overfitting.
3. Tuning Key Hyperparameters:
   • Learning Rate: SFT typically uses smaller learning rates than pretraining, usually ranging from 1e-5 to 5e-5.
   • Epochs: Usually, 1 to 3 epochs are sufficient. Too many epochs can lead to overfitting on small datasets, causing the model to “forget” the general knowledge learned during pretraining.
   • Batch Size: Within the memory limits, increasing the batch size appropriately helps stabilize training.
4. Evaluation and Iteration:
   • Comprehensive Evaluation: Do not rely solely on the loss function. Combine it with objective evaluation benchmarks (such as MMLU) and subjective human evaluation for a more thorough assessment of model performance.
   • Iterative Optimization: SFT is an ongoing iterative process. Based on evaluation results, continuously clean the data, adjust hyperparameters, and optimize the model.

4.1.5 Relevant Paper Repositories

• LLM4NLP

4.2 Reinforcement Learning

4.2.1 Reinforcement Learning

• West Lake University’s “Mathematical Principles of Reinforcement Learning”
  • Features: Starts with MDP and the Bellman Equation, derived using the policy gradient theorem.
  • Prerequisite Knowledge: Linear algebra, probability theory.
  • Focus: The mathematical essence of value iteration and policy optimization.
• Book-Mathematical-Foundation-of-Reinforcement-Learning (Beginner-friendly)

4.2.2 Core Algorithms of Reinforcement Learning

Authoritative Courses

Course	Lecturer	Features	Resources
Foundations of Deep RL	Pieter Abbeel	6 concise lectures (Q-learning → PPO)	YouTube
UC Berkeley CS285	Sergey Levine	Includes SAC/Inverse Reinforcement Learning and other advanced topics	Course Website
Reinforcement Learning by Hung-yi Lee	Hung-yi Lee	In Chinese + Practical Exercises with EasyRL	Bilibili
Reinforcement Learning: An Overview	Kevin Murphy	Continuously updated resources on Deep Reinforcement Learning algorithms	Arxiv

Essential Basic Algorithms

• Basic Reinforcement Learning Algorithms

  • DQN: The beginning of Deep Reinforcement Learning
  • PPO: A key method in policy optimization, widely used in industrial applications
  • SAC: Incorporates exploration entropy, robust for continuous action spaces
  • TD3: Improves off-policy reinforcement learning using double delayed networks

• Model-Based Reinforcement Learning Algorithms

  • dreamer: A model-based reinforcement learning algorithm
  • tdmpc2: Significant advancement in model-based reinforcement learning algorithms

• Offline Reinforcement Learning Algorithms

  • CQL: Introduces conservative constraints, foundational work in offline reinforcement learning
  • decision-transformer: Introduces autoregressive models into offline reinforcement learning

• Large-Scale Model Reinforcement Learning Algorithms

  • PPO: Applying classic PPO to large language models
  • DPO: Preference optimization without rewards, an offline reinforcement learning algorithm for large models
  • GRPO: Group Relative Policy Optimization, core algorithm in DeepSeek-R1

Cutting-Edge Algorithms for Large-Scale Model Reinforcement Learning

• DAPO: Four improvements on GRPO
• LUFFY: Off-policy version of GRPO, introduces high-quality external trajectories
• Absolute-Zero-Reasoner: A large-scale reinforcement learning algorithm requiring no annotations
• One-Shot-RLVR: One-shot optimization for large model inference
• SPIRAL: Reinforcement learning in self-play game environments, successfully enhancing mathematical reasoning abilities
• High-Entropy Minority Tokens Drive Effective RLVR: Reinforcement learning driven by high-entropy tokens (20%)
• Spurious_Rewards: Random rewards can also enhance LLM reasoning abilities
• SwS: Reasoning reinforcement learning driven by self-perceived weaknesses

4.2.3 Development Frameworks for Reinforcement Learning

Basic Reinforcement Learning Frameworks

• stable-baselines3 (Quick experimentation, with well-established and stable baselines)
• legged_gym (Quadruped robot control)

Large-Scale Model Reinforcement Learning Frameworks

• verl: A high-performance and user-friendly open-source reinforcement learning training library based on Ray, vLLM, ZeRO-3, and HuggingFace Transformers, with features like efficient resource utilization, scalability, and production readiness. (Complex structure, highly reusable, excellent performance)
• OpenRLHF: An open-source RLHF framework released by teams such as NVIDIA, based on Ray, vLLM, ZeRO-3, and HuggingFace Transformers. It supports algorithms like PPO, GRPO, and REINFORCE++, and provides dynamic sampling and asynchronous agent mechanisms to accelerate training.
• AReaL: Asynchronous reinforcement learning framework
• ROLL: Supports training large models with 600+ billion parameters
• Hugging Face TRL: An RLHF full-stack library maintained by Hugging Face, integrating SFT, GRPO, DPO, Reward Modeling, and other modules. It supports multiple model architectures and distributed scaling, making it one of the most active RLHF tools in the community. (User-friendly, quick start, active community)
• RL4LMs: An open-source RLHF library for language models, providing end-to-end tools for reward model construction and policy network training, helping researchers quickly build custom RLHF pipelines.

Additionally, there are some interesting extension repositories:

• Sachin19/trlp: An end-to-end RLHF library based on the TRL stack, supporting not only language models but also extending to Stable Diffusion models. It includes steps like SFT, reward modeling, and PPO, with example code for experimentation.
• OpenRLHF-M: An extension of OpenRLHF, optimized for multimodal models. It leverages DeepSpeed and HuggingFace Transformers to achieve higher throughput and richer training scenarios.
• HumanSignal-RLHF: An archived resource repository, gathering links and tutorials on RLHF data collection, system construction, and best practices, suitable for beginners to quickly understand the full RLHF pipeline.
• MichaelEinhorn/trl-textworld: A derivative version of TRL focused on performing RLHF experiments in the TextWorld environment, demonstrating how to train models like GPT2 using PPO to generate text that meets specific feedback requirements.

4.2.4 Test Environments

Classical RL Tests

• OpenAI Gym: Classic Control

Environment ID	Task Description	Features
CartPole-v1	Balance an inverted pendulum	4-dimensional state/discrete actions, termination if pole tilts > 12° or steps ≥ 500
MountainCar-v0	Swing car to the top	2-dimensional state/discrete actions, requires potential energy swing
Pendulum-v1	Control pendulum to stay vertical	3-dimensional state/continuous actions, no physical termination condition
Acrobot-v1	Swing double-link to touch target line	6-dimensional state/discrete actions, termination when target line is touched

• Atari 2600: Games

Environment ID	Game Type	Challenges
Pong-v5	Ping Pong	210×160 RGB input, requires image preprocessing
Breakout-v5	Breakout	Dense rewards, suitable for DQN training
SpaceInvaders-v5	Space Invaders	Multiple enemies coordinated attack, complex reward system

• Box2D: Physics Simulation

Environment ID	Physics System	Core Challenges
LunarLander-v2	Lunar Lander	8-dimensional state/discrete actions, fuel control and precise landing
BipedalWalker-v3	Bipedal Walker	24-dimensional state/continuous actions, balancing on complex terrain
CarRacing-v2	Car Racing	96×96 RGB input, vision + continuous control combined

• MuJoCo: Robotic Control

Environment ID	Robot Model	Task Type
HalfCheetah-v4	Cheetah Robot	High-speed running control (17-dimensional state)
Ant-v4	Ant Robot	Complex terrain navigation (111-dimensional state)
Humanoid-v4	Humanoid Robot	Bipedal balance walking (376-dimensional state)

• Other Special Environments

| Category | Example Environment |

README 内容过长，已截取前 64 KB。完整内容请前往 GitHub 仓库查看。