Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling

Test-Time Scaling (TTS) is an important method for improving the performance of Large Language Models (LLMs) by using additional computation during the inference phase. However, current studies do not systematically analyze how policy models, Process Reward Models (PRMs), and problem difficulty influence TTS. This lack of analysis limits the understanding and practical use of TTS methods. In this paper, we focus on two core questions: (1) What is the optimal approach to scale test-time computation across different policy models, PRMs, and problem difficulty levels? (2) To what extent can extended computation improve the performance of LLMs on complex tasks, and can smaller language models outperform larger ones through this approach? Through comprehensive experiments on MATH-500 and challenging AIME24 tasks, we have the following observations: (1) The compute-optimal TTS strategy is highly dependent on the choice of policy model, PRM, and problem difficulty. (2) With our compute-optimal TTS strategy, extremely small policy models can outperform larger models. For example, a 1B LLM can exceed a 405B LLM on MATH-500. Moreover, on both MATH-500 and AIME24, a 0.5B LLM outperforms GPT-4o, a 3B LLM surpasses a 405B LLM, and a 7B LLM beats o1 and DeepSeek-R1, while with higher inference efficiency. These findings show the significance of adapting TTS strategies to the specific characteristics of each task and model and indicate that TTS is a promising approach for enhancing the reasoning abilities of LLMs.

Host: Hey everyone, welcome back to the podcast! I'm Leo, your host, and I'm super excited about today's topic. We're diving deep into the world of Large Language Models, or LLMs as everyone calls them. But we're not just scratching the surface; we're going to be exploring some cutting-edge research that's really challenging how we think about scaling these models, especially during the inference phase – that's when they're actually being used to generate text, answer questions, and all that good stuff.

Guest: Hey Leo, great to be here! Yeah, LLMs are fascinating and changing so rapidly. It feels like every week there's a new model or a new technique that's pushing the boundaries. And inference, or 'test-time' as some researchers call it, is becoming a critical area. It’s no longer just about having the biggest model, but about how efficiently we can use that model to get the best results.

Host: It's a mouthful, I know, but the core idea is that smarter scaling strategies during inference can allow smaller models to actually outperform their massive counterparts. Think of it like this: it's not always about brute force; sometimes, finesse wins the day. It's like a perfectly tuned sports car against a monster truck, in some ways, that's how I think about it at least.

Guest: That's a perfect analogy! I think of it like a skilled chess player versus someone who just has more pieces on the board. It's the strategic thinking, the careful calculation of moves, that ultimately leads to victory. This paper is all about finding that strategic approach to inference.

Host: Right, so the paper is from a team at Shanghai AI Laboratory and Tsinghua University, along with some other collaborators. The lead author is Runze Liu, and the team also includes Junqi Gao, Jian Zhao, Kaiyan Zhang, Xiu Li, Biqing Qi, Wanli Ouyang, and Bowen Zhou. It's a pretty impressive lineup. They are really digging into something called Test-Time Scaling, or TTS, which is all about dynamically allocating extra computational resources during inference to improve performance.

Guest: Exactly. It's like giving the model a little extra brainpower when it really needs it. But the key is knowing when and how to allocate that extra compute. That’s the core of TTS. It's not just randomly throwing more resources at every problem; it's about tailoring the computational effort to the specific task at hand.

Host:

Guest: That's a pretty standard structure for a research paper. But within each of those sections, there are some really interesting insights. For instance, in the introduction, they highlight that while TTS is known to improve LLM performance, there hasn't been a lot of systematic analysis of how different factors like the policy model, the Process Reward Model (PRM), and the difficulty of the problem itself influence TTS. That's a huge gap they're trying to fill.

Host: Right, so what's a PRM in this context? Is that like a judge that tells the model if it's on the right track?

Guest: Precisely. A Process Reward Model essentially assesses the quality of the intermediate steps the LLM takes while solving a problem. So rather than just rewarding the final answer, it gives feedback on how the model arrives at that answer. This allows us to guide the model towards better reasoning paths. Think of it as a coach providing feedback during a training session, not just after the game is over.

Host: That makes a lot of sense. So, the paper highlights that current TTS methods rely on these PRMs to guide the generation process and select the final answer. But the authors argue there's a lack of understanding about how the choice of policy model, PRM, and the inherent problem difficulty all play into this. That's a complex interplay of factors.

Guest: Exactly. And that's where their two core questions come in. First, what's the best way to scale test-time computation given different policy models, PRMs, and problem difficulty levels? And second, how much can this extra computation actually improve performance on really tough tasks, and can it allow smaller models to compete with, or even surpass, larger ones? These are fundamental questions that have huge practical implications.

Host: So they're essentially asking, 'Can we get more bang for our buck by being smarter about how we use compute?' And the answer, based on their experiments, seems to be a resounding 'Yes!' They conducted experiments on MATH-500 and AIME24, which are challenging math problem datasets. They've found that the best TTS strategy is highly dependent on the policy model, PRM, and problem difficulty. It's not a one-size-fits-all solution.

Guest: That's right. And the most striking result is that, with their compute-optimal TTS strategy, smaller models can actually outperform larger ones. They give the example of a 1B parameter LLM exceeding the performance of a 405B parameter LLM on MATH-500! That's a massive difference in model size, yet the smaller model wins with the right scaling strategy. And in many cases the smaller model can outperform others while also having higher inference efficiency.

Host: That's incredible. It really underscores the importance of efficient inference. It means we might not need to keep chasing ever-larger models, which are expensive to train and deploy. Instead, we can focus on developing smarter algorithms that leverage existing models more effectively. The key is to adapt the TTS strategies to the specific characteristics of the task and the model.

Guest: It also opens up possibilities for running LLMs on resource-constrained devices, like mobile phones or embedded systems. If you can get similar performance from a smaller model with optimized TTS, it becomes much more feasible to deploy these models in a wider range of applications. That's democratizing access to powerful AI, in a way.

Host: Definitely. The paper also mentions that a 0.5B LLM outperformed GPT-4o, a 3B LLM surpassed a 405B LLM, and a 7B LLM beat o1 and DeepSeek-R1 on those same tasks. It's not just a marginal improvement; it's a significant leap in performance, achieved through intelligent scaling.

Guest: These are really strong comparative points. The fact that they’re benchmarketing agains models such as GPT-4o, o1 and DeepSeek-R1, really puts the scale of the work in perspective. The point here is that intelligently scaling compute provides an advantage in reasoning abilities for LLMs.

Host: Okay, so what are the main contributions of this work, according to the authors themselves?

Guest: They highlight three key contributions. First, a comprehensive evaluation of different TTS methods using up-to-date policy models, multiple PRMs, diverse scaling methods, and more challenging tasks. It's a really thorough empirical study.

Host: So, it's not just a theoretical exercise; they actually put these methods to the test with a wide range of models and datasets. What's the second contribution?

Guest: The second contribution is their analysis of the influence of rewards in the TTS process. They introduce the concept of 'reward-aware compute-optimal TTS' and demonstrate that the compute-optimal scaling strategy varies with different policy models, PRMs, and problem difficulty levels. This is a crucial point because it emphasizes that the PRM plays a critical role in guiding the scaling process, and you need to choose the right PRM for the job.

Host: So it's not just about throwing more compute at the problem; it's about having a good guide, the PRM, to tell you where to focus that compute. And the effectiveness of that guide depends on the specific model and the type of problem you're trying to solve. And what's their third major contribution?

Guest: Their third contribution is the empirical demonstration that smaller language models can significantly outperform larger models through TTS. They show that a 3B LLM can outperform a 405B LLM, and a 7B LLM can surpass o1 and DeepSeek-R1 on MATH-500 and AIME24. This really drives home the point that intelligent scaling can unlock the potential of smaller models.

Host: Okay, let's dive a little deeper into the technical details. The paper mentions something called a Markov Decision Process, or MDP. How does that fit into all of this?

Guest: Ah, yes. They frame the reasoning problem as an MDP, which is a common way to model sequential decision-making processes. In this context, the 'state' represents the current state of the problem-solving process, the 'action' is the next step the LLM takes, the 'transition function' determines how the state changes based on the action, the 'reward function' evaluates the quality of the action, and the 'discount factor' determines how much weight to give to future rewards versus immediate rewards. Using this MDP framework allows them to formally analyze and optimize the reasoning process.

Host: So it's like breaking down the entire reasoning process into a series of steps, where each step is a decision, and the goal is to maximize the overall reward. And the PRM is essentially providing the reward signal to guide the model through this MDP.

Guest: Precisely. The reward function, provided by the PRM, tells the model how good each step is, guiding it towards a solution. So, given a prompt 'x', the model, guided by the reward, generates an initial action, and the reward returns an evaluation.

Host: They then discuss different TTS methods, including Best-of-N (BoN), beam search, and Diverse Verifier Tree Search (DVTS). Can you give us a quick rundown of what these methods are and how they work?

Guest: Sure. Best-of-N (BoN) is the simplest approach. The policy model generates 'N' different responses, and then a scoring or voting method is used to select the best one. It's like having the model come up with multiple ideas and then choosing the best one.

Host: So it's a bit like brainstorming and then picking the best idea. And what about beam search?

Guest: Beam search is a bit more sophisticated. It maintains a 'beam' of the 'N' most promising candidate solutions at each step. At each step, the policy model generates multiple continuations for each candidate in the beam, and the verifier selects the top 'N' continuations to keep in the beam for the next step. This allows the model to explore multiple promising paths simultaneously.

Host: So instead of just picking the single best answer at each step, it keeps track of multiple good answers and explores them further. That sounds like it could be more robust than BoN. And what about DVTS, Diverse Verifier Tree Search?

Guest: DVTS takes beam search a step further by introducing diversity. It divides the search process into multiple independent subtrees, each of which is explored using beam search. This helps to prevent the search from getting stuck in local optima and encourages the exploration of a wider range of possibilities. DVTS can outperform beam search on some tasks, especially with a large computational budget.

Host: So it's like having multiple independent teams working on the problem simultaneously, each with its own beam search strategy. That sounds like it could be even more effective at finding the optimal solution. But the key, according to the paper, is to choose the right TTS method for the specific problem and model. It's not just about blindly applying the most complex algorithm.

Guest: Exactly. That's where the concept of 'compute-optimal test-time scaling' comes in. The goal is to select the hyperparameters for a given TTS strategy that maximize performance on a specific prompt, given a limited compute budget. This involves finding the right balance between exploration and exploitation, and it depends on the characteristics of the policy model, the PRM, and the problem itself.

Host: Okay, so it's like a tuning process, where you're tweaking the knobs and dials of the TTS algorithm to get the best possible performance for a given situation. And that tuning process needs to take into account the specific characteristics of the model and the problem. This is where they introduce the concept of 'reward-aware' compute-optimal TTS, right?

Guest: Yes. The authors argue that previous work on TTS has often overlooked the importance of the reward function provided by the PRM. They point out that using a single PRM as a verifier can be problematic because the PRM might be trained on a different policy model than the one used for TTS. This can lead to 'out-of-distribution' (OOD) issues, where the PRM is not well-suited to evaluating the responses generated by the policy model.

Host: So if the PRM is trained on a different kind of data, or a different model, it might not be a good judge of the current model's performance. It's like having a sports coach who's never seen the team play before – they might not be able to give the best advice.

Guest: That's a great analogy! And that's why they propose integrating the reward function directly into the compute-optimal TTS strategy. Their 'reward-aware' strategy ensures that the compute-optimal scaling adapts to the policy model, the prompt, and the reward function. This leads to a more robust and general framework for practical TTS. It makes the models more capable of being deployed in a practical setting. It also makes them more general, as they respond better to varying prompts, and provide the reward function.

Host: So the TTS methods should be more robust with this new reward-aware compute-optimal TTS strategy.