Host: Hey everyone, and welcome back to the podcast! I'm your host, Leo, and today we're diving into something that's been really fascinating me lately: how large language models, or LLMs, are evolving to tackle more complex problems. It's not just about generating text anymore; it’s about thinking deeper, and that's what we'll be exploring today.

Guest: Hi Leo, great to be here! Absolutely, the evolution of LLMs has been incredible. It feels like we're moving from simple question-answering to actually seeing these models strategize and plan. It’s like watching them learn to navigate a complicated maze, and it’s exciting to see how different approaches are pushing the boundaries.

Host: Exactly! And what really caught my attention is this idea of using evolutionary strategies to improve their problem-solving abilities. We’re not just tweaking parameters; we're talking about a process that’s almost like natural selection for ideas within these models. It’s a fascinating blend of AI and evolutionary biology, right?

Guest: It is indeed! It's a departure from the traditional methods of fine-tuning. Instead of explicitly programming every step, we’re creating an environment where the models can explore a wide range of solutions and refine the most promising ones. This concept of 'Mind Evolution,' as it's being called, mirrors how biological organisms evolve - through variation and selection. It’s not just a clever name; it’s a concept that could fundamentally change how we approach AI problem-solving.

Host: So, before we jump too deep, let's give our listeners a bit of background. When we talk about LLMs thinking deeper, we’re really referring to how they utilize their inference time compute, right? It's not just about spitting out the first answer that comes to mind, it’s about having the capacity to consider multiple possibilities, refine those, and then arrive at a better solution. It's like giving them more thinking time, but in a structured way.

Guest: Precisely, Leo. That inference time is crucial. The traditional methods, like just 'chain-of-thought' prompting, are useful for getting models to think through a problem step-by-step, but they don't necessarily explore a wide range of solutions. Think of it as giving the model more room to breathe and consider different angles. And these 'self-consistency' methods, where the model generates multiple answers and selects the most consistent one, are an improvement but still don't leverage the power of iterative refinement over a population of candidates.

Host: Right, and that’s where these sequential revision methods come in, where models refine their answers based on feedback. It's like a writing process where the first draft is reviewed, critiqued, and then revised. However, it seems these revisions tend to be more about fixing errors in a single line of thought, rather than exploring fundamentally different solutions. They're kind of like polishing a single gemstone, whereas we're aiming for a whole box of various gems.

Guest: Exactly, and this is where the concept of 'search strategies' becomes so important. Methods like 'Best-of-N,' where you generate a large set of solutions and pick the best one according to an evaluator, definitely helps with problem solving. You're casting a wider net to see what solutions the LLM can come up with. But, in many ways, it’s still a fairly brute-force approach. You're exploring breadth but without much depth, like trying a lot of keys on a door lock at once hoping that one opens it.

Host: That's a great analogy. So, now let's introduce this 'Mind Evolution' strategy. From what I understand, it's a bit different. It’s not just randomly generating solutions and picking the best one, nor is it just about revising a single line of thought. It actually evolves a whole population of candidate solutions, combining free-flowing exploration with iterative refinement. It’s like a more structured and intelligent form of trial and error, almost like nature’s way of finding solutions.

Guest: Absolutely, Leo. The core idea behind Mind Evolution is to blend 'divergent thinking,' that free-flowing exploration of different options, with 'convergent thinking,' the critical evaluation and selection of those ideas. It’s like having both a brainstorming session and a rigorous review process simultaneously. In traditional evolutionary algorithms, individuals are often represented by a string of bits or numbers, but here, we're working directly with natural language, which is a lot more flexible and intuitive.

Host: Okay, that’s a key point. So, instead of manipulating some abstract code, it directly manipulates and evolves solutions expressed in natural language, like sentences and paragraphs. This leverages the power of the LLM in understanding and generating human language, but in the context of solution generation. So, how exactly does this evolutionary process work in practice? What are the key components of 'Mind Evolution'?

Guest: Right, so let's break down the key components. Firstly, it begins with a population of initial solution candidates generated by the LLM. These are the starting points for our evolutionary journey. Then, it uses a genetic algorithm approach to evolve these candidates iteratively. Think of a typical genetic algorithm: selection, crossover, and mutation, but reimagined in the context of natural language processing. Selection involves picking the most promising candidates. Then, in the 'crossover' step, solutions are combined or re-combined in some way, creating new candidate solutions. And finally, the 'mutation' step introduces some random changes, which helps maintain diversity in the population.

Host: Okay, so instead of directly manipulating genetic code, it’s like combining ideas, tweaking them, and seeing what works. It’s like two people collaborating to refine ideas in a brainstorm. The LLM plays the role of both the idea generator and the combiner of ideas. But how does the 'selection' work in this setting? I mean, how do you know which solutions are 'good' and which are 'bad'?

Guest: That's where the 'fitness function' comes into play. The fitness function evaluates how well each candidate solution performs with respect to the target task. It essentially assigns a score to each solution. In this case, we're focusing on tasks where we can programmatically evaluate a solution. This means we have a set of rules that can be implemented in code that can check whether the solution satisfies given requirements. Based on the score from the fitness function, solutions are selected for reproduction, where better solutions are more likely to be selected, but less effective ones may still be selected to maintain diversity and provide unexpected paths for improvement.

Host: Okay, so you’re not just letting the LLM decide if something is good, you're using a separate evaluator that knows the rules and can objectively grade a solution. It’s almost like having an external judge that provides a score. So, what are some examples of these tasks that can be programmatically evaluated? I assume it's not something like abstract artwork generation?

Guest: You're right, Leo. We're focusing on what we call 'natural language planning tasks.' Think of things like travel planning, where you need to coordinate flights, accommodations, and activities. Or meeting planning, where you have to schedule meetings based on people's availability and location. In these tasks, the LLM generates a proposed plan in natural language, and this plan can be parsed and checked programmatically to see if it meets certain criteria. So, for example, the travel plan should actually follow travel rules, the budget should fit, and the meeting schedule should not have conflicts. We can write a piece of code that takes a proposed travel plan and verifies each constraint.

Host: That makes sense. So, the LLM generates a plan, but a separate program checks if it’s a valid and effective plan. This is where the search strategy is not in an abstract formal space but specifically in natural language, right? So, instead of just having the LLM propose some random words that happen to satisfy some abstract constraint in vector space, the LLM needs to propose solutions that are human readable and also follow certain criteria. And because those solutions are generated in natural language, they can be manipulated and evolved using prompts with the LLM.

Guest: Exactly. The key is that the representation of the solutions is in natural language, which allows us to leverage the LLM's ability to understand, generate, and manipulate human language in a very effective manner. Now, something else crucial for ‘Mind Evolution’ is the ‘island model’. To maintain diversity in the population, solutions are evolved independently across several 'islands', or sub-populations. Then, periodically, some solutions are exchanged between these islands. This approach prevents the whole population from converging on a single local optimal too early, therefore helping the search algorithm explore the solution space more thoroughly.

Host: Ah, so it's like creating different isolated ecosystems where each subpopulation evolves, and then there’s an exchange of genes, or ideas, between these ecosystems. This makes sense to improve diversity and not get stuck in a particular way of thinking. But how are these natural language solutions actually being changed during this evolutionary process? We talked about crossover and mutation, but what does that look like in the context of text?

Guest: That's a great question, Leo. Instead of traditional crossover and mutation operations that work on numerical representations, we're leveraging the LLM through a process we call 'Refinement through Critical Conversation,' or RCC. It involves a structured dialogue between a 'critic' character and an 'author' character, both powered by the LLM. The critic analyzes the current solution and the feedback from the fitness function, and then suggests improvements. The author then uses this analysis to propose a new, refined solution. This dialogue is guided by carefully designed prompts that instruct the LLM to play these specific roles. It’s not just random alterations; it’s informed and directed changes.

Host: Okay, so you're basically using the LLM to critically analyze a solution and then refine it based on those insights. It's like giving the LLM the ability to not only generate solutions but also critique and improve those solutions. This 'critic' role seems incredibly important. It's not just generating more options, but critically assessing existing ones, similar to a person doing peer review on a paper. But, if you’re using the LLM to do the generation, the selection, and the review, isn’t there a risk of the LLM just converging to some local minimum, or some pattern of thinking?

Guest: That's a very insightful question, Leo, and you’re touching on a core challenge in these evolutionary approaches. Yes, there is a risk of the LLM getting stuck, which is why the diversity mechanisms, like the island model, are so important. Also, the way that crossover, mutation, and island resets are implemented also contributes. First of all, the 'crossover and mutation' is itself a complex process as the LLM is generating novel solutions by combining or altering existing solutions, which ensures that not every new solution is simply a variation of the existing solution. Also, the population reset is not always simply taking the best existing solution; it's often done using an LLM that considers diversity during the selection process. So, it's not like the LLM is just optimizing for some score, it's optimizing for the generation of highly diverse high-scoring solutions through a complex prompting process.

Host: Okay, so you're not just blindly choosing the 'best' solutions; you're also trying to maintain diversity by using the LLM to evaluate different solutions in a sophisticated way. It’s like selecting different types of experts to provide solutions, which reduces the risk of a population converging on only one solution. Now, this sounds like a complex process with many moving parts. How well does this 'Mind Evolution' actually perform in comparison to other methods? I'm curious to know the results.

Guest: Great question, Leo. The experimental results are actually quite impressive. Across different benchmarks, like the TravelPlanner benchmark and the Natural Plan benchmarks for both Trip and Meeting planning, Mind Evolution significantly outperforms the traditional methods like Best-of-N and sequential revision. In the TravelPlanner and Natural Plan benchmarks, Mind Evolution often achieves over 95% success rate, whereas the traditional baseline methods could barely pass 80% in the best case. This shows a clear advantage of combining broad exploration with iterative refinement when trying to solve complex reasoning tasks. Also, when you look at the computational costs, such as the number of LLM calls, Mind Evolution is often more efficient than methods like sequential revision which makes many refinement steps on a single plan.

Host: That's a significant improvement. So, not only is it solving more problems but it's doing so with better efficiency. It makes sense that such a complex and carefully designed approach would lead to better outcomes, but it's always great to see experimental evidence that supports such claims. But it also makes me wonder, what are some limitations of this 'Mind Evolution' approach? I mean, surely there must be some scenarios where it's not the best approach?

Guest: You're right, Leo. Like any method, Mind Evolution has its limitations. Currently, it relies on having a solution evaluator that can be implemented programmatically. This means it can’t be directly applied to tasks where it’s hard to define clear rules or an objective score to judge the quality of a solution. In many ways, that is why this method was demonstrated on planning tasks where there is a well defined objective to evaluate the quality of a solution, like trip planning where each constraint can be mapped into a piece of executable code. So, the main limitation right now is the reliance on programmatic evaluators. The next challenge, and one we're actively working on, is extending this approach beyond these types of evaluations. That will require creating methods for LLMs to also assess the quality of a solution.

Host: That makes sense, so it works best when you can clearly define a way to evaluate whether or not a solution is good. And it seems like future research will focus on how to extend it to cases where evaluation is not easily done by a computer program. It's like developing ways for the LLM to grade its own homework, which is a pretty ambitious goal. But, before we wrap up for today, this whole discussion really made me curious about a new benchmark introduced in this research. I believe it is called 'StegPoet'?

Guest: Yes, that’s right. StegPoet is a very interesting benchmark we designed to push the boundaries of what these LLMs can do. It involves encoding a hidden message within a piece of creative writing. This form of stenography, where you hide a message in plain sight, is inherently difficult to formalize, yet a detector can still be implemented to programmatically assess the solution. We want to show that these search strategies can also be applied to less obvious problems, problems that are not as straightforward to turn into code.

Host: Okay, so it’s not just about finding the shortest path or creating the most optimal plan; it's about weaving secret messages into creative writing. That is a really challenging problem because it needs to satisfy the constraint that there's a hidden message within, and also that it reads naturally like a piece of creative writing. So, how does 'Mind Evolution' perform on this kind of task compared to the baseline methods?

Guest: Well, as you might imagine, the baseline methods like Best-of-N and Sequential Revision perform terribly in this new setting. It’s difficult to randomly generate text that also encodes a specific message. But the results for Mind Evolution are pretty good. They shows that it has the ability to successfully encode hidden messages within creative texts at a much higher rate. And importantly, the ‘two stage’ approach, where we use Gemini 1.5 Pro to tackle the problems that Gemini 1.5 Flash was not able to solve, shows even better results, suggesting a strong direction for further development.

Host: That’s fascinating, it shows the versatility of the search strategy beyond simple planning tasks. It's not just about optimal solutions, it's about creatively encoding information. This shows the power of this evolutionary approach when a programmatic evaluator can be created, even if it is a completely new setting with its own challenges. So, it's not just about finding the right key, but crafting the key while fitting a hidden message on the key itself. It really does show that 'Mind Evolution' is pushing the boundaries of how we approach AI problem-solving. But I feel that is a good point to stop this discussion for today. Thank you for coming and explaining this interesting work to us. This is all the time we have for today.

Evolving Deeper LLM Thinking

Summary

Discussion