GameFactory: Creating New Games with Generative Interactive Videos

Generative game engines have the potential to revolutionize game development by autonomously creating new content and reducing manual workload. However, existing video-based game generation methods fail to address the critical challenge of scene generalization, limiting their applicability to existing games with fixed styles and scenes. In this paper, we present GameFactory, a framework focused on exploring scene generalization in game video generation. To enable the creation of entirely new and diverse games, we leverage pre-trained video diffusion models trained on open-domain video data. To bridge the domain gap between open-domain priors and small-scale game dataset, we propose a multi-phase training strategy that decouples game style learning from action control, preserving open-domain generalization while achieving action controllability. Using Minecraft as our data source, we release GF-Minecraft, a high-quality and diversity action-annotated video dataset for research. Furthermore, we extend our framework to enable autoregressive action-controllable game video generation, allowing the production of unlimited-length interactive game videos. Experimental results demonstrate that GameFactory effectively generates open-domain, diverse, and action-controllable game videos, representing a significant step forward in AI-driven game generation. Our dataset and project page are publicly available at https://vvictoryuki.github.io/gamefactory/.

Host: Hey everyone, and welcome back to the podcast! I'm your host, Leo, and I'm really excited about today's topic. We're diving into some cutting-edge research that's blurring the lines between AI, video generation, and, get this, game development. It's a really fascinating area, and I think it’s something a lot of people haven’t even considered the possibilities of. We have a very cool discussion to share with you all, so let's jump straight into it!

Host: Alright, so today we're going to be unpacking a fascinating paper titled 'Creating New Games with Generative Interactive Videos'. It's from a team at the University of Hong Kong and Kuaishou Technology. This research really pushes the boundaries of how we think about game creation, moving beyond traditional methods and embracing AI-driven content generation. We’ll talk about how they are achieving this by leveraging video diffusion models that have been pre-trained on vast open-domain video datasets, which is already quite a mouthful and a big step in itself.

Host: So, to give a bit of context, the fundamental idea here is to build a generative game engine. This isn’t about making minor tweaks to existing games but rather creating entirely new ones, automatically, and with a focus on scene generalization – that's creating completely new game environments, not just reskins of what we already have. It's about enabling the creation of games that aren't limited by fixed styles or scenes. Traditional game development is usually manually intensive with a lot of human work, from art design to level creation, which is a real resource hog. So if they can do this, it’s a pretty big deal.

Host: The team calls their framework 'GameFactory,' and it's designed to generate diverse and interactive game videos with action controls. Instead of relying on massive datasets of game-specific footage, which can be both expensive and limiting, they've cleverly used video diffusion models that have been pre-trained on a lot of varied, open-domain video data. Think about all the videos that exist on the internet – that's their starting point, and it’s a great idea. They’re trying to bridge the gap between the open-domain knowledge of these large models and the specifics of game mechanics using a really clever training approach.

Host: This is all powered by the idea of 'scene generalization.' This is where the system can create new game scenes that look totally different from existing ones, which hasn't been easy so far. Most existing methods are trained on specific games like DOOM, Atari, or Minecraft, and that means they're kind of stuck in those styles and environments. GameFactory tackles this by using pre-trained video models, and it is very interesting how they do it, they use a multi-phase training strategy to keep the open-world generation strong while also learning action control. They've also introduced a new dataset called 'GF-Minecraft' for action-annotated video, and they’ve done it in a really cool way that we'll touch on later.

Host: Okay, let’s start digging into the technical stuff a bit, shall we? The core concept is that video diffusion models are really good at generating high-quality videos and even simulating some real-world physics. The models basically learn to remove noise from an image or video until it becomes a coherent visual. So they propose to use this to generate the physics of games, making them a good choice for building game engines, as if the world itself has its own set of physics rules that the model is using to generate the video. This can dramatically reduce the manual work involved in traditional game creation, because you are not scripting every little interaction, and the model will understand some of those interactions on its own.

Host: Now, these generative game engines typically work by allowing user actions like keyboard and mouse inputs to control the video generation, which is how you make the game 'interactive.' But the big challenge, like we touched on earlier, is scene generalization. Existing systems, as we've seen, mostly stick to familiar game styles and scenes, so the real breakthrough with GameFactory is pushing beyond that. The big problem is that training a model to generate content for, say, Minecraft, is great for Minecraft-like games, but it's not going to generate anything that looks like, say, a racing game. This is a key limitation this paper addresses. They're thinking bigger and want the engine to create all sorts of games and not get trapped within the visuals of Minecraft, or similar.

Host: They highlight that collecting large-scale datasets with action annotations for a wide range of environments would be ideal for this sort of thing, but that’s super expensive and probably not really possible. You’d need to manually label every action and every possible game scenario, which, if you think about it, is an immense amount of work. Instead, the approach they are taking is to tap into the vast amount of unlabeled video data on the internet, where the model has already learned a bunch of stuff about how things look and move. They use this knowledge and just train a small action control module, so they can keep the pre-trained model’s rich generative knowledge and still have a way of controlling it. So, the action control is trained on a smaller dataset with annotations, but they can create a whole universe with the open-domain video dataset, that’s a clever approach.

Host: And that brings us to why they chose Minecraft for the action data. Minecraft is actually a great choice because it's very customizable, meaning you can control a wide range of actions, and can get frame-by-frame action annotations. It is also not so difficult to generate game data without human bias, which is actually very useful when you want a model to learn everything, and not just the things human players tend to do, which is important to get the full capabilities of your model. It allows you to explore things that you may not have thought of, and it is a big help in creating a more robust model.

Host: Now, a key part of GameFactory is this multi-phase decoupled training strategy. This is what lets them separate the 'look' of the game from the 'control' of the game. If you just try to teach the model everything at once on, say, Minecraft footage, the generated videos will look very Minecrafty, that pixelated block look, because it has learned to connect those visuals to the gameplay. But that’s not ideal if you want to make different kinds of games. So, their method is to train in stages; first they tune a model to understand the general style and feel of a game, and then they train a separate module to control the actions in the game using keyboard or mouse inputs, keeping the look and gameplay separate, which is pretty key for generalizing to different styles of games.

Host: Another crucial element is how they handle the differences between mouse movements, which are continuous, and keyboard inputs, which are discrete. They've designed control mechanisms specifically for each, which is something that most approaches miss. They also extended their model to handle long, continuous videos, which is critical for actual games, where you don’t want a short clip, you need long-form gameplay. They achieve this through an autoregressive generation method which is what allows them to generate the longer game videos.

Host: So, to sum up the key contributions here, GameFactory introduces a new way of generating open-domain game videos by decoupling style from control, they also release their GF-Minecraft dataset of annotated video for research, which is cool, and they also have implemented autoregressive generation which makes unlimited length interactive videos possible. It's all about moving towards AI-driven game engines that can create diverse and new experiences without being limited to existing game styles. This is quite a significant jump!

Host: Now, let's zoom in a bit on some of the technical details. The paper uses a transformer-based latent video diffusion model as a backbone. This model takes a video sequence and compresses it into a latent space, think of it as a kind of condensed representation of the video, then it learns to generate the video from this condensed form. The model is trained to predict noise; it adds noise to clean video data, then learns how to remove it, and by reversing the noise removal process, the model is able to generate new videos. This is key to using the pre-trained video data because it allows the model to learn from any video data.

Host: When you introduce action control, they then add in action data, like keyboard presses and mouse movements. The system takes the action taken between frames, calculates what changed, and uses it as a new condition for noise prediction, enabling the model to control the output using these actions, instead of the model just generating random content. Then when it generates a video, it not only creates the content but makes sure it corresponds to the actions being provided.

Host: They address the challenge that the number of actions doesn't directly match the number of video features because of how they compress the video sequence for efficiency; to tackle this, they group actions using a sliding window. This is to take into account that an action like a jump isn't necessarily fully shown in one frame; it has an effect over multiple frames. The size of the window is what they adjust to account for this, which is also a really key implementation to get the model to understand the impact of user inputs correctly. This action grouping method is crucial for mapping actions to the video output.

Host: For mouse movements, which are continuous inputs, they use concatenation to combine the action data with the video features, it allows the model to take into account the magnitude of movements. For keyboard inputs, which are discrete, they use cross-attention, which is a great way to process categorical inputs, where the model pays attention to only the parts of the video that correspond to certain keyboard inputs. They found this combined method works better than using just one method for both actions. They also found that mouse movement had more impact on visuals than keyboard input, so the model is able to generate visuals that are relevant to the different inputs, that’s really cool.

Host: Now let's get back to this idea of leveraging pre-trained models. Existing game generation methods often just reproduce existing games, but the real goal is to create entirely novel ones. They've been able to achieve this by using a model pre-trained on open-domain videos, and they have been able to do it by separating the control and visuals in the multi-stage training process we discussed earlier. This allows the model to create game scenes that don’t look like existing games. This is what gives them the open-domain scene generalization.

Host: They highlight the multi-phase training strategy again, with a first phase where the model learns the style of a specific game, and a second phase where it trains just the action control module. This decoupling is the key to avoiding style bias. In essence, Phase #1 learns a game's visual 'look,' while Phase #2 focuses on the interactive part, which are two different things for the model, which allows for open-domain action-controlled video generation that doesn't inherit the style from the training data. Then for inference, they remove the style weights from the model. The action control will then work independently of specific game styles, which is what allows them to generalize to open-domain scenarios. It’s quite an innovative approach to a tricky problem.

Host: And finally, to create those long videos necessary for a full game experience, they have an autoregressive generation approach. Rather than generating the full video at once, they generate video segments and feed the previous output back into the model as a condition. In training, they randomly choose segments of the video and only predict the noise of the predicted frames. In inference, they repeatedly generate video sections, using the most recently generated frames as a starting point and combining them to build longer videos; this is really important if you want an interactive, engaging game, as you'd otherwise have very short clips.

Host: The beauty of this is that it's very efficient, and they are also able to generate multiple frames in one step, which is a major improvement compared to previous methods that could only create one frame at a time, and this also allows them to generate an unlimited number of frames. They also add tiny amounts of noise to the condition frames, which is a clever trick they employ that helps prevent error accumulation when generating long videos.