The GAN is dead; long live the GAN! A Modern GAN Baseline

There is a widely-spread claim that GANs are difficult to train, and GAN architectures in the literature are littered with empirical tricks. We provide evidence against this claim and build a modern GAN baseline in a more principled manner. First, we derive a well-behaved regularized relativistic GAN loss that addresses issues of mode dropping and non-convergence that were previously tackled via a bag of ad-hoc tricks. We analyze our loss mathematically and prove that it admits local convergence guarantees, unlike most existing relativistic losses. Second, our new loss allows us to discard all ad-hoc tricks and replace outdated backbones used in common GANs with modern architectures. Using StyleGAN2 as an example, we present a roadmap of simplification and modernization that results in a new minimalist baseline -- R3GAN. Despite being simple, our approach surpasses StyleGAN2 on FFHQ, ImageNet, CIFAR, and Stacked MNIST datasets, and compares favorably against state-of-the-art GANs and diffusion models.

Host: Hey everyone, and welcome back to the podcast! I'm your host, Leo, and I'm really excited about what we're diving into today. It’s something that's been buzzing in the AI world, especially if you're into generative models. It’s kind of like when you hear a new song and can't wait to share it with your friends, that's how I feel about this topic. We're going to be talking about some really interesting research in the field of GANs, or Generative Adversarial Networks. I know that might sound a little technical, but trust me, we'll break it down into something that we all can understand and enjoy.

Guest: Hey Leo, thanks for having me! Yeah, GANs are definitely one of those areas where there's always something new happening, which is really cool. It's not just about making pretty pictures, you know? It’s about pushing the boundaries of what we can do with AI, how we understand the complexities of data, and how to create entirely new things. I think that’s something that will surprise even people who aren't deep into the tech world.

Host: Absolutely! It's easy to get caught up in the technical jargon, but at the core of it all, it’s all about creation and innovation. So, what's got us so excited today is a paper that’s doing the rounds, it’s called 'The GAN is Dead, Long Live the GAN!'— a pretty bold title right? It's a kind of paper that makes you sit up and pay attention. Basically, this research challenges a lot of what we thought we knew about training GANs and offers a very modern take on it. We've got a bunch of interesting angles to cover.

Guest: Totally. The title alone is a great conversation starter. It’s like a headline from a dramatic movie! And the substance lives up to that hype, actually. It questions this long-standing perception that GANs are just inherently tricky to train, you know, all those little tricks and workarounds. They're arguing that with the right approach, it’s not as unstable and unreliable as people often say. It's really kind of a fresh perspective that, I think, a lot of us have been looking for. Plus, this paper also touches on how the backbone architectures used in GANs are actually outdated, which is something that probably many people aren’t aware of.

Host: Exactly! It’s like they're saying, 'Hey, let's take a closer look at the foundations, not just keep piling on patches.' What’s also grabbing my attention is how they’ve simplified the entire thing, which is something we always talk about as being the key to better understanding, right? This whole idea of moving away from ad-hoc tricks to a more principled approach, it's like Marie Kondo-ing GAN training, getting rid of all the clutter. They've streamlined it, and that has resulted in something they call 'R3GAN', which is 'Re-GAN'. I love how they're basically rebuilding it from the ground up.

Guest: Yeah, that minimalist approach is what makes it so compelling. It’s like they’ve gone back to the drawing board and really re-evaluated everything. You know, how much of the complexity was actually necessary and what was just kind of…legacy stuff. It’s funny, but sometimes we get so caught up in the way things have always been that we overlook really simple, elegant solutions. Plus, by simplifying, they've also opened up doors for integrating more modern architectures. It’s not just about simplifying for simplicity’s sake, it’s about making it adaptable and scalable too, which is super important. And let's be honest, it’s also easier to understand and iterate on when you don't have to wrestle with a tangled mess of ad-hoc tricks.

Host: Right! It's like cleaning up your room, you find things you didn't know you had and it’s far easier to move around and do your work. And when you talk about the backbone architectures they talk about, it is so fundamental and they mention that some of the GANs still use backbones similar to DCGAN from 2015. It’s fascinating to see how some models stay with older technologies despite all the recent advancements. This paper is advocating that we should ditch those outdated backbones and bring in some more modern methods, like the ones we often see in diffusion models. They’re talking about stuff like multi-headed self-attention, ResNets, U-Nets, and vision transformers. It’s like giving GANs a much-needed upgrade!

Guest: It’s almost like a call to action for the GAN community! It’s saying, ‘Hey, we can do better. We don’t have to be stuck in the past.’ And these modern architectures really do seem to be the key. Diffusion models, for example, have shown just how powerful these newer approaches can be. By integrating things like multi-headed self-attention, which allows models to focus on different parts of an image when generating it, GANs can really improve. And also, things like ResNets, which allow for training of much deeper networks, is something crucial to improving performance. It's not just about swapping out old parts with newer ones, it’s also about ensuring the GANs can scale up and compete with current models that are on the frontier of generative modeling. It kind of begs the question: Why did GANs stagnate with older tech for so long? It’s probably due to the widely held belief that GANs are tricky and unstable that prevented people from really exploring the modern architectures.

Host: That’s such a key point you bring up, the perception that GANs are difficult. I think that's what's held them back. But what’s so cool is that this research provides actual evidence against that claim, which is a really big deal. They didn't just say it; they've backed it up with math. They’ve derived a well-behaved regularized relativistic GAN loss. I had to read that twice when I first came across that term. This loss addresses key issues like mode dropping and non-convergence, which have plagued GANs for ages. They go on to prove that this loss even admits local convergence guarantees, unlike the existing relativistic losses. That's quite impressive!

Guest: Oh yeah, the math behind this is really important. It's not just about throwing stuff at the wall and seeing what sticks. It's about having a solid theoretical foundation. That's what makes this work so compelling. And you’re spot on with the mode dropping and non-convergence issues. Those have been the Achilles' heel of GANs. Mode dropping, where a GAN only produces a limited variety of outputs, and non-convergence, where training just goes haywire, are things that have driven many people nuts. The fact that they've come up with a loss function that tackles these specific problems head-on and proved that they can achieve local convergence, is amazing. It gives us a roadmap for stable and reliable GAN training. It’s also saying that the tricks that people have been using to work around these issues are actually unnecessary when you have a well-defined loss function.

Host: Exactly! It’s like, if you get the fundamentals right, you don’t need all those band-aids. And this also ties in with the minimalist design approach they’ve taken with R3GAN. It’s a new GAN baseline, and it's designed to be as simple as possible, while also being extremely effective. They started with StyleGAN2, stripped it down to the core, and then rebuilt it with modern backbone architectures that I mentioned earlier. It's interesting how they went from StyleGAN2, which is considered quite good, to something they call their minimum baseline. This shows that a lot of things in StyleGAN2 were actually just adding complexity without necessarily providing huge benefits. It’s fascinating to see how they carefully removed things like the mapping network, the style injection, the weight modulation/demodulation and a whole host of other tricks, and how they ended up with a system that’s both simpler and performs better.

Guest: Yeah, the process they went through with StyleGAN2 is like an archeological dig, removing layer after layer of added complexity until they got to the very core of what’s needed for image generation. And it's really telling that the removed features are kind of in three categories: style-based generation techniques, image manipulation enhancements, and what they call tricks. Each of these categories serves a purpose of course, but this paper highlights that there are far more fundamental areas that one needs to address to achieve high quality image generation. By systematically removing all those bells and whistles, and then using their new well-behaved loss function, they were able to actually improve on StyleGAN2’s performance. It really drives home the point that sometimes, less is more and simplicity is key. I also like how they didn’t just jump to the newest cutting-edge architecture, but instead took time to carefully evaluate each step they made along the way. It’s a very principled and methodical approach. They have also inherited training hyperparameters from StyleGAN2, which adds a point of reference to this research.

Host: Exactly, it’s not just about blindly adopting the newest and shiniest tech; it’s about taking a principled, methodical approach, understanding the ‘why’ behind the ‘what.’ It’s something that I think is critical in research. Now, what I found quite interesting is this 'ResNet-ify' process they go through. They basically replace the traditional StyleGAN2 backbone with a more modern ResNet architecture, which is the direct ancestor of all modern vision backbones. And this is a key point: it’s not just any ResNet, but a properly designed ResNet, and that is so critical in this whole transformation process. The idea that simply changing to something that looks new will magically improve things, that's not it at all. They were also very keen to incorporate the findings of the ConvNeXt paper. I read this paper and you really can see a lot of common themes there and what they have taken from that research. It's so important, I think, that they really pay attention to which elements are beneficial in ConvNeXt and which aren’t.

Guest: Oh, that 'ResNet-ify' section is where things really get interesting from a technical point of view. It’s like they are building the model, piece by piece, right? Instead of just swapping in a ResNet, they went back to those first principles of what makes ResNet work so well, and made sure to incorporate it correctly. And it's so key they don’t just do it for the sake of ‘new,’ but also to understand what’s truly working. They emphasize the importance of things like the 1-3-1 bottleneck architecture, which is a very effective way to manage the number of parameters in the network. They also discuss those beneficial elements that they took from ConvNeXt, which includes things like increased width with depthwise convolution, and that inverted bottleneck design that improves the capacity of the model. It really shows how carefully they were thinking about each architectural choice, rather than just blindly following the trend. The way they categorize the ConvNeXt elements into ‘consistently beneficial’ ‘negligible performance gain’ and ‘irrelevant to our setting’ is also something I think more research should do. It helps streamline not just their own work, but helps others know what is crucial and what not so much.

Host: That categorization is so important! It shows that not all novel ideas translate into real performance gains for all scenarios and it’s important to have that critical eye when looking at new architectures. And what really got me thinking is how they’re not just trying to mimic the vision transformers. They’re actually taking the useful ideas from those architectures and applying them in their own way. The fact that they maintain a fully symmetric design for the generator and discriminator also highlights their pursuit of simplicity. Their goal was not just to make things modern, but to make things effective, and they kept the parameter counts relatively similar to the original StyleGAN2 while having something fundamentally different under the hood. They also emphasized that things such as the output skips from StyleGAN2 are no longer useful in this new architecture, which I find fascinating.

Guest: Yeah, the architecture details they provide really emphasize that well-thought-out and principled approach. The fully symmetric generator and discriminator design makes it easier to analyze and optimize for, which goes right back to their minimalist approach. And that point about output skips in StyleGAN2 not being useful in this context, it’s like they’re saying, ‘We've solved the gradient dynamics issues in a more fundamental way, so we don't need those workarounds anymore.’ It’s also important how they went into the tiny details like the ResNet block designs, which each have a transition layer and two residual blocks, which are more sophisticated than what was previously used. And again, they have taken principles from Config B, the minimum baseline architecture, such as avoiding normalization layers, having proper resampling with bilinear interpolation and using a leaky ReLU activation functions. It's clear they’ve given the process a lot of thought.

Host: They have also gone into the details of initialization, using fix-up initialization to avoid variance explosion. I know that's often quite a common hurdle for many and it’s something they have specifically addressed. It’s that kind of painstaking attention to every part of the process that really sets the R3GAN apart. It’s not just about high-level ideas; it’s also about making sure that everything works smoothly at the nuts-and-bolts level. Also when we talk about these inverted bottlenecks in the model, they make sure to use grouped convolutions which really enhance the capacity of these convolutions. And these are all really detailed points, but these are exactly the details you need to pay attention to, and this research really touches upon that. The comparison in Figure 3 really shows how simple and clean the R3GAN design is in comparison to StyleGAN2. It’s quite striking when you see the difference laid out visually.

Guest: Absolutely. The fix-up initialization is crucial. It's something they take seriously, and it really highlights how important all these little details are. And, as you rightly pointed out, the way they make use of grouped convolutions to reduce the computational cost while also increasing the bottleneck width and enhancing the capacity of grouped convolutions by inverting the bottleneck width and stem width, all these decisions show a deep understanding of modern CNNs. This is not just a random selection of methods, they’ve put together a design that’s both powerful and efficient and really optimized for GAN training. And the comparison in Figure 3 really emphasizes the differences: you see the old, complex design of StyleGAN2 next to their sleek new design. It really highlights the focus on simplicity and efficiency and also showcases how much they have managed to modernize the whole architecture of a GAN.

Host: And all that theoretical work, all that attention to detail, it pays off when you look at the experimental results. It was amazing to see how much the R3GAN improves on the original StyleGAN2. In fact, in their experiments on the FFHQ-256 dataset, the R3GAN actually outperforms the original StyleGAN2, and this is not even with optimized hyperparameters. They also achieve really high mode coverage on StackedMNIST and they also tested R3GAN on multiple datasets like CIFAR-10 and ImageNet at various resolutions. It's pretty telling that their model outperforms even more recent diffusion-based models as well. And they also make a clear point that their model does not use any ImageNet pre-training, which avoids a lot of issues that has to do with leaking information. I have to say I was very impressed with their performance on ImageNet, especially given the small size of the model. I find it really intriguing that even though they have a much smaller model than diffusion models, they actually surpass them in terms of the Fréchet Inception Distance(FID), which is a key metric used in assessing the quality of generative models.

Guest: Yeah, the experimental results are really the proof in the pudding, right? It’s one thing to talk about theory and architecture, but it’s another to actually demonstrate its practical application and how well it performs on different datasets. And the fact that their model achieves a lower FID on FFHQ-256 than the original StyleGAN2 without even optimized hyperparameters and then further improves by optimizing hyperparameters is really amazing. It truly showcases the impact of a well-behaved loss function combined with an effective architecture. And that mode coverage they’ve achieved on StackedMNIST, recovering all 1000 modes, is truly something special. It shows the diversity of images the model can create. It’s also really cool they tested it on ImageNet and are competing with diffusion models. The size of their model is also a crucial factor; they’ve shown that you don't need these huge, computationally expensive models to produce high-quality results. The fact they are achieving better results than diffusion models with one function evaluation shows how powerful GANs can be when they are approached in a principled manner.

Host: And on the discussion of the model sizes they have also touched upon the recall metric, and how this model is able to achieve a comparable recall rate to diffusion models, which is something that is often touted as being a strength of diffusion models. So all in all, it showcases that GANs, when rethought and properly constructed, can compete with and, in some cases, exceed the performance of other generative models. Their point about simplicity is not just about less code or less computational cost, but about making the model easier to understand, making it more versatile and adaptable and that I think is the real value in all of this. So all in all, this is a very impactful paper in the GAN community.

Guest: Absolutely. And that point about the recall metric is really important. It’s not just about creating images that look realistic, but also about creating diverse images that cover the spectrum of possibilities. That they have achieved comparable recall to diffusion models shows they have addressed a critical concern of GANs. And the impact extends far beyond just the numbers and benchmarks, though, because it’s providing a blueprint for future research and development in GANs. They've shown that there's still a lot of potential in GANs, and we shouldn't just give up on them because of some past challenges. It’s such a great piece of work, really highlighting that a strong theoretical foundation, combined with a minimalistic architecture, can result in very powerful models. And it's also a message that the AI field should embrace more often, to understand the principles behind what we do, rather than just chasing the newest trends. Their new baseline model, R3GAN, could really be a launchpad for even more exciting developments in GAN technology.