MARVEL-40M+: Multi-Level Visual Elaboration for High-Fidelity Text-to-3D Content Creation

Generating high-fidelity 3D content from text prompts remains a significant challenge in computer vision due to the limited size, diversity, and annotation depth of the existing datasets. To address this, we introduce MARVEL-40M+, an extensive dataset with 40 million text annotations for over 8.9 million 3D assets aggregated from seven major 3D datasets. Our contribution is a novel multi-stage annotation pipeline that integrates open-source pretrained multi-view VLMs and LLMs to automatically produce multi-level descriptions, ranging from detailed (150-200 words) to concise semantic tags (10-20 words). This structure supports both fine-grained 3D reconstruction and rapid prototyping. Furthermore, we incorporate human metadata from source datasets into our annotation pipeline to add domain-specific information in our annotation and reduce VLM hallucinations. Additionally, we develop MARVEL-FX3D, a two-stage text-to-3D pipeline. We fine-tune Stable Diffusion with our annotations and use a pretrained image-to-3D network to generate 3D textured meshes within 15s. Extensive evaluations show that MARVEL-40M+ significantly outperforms existing datasets in annotation quality and linguistic diversity, achieving win rates of 72.41% by GPT-4 and 73.40% by human evaluators.

Host: Hey everyone, and welcome back to another episode of 'TechForward'! Today, we're diving deep into the fascinating world of 3D model generation, specifically how we can create incredibly realistic 3D objects from just text prompts. It's a field that's rapidly advancing, and we've got a really exciting development to talk about today.

Guest: That's right, Leo. It's amazing how far we've come. Remember when even simple 3D models took hours of painstaking work? Now, we're talking about generating complex, textured meshes in seconds, all from a simple text description.

Host: Exactly! And the key to unlocking even higher fidelity and speed in text-to-3D generation lies in the data. Today we're discussing MARVEL-40M+, a massive new dataset that's pushing the boundaries of what's possible.

Guest: MARVEL-40M+ is huge. We're talking about 40 million text annotations for almost 9 million 3D assets! That's an unprecedented scale. Before this, the datasets were just too small and lacked the diversity needed to train truly sophisticated models. The lack of annotation depth was another massive hurdle.

Host: Right, and it's not just the sheer size, it's the quality and depth of the annotations. They've used a clever multi-stage annotation pipeline. They're not just simple tags; these are detailed descriptions, sometimes 150 to 200 words long, giving incredible detail about the 3D objects. They also have concise semantic tags for quick prototyping, providing a flexible framework for different use cases.

Guest: That multi-stage pipeline is brilliant. They combined open-source pretrained multi-view vision-language models (VLMs) and large language models (LLMs) which is a big step towards making this technology more accessible. No more expensive proprietary models needed for everyone to get in on this action. Using InternVL2 and Qwen, they're able to generate descriptions covering object names, components, shape, geometry, texture, materials, colors, and the whole contextual environment. It's comprehensive.

Host: And what's really clever is how they incorporated human metadata from the original datasets. This helps to add domain-specific information and to reduce those pesky hallucinations that plague these models. Imagine trying to generate a 3D model of a historical artifact—human input ensures accuracy and context that a VLM alone might miss.

Guest: Exactly! The human metadata acts as a kind of ground truth, guiding the VLMs and LLMs toward more accurate and detailed descriptions. It's like adding a layer of expert knowledge to the process, reducing the chance of errors. They cleverly filtered this human data to remove any noise or irrelevant bits to maintain data integrity.

Host: The hierarchical structure of the annotations – five different levels of detail – is also a game changer. It means you can easily switch between detailed descriptions for fine-grained control and simpler tags for faster prototyping. It's adaptability in action, so you get the best of both worlds.

Guest: Absolutely. This is a massive step forward in terms of both the quality and efficiency of 3D model annotation. Think about the implications; training data is no longer a bottleneck. This means more realistic, more detailed, and higher-quality 3D models, at speed.

Host: And they didn't just create the dataset; they also developed MARVEL-FX3D, a two-stage text-to-3D pipeline that uses this data. It's pretty fast – generating textured meshes in just 15 seconds! They fine-tuned Stable Diffusion to generate high-quality images suitable for 3D reconstruction, and then used a pre-trained image-to-3D network.

Guest: The speed is astonishing. It overcomes a major limitation of existing methods that rely on slow optimization processes. This is really where the combined power of the dataset and this new pipeline shines. It addresses the 'Janus problem'—the geometric inconsistencies that often plague text-to-3D generation – leading to more realistic and consistent 3D outputs.

Host: And the results speak for themselves. Their evaluations show that MARVEL-40M+ significantly outperforms existing datasets in terms of annotation quality and linguistic diversity. Both GPT-4 and human evaluators gave it very high marks. They also showed that MARVEL-FX3D outperforms other state-of-the-art text-to-3D methods in terms of prompt fidelity and overall preference.

Guest: This is a really exciting development, Leo. It's a huge leap forward in the field of text-to-3D generation. The availability of this dataset and the associated pipeline opens up a whole new world of possibilities for 3D modeling. It will certainly speed up adoption of this tech. The fact they used open-source models too is a big plus, meaning this innovation can be widely accessible and used to benefit different sectors and industries.

Host: Definitely. It's not just about gaming and VR anymore; this has applications across design, architecture, manufacturing, and even scientific visualization. This is just the beginning, and I can’t wait to see what incredible creations emerge from this breakthrough.