SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines

Large language models (LLMs) have demonstrated remarkable proficiency in mainstream academic disciplines such as mathematics, physics, and computer science. However, human knowledge encompasses over 200 specialized disciplines, far exceeding the scope of existing benchmarks. The capabilities of LLMs in many of these specialized fields-particularly in light industry, agriculture, and service-oriented disciplines-remain inadequately evaluated. To address this gap, we present SuperGPQA, a comprehensive benchmark that evaluates graduate-level knowledge and reasoning capabilities across 285 disciplines. Our benchmark employs a novel Human-LLM collaborative filtering mechanism to eliminate trivial or ambiguous questions through iterative refinement based on both LLM responses and expert feedback. Our experimental results reveal significant room for improvement in the performance of current state-of-the-art LLMs across diverse knowledge domains (e.g., the reasoning-focused model DeepSeek-R1 achieved the highest accuracy of 61.82% on SuperGPQA), highlighting the considerable gap between current model capabilities and artificial general intelligence. Additionally, we present comprehensive insights from our management of a large-scale annotation process, involving over 80 expert annotators and an interactive Human-LLM collaborative system, offering valuable methodological guidance for future research initiatives of comparable scope.

Host: Hey everyone, and welcome back to the podcast! Today, we're diving into the fascinating world of scientific research and how it's disseminated. It’s something we all benefit from, even if we don't always realize it, because it's the bedrock of progress. I'm your host, Leo, and I'm thrilled to have you join me.

Host: Specifically, we’re going to be talking about arXiv, that's A-R-X-I-V. It's an amazing resource, but I think a lot of people outside of academia might not be super familiar with it. So, we're going to explore what it is, why it's important, and some of the challenges it faces.

Host: Okay, so to get started, let's talk about arXiv in general. It's basically an open-access repository for electronic preprints of scientific papers. That might sound like a mouthful, but essentially it's a place where researchers can upload their work before it's formally published in a peer-reviewed journal. It's a digital archive, think of it like a giant library but specifically for cutting-edge research papers in fields like physics, mathematics, computer science, quantitative biology, quantitative finance, statistics, electrical engineering and systems science, and economics. I mean, that's quite a range!

Host: And you know, what's really cool is that it's completely free for anyone to access. So, researchers all over the world can see the latest findings in their fields, and even people outside of academia, like us, can take a peek behind the curtain and see what's happening at the forefront of science. It democratizes access to knowledge in a really powerful way. It's funded by Cornell University, the Simons Foundation, member institutions, and contributors, which shows the collaborative spirit of the scientific community.

Host: Alright, I just pulled up a specific arXiv entry here. Apparently there's “No HTML for '2502.14739'.” Which already opens up a can of worms about accessibility and formats, doesn’t it? So the entry is a placeholder for a paper presumably submitted under that identifier, 2502.14739, but the HTML version, which is what a web browser needs to properly display the content, isn’t available.

Host: This can happen for a few reasons. The most common one is probably that the original source files submitted by the author weren’t in HTML, LaTeX, or some other format that arXiv can automatically convert to HTML. LaTeX is a document preparation system that's widely used in the scientific community, especially in math and physics, because it's really good at handling complex equations. If the author submitted a PDF, or some other format that isn't easily convertible, arXiv might not be able to generate an HTML version. This is important to note because while PDFs are accessible, HTML provides a more streamlined, accessible and often faster way to read the paper online, especially on different devices.

Host: The system also points out that if you are the author, you can learn how to improve HTML conversions for your papers. So, it's actively encouraging authors to contribute to the accessibility of their work. That's a good sign. Because accessibility is super important for scientific research, so more people can access your results.

Host: Let's delve into that accessibility aspect a bit more. When a paper isn’t available in HTML, it creates a barrier for some users. People with visual impairments, for example, often rely on screen readers to access online content. Screen readers work best with HTML because it's structured in a way that allows the reader to navigate the content logically. Trying to use a screen reader with a PDF can be much more difficult, especially if the PDF isn't properly tagged for accessibility. Similarly, users with older devices or slower internet connections might find it easier to load and view an HTML version of a paper than a large PDF file.

Host: Beyond accessibility for individuals with disabilities, HTML also facilitates other forms of accessibility. For instance, it makes it easier to copy and paste text from the paper, which is useful for researchers who want to quote passages in their own work or analyze the text using computational tools. HTML is also more easily indexed by search engines, which means that papers available in HTML are more likely to be discovered by researchers searching for relevant information. So, making sure papers are available in HTML really enhances their visibility and impact.

Host: The page also includes links to sections like 'About', 'Help', 'Contact', 'Subscribe', 'Copyright', 'Privacy Policy', and 'Web Accessibility Assistance'. It's got all the bases covered when it comes to transparency and support, which builds trust in the system, right? Because ultimately, arXiv is only as good as the community that uses and supports it.

Host: Let’s think a bit about the operational side of arXiv. It mentions an 'arXiv Operational Status' page where you can get status notifications via email or Slack. I think that’s pretty key for researchers relying on the platform. If there are server issues or maintenance going on, knowing about it in advance can save people a lot of frustration. It shows they are thinking about the user experience and providing consistent service. It also hints at the scale of the infrastructure behind the scenes, to keep a repository like this up and running for millions of users globally.

Host: And let's be honest, arXiv has become so ingrained in the research workflow in many fields that it's almost unthinkable to imagine science without it. It's a primary way that researchers share their findings and get feedback from the community before formal publication. It speeds up the pace of scientific discovery immensely. Think about it, before arXiv, researchers had to wait months, sometimes even years, for their papers to be published in a journal. With arXiv, they can share their work with the world within days.

Host: However, this speed and openness also come with challenges. One of the biggest ones is quality control. Because arXiv is a preprint server, papers aren't subject to the same rigorous peer review process as they would be in a traditional journal. This means that there's a greater chance of errors, or even outright fraudulent research, being posted on the site. While arXiv does have some basic screening processes to prevent obviously problematic submissions, it's largely up to the community to identify and flag issues.

Host: This reliance on community feedback can be a double-edged sword. On the one hand, it allows for a more democratic and collaborative approach to quality control. Researchers can quickly point out errors or flaws in a paper, and the author can then revise their work accordingly. On the other hand, it can also lead to biases and inequalities. For example, papers by well-known researchers or from prestigious institutions might receive more attention and scrutiny than those by less-established researchers or from less-known institutions.

Host: And speaking of biases, another potential issue with arXiv is that it may not be representative of all research. Researchers in certain fields, or from certain countries, may be more likely to use arXiv than others. This could lead to a skewed picture of the overall landscape of scientific research. It’s worth considering, for instance, if the dominance of English-language publications creates a barrier for researchers who aren’t native English speakers. Are important findings being missed simply because they aren't easily accessible to the majority of the community?

Host: Another element I think we should touch upon is the impact of arXiv on traditional academic publishing. Journals are still seen as the gold standard for scientific publication, and they play a critical role in evaluating research for funding and promotion decisions. So how does arXiv fit into this ecosystem? Does it complement or compete with traditional journals?

Host: Well, I think it's a bit of both, actually. On the one hand, arXiv can serve as a valuable supplement to traditional journals. It allows researchers to get their work out there quickly and receive feedback from the community before submitting it to a journal. This can help improve the quality of the final published version. Plus, many journals now allow or even encourage authors to post preprints of their papers on arXiv.

Host: On the other hand, arXiv does pose a challenge to the traditional publishing model. If researchers can freely access papers on arXiv, why would they pay for a journal subscription? This is a question that publishers are grappling with, and they're exploring different ways to adapt to the changing landscape. Some are experimenting with open-access publishing models, where authors pay a fee to have their papers made freely available. Others are focusing on providing value-added services, such as peer review and copyediting, that aren't available on arXiv.

Host: I think the rise of arXiv has really forced the scientific community to rethink the way we disseminate and evaluate research. And that's a good thing. It's pushing us to move towards a more open, accessible, and collaborative model of science. But it also raises some important questions about quality control, bias, and the role of traditional publishers.