LTI and Amazon AGI Launch MMU-RAG Competition to Advance AI Systems for Complex Information Tasks
The MMU-RAG competion debuts at NeurIPS 2025
By Bryan Burtner
The Language Technologies Institute (LTI) at Carnegie Mellon University, in partnership with Amazon AGI, has announced the launch of MMU-RAG, a new competition aimed at evaluating the next generation of Retrieval-Augmented Generation (RAG) systems.
The inaugural MMU-RAG competition will take place at the 39th edition of the Annual Conference on Neural Information Processing Systems (NeurIPS 2025).
RAG systems combine traditional information retrieval, such as searching large text corpora, with AI text generation, enabling them to produce informed, contextually relevant responses. These systems are increasingly used in applications like advanced chatbots, digital assistants, and research tools.
While current benchmarks typically focus on straightforward queries, MMU-RAG is designed to reflect the complexity of real-world information needs. The competition introduces multi-turn, user-driven queries that are open-ended, ambiguous, and exploratory, posing a greater challenge to participating systems.
In addition to conventional evaluation metrics, MMU-RAG will incorporate real-time human feedback through RAG-Arena, an interactive online platform where users directly compare system responses. This approach is intended to better simulate practical performance and usability.
Participants will have access to ClueWeb22-B, a large-scale web corpus containing more than 87 million documents, to support system development. The competition includes two tracks: Text-to-Text, focused on answering open-domain questions using textual data, and Text-to-Video, requiring systems to generate video-based responses grounded in retrieved information.
Submissions will be judged using a combination of automated metrics, evaluations by large language models, and live user ratings collected through RAG-Arena.
MMU-RAG is open to researchers working on information retrieval, language generation, and multimodal AI. The competition officially opens on August 1, 2025.
More details and registration information are available at: https://agi-lti.github.io/MMU-RAGent/