Termsslug: mmlu-pro

MMLU-Pro

An enhanced multitask language understanding benchmark using harder multiple-choice reasoning questions.

Curated Intelligence

2 items · Page 1/1
Rohan Paul (@rohanpaul_ai)T2·Models
58 pts
Australia Launches Agentaus 1, a 220B Sovereign AI Model

Australian firm Trellis Data has released Agentaus 1, a 220B-parameter model built around data sovereignty: prompts stay on Australian servers and are never used for training. The company reports 84.4% on MMLU-Pro, ranking 43rd among foundation models, and says it runs on two GPUs where comparable models need four.

One Useful Thing (Ethan Mollick)T2·Models
68 pts
Why AI Benchmarks Like MMLU-Pro Fail to Measure Intelligence

The article argues that mainstream AI benchmarks like MMLU-Pro are flawed due to data contamination, irrelevant questions, and poor calibration, failing to measure true intelligence. It advocates moving away from static 'human-like' testing towards dynamic, interview-style evaluation methods.