Google and Artificial Analysis Launch MKA Benchmark for Multimodal Agents
Google and evaluation firm Artificial Analysis released MKA, a benchmark of 200 multimodal knowledge-work tasks spanning 10 industries that mirror 3-8 hours of expert human effort. Gemini 4 Carbon placed second while costing roughly one-sixth of the top model per task, and the benchmark is now open to developers via Artificial Analysis.
Full Report
IT之家 November 11 news — on November 10 local time, Google announced a partnership with AI model evaluation organization Artificial Analysis to launch the Multimodal Knowledge Agent benchmark, MKA (Multimodal Knowledge Agent), designed to evaluate the ability of AI agents to handle visually complex documents and complete real-world knowledge work tasks.
Google stated that its newly released Gemini 4 Carbon achieved second place among the participating models in this test, with an average task completion cost of only about one-sixth that of the top-ranked model.
Rohan Doshi, Senior Product Manager at Google DeepMind, and Ben Caine, Principal Engineer, explained that many existing knowledge work benchmarks can be completed using text alone, whereas real-world work often involves a large amount of visual information. For example, architects need to cross-check building blueprints, engineers need to analyze engineering diagrams, and professionals also need to organize information from different sources into presentations, reports, and spreadsheets.
The MKA benchmark is designed to evaluate the performance of AI agents in these complex work scenarios. Google collaborated with domain experts from 10 industries and 83 occupations to build an evaluation set containing 200 tasks, covering fields such as business, architecture, education, engineering, finance, healthcare, law and government, logistics and maps, science, and software and data.
Each task simulates a multi-step professional project, including real-world work prompts and a set of multimodal input files. Google says these tasks typically require an experienced professional 3 to 8 hours to complete. IT之家 noted that each task is also accompanied by expert-written scoring criteria, averaging about 70 true/false check items, of which factual accuracy accounts for 75% and visual design accounts for 25%.
In terms of the evaluation mechanism, all participating models run on the same open-source agent framework, Stirrup, using the same tools, sandbox environment, and interaction turn budget. The final deliverables are scored by an agent-based review system built on non-Google models, which opens all files, renders pages, checks spreadsheet formulas, and verifies results according to the expert scoring criteria. Google says that when domain experts manually score model outputs, their agreement rate with the automated review system reaches 96%.
To prevent test questions from leaking and entering model training data, Artificial Analysis has kept the 200 official evaluation tasks private. At the same time, the project has also publicly released another 10 tasks on Hugging Face, allowing developers to understand the test format and evaluate their own agents.
The test results published by Google show that Gemini 4 Carbon achieved second place in quality among the participating models, but its average task cost was only one-sixth that of the top-ranked model. Google also pointed out that there are significant differences in efficiency among different models when completing the same batch of tasks, with the average number of interaction turns consumed per task ranging from 30 to 130.
Currently, the MKA benchmark is available through Artificial Analysis, and developers can use the platform to evaluate their own models. Gemini 4 Carbon is now available in Google AI Studio through the Gemini API.