Artificial Analysis has launched the first benchmark index specifically for "Decision Models," evaluating their ability to output structured options with probabilities in a single call. This index defines a new category distinct from generative LLMs, focusing on five dimensions: classification, tool calling, knowledge retrieval, hallucination suppression, and reasoning.
Cross-verified data indicates that specialized models like Jev offer significant speed and cost advantages. While general-purpose models like Claude Haiku 5.5 achieve slightly higher accuracy, they incur approximately 4.4 times the cost and higher latency. Key test sets include AA-Omniscience-Decision, HLE-MultiChoice, and When2Call, covering scenarios from basic classification to complex logical judgment.
This release marks a deepening of AI evaluation into niche areas, filling the gap for standardized assessment of structured probabilistic outputs. It enables developers to precisely select cost-effective underlying models for Agents and automation workflows, shifting industry focus from parameter scale to efficiency and cost balance in specific tasks.