Termsslug: calibration-error

Calibration Error

校准误差

Metric measuring the gap between predicted confidence and actual accuracy; lower values indicate more reliable probability estim

Curated Intelligence

4 items · Page 1/1
Marktechpost (@Marktechpost)T2·Models
78 pts
Liquid AI Releases d1.2 Decision Models With Multimodal Probability Output

Liquid AI has released d1.2 and d1.2-ultra, decision models that skip token generation entirely: given a state plus typed questions, they return calibrated probabilities for every option in a single call, adding five new answer types including value reading, span finding, box drawing, range estimation and multi-select. d1.2-ultra scores 75 on the Artificial Analysis Decision Index v2.1, two points ahead of rivals at 73, with a 0.8-point calibration error versus 3.7 for Perplexity Decider v1.3. The trade-off: d1.2 scores just 69, trailing eight models, and parameters, latency and weights remain undisclosed.

新智元 (微信公众号)T2·Models
78 pts
Google Launches Gemini Decision 1 with Free Output Tokens

Google has launched the Gemini Decisions API, powered by Gemini Decision 1, its first multimodal decision model built for closed-ended choice tasks. It natively supports Yes/No, multiple-choice with up to 255 options, and rubric-based scoring, handling 1,000 questions per call at $0.10 per million input tokens with output tokens free of charge. The model leads benchmarks with the lowest calibration error, marking a shift from generative output toward structured decision interfaces.

InfoQ 中文T2·Models
88 pts
Google Unveils Gemini Decision 1, a Non-Generative Model Built for Fast Decisions

Google launched Gemini Decision 1 and the accompanying Decisions API, which skip token-by-token generation and instead score a predefined set of candidate answers in a single forward pass, returning the decision, probability and confidence. The model handles yes/no judgments, multiple choice with up to 255 options and rubric-based scoring, processes up to 1,000 questions per request with 150-250 ms end-to-end latency, and is priced at $0.10 per million input tokens with free output. Google claims first place on 7 of 9 benchmarks with up to 92.2% accuracy, and says internal teams already use it in Search and product features.

Marktechpost (@Marktechpost)T2·Models
78 pts
Google Launches Gemini Decisions API for Single-Pass Probability Scoring

Google released the Gemini Decisions API, powered by Gemini Decision 1, shifting the LLM from text generation to structured scoring: a single forward pass scores up to 1,000 yes/no, multiple-choice or rubric questions with calibrated probabilities and confidence values, always emitting zero output tokens. It leads 7 of 9 accuracy tests on Google's own benchmark table, hitting 92.2% on JevBench with a calibration error of 0.0250, roughly half that of TypeSafe Jev 1.14, though it trails on When2Call at 79.9% versus 82.0%. Pricing is $0.10 per 1M input tokens with free output and 150-250 ms end-to-end latency.