Google Releases Android Bench 2.0 with Long-Horizon Tasks and Continuous Scoring

Android Bench 2.0 is an upgraded AI agent benchmarking framework from Google, designed for Android development scenarios. It focuses on evaluating long-horizon, multi-step real-world engineering tasks.

The update introduces Long-Horizon Tasks (LHTs), covering complex engineering practices that take days to complete, such as dependency upgrades, adding new features, building apps from scratch, and cross-platform migrations. It adopts an agent-based evaluation mechanism, integrating native agents from model providers for closed-loop testing. Furthermore, it replaces binary pass/fail judgments with continuous scoring to improve the precision of identifying partial successes.

This content is suitable for Android engineers, AI agent developers, and MLOps architects.

Google has released Android Bench 2.0——a benchmarking framework used to evaluate AI models and agents in Android development tasks. This major update introduces Long-Horizon Tasks (LHTs), agent-based evaluation, and a continuous scoring mechanism to better assess performance in complex, multi-step development workflows.

Android Bench was launched a few months ago to evaluate AI models through a series of common development tasks, incorporating Android best practices related to permissions, navigation, and network connectivity. With this latest release, Google has expanded the benchmark's coverage to include a broader range of development tasks.

Today, we are releasing the first batch of Long-Horizon Tasks (LHTs). These tasks are highly complex, requiring engineers days or even a week to complete. Additionally, we have introduced an agent-based evaluation mechanism, initially utilizing agents from respective model providers.

While the initial version primarily focused on incremental modifications to existing codebases, Android Bench 2.0 introduces Long-Horizon Tasks (LHTs)—which Google describes as work that engineers might need "days or even a week" to finish. These tasks include upgrading dependencies, adding new features, building applications from scratch, or converting cross-platform apps into Android apps.

A significant change in version 2.0 is the shift in evaluation methodology from a binary "pass or fail" system to a more nuanced scoring system. Previously, a complex task could be marked as failed due to a single edge-case assertion failure, even if the agent had successfully met dozens of other requirements.

We calculate completion rates by considering factors such as functionality, visual fidelity, and regression avoidance. Additionally, we apply objective penalties for deviations from evaluation instructions or structural constraints.

The results provided by Android Bench 2.0 also help identify which tasks are more likely to succeed with AI assistance. For instance, Google reports that "AI performs better at writing new code than refactoring existing code," which may reflect that refactoring and migration require a deep understanding of architectural complexity within the codebase.

Similarly, even in large-scale codebases, AI excels at several "mature deterministic transformation" tasks. Examples include converting Java to Kotlin, replacing Retrofit with Ktor, or introducing a ViewModel layer.

Conversely, large models still face difficulties in many scenarios, including tasks requiring runtime validation (such as missing dependency injection graphs), those involving breaking framework changes, or cases where knowledge gaps arise due to unreleased libraries. Notably, when porting cross-platform apps to Android, even top-tier models achieve only an 80% completion rate, indicating that this remains largely an "unsolved problem."

The updated Android Bench 2.0 dashboard includes the latest models, such as Gemini 3.8 Flash, Gemini 3.7 Flash, OpenAI GPT-6, Anthropic Fable 5.1, Kimi K3, and Qwen 3.8 Max. As of this publication, Claude Opus 5.5 leads the leaderboard with a 32% LHT pass rate, followed by GPT 6 Astra with a 28% pass rate.

Original article: https://www.infoq.com/news/2026/10/android-bench-2/