The July launch of Android Bench — its LLM benchmark leaderboard for Android growth duties — is now primarily based upon the Harbor framework for evaluating fashions, Google has introduced.
In its blog post asserting the updates, the corporate famous that when Android Bench was launched in March, the objective was to supply insights into mannequin capabilities, supply AI choices and to induce enhancing the fashions. On this launch, eight fashions have been added to the leaderboard — Claude Fable 5, Sonnet 5 and Opus 4.8; GLM 5.2; Kimi K2.7; MiniMax M3; and Qwen 3.7 Plus and three.7 Max.
“From the start, we’ve valued an open and clear strategy, which is why we made our authentic methodology and take a look at harness publicly obtainable on GitHub,” the corporate wrote within the weblog. “Now we’re taking collaboration a step additional by supplying you with, the Android developer group, an opportunity to form Android Bench.”
Builders can now design and submit Android growth duties and see how the fashions deal with the situations necessary to them, and now can truly run benchmark evaluations to check the fashions they use in opposition to Android’s dataset of duties. These duties could be discovered on this GitHub repository and builders can submit evaluations or discover Android’s information at Harbor Hub.
“With increasingly choices for agentic growth,” the weblog concluded, “sustaining a cutting-edge benchmark ensures that the AI help you depend on retains getting smarter, extra useful, and more practical.”
