r/androiddev • u/Hubastard • 7h ago
Speedrun, an open-source mobile E2E test runner that runs your Maestro flows up to 8x faster
I've always been bothered by how slow mobile UI tests are. Most of a test run is spent waiting: fixed sleeps, polling, comparing screenshots to guess whether the app is done animating. Coming from a game dev background, where you fight for every millisecond of a frame, that felt like a solvable problem. So I spent the last while building my own runner from the ground up, and it turns out UI tests can be a lot faster.
Speedrun is a black-box E2E test runner for Android and iOS. It's open source (Apache 2.0) and it runs Maestro's YAML flow format so if you already have Maestro tests, trying it is one word:
maestro test flows/ # before
speedrun test flows/ # after
Numbers
One user journey (sign in with a code, search a shop, fill a cart, set a PIN under a loading overlay, scroll a long list), written for each tool the way its tests are usually written. Median of 3 runs after a warm-up, start-up included:
| Tool | Android emulator | iOS simulator |
|---|---|---|
| Speedrun | 20.4 s | 27.2 s |
| maestro-runner | 65.9 s | 49.6 s |
| Appium | 59.4 s | 94.1 s |
| XCUITest | - | 81.2 s |
| Maestro 2.11 | 126.5 s | 113.1 s |
On the native Settings app benchmark it's 8x faster than Maestro on Android and 2.8x on iOS. On a 19-flow React-Native test suite it passes all 19 on both platforms, and is 6.1x faster on Android and 2.6x on iOS. Android is where it shines most, which is nice for CI: Android runs on cheap Linux runners.
Full disclosure: the main benchmark app is mine, built in React Native to be hard to test (endless animations, late popups, loading overlays, lists that load as you scroll). The flows, per-flow results and scripts to rerun everything are in the repo, and I'd love to see numbers on other apps.
Why it's fast
- It waits for the app, not the clock. A small on-device agent knows when the app has actually settled (accessibility events on Android, XCTest's idle state checked against the screen on iOS), so each step starts the moment the app is ready. No sleeps, no screenshot diffing.
- Reading the screen is cheap. The agent reads the UI tree in about 10 ms on Android and talks to the host over a compact binary protocol.
- Screens that never stop moving don't block it. Spinners, tickers and carousels are left out of the "is it done?" check.
- Nothing to warm up. One native binary, no JVM. The iOS agent stays running between runs, so later runs start in about 0.2 s.
It's also less flaky: it won't tap a popup that's still sliding in or a button under a loading overlay, and it scrolls an element out from under a sticky footer or the keyboard before tapping it.
It works with AI agents too
speedrun mcp exposes it as an MCP server (Claude Code, Cursor, etc.). Your agent can explore the app on its own: tap around, read each screen in a few lines (cheap on tokens, not a whole accessibility tree), figure out how things work, then save the steps that worked as a YAML flow you can drop into CI.
claude mcp add speedrun -- ~/.speedrun/speedrun mcp
| Android emulator | iOS simulator | |
|---|---|---|
Speedrun (speedrun session) |
4.2 s, ~1,560 tokens | 9.4 s, ~1,230 tokens |
| agent-device 0.21 | 9.1 s, ~2,240 tokens | 14.0 s, ~1,620 tokens |
| Argent 0.31 | 11.7 s, ~3,630 tokens | 12.2 s, ~2,650 tokens |
Limitations (so you don't have to find them)
- Android emulators and devices, iOS **simulators** only (no real iOS devices yet)
- macOS (Apple silicon); the Linux x86_64 build is Android-only
- Maestro's AI commands (`assertWithAI` and co.) aren't supported; the few other differences are documented
- It's young (v0.1.10) and moving fast
Try it
Repo, docs and benchmark details: https://github.com/hubastard/speedrun
If you have a Maestro suite, I'd really like to hear how it does on yours, especially where it breaks or is slower. Happy to answer questions about how it works.
edit1: added maestro-runner to the benchmark.
edit2: added ai agent benchmark.