UI Tests: First Test 104s -> 8.7s

I have wanted to get off of Firebase Test Lab for a while now, and Google finally made the decision for me. Firebase Test Lab is deprecated, test executions are supported until September 30, 2027, and then it's gone.

When I started digging into why our UI test step took so long, I realized the tests themselves weren't really the problem. It was how long they took to start. You push a PR, the UI test step kicks off, and then it sits there for almost two minutes before a single test case runs... only for it to fail in the first second 🤦.

So I started measuring emulator.wtf against FTL with my experimentation framework, and the first results were so lopsided that I didn't believe them. FTL took 99s to start a test at the median, and its last shard didn't start until 249s. emulator.wtf took 7s, and its last shard started at 17s. That was one machine per vendor though, and I wanted to see it at scale before I said anything to anybody.

Long story short, it held up:

Fig. 1 — Every module from every full run of about a hundred modules. Thin lines are individual runs.

That's 42 full runs on emulator.wtf and every one of them had its median between 8.3s and 9.2s. The FTL runs were all over the place. And the module that was slowest to start in each run got going at 12.8s on emulator.wtf, against 158s on FTL.

Two numbers

I learned the hard way on an earlier investigation that if you measure totals you will talk yourself into the wrong answer. Job duration, device minutes, end-to-end build time... all of it hides the thing developers actually feel. So I held myself to two numbers, first and last:

Time to first test (TTFT) = seconds from the vendor accepting a module's test run to that module's first test case starting.

Time to last test (TTLT) = the same thing for the module that is slowest to start in a run. A great median doesn't help much if one shard sits in a queue for two more minutes, because the step isn't done until that shard is.

• FTL "accepted" = the test matrix is created (after upload)
• emulator.wtf "accepted" = ew-cli's "Triggered test run" log line (after upload)
• "Starting" = the first TestRunner: started in the device logcat

Upload time is left out on purpose. That's our network, not the vendor.

A switch, not a migration

I didn't want a big scary migration, so I made the vendor a pipeline variable:

UI_TEST_VENDOR=ftl          # or emulatorwtf

Every vendor has to leave the same things behind: JUnit XMLs, a map from each result to a link in the vendor's UI, coverage files per module, and a non-zero exit if anything failed. Everything after that (failure annotations, coverage reports, archives) doesn't care who ran the tests.

Rolling back is just flipping the variable back. I love this, because it meant I could run both vendors side by side on throwaway branches for days before touching production.

Results

Medians across full runs:

100+ lottery tickets

Here's the thing I didn't expect. emulator.wtf's median was rock solid, but a lot of runs had a tail: an emulator that started noticeably later than the rest. In 19 of the 42 runs, the last module to start took more than 1.5x the median.

When I dug into the worst runs, it was always exactly ONE emulator. The slowest module would be at 36s or 69s while the second slowest was sitting around 12s. It was a different module every single time, so it wasn't anything in our APKs. And the delay came in chunks: five outliers landed at 36-39s, which is about 28s on top of a normal ~9s start, and the 69s one was about two of those chunks.

The folks at emulator.wtf floated that it might be an emulator crashing and restarting, and fixed chunks like that sure look like paying for an extra boot. I haven't confirmed it yet, but either way it made me realize that every shard is a lottery ticket. If any single emulator has a small chance of starting slow, and we run 100+ of them per build, then we are basically buying 100+ tickets for the "slow TTLT" prize on every PR. Say 1 in 180 emulators starts slow. With 100+ shards, at least one of them is slow in almost half of builds, which is pretty much exactly what I saw.

emulator.wtf has talked about running multiple APKs per shard, and I love it because it fixes two things at once:

• Fewer lottery tickets. Put 4 modules on each emulator and 100+ shards become about 25. Same 1-in-180 odds, and now only about 1 in 8 builds gets a slow straggler.
• Way fewer wasted billable minutes. More than 80% of our modules finish their entire run in under a minute, but every one of them gets billed for at least a full minute. A lot of our tests run in milliseconds! Packing them onto fewer emulators means we stop paying for 100+ mostly-empty minutes.

Wait, why isn't the whole step faster?

This one almost got me. When I plotted the entire UI test step for every run, some of the emulator.wtf runs were SLOWER than FTL. That made zero sense until I split each step into phases:

Fig. 2 — Building the test APKs, the vendor run, then coverage reports and results.

The vendor part was about 3.5 minutes shorter on emulator.wtf in every single run. The thing jumping around was the Gradle build of the test APKs, by as much as ten minutes. Most of my emulator.wtf PR samples came from one stack of PRs that bumped the dependency version catalog, which rebuilds pretty much every module. My FTL baseline runs were cut from a fresh main with nothing to rebuild.

Main against main, where the builds do similar work, the whole step went from ~23 min to ~17 min.

A faster vendor can't fix a slow build. And a noisy build will hide a real vendor win if you don't split the phases apart.

What about cost?

The minutes came out almost identical, because both vendors bill test time rounded up per module. At list prices emulator.wtf charges more per minute, so the vendor bill is a bit higher. But we upload less than half the data, and the CI agent gets freed up sooner.

And same as I argued with our CI agents, engineer time costs way more than compute. In my numbers, emulator.wtf comes out cheaper overall as soon as an engineer's time is worth more than about $15/hour. That only counts when someone is actually sitting there waiting, so it's an upper bound... but $15/hour is way below what any engineer costs, so it doesn't take much waiting to tip it.

So much more to say on this topic, but that's the gist of it. The test APK build is now the biggest chunk of the step, so I guess that's next haha.