Answer
No, an emulator alone is not enough before a release: keep the automated suite on emulators, and run the main flows on one or two real phones before each store release.
Verdict on the code under testYes
Why
- Blast radiususers
- Change frequencyregularly
- Detectabilityeventually
- Reversibilitycostly
- Test costheavy
No, an emulator alone is not enough: keep the suite on emulators, and run the main flows on one or two real phones before each release. The typical case is an Android and iOS app whose UI tests run on emulators in CI. Blast radius is users, and Change frequency is regularly, with a store release about once a month. Detectability is eventually, because a failure that only real hardware shows, such as an upload that a maker's battery saver stops, raises no crash. Reversibility is costly, since a fix needs a new store release, and Test cost is heavy, because real phones must be bought, updated and kept connected, so rule R12 gives Test minimally.
| When | Decision | Why |
|---|---|---|
| The instrumented tests already run in a device cloud with physical phones, such as Firebase Test Lab | Test: run the main flows on three physical models for every release build | Test cost falls to moderate, because the existing tests run unchanged on phones the cloud maintains |
| The main flow sells a subscription through an in-app purchase | Test mandatory: complete a sandbox purchase on a real phone of each platform before every release | Blast radius rises to money and Reversibility stays costly, because sales lost while the purchase fails do not come back |
| The screens are a web view, so most fixes ship from the server without a store release | Test it differently: alert when errors or finished main flows on one phone model drift from that model's usual rate | Reversibility falls to with-effort, while Detectability stays eventually and Test cost heavy |
| Staff use one company-issued phone model, and device management installs updates without store review | Do not add real-phone test runs; keep the emulator suite and fix the failures staff report on the issued phones | Blast radius falls to internal, Detectability to same-day and Reversibility to with-effort |
| You want each release checked on every Android model your users own | Test it differently: roll the release out in stages, watch crashes per model, and halt the rollout when crashes on one model rise | Test cost rises to prohibitive, because nobody can buy or rent every model for each release |
What breaks if you don't test
An Android emulator runs Google's build of Android on a fast computer, while a real phone runs its maker's build on a slower chip. The iOS Simulator uses the Mac's processor and memory, so Apple's archived Simulator guide calls it an inaccurate test of performance and memory use. A green emulator suite misses a battery saver that stops uploads, and users find it first.
What you lose if you over-test
A phone attached to a CI machine shows system dialogs, installs updates overnight and drops its USB connection, and each of those turns a run red with no bug behind it. Running every pull request on such phones teaches the team to rerun red builds, including the one that found a real fault.
How to test
- Keep the UI and instrumented tests on emulators in CI, with Gradle managed devices on Android and the Simulator in Xcode.
- Pick one or two phones from your analytics: the model most users own, and the slowest model you support. On Android, take one from a maker other than Google.
- Before each release, run the main flows on those phones, plus anything that uses the camera, Bluetooth or background work with the screen locked. The same instrumented tests run on a phone connected by USB or Wi-Fi, and Xcode runs them on an iPhone.
- In CI, force Doze on the emulator; the real phone covers the maker's own power rules.
Procedure and references
When the answer changes
- The main flow takes money.
- Fixes ship from a server, as with a web view.
- Staff use one company phone model.
- Your tests already run in a device cloud.
Real incident
Photos that waited for the courier
On a delivery app I worked on, couriers photographed each delivery, and WorkManager uploaded the photo in the background. The suite passed on Pixel emulators, including under forced Doze. Then the courier company bought Xiaomi phones. MIUI stops background work of apps without its autostart permission, as Don't kill my app documents, so photos waited until the courier next opened the app. Nothing crashed. Eleven days later a dispatcher asked why proof of delivery from one depot arrived hours late. The fix ran the upload as a foreground worker and needed a store release. Since then each release runs the upload test on a fleet-model phone with the screen locked.
Sources
- stackoverflow.com/q/2979718
- developer.android.com/studio/run/device
- developer.apple.com/library/archive/documentation/IDEs/Conceptual/iOS_Simulator_Guide/TestingontheiOSSimulator/TestingontheiOSSimulator.html
- dontkillmyapp.com/xiaomi
- developer.android.com/develop/background-work/background-tasks/persistent/how-to/long-running
- support.google.com/googleplay/android-developer/answer/6346149
Related questions
FAQ
- Emulator or real device for testing?
Use both: emulators for the automated suite on every change, and one or two real phones for the main flows before each release. Google's Android Studio guide says: "Always test your Android app on a real device before releasing it to users."
- Is the iOS Simulator enough to test an iPhone app?
The iOS Simulator is not enough before a release, because it runs on the Mac's processor and memory and has no camera. Run the main flows on the oldest iPhone you support before each release.
- Do I need a device farm for mobile testing?
A typical app does not need a device farm, because one or two real phones cover the main flows before each release. A device cloud such as Firebase Test Lab pays off once instrumented tests exist and you want them on several physical models for each release.