Verdict
Yes
Test minimally on low-end devices: before each release, run one automated test of the main flow on one real Android phone with 3 GB of RAM or less, and stop the release when the app crashes, stops responding, or starts slower than a set limit.
Why
- Blast radiususers
- Change frequencyregularly
- Detectabilityeventually
- Reversibilitycostly
- Test costheavy
Test minimally on low-end devices: one automated check of the main flow on one cheap phone before each release. My typical case is a consumer Android app, released monthly, that the team builds on recent phones. Blast radius is users, who get slow starts and frozen screens. Change frequency is regularly, since each release can add a heavier screen or library. Detectability is eventually: a slow start on a phone nobody on the team owns raises no alert, and users uninstall without writing to support. Reversibility is costly, because the fix needs a new store release. Test cost is heavy, because the test needs a real device and startup times vary between runs, so rule R12 gives Test minimally.
| When | Decision | Why |
|---|---|---|
| The main flows already have an instrumented suite that CI can run on one low-end physical phone | Test: run the whole suite on that phone for every release build | Test cost falls to moderate: one more device reuses the existing tests, and a crash or an ANR fails the run the same way each time |
| Buyers pay in the app, and a checkout screen that freezes on a 2 GB phone loses the order | Test mandatory: run the checkout test on a low-end phone before every release | Blast radius rises to money: a sale lost to a frozen screen does not come back |
| A remote flag can switch the heavy feature off for low-memory phones without a store release | Test it differently: alert on ANR rate and slow starts per device model, and switch the flag off where they rise | Reversibility falls to with-effort: the remote flag switches the feature off within minutes |
| The product is a website that visitors open in phone browsers | Test it differently: record load time and input delay from real visitors by device memory, and alert when low-memory visits slow down | Reversibility falls to trivial: a redeploy fixes the page for every visitor, and a slow page stores nothing |
| A staff app runs only on the one phone model the company issues | Do not test on other low-end devices: try each build on the issued model | Blast radius falls to internal, Detectability to same-day and Reversibility to with-effort: staff report a frozen screen that day, and device management installs the fix without store review |
What breaks if you don't test
On a 2 GB phone, Android kills background apps early. A user fills in a long form, taps Add photo, and the camera pushes your app out of memory, so the form comes back empty. The team's 8 GB phones never show it. Freezes also cost store reach: above an 8% user-perceived ANR rate on one phone model over 28 days, Play may reduce the app's visibility and warn users on the listing.
What you lose if you over-test
Cheap Android phones come from many makers with different chips. Firebase Test Lab charges $5 per physical device hour beyond 30 free minutes a day, so a 15-minute suite on ten phones for 20 pull requests a day costs about $250 a day. Strict time limits fail at random on slow phones, and the team reruns until green.
How to test
- Pick one phone: in Android vitals by device model, take the most used model among those with the least RAM.
- Keep it on a CI runner, or book it in Firebase Test Lab, whose free plan allows 5 physical device runs a day.
- On each release build, run the main-flow instrumented test there; a crash or an ANR fails it.
- Measure cold start with
am start -S -Wfrom adb, take the median of several runs, and fail above a limit well below the 5 seconds that Android vitals counts as an excessive cold start. - Roll out to a small share of users first and read vitals per device model.
When the answer changes
- Money moves in the flow that can freeze.
- The product is a website, so a redeploy fixes every visitor.
- The app runs only on phones the company issues.
Real incident + Code example
The video preload that froze cheap phones
On a recipe app I worked on, one release preloaded three step videos when the recipe screen opened. On the team's 8 GB phones the screen opened in under a second; on 2 GB phones it froze into the "App isn't responding" dialog. A week later Android vitals flagged the two most common cheap models, and one-star reviews said "freezes on every recipe". The fix, loading a video on tap, needed a new store release. We then plugged a used 2 GB phone into CI and ran this on every release build:
#!/usr/bin/env bash
# Median cold start on the low-end phone; fails the release above the limit.
set -euo pipefail
ACTIVITY=com.example.recipes/.MainActivity
LIMIT_MS=2500 # our median on this phone before the video release, plus a margin
times=()
for i in $(seq 9); do
# -S force-stops the app first, so every run is a cold start
ms=$(adb shell am start -S -W -n "$ACTIVITY" | tr -d '\r' | awk '/TotalTime/ {print $2}')
times+=("$ms")
sleep 3
done
median=$(printf '%s\n' "${times[@]}" | sort -n | sed -n 5p)
echo "median cold start: ${median} ms, limit ${LIMIT_MS} ms"
if [ "$median" -gt "$LIMIT_MS" ]; then
echo "cold start too slow on the low-end phone"
exit 1
fi
Sources
- developer.android.com/topic/performance/vitals
- developer.android.com/topic/performance/vitals/launch-time
- developer.android.com/tools/adb
- firebase.google.com/docs/test-lab/usage-quotas-pricing
- developer.android.com/guide/topics/androidgo
- support.google.com/googleplay/android-developer/answer/9844486
- developer.mozilla.org/en-US/docs/Web/API/Navigator/deviceMemory
Related questions
- Is testing on an emulator enough?Code under test: Yes
- Should I test my app on a slow network?Yes
- Should I write performance tests?Test it differently
- Should I test every screen size?Test it differently
- Should I test in every browser?Test it differently
FAQ
- Which low-end Android phone should I test my app on?
Test on the model with the most of your users among those with the least RAM, which Android vitals shows by device model. With no users yet, buy a current Android Go phone: Android 13 Go edition runs on 2 GB of RAM.
- Can the Android emulator simulate a low-end device?
The Android emulator can set a small RAM size, which shows memory crashes, but it runs on your laptop's processor and hides slow starts. Measure startup and freezes on a real low-end phone.
- Should I test my website on low-end phones?
Watch a website on low-end phones instead of testing on them: record load time and input delay from real visitors, and alert when low-memory visits slow down. Safari and Firefox do not report
navigator.deviceMemory, so the grouping covers Chromium browsers only.