Offline Testing of an Online App - What, why and how!?

Offline testing of an online app!? If an app expects to be online, then, of course, it’ll not work offline! Whatever is there to test offline!?

Like I’d written recently, Testing is much more than merely validating Business Acceptance Criteria. A Tester helps see the app from the end user’s point of view and provides feedback to improve the quality and helps polish the user experience.

Having a great experience online and offline is one of the key points of engaging users in-app.

So, the user is on a holiday or as users keep roaming around having different cellular ranges, trying to get something done with the help of the app. And the spinner is spinning endlessly. Maybe it’s a poor network. The app is making the user wait for longer times. What happens next… The user keeps trying or waiting but in vain. Fed up, the user closes the app or uninstalls it, switches to another similar app or the website where they can get their work done with their network connectivity constraints. And this results in 1-star reviews on Stores. Also, when online, the user can check on the website as well and decide why to waste the space for the app! Such poor experiences often result in the permanent loss of a potential customer!

If it happens to be an Enterprise app (from a third party or developed in-house), there may be other frustrations and internal dissatisfaction culture building up which could impact their actual effectiveness and productivity.

Unlike the website, the app should work offline (cached data, maybe) and online both. There is no added value of having an App if you already happen to have a fully functional website and your app does not support offline display, saving of data etc. You might as well make your website mobile-responsive and with offline-mode capabilities and continue with the business.

There are two ways when the user is offline.

1. When a user does it knowingly e.g. Aeroplane mode or switching internet off, or not allowing on either Mobile data or WiFi.

2. When it happens to a user unknowingly. For e.g.: forgot to exit Airplane mode, server-side API taking longer to respond, Backend is down, Server-side slowness, the third party API being offline, mobile OS slowness, app slow-ness, RAM not available.

What being ‘Offline’ while using the app means:

• Poor network - just 1 bar - the spinner spins endlessly. Important - you may very well be connected to the Internet but actually having really poor connectivity! This is one of those non-binary conditions that we don’t consider!
• The transition between ‘No network and network’. For example When on the train/ underground transport/ Tube the mobile OS is trying to look for a network.
• Aeroplane/Airplane mode - the user may have activated this when in flight or when at a meeting and then forgot to switch out of it.
• Connected to Free-WiFi - On Airports, in Hotels.. - You have a connection but only to the local Wifi system and not to the Internet. The Internet can’t be accessed unless you sign up, but you realise this only when you try to surf a website and not when you use the app directly without surfing the website.
• The App’s Server Side or Third Party may not be accessible via a particular network for various reasons (blocked, network congestion, networking mistake)
• The App is trying to fetch the data and spinning up - User wants to switch to another app while this one loads the data. When the user wants to return to this app, the user feels it is unresponsive and has to kill the app.
• Switching between mobile data and WiFi. What happens when the app transitions between different bandwidths and public IP addresses?
• Apps having no access to use the network when on Mobile Data - (For example, I usually don’t allow downloads, updates, Prime/Netflix streaming on Mobile Data).

Making the User wait is a bad idea! We should not expect a network connection to be always fast and stable.

So, always test your App offline!

Some suggestions on what to check when offline and how to make the app Fail gracefully:

• Ensure that the app doesn’t crash upon going offline or when the user performs any action on the app while being offline. This may seem obvious, but we are habituated to testing an app when online. As Testers, we’re supposed to explore the unknown!
• An unexpected exit may seem like a crash to the user if the app closes itself when it detects no network.
• Similarly, check if the app happens to close itself when it detects that it’s offline whether at startup or during use. Depending upon how the app has been coded or the libraries that it uses, such an unexpected exit is possible.
• Suppose a user is using the app offline, and guess what, the dev caught the ‘Network Connection Exception’ but happened to ignore it and moved to the next screen. This leads to a business logic bypass!
• Keep an eye out for all the workflows in the App which matter to the User. Assess what happens or how those processes behave when the app goes offline.
• The User performs an action, and the app is waiting for a response and then gets disconnected. What happens to the transaction?
• Consider disabling certain controls in the absence of network connectivity and indicate poor/zero network connectivity. This may reduce frustration and error-reports.
• Does the app show a visual progress indicator when “pull to refresh” is in progress?
• If your app is network dependent, - then Fail nicely when offline. Meaning - show the right Error or “Retrying” messages rather than just showing a spinner.
• Check that nothing unexpected happens when the user is trying a ‘pull to refresh’ or navigate around the app while offline. E.g. bad UI, overlapping content, half displayed sections, showing wrong data to user from hard-coded default values or from a stale cache, or a test URL (for example since Prod is not accessible during testing).
• When Offline, when the user is trying a pull to refresh- Does the app wipe out all data on the screen and return a sorry message, thereby making the user fill in the form all over again? Or do you prefer showing data from the local cache (if any!)? What are the repercussions of such a decision?
• Consider displaying data quickly (from local storage) while the app displays the “Internet not available snack bar”, so that the user still has access to the cached data. Ensure that the user has something to look at always, Offline or online!
• While you display cached data, do show a small snack bar to the user stating Network not available/check your internet connection, etc.
• Show the right message to the user while data loading happens or when there is poor network /no network or when data loading fails - instead of a blank page or just the “Sorry” message.
• Consider asking the devs to use Shorter timeouts to detect loss of Network Connectivity. This will help prevent uncertainty.
• How are network retries decided? To spread out the server-side load while the app keeps on retrying in case of poor network, the app should consider using exponential timeouts/backoffs so that the connection retries from various users are spread and doesn’t cause a surge in the load on the server-side (thundering herd problem).

Somethings to examine when the App returns Online :

• Does the app continue to work smoothly when network connectivity is restored?
• How correctly is the app syncing the data and getting the updates/updated data? When online, avoid displaying stale data to the user.
• Does the app allow Saving the data/posts offline just in case so that the user can see it and work on it until the network returns? For e.g., Google Docs allows us to mark certain documents for offline availability.
• Should app have an offline mode similar to how Google Maps lets us save and view Maps offline?
• When the network returns, what happened to all the local addition, modification and deletion of the data that you did when offline?
• What happened to the transactions that you initiated when online earlier but which the app didn’t get to show the status of when connectivity was lost?
• Does the user continue to have access to all the offline saved comments, images, notes in the app? Did the app preserve those or wipe all of those out upon a reconnection?
• Does your test plan account for duplicate data when the network connectivity is restored?
• Do the image and the name/caption that you saved while offline match the online version?
• What happens to offline saved data when the user does “pull to refresh”? Does it take away all offline data?
• Also, when the network reconnects on its own, the app should pull new data (like Gmail does on its own) rather than making the user gesture a pull to refresh (Scroll the screen to get the new data). You can have in-app visual (and audio, for those who need it) indicators to inform the user that the app is online /offline.
• What happens when the mobile OS finishes switching between cellular data and Wifi? Does the change in public IP result in traffic/API calls being blocked? Is there a difference in the response times or the data load times? Do the workflows continue or is the traffic from a different IP treated as a security issue by the server-side?
• Do the authenticated sessions persist across online-offline-online states?

When planning your testing, assess the business needs, check the risk to the business of making or not making some features available offline, brainstorm with the team on what / how to respond when offline to avoid the user seeking alternatives.

Offline operations and behaviour isn’t just a nice-to-have feature but instead serves as behind-the-scene support for polished and data-safe user experience. It may not give the app 5-star reviews but it will definitely lessen down the 1-star reviews and unhappy/potential customers.