The digital market is thriving, with a large amount of development being targeted at mobile devices. But this is a complex environment to navigate, as project teams need to ensure quality of delivery across a multitude of operating systems and an array of mobile devices from multiple device vendors.

With so many mobile applications to choose from, it is easy for a lesser quality application to fall through the cracks. If your app is unable to fulfill users’ expectations, they will quickly switch to an alternative solution.

To mitigate the risk of your application failing and driving customers away, it is important that adequate testing is employed.

Emulators are a good starting point for testing your application and can be particularly helpful during development stages or for testing hard to reproduce scenarios. They are inexpensive and easy to access. But they should never become a substitute for testing on real devices.

Benefits of Testing on Real Devices

There are a lot of benefits of testing mobile applications on real devices. Some of the key benefits are listed below:

• See actual results without needing to worry about “false positives” or “false negatives”.
• Understand the real-world impact of your app on device memory usage and see how it works when competing with other apps for memory.
• Factor in network considerations, as an application may not behave the same when it is connected to a 3G network or a public Wi-Fi.
• Perform interoperability testing in a live network, as information/data transferred by one app over a cellular network to another carrier’s network to a different app may not turn out to be the same as what was actually transferred.
• Confirm actual content rendering results in a live user interface.
• Gain a true user experience and allow for early feedback as the application can be tested for useability and accessibility by different kinds of users who will probably be a subset of the end user of the application.
• Measure performance in real life scenarios, noting the impact of handset or environmental factors.

Managing the Challenges of Testing on Real Devices

Testing on real devices comes with its own set of challenges, which, if properly managed, should be insignificant in comparison with the potential advantages. The two main challenges are:

1. Logistics

What devices should an application be tested on? The Developer will have a view on this, so will the Business, as too will the Tester. Each of these views will likely vary.

A useful way to select your top priority devices is to investigate what mobile devices your target audience is using by referring to your Google Analytics or to websites such as http://gs.statcounter.com/.

You can then create a matrix of devices factoring in operating system, OS version and screen size. As an example, see below graph sourced from Google:

Data collected during a 7-day period ending on January 5, 2015.
Any versions with less than 0.1% distribution are not shown.  

2. Costs

Rather than trying to cover the endless combination of devices, OS and screen sizes that your audience may be using, it is essential that project teams prioritise their testing on the top items in your device matrix to minimise the costs associated with purchasing or renting, not to mention testing effort.

There are also costs for maintaining a cellular operator specific plan. What happens to the current top device 1 year down the line, when it doesn’t form part of your device matrix? For these reasons, purchasing unlocked devices can be a wise choice. Also look for buy-back options or consider re-selling.

If outsourcing your testing, you may be able to leverage the device library of your testing consultancy. At Planit, projects outsourced to our Test Factory benefit from our extensive lab of devices, ensuring quality outcomes.

Conclusion

While emulators are useful aids for mobile application testing, we should not assume that just because an application works well on an emulator that it will work well on a real devices. Emulators should never be used to replace testing on real devices.

Real devices provide far more reliable test results, allowing you to ensure interoperability, gauge performance and check true user experience.

Given the significant investment that companies put in developing and marketing apps they hope will be engaged by their target market, the costs associated with testing on real devices can be easily justified by their many advantages, where quality is key to application success.