IDC predicts that the volume of smartphones is estimated to reach over 500 million in shipments by 2014. These smartphones will run millions of applications embedded in them. A lot of these applications will communicate with complex ERP and CRM systems which will require rigorous security testing and verification. And, all these applications will be used on many OS’s which then will need a good look at their functionality and integration capabilities. Unlike the PC based environment, the mobile world comprises of a zillion devices with diverse hardware and software configurations presenting a unique challenge in QA and designing testing strategies.
Testing is and will be quintessential for mobile applications and testing spend will exponentially grow along with the volume of smartphones — almost to about 20billion by 2015 per IDC. The first question that then comes to mind — do we need to test an application on all these devices or can we use emulators/simulators to simulate the real device experience. Diversity in this environment with multiple device types, browsers, hardware configurations and network related challenges might make actual device testing a little difficult, but some key attributes of applications have to be tested on real devices only. Emulators can only be used to test some device independent features.
Which route should you go? Well, it depends — on the type of application, network and firewall configurations, on the QA strategy, current and immediate needs, cost and resource availability. But let’s take a look at the comparison matrix to analyze the emulator vs. the real device experience to arrive at a more definitive conclusion
|
Emulators |
Real devices |
| The application can be installed through apk file backup which is a time consuming process and it requires a real device | The application can be directly installed in the device. |
| Real time applications (i.e. GPS based, server based, motion, multitouch or tilt control based) cannot be tested on emulators | All types of application can be very accurately tested on a real device. |
| Emulator is slower than a real device. So, the app performance cannot be measured accurately and slower speed of emulators effects productivity | A real device is always more accurate than an emulator |
| Communication with the emulated device may be blocked by a firewall program running on computer because the emulator can act like a normal application on workstation. | There are no firewall restrictions on a real devices |
| There is no support for USB connections, SD card insert/eject, battery charge level, device-attached headphones etc. | Real devices supports all of this without any restriction |
To firmly establish this premise, we conducted an experiment by testing top 75 apps in the market on emulators as well as real devices. Here’s what we found, emulators are suitable for initial stages or only to test some device independent features. But to ensure the release of a zero defect application, real devices are important and a very critical part of the testing cycle.