Implementing the optimal mobile testing strategy

White paper

Mobile Application Quality

The Mobile Market is Booming
mobile testing continues to grow

The mobile market is booming, the app economy is exploding and the need for mobile testing continues to grow. But mobile testing is very different from traditional enterprise, web and packaged application testing. There are various devices with numerous form factors, fast changing mobile OSs, and growing demand to support mobile applications globally.

mobile quality challenges

Each phase of mobile quality brings its own set of unique challenges to be addressed. For example, in Planning, there needs to be a focus on device selection. In Definition, specific types of tests and processes need to be well defined. In Execution, remote access and conditioning issues for devices need to be covered. With Automation, users have to account for ROI, tools, types of test cases, etc. And in the last phase of mobile quality, Optimization, post release support and upgrades need to be measured and addressed.

mobile testing strategy

To do all this, the overall testing strategy needs to be well defined with the right tools, framework and technology. This white paper discusses the different considerations for mobile testing and lays out a blueprint for creating an optimal mobile testing strategy with risk mitigation plans and streamlined processes.


The challenge is to make your application work and work well across this constantly evolving ecosystem.

This means quality mobile app development needs:

The mobile landscape is changing real time – new devices, new versions of operating systems, new carriers and of course, new applications.

Global testing coverage
Global testing
Cross-platform testing capabilities
testing capabilities
Ongoing support & user experience management
Ongoing support & user
experience management
The mobile boom by the numbers
The mobile boom by the numbers
  • There are 1.6 million apps in Google Play and 1.5 million apps in Apple’s App Store (Source: Statistica, June 2015)
  • 15 billion has been paid to developers by Google and Apple in 12 months (in the 12 months up to June 2015) (Source: Benedict Evans, June 2015)
  • Apps account for 77% of time spent consuming digital media (Source: comScore, 2015 Mobile App Report)
  • By 2020, around three-fifths of the global population will have a mobile subscription (Source: GSMA Global Mobile Economy Report 2015)
  • By 2020, there will be 4.6 billion unique mobile subscribers (Source: GSMA Global Mobile Economy Report 2015)
  • Mobile app revenues will reach 76.52 billion by 2017 (Source: Statistica, June 2015)


Mobile testing is different from traditional enterprise QA. With a zillion device / OS combinations challenging testing coverage, security issues, and varied device connection speeds impacting application performance, mobile testing quickly becomes the largest elephant in the room.

Apart from adding device provisioning and device level validations, there are issues related to memory, network bandwidth, cross platform, screen size and resolution compatibility. There are functional issues related to swipes, UI and localization. There are security concerns such as caching, uploading personal information to servers, virus scanning, memory corruption and more. All this makes mobile QA a very different animal.

mobile application quality assurance phases

Mobile Testing Challenges – At a Glance

Industry challenges
Industry challenges
  • BYOD Mobile Device Management
  • Enterprise Level Security
  • Testing on the Cloud
  • Consumer and Enterprise Mobility
  • Native vs. Hybrid vs. mWeb vs. HTML5
  • Mobile Test Automation, Security Tools
Network Service Providers
Network Service Providers
  • Network Delay and Jitter
  • Network Disruption Testing
  • Network Performance Testing
  • Network Congestion Testing
  • Multi-network/Trace Testing
  • Enterprise Carrier Network Switching
Networking Connectivity/Varying Speeds
Networking Connectivity/
Varying Speeds
  • 2G, 2.73G (GPRS, EDGE), 3G, 4G
  • WiFi (Secured, Unsecured)
  • WiMax
  • VPN, Hotspot, Tethering
  • Cellular Internet, Telecom Carriers
  • Data Loss Over Network
  • Bandwidth, Signal Loss
Device Diversity
Device Diversity
  • Device Makers, Models
  • Smart Phones, Phablets, Tablets
  • Device Density (ldpi, mdpi, hdpi, xhdpi)
  • Device Screen Sizes (Small, Normal, Larger and Extra Large Screens)
  • Device Orientation Challenges
  • Device Browsers, Rendering Differences
Device OS Fragmentation
Device OS Fragmentation
  • iOS (3, 3GS, 4, 4GS, 5.x, 6.x)
  • Android (2.x, 3.x, 4.x)
  • Blackberry (4.x, 5.x, 6.x, 7.x)
  • Windows Phone (2003, 5.x, 6.x, 7.x)
  • Symbian, BREW
  • J2ME, BADA, others
Device Hardware
Device Hardware
  • Limited Processing Capability / Power
  • Locations and Languages
  • Limited Memory Size
  • External Memory SD Card Availability
  • Device Comm Protocols (WAP, HTTP)
  • Device Configuration
Device Data Input Methods
Device Data Input Methods
  • Touch Screen Keypad
  • Non Touch, Soft/Physical Keypad
  • Track Balls, Qwerty Keypad
  • Voice (SIRI, Skyvi)
  • Split Keypad
  • Copy and Paste
  • Blink of an Eye
Device Capability & App
Device Capability & App
  • Functionality Bandwidth Connectivity
  • Camera
  • Phone / SMS / MMS
  • Contact List
  • Rapid Rollouts, Deployment Velocity
  • Mobile Context, Usability, Performance
  • Energy Consumption Monitoring
  • Testing with Emulators / Simulators
  • Device Interruption Scenarios
  • Field Testing, Test Execution Challenges

Mobile Demands a Comprehensive Test Strategy

Interaction with Device/OS native capabilities

Interaction with Device/OS native capabilities

Optimal combination of devices and OSs

Optimal combination of devices and OSs

Effective combination of manual and automated testing

Effective combination of manual and automated testing

Need for faster agile adoption

Need for faster agile adoption

On-time and off-line connectivity

On-time and off-line connectivity

best QA strategy to test a mobile app
Mobile Native Apps vs. Mobile Web vs. Hybrid

The best QA strategy to test a mobile app depends on the type of app. If it is a native app, then there are certain conditions which need to be tested.

The tester needs to ensure that the app can be successfully downloaded, executes on the device and interacts with the backend infrastructure. Apps render differently across devices, so each app needs to be thoroughly tested on all relevant devices. We must also ensure backward compatibility with selected older generation devices and OSs. Creating test plans for manual testing on each of these devices can be a cumbersome process.

Mobile web apps pose a different set of challenges. These apps are viewed by users across the globe, presenting unique challenges in terms of networks, scripting and devices. No matter what type of app it is – native, hybrid or web – manual testing of these apps has become more rampant. Testing on simulators does not accurately replicate real user experience. And, testing on all device/OS combination is not practical .The other issue that we have already touched on is the lack of a standardized mobile SDLC, which results in an inconsistent testing methodology.

The right testing approach for your enterprise

Selecting the right mobile testing methodology is critical to the success of any mobile initiative. You need a partner who understands the intricate details and brings the required knowledge as well as the right infrastructure and enabling technology.

mobile testing methodology
testing approach for enterprise applications
key quality dimensions to mobile testing

It’s imperative to link business goals and quality goals, leveraging key quality dimensions specific to mobile testing.

This means:

  • Understanding the quality dimensions (see graphic)
  • ligning quality dimensions with a tactical testing plan
  • Selecting the right service partner to deliver these dimensions
Key Activities & Outputs from Mobile Testing
Test strategy
Test strategy

Define key policies, components, review criteria, approach, testing types, scope, communication and resource planning necessary to control and manage risks throughout the Software Testing Program.

Test plan
Test plan

Address key areas such as test approach, scope, test types, review criteria, test data, incident management, resources and supplementary materials.

Test case/specification
Test case/specification

Identify and outline key objectives of the test validation, related requirements and testable items. All required test pre-conditions and processing in detail with associated supplementary materials.

Test summary report
Test summary report

Provide dynamic snapshot of progress throughout the testing life cycle, providing visibility to management and stakeholders. The overall summary, scope deviation, requirements coverage, analysis/design progress, execution progress, review criteria and outstanding defects are areas that will be addressed throughout the life of the project for review, approval and sign-off.

Test automation

Test automation

Define repeatable process and help you streamline your testing to support on-demand releases and critical software changes. A hybrid automation framework is deployed for immediate use, eliminating development ramp-up time.

Traceability matrix
Traceability matrix

Provide a relational model of your software testing validation framework. Requirements and User Stories are mapped to Test Cases and Test Plan and are executed against target software build releases, configurations and environments.

Test results
Test results

Each of the test result is well supported with tester comments and screenshots along with the required logs for the developer to understand the issue, and be able to reproduce the same without any trouble.

Test assessment
Test assessment

Provide an overall review of the efficiency and effectiveness of testing activities. Assessment methodology uses interviews with key testing leaders and an analysis of your testing deliverables in a short time frame to provide recommendations.

Different testing types: what you need to know

There are different types of testing that need to be addressed – independent of the platform – to ensure mobile app quality. Following is a snapshot of the various testing types and test cases.

Sign-up & Login
Make sure the app is easily accessible to the users.
Menu Options
Ensure menu items like Help, About, etc. are easy to find and navigate.
Make sure key functionality is clear and consistent. Assure the app will run with a keyboard and/or touch-screen.
Data Handling
Managing data handling, data deletion and unintended data deletion (important for games in particular) in the testing scope.
Connection Speed / Carrier
Incorporate most popular and likely carriers in most basic app tests to ensure performance.
Operating System
Ensure app performance on different mobile OSs.
Screen Size
Testing for screen size discrepancies.
Test mobile app behavior against device status variations such as battery strength (full, medium, low), incoming calls, SMS, MMS and video calls, etc.
Error Messages
Ensure error messages are clear, concise and actionable.
Usability TESTING
Check for slow and unreliable wireless network connections with low bandwidth taking into account various network conditions.
Small Screen Size
Check for physical constraints of mobile devices, especially small screen sizes which might impact app usability.
Different Display Resolution
Check for low resolution that can degrade the quality of the multimedia information.
Test Case
Page Layout
Evaluate the overall effectiveness of the page layout.
Color Schemas
Evaluate use of colors in the background, text, links, icons and buttons.
Evaluate visibility and accessibility of key items like Help, About, Instructions, search.
Evaluate app consistency in terms of text, message and symbols in all supported languages.
Test for different touchscreen scenarios: multi-touch vs. single touch; long touch vs. short touch; button size, workflow and any other supported input methods.
Trackballs, Track Wheels and Touchpads
Confirm user ability to move from object to object regardless of navigation tool.
Soft Keyboards
Check user interaction with, and accessibility of, the soft keyboard.
Hard Keys
Check the available device hard keys; e.g., Start, Home, Menu, Back – interact with the application in similar way.
Short Cuts
Check for expected shortcuts for the device and test their use within the application.
localization TESTING
Test Case
Check user interface, content files, keyboard, text filters, hot keys, spelling rules, sorting rules, upper and lower case conversion rules.
Format Testing
Check for date formats, currencies per country.
Local Market Compliance
Check app compliance with local laws and regulations.
Content Testing
Check static and dynamic content like video, catalogs, search results, metadata.
Check different languages and character sets.
Postal Codes
Verify if the target countries support only number or alphanumeric.
Language Direction
Verify the direction for the way language is written – left to right or right to left.
Currency Conversion
Test currency conversion for Internet retail applications.
Tax Calculation Verification
Test rules around the calculation of sales tax and VAT.
Translation Validation
Test for on-demand translation validation and accuracy in including accessibility from any market location.
Web Applications
Test with common web security and testing tools such as Firefox web developer. Ensure that web proxy is used to intercept all mobile device traffic to monitor data and test for security issues.
Format Testing
Test for password/usernames being left unencrypted on device. Ensure footprint and fingerprint analysis can be used to make hash files.
Local Market Compliance
Research permissions configuration on the device; test to ensure that app can get access to the device areas it needs in order to function properly, and it provides useful error messages directing the user to set the permissions properly, if it cannot. Verify that the app can be successfully installed at all expected device security level settings. After installation, ensure it is possible to change the security settings of device or application permissions. Include some test cases to cover these scenarios as well.
Test Infrastructure Management & Reporting
Ensure there is enough research on app to decide upon the test infrastructure on live or simulated loads. Feed the mobile app under test with the largest tasks it can handle. Make sure to increase the load on the system via automated tools to simulate real time scenario with virtual users. Set up report management using internal and external bug reporting and test management tools.
Interruption TESTING
Incoming Calls and SMS, Battery & Cable Removals
Must be tested for: Incoming and Outgoing SMS and MMS Incoming and Outgoing Calls
Battery Removal Cable Insertion and Removal Network Outage and Recovery
Media Player On/Off Device Power Cycle
Test automation

Because mobile testing must be performed on so many different combinations of devices, browsers, and operating systems, doing it manually quickly becomes impractical in terms of both time and expense. Test automation can bring significant efficiencies to the process – dramatically accelerating go to market plans and reducing costs.

Why Automate?
The device os maze
The device
os maze

The virtually endless combination of devices, OS models, and device sizes. Test automation enables QA teams to test the same functionality on different devices, OSs and form factors.

Changing industry trends
industry trends

Constant OS updates, upgrades, new devices, and new manufacturers. In order to ensure that the application is up-to-date and supported by new trends, testing it in a timely manner becomes important.

Rapid upgrades to apps
Rapid upgrades
to apps

Apps are regularly updated with new features in order to keep up with market demand. In order to introduce upgrades into the market quickly, automated testing is crucial.

The apexon approach to automation

It’s imperative to link business goals and quality goals, leveraging key quality dimensions specific to mobile testing.

This means:

Apexon approach to mobile test automation

Picking the right mobile qa partner

Mobile testing can be complex but app quality is critical to success in today’s business world. The traditional rules of testing do not apply to mobile due to the fast changing landscape, unique features of apps and devices, limited bandwidth and unreliability of wireless networks.

There is a need to develop mobile specific test methodologies and guidelines and therefore a need to rely on a testing partner. This partner should be able to provide insights to modern approaches in testing and suggest the right QA strategy and coverage based on specific business needs.

After defining the qa strategy, your QA partner should provide a well-defined plan on what types of testing and what attributes need to be tested, validated and verified. Ideally, they would have executed testing strategies for enterprises across different verticals – retail, travel, banking and insurance, and digital entertainment – so they understand all the different possibilities and use case scenarios.

What to Look For When Selecting a Vendor
real device testing

The ability to use the right combination of emulator and real device testing to ensure predictability; emulators with scripting help to test many aspects at an effective cost

application testing on multiple devices

The ability to provide intelligence behind use of select devices for your application

automation and agile solutions

The ability to provide automation and agile solutions for faster time-to-market by building the right user stories, increasing use of test sprints and creating script reusability

access real devices remotely

The ability to have access to the real devices remotely when it’s not possible to run physical test on a slew of mobile devices

increase the app store star ratings

The ability to increase the app store star ratings of the app by closely monitoring it and providing reactive and proactive remediation

adoption and monetization of the application

The ability to optimize the adoption and monetization of the application post-launch