Implementing the optimal mobile testing strategy

White paper

Mobile Application Quality

IDENTIFYING AND IMPLEMENTING THE OPTIMAL MOBILE QA STRATEGY

The Mobile Market is Booming

The mobile market is booming, the app economy is exploding and the need for mobile testing continues to grow. But mobile testing is very different from traditional enterprise, web and packaged application testing. There are various devices with numerous form factors, fast changing mobile OSs, and growing demand to support mobile applications globally.

Each phase of mobile quality brings its own set of unique challenges to be addressed. For example, in Planning, there needs to be a focus on device selection. In Definition, specific types of tests and processes need to be well defined. In Execution, remote access and conditioning issues for devices need to be covered. With Automation, users have to account for ROI, tools, types of test cases, etc. And in the last phase of mobile quality, Optimization, post release support and upgrades need to be measured and addressed.

To do all this, the overall testing strategy needs to be well defined with the right tools, framework and technology. This white paper discusses the different considerations for mobile testing and lays out a blueprint for creating an optimal mobile testing strategy with risk mitigation plans and streamlined processes.

AND CONTINUES TO EVOLVE

The challenge is to make your application work and work well across this constantly evolving ecosystem.

This means quality mobile app development needs:

The mobile landscape is changing real time – new devices, new versions of operating systems, new carriers and of course, new applications.

Global testing
coverage

Cross-platform
testing capabilities

Ongoing support & user
experience management

The mobile boom by the numbers

There are 1.6 million apps in Google Play and 1.5 million apps in Apple’s App Store (Source: Statistica, June 2015)
15 billion has been paid to developers by Google and Apple in 12 months (in the 12 months up to June 2015) (Source: Benedict Evans, June 2015)
Apps account for 77% of time spent consuming digital media (Source: comScore, 2015 Mobile App Report)

By 2020, around three-fifths of the global population will have a mobile subscription (Source: GSMA Global Mobile Economy Report 2015)
By 2020, there will be 4.6 billion unique mobile subscribers (Source: GSMA Global Mobile Economy Report 2015)
Mobile app revenues will reach 76.52 billion by 2017 (Source: Statistica, June 2015)

MOBILE TESTING IS DIFFERENT

Mobile testing is different from traditional enterprise QA. With a zillion device / OS combinations challenging testing coverage, security issues, and varied device connection speeds impacting application performance, mobile testing quickly becomes the largest elephant in the room.

Apart from adding device provisioning and device level validations, there are issues related to memory, network bandwidth, cross platform, screen size and resolution compatibility. There are functional issues related to swipes, UI and localization. There are security concerns such as caching, uploading personal information to servers, virus scanning, memory corruption and more. All this makes mobile QA a very different animal.

mobile application quality assurance phases

Mobile Testing Challenges – At a Glance

Industry challenges

BYOD Mobile Device Management
Enterprise Level Security
Testing on the Cloud
Consumer and Enterprise Mobility
Native vs. Hybrid vs. mWeb vs. HTML5
Mobile Test Automation, Security Tools

Network Service Providers

Network Delay and Jitter
Network Disruption Testing
Network Performance Testing
Network Congestion Testing
Multi-network/Trace Testing
Enterprise Carrier Network Switching

Networking Connectivity/
Varying Speeds

2G, 2.73G (GPRS, EDGE), 3G, 4G
WiFi (Secured, Unsecured)
WiMax
VPN, Hotspot, Tethering
Cellular Internet, Telecom Carriers
Data Loss Over Network
Bandwidth, Signal Loss

Device Diversity

Device Makers, Models
Smart Phones, Phablets, Tablets
Device Density (ldpi, mdpi, hdpi, xhdpi)
Device Screen Sizes (Small, Normal, Larger and Extra Large Screens)
Device Orientation Challenges
Device Browsers, Rendering Differences

Device OS Fragmentation

iOS (3, 3GS, 4, 4GS, 5.x, 6.x)
Android (2.x, 3.x, 4.x)
Blackberry (4.x, 5.x, 6.x, 7.x)
Windows Phone (2003, 5.x, 6.x, 7.x)
Symbian, BREW
J2ME, BADA, others

Device Hardware

Limited Processing Capability / Power
Locations and Languages
Limited Memory Size
External Memory SD Card Availability
Device Comm Protocols (WAP, HTTP)
Device Configuration

Device Data Input Methods

Touch Screen Keypad
Non Touch, Soft/Physical Keypad
Track Balls, Qwerty Keypad
Voice (SIRI, Skyvi)
Split Keypad
Copy and Paste
Blink of an Eye

Device Capability & App

Functionality Bandwidth Connectivity
Camera
Phone / SMS / MMS
Contact List

OTHERS

Rapid Rollouts, Deployment Velocity
Mobile Context, Usability, Performance
Energy Consumption Monitoring
Testing with Emulators / Simulators
Device Interruption Scenarios
Field Testing, Test Execution Challenges

Mobile Demands a Comprehensive Test Strategy

Interaction with Device/OS native capabilities

Optimal combination of devices and OSs

Effective combination of manual and automated testing

Need for faster agile adoption

On-time and off-line connectivity

Mobile Native Apps vs. Mobile Web vs. Hybrid

The best QA strategy to test a mobile app depends on the type of app. If it is a native app, then there are certain conditions which need to be tested.

The tester needs to ensure that the app can be successfully downloaded, executes on the device and interacts with the backend infrastructure. Apps render differently across devices, so each app needs to be thoroughly tested on all relevant devices. We must also ensure backward compatibility with selected older generation devices and OSs. Creating test plans for manual testing on each of these devices can be a cumbersome process.

Mobile web apps pose a different set of challenges. These apps are viewed by users across the globe, presenting unique challenges in terms of networks, scripting and devices. No matter what type of app it is – native, hybrid or web – manual testing of these apps has become more rampant. Testing on simulators does not accurately replicate real user experience. And, testing on all device/OS combination is not practical .The other issue that we have already touched on is the lack of a standardized mobile SDLC, which results in an inconsistent testing methodology.

The right testing approach for your enterprise

Selecting the right mobile testing methodology is critical to the success of any mobile initiative. You need a partner who understands the intricate details and brings the required knowledge as well as the right infrastructure and enabling technology.

testing approach for enterprise applications

key quality dimensions to mobile testing

It’s imperative to link business goals and quality goals, leveraging key quality dimensions specific to mobile testing.

This means:

Understanding the quality dimensions (see graphic)
ligning quality dimensions with a tactical testing plan
Selecting the right service partner to deliver these dimensions

Key Activities & Outputs from Mobile Testing

Test strategy

Define key policies, components, review criteria, approach, testing types, scope, communication and resource planning necessary to control and manage risks throughout the Software Testing Program.

Test plan

Address key areas such as test approach, scope, test types, review criteria, test data, incident management, resources and supplementary materials.

Test case/specification

Identify and outline key objectives of the test validation, related requirements and testable items. All required test pre-conditions and processing in detail with associated supplementary materials.

Test summary report

Provide dynamic snapshot of progress throughout the testing life cycle, providing visibility to management and stakeholders. The overall summary, scope deviation, requirements coverage, analysis/design progress, execution progress, review criteria and outstanding defects are areas that will be addressed throughout the life of the project for review, approval and sign-off.

Test automation

Define repeatable process and help you streamline your testing to support on-demand releases and critical software changes. A hybrid automation framework is deployed for immediate use, eliminating development ramp-up time.

Traceability matrix

Provide a relational model of your software testing validation framework. Requirements and User Stories are mapped to Test Cases and Test Plan and are executed against target software build releases, configurations and environments.

Test results

Each of the test result is well supported with tester comments and screenshots along with the required logs for the developer to understand the issue, and be able to reproduce the same without any trouble.

Test assessment

Provide an overall review of the efficiency and effectiveness of testing activities. Assessment methodology uses interviews with key testing leaders and an analysis of your testing deliverables in a short time frame to provide recommendations.

Different testing types: what you need to know

There are different types of testing that need to be addressed – independent of the platform – to ensure mobile app quality. Following is a snapshot of the various testing types and test cases.

Function Testing

Scenarios

Description

Sign-up & Login

Make sure the app is easily accessible to the users.

Menu Options

Ensure menu items like Help, About, etc. are easy to find and navigate.

Keys

Make sure key functionality is clear and consistent. Assure the app will run with a keyboard and/or touch-screen.

Data Handling

Managing data handling, data deletion and unintended data deletion (important for games in particular) in the testing scope.

Connection Speed / Carrier

Incorporate most popular and likely carriers in most basic app tests to ensure performance.

Operating System

Ensure app performance on different mobile OSs.

Screen Size

Testing for screen size discrepancies.

Interruptions

Test mobile app behavior against device status variations such as battery strength (full, medium, low), incoming calls, SMS, MMS and video calls, etc.

Error Messages

Ensure error messages are clear, concise and actionable.

Usability TESTING

Scenarios

Description

Connectivity

Check for slow and unreliable wireless network connections with low bandwidth taking into account various network conditions.

Small Screen Size

Check for physical constraints of mobile devices, especially small screen sizes which might impact app usability.

Different Display Resolution

Check for low resolution that can degrade the quality of the multimedia information.

Layout & design TESTING

Test Case

Description

Page Layout

Evaluate the overall effectiveness of the page layout.

Color Schemas

Evaluate use of colors in the background, text, links, icons and buttons.

Findability

Evaluate visibility and accessibility of key items like Help, About, Instructions, search.

Localization

Evaluate app consistency in terms of text, message and symbols in all supported languages.

Touchscreens

Test for different touchscreen scenarios: multi-touch vs. single touch; long touch vs. short touch; button size, workflow and any other supported input methods.

Trackballs, Track Wheels and Touchpads

Confirm user ability to move from object to object regardless of navigation tool.

Soft Keyboards

Check user interaction with, and accessibility of, the soft keyboard.

Hard Keys

Check the available device hard keys; e.g., Start, Home, Menu, Back – interact with the application in similar way.

Short Cuts

Check for expected shortcuts for the device and test their use within the application.

localization TESTING

Test Case

Description

General

Check user interface, content files, keyboard, text filters, hot keys, spelling rules, sorting rules, upper and lower case conversion rules.

Format Testing

Check for date formats, currencies per country.

Local Market Compliance

Check app compliance with local laws and regulations.

Content Testing

Check static and dynamic content like video, catalogs, search results, metadata.

Characters

Check different languages and character sets.

Postal Codes

Verify if the target countries support only number or alphanumeric.

Language Direction

Verify the direction for the way language is written – left to right or right to left.

Currency Conversion

Test currency conversion for Internet retail applications.

Tax Calculation Verification

Test rules around the calculation of sales tax and VAT.

Translation Validation

Test for on-demand translation validation and accuracy in including accessibility from any market location.

Security TESTING

Web Applications

Test with common web security and testing tools such as Firefox web developer. Ensure that web proxy is used to intercept all mobile device traffic to monitor data and test for security issues.

Format Testing

Test for password/usernames being left unencrypted on device. Ensure footprint and fingerprint analysis can be used to make hash files.

Local Market Compliance

Research permissions configuration on the device; test to ensure that app can get access to the device areas it needs in order to function properly, and it provides useful error messages directing the user to set the permissions properly, if it cannot. Verify that the app can be successfully installed at all expected device security level settings. After installation, ensure it is possible to change the security settings of device or application permissions. Include some test cases to cover these scenarios as well.

Load & stress TESTING

Test Infrastructure Management & Reporting

Ensure there is enough research on app to decide upon the test infrastructure on live or simulated loads. Feed the mobile app under test with the largest tasks it can handle. Make sure to increase the load on the system via automated tools to simulate real time scenario with virtual users. Set up report management using internal and external bug reporting and test management tools.

Interruption TESTING

Incoming Calls and SMS, Battery & Cable Removals

Must be tested for: Incoming and Outgoing SMS and MMS Incoming and Outgoing Calls

Battery Removal Cable Insertion and Removal Network Outage and Recovery

Media Player On/Off Device Power Cycle

Test automation

Because mobile testing must be performed on so many different combinations of devices, browsers, and operating systems, doing it manually quickly becomes impractical in terms of both time and expense. Test automation can bring significant efficiencies to the process – dramatically accelerating go to market plans and reducing costs.

Why Automate?

The device
os maze

The virtually endless combination of devices, OS models, and device sizes. Test automation enables QA teams to test the same functionality on different devices, OSs and form factors.

Changing
industry trends

Constant OS updates, upgrades, new devices, and new manufacturers. In order to ensure that the application is up-to-date and supported by new trends, testing it in a timely manner becomes important.

Rapid upgrades
to apps

Apps are regularly updated with new features in order to keep up with market demand. In order to introduce upgrades into the market quickly, automated testing is crucial.

The apexon approach to automation

It’s imperative to link business goals and quality goals, leveraging key quality dimensions specific to mobile testing.

This means:

Apexon approach to mobile test automation

Picking the right mobile qa partner

Mobile testing can be complex but app quality is critical to success in today’s business world. The traditional rules of testing do not apply to mobile due to the fast changing landscape, unique features of apps and devices, limited bandwidth and unreliability of wireless networks.

There is a need to develop mobile specific test methodologies and guidelines and therefore a need to rely on a testing partner. This partner should be able to provide insights to modern approaches in testing and suggest the right QA strategy and coverage based on specific business needs.

After defining the qa strategy, your QA partner should provide a well-defined plan on what types of testing and what attributes need to be tested, validated and verified. Ideally, they would have executed testing strategies for enterprises across different verticals – retail, travel, banking and insurance, and digital entertainment – so they understand all the different possibilities and use case scenarios.

What to Look For When Selecting a Vendor

The ability to use the right combination of emulator and real device testing to ensure predictability; emulators with scripting help to test many aspects at an effective cost

The ability to provide intelligence behind use of select devices for your application

The ability to provide automation and agile solutions for faster time-to-market by building the right user stories, increasing use of test sprints and creating script reusability

The ability to have access to the real devices remotely when it’s not possible to run physical test on a slew of mobile devices

The ability to increase the app store star ratings of the app by closely monitoring it and providing reactive and proactive remediation

The ability to optimize the adoption and monetization of the application post-launch

Mobile Application Quality

MOBILE TESTING IS DIFFERENT

Mobile Testing Challenges – At a Glance

Mobile Demands a Comprehensive Test Strategy

The right testing approach for your enterprise

Test automation

The apexon approach to automation

Picking the right mobile qa partner

Download Resource