SaaS (Software as a Service) delivers software over the Internet without the end users ever needing to install any software in premise, behind their own firewalls. On the surface doing software QA for a SaaS firm should be no different. After all, you are doing same old testing for an ISV (Independent Software Vendor), it is same old software, with a development life cycle, (well written, planned ☺ ) requirements, release plans, defects, triages, wrangling with the developers, resource crunches and so on; so what can be different?
There is a difference and it is significant. Worse, not knowing it, can impact the quality of your product and credibility to your customers. To understand it first lets see how SaaS is different (For a detailed account of how SaaS is different please read our “SaaS Operational Efficiency” white paper).
How is SaaS different?
- SaaS is more of a service than a shrink-wrapped software
- SaaS solution has more stringent performance and security implications
- SaaS business model enforces an Agile methodology of software engineering
- There are significant operational costs (infrastructure, software and deployment)
- A new release (and its defects) is immediately available to your customers, with a very high risk of an error
- Your customers have a lower risk, cost and impact to switch you and move to a different provider.
The above impacts SQA in more ways than what meets the eye. Following is a list of key areas QA gets impacted:
- Agile Methodology: SaaS firms have so stringent time to market pressures that they are forced to adapt an Agile methodology. With a release happening every month or less, QA teams have to turbo charge their productivity, in order to ascertain good quality level.
- Performance and Scalability Requirements: SaaS firms deploy their software on their own hardware. Poor performance exponentially increases the operational costs and in worst situations becomes a death bed for the company. QA teams have to validate the principles of multi-tenancy and test each release for its scalability.
- Web Services and SOA: As SaaS software gets deployed behind its own firewall, for most enterprise applications, there are strong needs for data integration. Most SaaS products have a rich WS layer that enables them to integrate with third parties. This actually is a blessing in disguise as it provides an easy interface for QA to test the key business functionality through an easy to automate interface. QA teams however need to know how to best test the SOA platform. Please read about some specific work we have done around SOA Testing here, here.
- Test Automation: I believe SaaS QA teams (or for that matter, any agile lifecycle) can not survive without significant level of automation. This includes (but is not limited to) good unit testing, GUI automation, functional tests of key application interfaces (such as your main controllers), SOA interfaces, performance etc. Having a good framework that extends itself to meet ongoing automation needs is critical. We have blogged about automation here and here.
- Security: As a SaaS company you are taking ownership of safekeeping of critical customer data both morally and legally. Any error with security can shut your company down. As a QA team, you have to know and test for major web exploits and plug them.
- Release Automation: Just like test automation, any agile methodology can not survive without automated builds, releases and deployments. QA teams have to understand and implement continuous integration process that builds regularly and reports failures in an automated way.
- Test Environments and clean Deployments: While the SaaS model is a boon for customer support (they are no longer required to support the unforeseen mess weird combination of OS, server and software environments in customer IT, they just support what is in their production); it is yet another pain for QA. How do you know what you are testing will work in production? QA needs to have very tight control over deployment scripts and deploy the software in the test environment. The test environment itself needs to replicate production environment closely including all the load-balancing and clustering infrastructure.
- Testing for SLAs: So what if the code functionally works and passes performance criteria? Does it meet your SLAs? How do you guarantee your software will not breach a functional, performance or uptime SLA in production? Again it is QA’s extended role to validate for conformance with SLAs.
QA Managers working for SaaS firms have to keep the above points in mind when creating a test strategy for their SaaS product.
