Transform from QA to QE the Right Way – A Case Study

Ram: Let’s talk about quality engineering or QE. QE drives, the quality development of product and processes while enabling effective testing in parallel. The idea is that during the software development life cycle, you design your test automation strategy beforehand and then the developers, operations and QE teams, all work together as a single unit. You define the automation in your test cases beforehand, you script them, and make the executables ready as part of your development process. This prepares your team – so they are ready to validate the functionality assets being developed.

The software paradigm is called shift-left wherein the attention to quality is paid, not towards the end of the software development lifecycle, but at the earlier stages – during conceptualization and development. Here you define the executable requirements as part of the behavior defined development process and have it ready as part of the distribution.

As the product is being developed, you implement and verify your executable code and verify the functionality across multiple devices. Once you have your automated testing ready, you have the ability to get quicker feedback on the devices and the viability of the operating systems that you want to verify. You incorporate that feedback back into the software development, and during this process it’s important that you incorporate the continuous integration and services into your overall test automation strategy. That helps you with respect to the overall software quality being verified at the early stages of development.

With QE, the emphasis is on creating a lot of automated test cases closer to your development code. So, you develop a large set of unit test cases closer to development code and you have lots of area for automated test cases and the competent technician at the API test level and you do lots of UA-related test cases. This enables you to do rapid development or verification very easily in a heightened manner, so you can complete your validations very quickly. When you compare this with legacy software development lifecycles, there is more manual testing done and less emphasis and data on unit tests. The result is additional time and cost for you to verify across multiple devices.

In the new model, you spend a lot less time on manual testing, and have more automated test cases – closer to the development code. This should provide tighter integration of your automated test cases closer to your development code.

The Difference Between QA and QE

What is the difference between QA and QE? Quality Assurance “assures” quality of the product, but quality engineering drives development of quality product and the processes. This includes quality, maturity of the quality team itself, but it’s also a cultural shift within the teams. Quality engineers focus on the quality right from ideation. And it involves the processes, so it includes Agile based processes and uses test-driven development and behavior-driven development processes to define requirements that are meaningful to the business developers and the testing teams.

With QE, everyone on the project development teams understand the common business requirements and they develop and validate those requirements. You also need a common set of tools in your test framework so any test data manager can assist you on running the tests across multiple devices simultaneously. These automated tests are being culled from your continuous integration and continuous deployment processes, and being validated as part of your build deployment.

Sridhar: Hi everybody. This is Sridhar Narayanan, from Scholastic. I’m going to talk to you a little bit about our journey as we transitioned from QA to QE. We had some unique challenges, but certainly some challenges that are common across the industry as well. So, I’ll walk through and tell you what we did and am happy to answer any questions you might have.

Scholastic is a very complex, very established organization — almost a hundred years old. We’re in the publishing industry, which by all definitions, should not really exist today. At least that’s what people have predicted. However, we are in the children’s publishing industry. There are lots of children who have interest in reading and books, driven by things like Harry Potter. So, we’re alive and thriving and doing very well. Because we are a very old organization we had, a huge number and variety of legacy applications. And we prefer to move all those applications in terms of resource farming, in terms of the cloud, in terms of new initiatives, so a fair amount of real transformation was a backdrop against which we attempted the QA to QE transformation.

About a year ago, we started re-platforming our very large e-commerce application. The shift was to move the application into the cloud, away from our enterprise e-commerce application. A very complex application covering workflows across multiple, multiple customer categories, complex checkout process, going from a non-responsive to responsive application. So, we were looking – in our background, we sort of started with a faulty QA transformation when this was already in play.

This initially meant we had some constraints because the application was in the cloud. With a classic application testing environment, you could hit the API, you could hit a lot of back-end processes with a large number of tests and we could reduce the number of UI tests. But because we were constrained by a cloud vendor, we had very limited access to the back end. So, we couldn’t really do a lot of testing in the API back ideologically. So, because of that, we are constrained to run a very large number of tests at the UI layer. This is not highly recommended, but this is a constraint we are facing.

So, as we were building out the test automation framework for this, what I wanted to do was essentially build out an automation framework that we could then leverage across many other product categories of the company. So, as part of this, we laid out some rules as to how we wanted to build a framework, and built it to meet a lot of those requirements. It was then implemented on the UI tests, on this platform. Later, we asked for a frame of what was being built out. We were able to take the framework and start having other things start utilizing the framework, and take advantage of that. As a result, we could see a lot of commonality in terms of how we could address the problem; we could look at common ways of defining the tests, executing the tests, and publishing the test results.

We started the process about 16 months ago and we had a fair amount of experience under our belt with the framework as it applies to this platform, and other applications. Our goal was to use BDD to help us get a strong connection to what a customer really wants out of an application. We had already decided to use BDD in the context of writing a regression test that was not quite as powerful, but we wanted to make sure in this particular case that we were going to have a framework that could still define a test using the syntax and that any newer tests could fit the same pattern and would actually be the way it is supposed to be done.

A lot of other teams which are starting with new features, are able to take advantage of BDD, by defining the spec way ahead of time and using that as a spec to create other wave tests. I’m not a big fan of renovate tooling. There are cost issues, licensing issues, and connectivity issues, in terms of how well you can adopt it to fit what you really need. Those are things that are left lacking. Now, it certainly applies that you need to have a team that can deal with open source and take advantage of it, and somehow, they can use it to work the problem that you need to solve.

It’s a double-edged sword in some way, but we our preference was to use open source. I think CI (continuous integration) is very common across the industry now, but our goal was to leverage CI, and for teams that are ready for CDV, we wanted to push them towards CD or at least enable CD such that if teams needed to take advantage of it, our framework was ready for that.

One of the things we did do, is lay some ground rules for how that’s got to be done and avoid expensive new tests. For example, rules saying, “Hey, your test should not take more than 10 minutes.” And 10 minutes is really the upper limit. We’d like to see tests run in, preferably, less than a minute, maybe three minutes, tops. But just to set a guideline for complex tests, we assume your test will not take more than ten minutes. And one of the reasons for this is we wanted to ensure that all tests could scale; that we could have many tests and execute that in a very efficient amount of time by leveraging parts of this. So, this is one of the rules that we used going into that.

Running multiple tests at the same time, there were several options for internal grade versus very few options available for the cloud-based environments we preferred. But we went for a cloud-based solution and that has worked very well for us. There are some questions about performance but I think overall the data does well enough for us that we don’t have any specific complaints. We still need to keep our options open in terms of being able to switch back from cloud provider or to same grade if we need to. And we built the framework with that in mind, keeping the cloud component fairly “switch out-able.” So, I could switch to a different provider, or I could switch to internal grade if I wanted to.

The use of a test manager is important here. If tests were public, we wanted to make sure that every test was accounted for. For example, the check-out test: if you want to run a hundred scenarios with check-out, we want to make sure each test running a different browser, a different situation, that each test could be independent. We didn’t really have a way of controlling the sequencing of those tests. To address that, we created a very large test data manager that would allow the testers to grab a particular test data set on demand and be assured that no other test was going to use that. So, we had a reservation system and because of that we were able to run multiple checkout scenarios that did not really stop each other.

This is just one example. There were many examples – where allowing the test data on demand, allowing the test to be padded, enabled us to complete many tests in operation. Back to the ten-minute guideline/slogan: “Each test shall last no more than ten minutes.” We said, look, given each test is small and short enough, and achieves the objectives; in examining the build, I could build out in that environment and completely be finished in ten minutes, right? 16 is the number of environments I would have access to, and in the cloud, we’re speaking short, limited time that you have. So, we put our slogan out there to say what was possible. And it’s something that has elevated the status of QA. We’ll say; “Hey, this is something that we can do. If you can finish your coding quick enough, we can tell you the regression is solved in ten minutes.” That is a conversation worth listening to.

All of this helps us run a large regression suite and reduce the amount of time needed to validate new features. This is just in terms of how we went about the solution: looking at existing tests, creating the framework components and CI using Agile within the regression projects. All of this was standard Agile process for automated test development, and this works well even as the application was being built out. We were able to complete the regression in about the same time as the application was complete. In the end, the build out framework was used across the company, and we were able to run extremely large suite of data. Again, it’s largely good at this point.

The goal for our division is to be able to run a core suite of tests, about 150 tests in about 20 minutes. This was extremely useful in being able to push out multiple tests a day. We could give them their test results back extremely quickly, and we could complete our entire regression suite in about three hours. With the current constraints, the team continues to optimize the test, to make each test run faster. We are also looking at expanding so we could really hit a goal that’s possible, maybe keep our eye on the ball for that moment.

From a process perspective, we were able to get the QE engaged very early in the game, to create and demonstrate this. The lessons learned: Automation is hard work; everybody knows that, especially if automation is against QA. Just getting them to change their mindset; changing others of the team’s mindset about QA is not easy. When we elevate quality engineering and bring them on equal footing when they’re creating automation code, they’re not doing any less work than somebody filling out the applications.

Sridhar: The technology is all there. If not, there are so many other ways of doing things that there really isn’t a gap in terms of solving the problem today. If you’re doing BDD you need to pay attention to how the Gherkin is respected, is graded. That is, how the respect is created. If you just try a functional test, you basically are not taking advantage of BDD. Maintenance is a huge aspect of how you test rapidly. You need to pay attention to how the application changes, and how the test changes as well. Upper management support is also important. We had a lot of support. We needed to make sure that is there and it helps take your automation strategy all the way through to the end. A good partner, I think, if you don’t have the skillsets in-house, we had Ram and team bring on some support and help us solve the early problems.

Certainly, you need to have your partner participate formally in the game, but a good partner can help you get some lead with this. And, that’s it, thank you.

Ram: I would like to talk about QA to QE technology. The key focus areas for QA to QE transformation include those that enclose your role transformation. The quality assurance engineers must make the transition to quality engineers, with the mindset that they want, not just the quality of the product, but the quality of the process, the collaboration that happens with the developers and the development house, and the business analyst, and the business team. And they have to validate the end user stack and then look at the product.

So, they have to be more proactive looking at what the upcoming changes are in the industry, including any new devices and operating systems that are being planned for release to the market. They also have to consider how the end users are actually using the test applications of the product. So, they have to be a little more proactive about the overall process and make sure that the business team, developers, and the developer team follow the process as well. So, that transition is necessary to make sure the goals are achieved.

And that’s part of the process transition of the earlier days of the waterfall model of SDLC development and lifecycle. In that scenario, QE has a very limited role in software development. They play only towards the end of the development. But here, the quality engineers walk hand-in-hand with the business and DevOps teams, so they not only just define the processes, they also create the automated code at the same time as the developers — create the development application code.

So, as a result, the end of the development is complete within in a day or two. Nowadays, we work in Agile teams. Where it’s been a week, or two weeks, or three weeks. So, by the time you are usually ready for testing, you should be ready to do an automated test case, and be ready to re-execute them and validate them. So, the problem of transformation methodology has created smaller teams of developers defining, validating and releasing quickly to market. That’s part of the transformation metric. In the olden days, you used to have your testing distribution cycle, where you’re faced with the project development cycle, SIT or System Integration Testing.

User acceptance testing is another area for change. These days you don’t hear anything about any system testing or user acceptance testing phases. Within the two-week cycle of the weekly cycle of sprint completion, you validate. At the end of the two weeks, you also give a demonstration to the project stakeholders that includes the business owners and the product owners, and you quickly get back the feedback from them on a continuous basis. And then you release this to the market.

And so, as part of this transition, you no longer can wait for your end of cycle report. Instead, you want a real-time test result dashboard and quality metric suitable for your current agile process. And you also need a set of tools and infrastructure that will help you with continuous integration, continuous deployment, and usage, utilization. And, any on-demand environment schedule that will help you with building test developments on the fly, not innovation to test frameworks and actual test code. So those were some of the key areas for you to transition; and in the overall engineering landscape, during all these conceptualizations and building assignment areas, there are tools in the market that we could leverage. Some of the tools that we are using are open source and many of these tools have been used in Scholastic for our QE organization.

Ram:So, what are the benefits of moving to QE? What is the cost? You can validate your application in more quickly, and that means you are able to deliver to your end customers by quickly releasing to the market. You are saving cost and time to market and accelerating the release cycle. And you are able to incorporate the user feedback back into the application faster, releasing it back to the customers. You’re also able to validate across multiple devices and browsers thoroughly. That means now you are getting higher user satisfaction. That brings customer loyalty back to you.

Josh: Excellent, thank you so much. I think that concludes our webinar. We want to give you guys an opportunity to go ahead and submit some questions.

The first question is for Sridhar.

“What are some of the challenges in creating and using a shared service testing framework?”

Sridhar: I think that the biggest concern, is it a macro service? In this case, you’re primarily looking at a framework that is hitting the API and you’re testing for that. If the application is primarily UI and not in the cloud, you’re constrained from hitting that layer. Then there are applications that cut across both where you are initially at the transaction of the UI layer, and then in the backend using an API. So, now you have a sort of mixed mode test. So, understanding what is required and building your response and framework around that, that’s the challenging part.

In terms of commonality, everybody still requires a way to define the test. So, having the ability to define the test in a format that works across all the audiences is huge. And like I said before, we are using Gherkin for many reasons, but one of them is getting to have a common language that’s cutting across the teams and now different stakeholders. The way we use test data, that’s a commonality in framework that helps the way we run the tests in the cloud, is certainly common, and we can leverage that on all the teams. And the way we dispute test results within our dashboard, that is also a shelter of framework, again with the context, we carry out that your test results needed to be set in the context or set in your particular domain. The team that turns a macro service needs to look at this and say, “Hey, this makes sense to me.” Similarly, a team that runs more tests, makes UI and API, the test results need to make sense to them, right? So, that’s what I look at as being a challenge.

Josh: Speaking of Gherkins versus Cucumber, do you have a preference?

Sridhar: So, Cucumber takes Gherkins specifications and translates that into a state and code for your different application languages. You can use this to integrate the Java code or other similar tools to create – Python, Loki – other code which you can then fill in with the test automation. Gherkin is the name of the format and Cucumber is one of the tools that implements it.

Josh: Another question for you Sridhar is around open source and how Scholastic used open source technology.

Sridhar: So almost all the tooling in our stack is based on open source. Cucumber by itself is free. There’s no cost to use it. We are writing our code using Java. We use Jenkins for continuation. There are certain components which have a cost. We use a plug-in called Behave Co to manage all of our user scenarios for BDD. That is a plug-in, again, a very small cost. Nobody will pay for a classic test management tool.

We pay for the cloud-based, test execution environments. Those are an expense, even if you used the same grid. There is an expense to maintain that and your expense to host that, so it is a wash. For a database example, you use MySQL for a test data manager; we use Tomcat when we need a front-end for quality test data manager. So, all the pieces – every problem that we had — were solved by picking an open source component that we used to build for the solution.

Josh: A question for Ram is how do you ensure that your QA cycles are stable enough?

Ram: That’s a good question. This goes back to what Sridhar just mentioned, you define your scenarios and behavior, it’s not tied to any specific UI element or any other component. So, you have your behavior scenarios defined as part of your test automation framework. That’s the way you develop that. You have a strategy wherein the changes to your application can be handled by means of configuration, or dynamically loading based on the changes that are happening to the product itself.

Sridhar: If I can add to the question about test cycles being stable. I think the answer to that doesn’t lie specifically in the QE space, but in how you approach application development. Are you breaking the application into small enough chunks where you can basically build it out, within your script period and where you have automation that supports it? You have your registration feature that’s built out as you build each feature into big chunks. That’s when you have all of your tests automated. You can run regression at any time or several times a day. And then, you can have your quality engineers focus on the exploited testing, focus on the experimental testing and revise testing.

From that perspective, your QA cycles are really critical because if you’re able to run your regression, in a matter of minutes or hours, you can tell the developer they broke something as soon as they break something.

Josh: The last question for you guys before we go is a great question from Ikta which is, “where would good tools be available on the market today that give you a real-time dashboard for quality analytics?

Sridhar: We looked at 30 Open Stack tools that leverage Grafana, Kibana, Logstack, and strung them together to create a dashboard. Our initial contract was to leverage something of that sort. We didn’t look at commercial tools on the market today, and we haven’t found everything that sort of plugs in entirely into the system yet, but what we are using is a plug-in called QMetry Wisdom from Apexon. It allows you to post test results and provide the dashboard view of the tests that ran. That has worked well for us so far. It’s not complete and we still intend to build on top of that, but that’s what we use today.

Josh: Great. Okay, I think that concludes our Q & A. I appreciate your time, both you, Ram, and Sridhar’s time. Thanks everyone for joining us.

We also have a great promotion running right now, which you can take advantage of a free, no-obligation assessment with one of our QE experts. Take a free QE Assessment. That concludes our webinar for today, if you have any questions or any input, e-mail us or contact us info at apexon.com. Thanks for joining us today, we appreciate your time.