How to Harness the Power of Data Analytics

Get more efficient test automation with QMetry wisdom

Josiah Renaudin: Hello, and welcome to today’s web seminar: Shift left or Get Left Behind – Harness the Power of Data Analytics for More Efficient Test Automation, sponsored by Apexon and featuring Vijay Shyamasundar, Senior Automation Architect at Apexon. I’m your host and moderator, Josiah Renaudin. Thank you for joining us today. Before I hand it off to the speaker, let me explain the controls on this console. If you experience any technical issues, please send a note through the question field located beneath the panels, and we’ll be on-hand to help as quickly as possible. Our responses can be read in the Q&A panel to the left of the slides. This is also the same way to submit questions. We welcome all comments and inquiries during today’s event, so feel free to ask as many as you like and we’ll answer as many as possible during the Q&A session after the presentation. Today’s event contains polling questions as well as a live demo, which can be seen within the slide view. To optimize your bandwidth for the best viewing experience possible, please close out any unnecessary applications that may be running in the background. At the bottom of your console’s a widget tollbar. These widgets open panels. The console opens with three on your screen: The speaker, Q&A, and the slide panels. All of these panels can be moved around and resized to your liking. If you accidentally close one, click the respective widget to reopen the panel in its original place. Hover your cursor over these icons to reveal labels identifying each widget. Through these icons, you can download the speakers slides as well as share this event with friends and colleagues. This event is being recorded and will be made available for on-demand viewing. Once the recording is ready, you’ll receive an email with instructions on how to access the presentation and slides on-demand. With that said, I would like to introduce and pass control to our speaker, Vijay. Vijay Shyamasundar: Thank Josiah. Hello everyone, welcome to this webinar today. I’m Vijay Shyamasundar, an Automation Architect at Apexon. I’m excited to talk to you all about how we can leverage and analytics to do effective test automation today, alright? Without much further ado, I’m gonna get into the topic for the day. Alright. So, of late, we are seeing a lot more companies getting into digital transformation, and we’re being fortunate to be a part of the journey with some of the clients that we’ve been working with. And we’ve learned some of the challenges that exist in traditional automation techniques and built tools around that to overcome those challenges. We’re going to talk today specifically about how we can leverage data. Of late, with the modern way of building software, there’s a lot more data that’s being collected. Instead of that just sitting somewhere on your storage, how can we put that data to work? That’s kind of what we’re trying to look at in today’s webinar, and also a sneak peek of – a demo, right? So, basically, what we are seeing is a paradigm shift in terms of the type of applications that are being built. It’s moving from more of a system of record to a system of engagement model. What we mean by that is – a system of record – is like traditional, single, monolithic-type of applications, where multiple teams are building and working towards delivering one product, and where typically the release cycles are pretty large – in months, sometimes quarters. And I want is more of a system of engagement model – where people are moving towards these days – where it is very user-centric and customer-experience-focused, and fast feedback, and omnichannel, and things like that. So, this type of application, in order to get fast feedback, demands that you deliver it much faster to the customers as much as possible. And the challenge with omnichannel experience, in terms of delivering it to different devices, one factors, etc., the amount of testing required and what kind of testing required will all be different than traditional system of records model. So, if you look at the landscape for this type of new and modern application, it’s not just mobile or desktop web anymore, you have much newer user interfaces like conversational UIs, like chatbot, wifebot, etc., right? And applications are being a lot more contextual, leveraging a lot more artificial intelligence and machine-learning type of data. And also in terms of the way it is being delivered, it’s a microservices model, where teams are focusing more on a singular focus, building one functionality, and there are multiple teams doing that. And there’s a level of orchestration that’s necessary in order to make all of this come together as one product and cater to all different – the channels that it’s being served to. And if management comes in, you still have your enterprise systems, and a lab that’s necessary for them to build it. So, the way in which we build software is changing in terms of the methodologies, and in terms of the architecture and in terms of the channels, etc., right? So, what this means from the quality aspect of it is it’s not about just testing functionality anymore, right? It’s not – It’s about testing the user experience, especially when you have multichannel dividers – how you would address the testing aspects of that. So if you’re – like we said, why some of the QA practices might not suffice. So, when we have to get faster feedback, there’s need for shipping the products much faster with higher confidence. And so, you’re kind of leveraging continuous integration/continuous delivery, and you have probably success-specialization-type thing that’s necessary to be incorporated in your delivery pipeline leveraging master data management because you’re piping to multiple omnichannels, you’re trying –kind of like streamline your data. So, basically, computerization and watchelization, and in terms of the deployment, and etc. so, what’s happening with all this – There is a need for going to market much faster but the complexity of the testing has been increasing. And, typically, what we used to see – in terms of the efforts between development, testing, and operations, it used to be a pretty even split between dev and ops and testing but what we are oddly seeing today is that the effort of the time allocated for testing is kind of shrinking, and it’s become more of an integral part within the development side or on the operations side. And in the very near future, we see that becoming a really integral part of dev and ops, and that’s why these dev-ops are not really dev-test-ops. And one thing with some of the Fortune 1,000, Fortune 500 companies that we are working with, we have seen that the automation adoption has increased by over 200 percent, and some of the clients who we have been working with who leverage continuous integration/continuous delivery are being able to push twice as fast as they were doing before they could adopt these methodologies. So, there is a lot of companies who are getting into doing predictive QA and leveraging data to kind of look at what the next test icon would look like and where they need to channel their efforts, etc. And so, in a nutshell, there is – The enterprises need quality at speed, and like I said before, the work that needs to be done is increasing significantly, and the time we have to do that is reducing very much. So, at this point, I’d like to ask our first poll question. So, I’m assuming most of the guys we have here are doing some type of automation, and I’m curious to know: How of you are analyzing your results manually? How much – How many of you are doing it automated, using any predictive analysis? Gonna give a few more seconds to everyone to get a chance to answer the poll. Those few responses coming, probably one more second. Alright. Alright. So, this is what we got as the results from the poll question, and it looks like a good portion – 76 percent – responded that they’re doing still manual, and it’s good to see about 24 percent who’re doing automated, and it looks like none of the attendees we have today are doing any predictive analytics. Alright, that’s good to know. So, this is a very simplified version of continuous delivery pipeline; Of the stages you would have in a typical continuous delivery pipeline: where you are checking all your code, and building it, and you might be running some units and static code analysis. And depending on the environment, and depending on if it is a double-up branch, and it’s a feature branch, and you may not have as much of a complicated pipeline as you would on a QA environment, where you’re probably getting the code artifacts and running your functional automation regression, or even pre-fermenting a pre-prod kind of environment, etc. But at that high level, this is kind of the typical CD pipeline that is would look like. So, it’d code, build, test on the fly. So, this is an example of a snapshot from one of the CI server – one of the sample projects that we have, right? So, what you see here is – in the rows, horizontally, that kind of represents every commit from the developers. So, in this case, the pipeline is set to trigger every time somebody commits a code into a particular repo that we are monitoring. So, you see in the bottom, everything is green, everything passed. We checked out bell, unitas, static code, we uploaded to – our artifacts to a repo, and you downloaded it on some test environment. You ran some smoke tests and regression tests, and you deployed. So, everything was good. And – But if you see in build number 133, it – things failed at – during the regression test and test environment, right? So – And then, if you see, there are other bells which failed earlier. So, basically, here, there are two things that I wanna talk about: 1) When this failure happened, somebody has to – if we are doing the analysis manually, somebody will have to go and look at whether this was a script issue or it was a true failure – to do whatever application issue or data issue, whatever it is, somebody has to go and do that analysis, and that’s kind of a bottleneck, and that kind of slows down your delivery pipeline. Now, the second aspect of this is – and if you see the number of commits that are happening, and number of – amount of tests that’s being run, you’re collecting a lot of data. And like I said earlier, is it just occupying some storage, or are you really making use of it? What is it that we can do – and we are getting and generating this much data – what can we do with that? So, those are the two things that we wanna look at, right? And how we can put that to work. So, like I was saying, our delivery pipeline – if you’re really not automated, it is kind of more looking like this. It’s not just code, build, test, and deploy, it is really code, build, test, analyze, and then deploy, and you have everything automated in your pipeline but the analysis part is still manual, which is a bottleneck and you’re not doing anything with the data. That’s kind of the problem we are trying to solve – so, traditional automation technique key challenges. So, if you – it is very critical – that analysis piece is required. Even if it slows down, there is no way you can get around it. Somebody has to look at it and identify if it’s a script issue or a real failure, and show that – and false negative is even more – bad scenario: you will let something pass through if you don’t have that analysis phase. So, some – a lot of organizations, they have – especially regulatory-compliance-type requirements – they have to analyze and have – more than just analyzing, they have to have the proof of execution and proof of analysis as well, right? And on the next level, once the analysis is done, you have to also get into categorizing the real failures: like some of it could be due to data, some of it is a functional defect; some of it is a network-time or a connectivity issue, etc. So, all of that has to happen. And if – how can we do some of that automated? How can we leverage the data and use the power of analytics to do some of that work for us? And another challenge that we have with agile methodologies, where teams are trying to achieve in-sprint automation, and there is constant effort required to make sure that the scripts that run successfully in the previous sprint will again run successfully in this sprint, and if – especially if you’re automating at a UI level, there may be a lot of maintenance work required. And also, the new development for the features that are being built as part of this sprint. Now, what happens typically in the real-world scenario is you keep – tend to prioritize the functionality – building the functionality and the scripts, washes, looking at some housekeeping aspects like: How long is my script taking? Why is it taking that long? How can I make it faster? And by not doing that, your test execution cycles are longer, and as and when you add more tests, it keeps growing. And there are – There have been cases where, I’ve seen before, this type of optimization regression cycle, in spite of it being automated, takes more than a day, which basically defeats the purpose of dong CICD and devops because you have huge a huge bottleneck and you can’t get all your tests done. And if you try to reduce that, then you’re not getting the coverage that you would want, and that’s kind of a double-edged sword. Another aspect of it is the more people today are using cloud-based watchel instances for executing some of these tests, and then it adds to the cost. You’re not considering optimization as much as you should and then it’s adding to the cost, and it quickly adds up and then the business side of the house calculates the autoIT impacts; How much are all your automation and – is generating. And also, from the CICD aspects, like I said: almost with every commit, you’re getting a lot of data but if that is failing at a particular stage, you may not even look at it and it’s just sitting somewhere. So, how can we actually – and like I said, the business impacts of it might leading to revenue loss or losing customers, etc. That kind of work transforms into the business side of issues due to some of these challenges. So, how can we actually put the data to work? Wouldn’t it be great if you could leverage analytics and get some insight on how your automation is progressing? How – And if your automation strategy is working or not; Is your product quality improving over time, and what’s causing some of the frequent failures? And how can I – How can it help me in prioritizing some of the work items I wanna take up in my next sprint? And what are some of my consistently underperforming test cases? And if I get insights on – let’s say, you optimized this particular test step, and it will help in this many scenarios, and it’ll make your cycle run this much faster – how – what can I – no, how can I get that kind of insight? And based on my actions – let’s say if I’m categorizing an issue as a data issue, and next time it happens again – is my analytics engine smart enough to do that categorization again for me so I don’t have to go do that every time? And also, hopefully it’ll learn to address which could be a false positive versus a negative, etc., and things like that. So, that’s kind of what we are looking to achieve, and in the at – and when you look for the right analytics tool: What are some of the key aspects that we should look for in order to look for that to be automated? So, basically, is it giving a high-level overview of where your automation is headed – where your automation strategy is headed, success rate, and if you have different omnichannels, is it giving your data – which is summarized for those specific platforms, for certain type of time duration and things like that. And most importantly, can it integrate with your existing systems that you use seamlessly? So, you may have your test management tools, you may have your automation prim – like, IDs that you have been using, certain type of format that – in which you generate your test results. So, is it compatible and can it integrate seamlessly with that, right? And different type of automation results. Like you may have XML, or JSON, or some custom format, etc. – And how well will it integrate and read that data, and is it customizable, configurable like some of the aspects that – something that you consider fast is relative and – versus someone else who might consider what fast or slow is – so how can you set some of those things? And what kind of insight it gives you in terms of improving your automation effectiveness, measuring the performance of your tests and things like that. These are some of the things that you would want to look for in an analytics tool if you’re looking to automate the analysis part of your test execution, and maybe when going a step ahead and doing some predictions on where you should focus for your future cycles, and where your problems could be and what module, and things like that. So, at this point, we’re gonna take our second poll question, and I would like to know if you’re using any kind of analytics tool – it was good to know about 24, 25 percent was doing some automated analysis, so… how many of you are actually using any analytic tools in your testing process. Right, we’ll give a few more seconds for people to respond. Alright. Okay, we see that about 15 percent do use the data analytic tools, and 40 percent – we have an even split, some of them are looking to use, some of – half of them are not looking at this point in time. That’s good to know. that’s mo – one, two… so, at this point I’d like to show a quick demo. So, here is the cumatory Wisdom dashboard, and once you come in, you select the project that you want to look at, and it will load the data in the dashboard view. And currently, it is showing for the last month. If I want to look at the stats for the last six mon – last three months, six months, last week, things like that, I – So, basically, it gives you overall summary of your success rate. Like right now, this is kind of the overall health of your project, so to speak. In the last three months, you have about 83 percent tests passing, and you have about 15 percent tests failing, and about 1.6 percent being skipped or not executed. So – and again, if I want to look at, say – for last week, I’ll say that my past percentage is low and – why is – a lot of my tests are failing, as opposed to my overall for the last three months. And you can kind of dig deep – and I’ll show you in a minute – on how we can get some insights on that. And then, you also have to look at the data per platform. Let’s say – If you’re building for multiple platforms, you can look at the success rate for each platform and also look at the coverage and how much of that particular platform is being automated and how much of it’s being executed, passing, failing, and all of that. And also, there is another view where I can go look at – irrespective of the platform, if I want to just look at what the health is for the latest test run, then you can kind of get to see – on your most recent, you have about 70 percent passing and 20 percent failing, and some skips. So, that’s kind of the high-level overview of it, and now coming to the execution trends. So, basically, again here, daily, monthly, weekly, yearly views, and you can filter it by if you just want to look at the failures or if you want to look at both, and kind of gives you the count as well. And if I’m looking at something monthly – this is just a high-level graph but if I want to – bear with me for a second – if I want to – and often – to kind of go – drill deep into that, it will show me all the test runs that I have for this, and I can filter by status if I wanna see all the runs or if it is just the failed runs. And for a particular run, if I select a failed scenario, it gives me the actual test steps for it, like how much it – how time it took for each test step. And if you notice the colors here, some of it is in red, some of it is in orange or yellow, or some of it is in green, and the legend is like your definition of the division, like if it’s fast, or if it’s moderate, or slow. So, anything in red is kind of slow, green is good, and this is – orange or yellow is moderate. So, once you get into it – and on this kind of – on the left-hand side that you see the color coding here – that if you see something in red, you know that was a failed test scenario, and if I want to know what exactly happened, I can click on that information tab there. And it looks like it was looking for a particular element by XPath and it could not locate that element, and it has some more console log and the stack trace, every information that will help in debugging. And apart from this, if I want to also go look at more details of that particular test step, then I can go navigate to that particular geaticad and then look at if there has been any changes, if somebody committed something to it, or if there is any info I need to put in from a tacking perspective, and things like that. So – And then, I can come back here into this view, and the next aspect is the performance bottlenecks. So, this is the analytics engine behind Wisdom that is doing this work. As and when you push results of – for every execution – every cycle, it kind of computes – takes the average and computes that – what are your worst performing test cases and gives you in this nice graphical view, and it also tells you on an average how long it takes to execute this particular test. And then, if – again, like other examples that we saw, we can drill down and it – in this case, it took about 30 minutes – and it gives you details of every single step, and how long each one took, and, again, for pass/fail in all the scenarios. And you can also sort it by division if you want to see which ones are the most performing – I mean worst performing tests here, and things like that. Alright, and then the next aspect is looking at what are the test cases that are failing really often. So, as an example on top, you have this test case, which was executed 11 times, and it failed 11 times, and you can basically – you can drill down, and you get all the details, and you can see why something was failed. In this case, looks like this was an issue. It failed, and looks like it – some – it waited for 106 milliseconds, maybe a time-out error or something like that, and then once it couldn’t’ get what it was looking for, it looks like other tests were skipped and it didn’t execute at all. Maybe in – this was case where it was waiting for something, and maybe the wait times are too long, and it gives you insights on, “Should I wait for that long? What is the delay from the service to get the response? Is there any room for me to tweak there?” or things like that. And so, it gives you insights on those kind of aspects. And the other – next aspect I wanna talk about is the categorization of errors. So, a lot of times in scenarios where if it’s a data issue or an environment issue, it kind of raises the alarm and increases the – especially during the tense situations, it could be a false alarm and cause a lot of hard work for a lot of people. So, it will help if – let’s say, if you know that you have categorized that, “Okay, this is a data issue, and the next time it happens, it’ll automatically categorize for you,” and instead of you going and doing it manually, it’s done, and you’re kind of reviewing it and forming it. So here is basically how it could look like when you’ve categorized those errors. It could be data errors, it could be false positives or negatives, it could be just bugs, and there could be some uncategorized errors as well. So, when you click on that, you can drill down and it will tell you how many scenarios are affected by this. So, how many test cases are affected by this particular issue, and how many times this has happened. It gives you that kind of an insight, and then you can also create new groups. So, for example, you – in your case, you may want to identify that this is an issue with a particular module, and you wanna create that, “Okay, I can add a new group, and I can say this is module B,” and then maybe there is an uncategorized error if I can go to this view, I see that these three are my uncategorized errors. And let’s say this issue I know is because of module B, and when I move it in there, it goes there. And next time it happens – and if it is, the analytics engine will determine if it is a similar issue, and in that case, it will bucket it in module B, and you will have the opportunity to come and review it in here when you are looking – of course you can move things around and it will learn and adapt. So, that is about the categorization, and another I want to show is – this is the QQ bot that we have, and the analytics that is happening behind the scenes. And it gives some very interesting insights based on its machine learning and artificial intelligence capabilities. So, basically, it says that most of your testing IUs used failed due to this following error, and if you solve this, it’ll get an additional 3.2 succ – 3.2 percent success rate. And then, it gives you other insights on, “This is the slowest in execution, and it affects these scenarios, and it took average 22.8 percent of the time, and if you – you will get an additional performance gain by this much percentage if you fix some of that.” And it also kind of gives you different insights on how fast/slow your things are running. And also, if automation was not running – if it was like, “Looks like automation was on vacation for these months, I don’t have any data for this.” So, this is some of – and also, you can check if there are new discoveries that it has made from the time that you last checked. You can refresh it and get any new data. So, that’s the analytics part of it… and there are some settings where, like I was saying earlier, the speed is relative, so if you want to set – for your specific project – like project A has a different time versus project B – then if you want to set that – anything under this threshold is fast or moderate or slow, you can do things like that, and you have some options to create filters and things like that. So, that’s pretty much what I had to show from a demo perspective, and if you are interested to learn more, you can go to the help section and it will take you to some reference videos that you can learn more about and the product and also its usage. There’s a documentation of available if you’re interested in integrating that with your test framework. Like I said, this is available as a Cheetah plugin on Apalachin Marketplace, and these are the toolings that will be very helpful if you’re interested in learning more. And the QMetry website itself also has a lot of information on that. Alright, so now I’m gonna switch back from the demo mode and – So, in the last light, for folks who are interested in talking to us about this analytics tool, test automation, devops, this – have the email address or the phone number, our product team would be very delighted to get in touch with you if you’re interested in doing a no-obligations strategy session. If you’re interested to do anything in particular with a proof of concept, for your particular implementation, I’d be happy to get in touch with you and discuss that further. At this point, Josiah, I would hand it over back to you for a Q&A. Josiah Renaudin: Alright, thank you so much, and remember, before we start Q&A, you can ask questions by typing them in the field beneath the panels and then clicking the submit button. We’ll try to get through as many questions as possible live but for those questions we’re unable to answer, some will be answered offline. We already have some good ones coming in, so let’s start right here. This person asks, “Would Keymetry’s test failure analysis be able to track Selenium test failure information?” Vijay Shyamasundar: Yes, it will get to s – track this information, yes. So, basically what – the way it would work is basically your test automation results are exposed to Wisdom through the API, and whatever console log or at the stack trace level – it could be BDD level, it could be a java level, and Selenium or Appium level – those things will be available within that, and that is also being con – used to do the analytics part of it as well. Josiah Renaudin: Alright, thank you so much for that clarification. This next person asks, “How is this comparable to SonarQube? Is there opportunity to add customization?” Vijay Shyamasundar: So, SonarQube is more a static analysis tool, where basically you are giving – defining your quality profiles, and your code smells, and walnut abilities, and quoting standards for a particular language, and things like that, and you are running your code against it. But what – So, if you look at it from the delivery pipeline, that kind of is something that you would apply in your – after your buil – you checked out the code, you built it, and you’re applying SonarQube – static analysis – you’re doing that but what Wisdom is doing is on your test automaton results. So, after your static analysis, you are executing your automation run, maybe regression or functional type of automation or APA level, whatever layer you’re – layer of the stack you may have automated – so we are taking that results, and analyzing it, more from how well your automation is run. It’s not necessarily at the static code level. Did that answer the que – I hope I answered your question. Josiah Renaudin:Alright, thank you very much Vijay. Next question here, this person asks, “Do you have any plans to support WATIR?” Vijay Shyamasundar:I’m sorry, can you come again? Josiah Renaudin:Yeah, this person asks, “Do you have any plans to support WATIR?” It’s No. 22 at the top, spelled W-A-T-I-R. Vijay Shyamasundar:I’ll have to defer that to a product team and we’ll definitely get back. I’m sorry; I’m not familiar with that. Josiah Renaudin:Alright, thank you very much Vijay. We can always reach out to that person individually. This next person asks, “How does Keymetry work with test run on Sauce Labs?” Vijay Shyamasundar:Yeah, so, basically, it’ll – like I said, this is an API integration. So, only thing that this really needs or depends on is you pushing your test results. It doesn’t matter where you ran them – it could be Perfecto, Sauce Labs, it could be local – wherever you ran them, as long as your providing the details of the test run. And the way it is done, it’s not looking a certain hardcover-type of fields, it is very customizable. Basically, whatever information you’re pushing through as part of test results, it will leverage that and it will build analytics and machine learning capabilities out of that and give you back some analyzed data. Josiah Renaudin:Alright, great, thank you so much. And we do have tome for a few more questions. This person has a really good one right here, it says, “Does the test steps breakdown assume you’re using Cucumber or Gherkin, or is it flexible with other formats such as Junit, TestNG, Jasmine, etc.? Vijay Shyamasundar:Yes, I don’t know if I can go back to the screen sharing mode but the API documentation that I showed on the – while I was doing the quick walkthrough – it gives you a very detailed – the framework and language support; Yes, we do support Cucumber, TestNG, JUnit, UFD, QAF, and in terms of the supported file formats, it does JSON, XML, and if you had any zip files containing JSON, that will work as well, and Selenium Appium. And also, basically, that’s the place to look for as you go to the documentation, and it’ll also give you some code samples there depending on what language you’re using to – and some really step-by-step instructions on how you can use that API for a specific client. Josiah Renaudin:Alright, thank you so much, and we should have time for this last one. This person says, “Does it support TShark or TFS?” Vijay Shyamasundar:Yeah, so, again, I think as long as your test result is in one of those formats – we do have this implementation on some custom frameworks based on .NET and the Team Foundation Server. So, if you – as long as you’re able to give the results in the format in one of the supported formats, I think which is most standard, like, JSON, XML type of formats, it is possible to do that. Josiah Renaudin:Alright, fantastic, and that’s actually going to end our event for today. I’d like to thank Vijay for his time; I’d like to thank Apexon for sponsoring this event. Also, a special thank you goes out to you, the audience, for spending the last hour with us. Have a great day and we hope to see you at a future event.