We chatted with Rob Thevenet our Sr. QA Manager about his experience and challenges with performance and load testing for enterprise software. Rob is a seasoned QA guru and has built automation and continuous frameworks to support performance automation and performance testing for HP, Virgin Mobile and Sun Run. At Apexon he leads the automation and performance competencies, rolling out new projects all the time.
Interviewer – Rob, thanks for taking the time to talk to us today. We wanted to do a technical deep dive into some of the issues that you come across repeatedly during performance and load testing preparation and planning.
Rob — No problem, I will be happy to answer any questions you might have. Thanks for having me.
Q1. What are Enterprise, Load and Performance (L&P) test objectives?
Rob – Some or all of the below can be included as test objectives and each require their own type of approach, metrics and tool sets
- Understand application or subsystem/component behavior under different operating conditions
- Disaster Recovery (DR) preparation and/or Business continuity (BC) assurance
- Model (mirror or extrapolate)
- Implement (tune)
- Benchmark (pre-test, pre-prod, etc.) specific subsystems or modules
Q2. What issues / constraints often need be overcome to setup, execute, measure, extend the test?
Rob – Technical gap or debt, environment contention and/or dependency
Q3. Which Non-Functional Requirements (*NFRs) and related conditions are in play for the test effort?
Rob – The following potential conditions apply to most if not all NFRs (most listed below) and serve as possible parameters for applying to common usage cases by particular user types including system and/or operational activities including database re-indexing, backups, replication, etc.:
(a) Minimal usage (standup, idle, acquiesce then activate minimally)
(b) Normal operation usage levels
(c) Peak operation usage levels
(d) Stress/Breakage usage levels
*List of NFRs – Availability, Business Continuity, Capacity, Configuration, Data Integrity (Data Currency, Data Integrity, Data Restoration, Data Retention
Disaster Recovery), Error Handling, Globalization, Internationalization, Localization, Logging, Longevity, Manageability, Messaging
Portability, Reliability, Scalability, Security, Testability, Usability, Volume
Q4. What are some Key Metrics to collect and what makes their analysis interesting,
Rob – Throughput, Latency, Transaction roundtrip time and Response times (for system, subsystems*, components); per SLA or Industry accepted thresholds**
*subsystems = Operating System, Application, Hardware, Software, Driver, Firmware, Memory, Disk I/O, Cache, Network, Processor, etc.
**SLA or industry accepted thresholds;
Q5. Which tools/methods can ensure a successful test effort?
Rob – Virtualization and Database snapshots help to provide s/w, h/w, f/w and state interchangeability, as well as decreased setup time enabling shorter Root Cause Analysis (RCA) cycles for issues found in test or production
-Full system infrastructure monitoring using Splunk, HP Mercury Sitescope, Topaz, IBM , Oracle Empirix, and/or Silk Central
-Unique Instance Report (UIR)
-Sorted Message type files (contain # of instances per source, message data, log entry of its containing log or event file)
Q6. What acceptance criteria are essential to a successful test with valid results?
Rob – Access to Application / System Under Test (AUT/SUT), REST, SOAP, etc. APIs to drive, check status, etc.
-Framework design and implementation supporting L&P test infrastructure client approval sign off
-Functional Stability
-NFR acceptance criteria defined (Agile, at least 1 NFR acceptance criteria per card)
-Implement an ongoing L&P test effort to reoccur monthly to quarterly in frequency supporting regular release cycle certification
Q7. What are protocols encountered in L&P testing and special conditions or methods to consider for handling?
Rob – Web(HTTP/HTTPS/HTML), LDAP, COM/DCOM, OBIEE, IEEE 802.11 (mobile standards) <OSI model>
Q8. What types of L&P issues manifest? How can failure analysis pinpoint issues?
Rob – Database deadlocks, failed/stopped transactions, refused connections, HTTP level 200, 300, 400 or 500 messages, load generation agent script/code compilation errors.