Only one thing could be more disappointing than paying $100 dollars for the lacklustre fight that was billed as the “Fight of the Century”. And that is, paying $100 for it and not being able to watch it.
On Saturday night, pay-per-view customers across the board had problems, ranging from error messages to blank screens to the wrong language. Twitter was bombarded with expressions of disbelief and anger at multiple cable service providers. The whole fight itself was delayed by 45 minutes as pay-per-view broadcasters struggled to get their services online.
So what happened? Why did so many service providers fail to achieve the basic service levels that paying customers expect?
What we saw on Saturday night was a failure of Performance Testing. Performance Testing requires planning for capacity and working out all the potential bottlenecks. Admittedly it was popular fight that was hyped beyond belief, which meant, more than likely, an unprecedented number of simultaneous streams. But it is the role of service providers to account for all the possibilities and make sure that the infrastructure is able to cope.
There are lots of tools that can be used – and that we use – to test for performance, from industry leaders like HP, IBM, SOASTA, PlatformLab and Keynote, allowing you to define and provide a seamless experience for end-users. Just as important as the tools that are used is the process and strategy that is put in place.
When we work on Performance Testing, we start with a performance test plan and a risk mitigation plan. We move onto establishing a performance test environment and performance test scripts. Then we move onto executing the performance test scripts, adding virtual users to identify bottlenecks. This allows you to tune your services and your infrastructure. It also allows you to cover capacity planning at the server level and database level. Then you deploy your risk mitigation plan and set up the monitoring parameters to highlight potential future issues.
All in all, it’s a big job, and frankly not everyone knows how to do this.
But right now, there are likely to be a lot of frantic engineering operations at cable companies. There will be a lot of tough questions from technical executives, the board, investors and subscribers. The biggest question they will face is “what went wrong?”
Then the next question, which will be tougher to answer is “how do we make sure that this never happens again?” Every environment is unique, but the steps required to get service providers and application providers to deliver the best possible service are not.