Earlier today we hosted a well-received webinar on implementing and executing more effective Performance Tests.
Our 30 minute webinar, presented by Sogeti’s Performance Testing Subject Matter Expert, Richard Badger, highlighted the importance of Performance Testing (and doing it earlier), how to implement and execute effective tests, and gave tips on overcoming some traditional and more recent problems faced in Performance Testing.
The replay of the full 30 minute webinar recording can be found here, and we have highlighted a few of the key points below.
It’s relatively easy to build a working bridge…
…but harder to cope with usage.
Although an integral activity for any business in terms of supporting both new and old programs and services, Performance Testing is still regarded as an afterthought in many organisations – and it is often squeezed in right at the end of the delivery cycle before the release goes live.
What Happens When We Don’t Execute Performance Testing Properly?
There are many examples that we can think of that show what happens when Performance Testing is either ignored, or not given as much thought as it should be. Here is just a handful:
- Ticketmaster struggled with the London 2012 Olympics tickets – the system was too slow to update ticket availability and the site was suspended on the day of launch
- Demand for Kate Middleton’s Reiss dress in 2011 crashed their website for 2.5 hours – at a time when the possibility of reaching a peak number of customers was the highest, neither the extra customers nor the regular, loyal shoppers could spend their money!
- Festival goers struggled to purchase tickets for Glastonbury in 2008, 2013 and 2014 due to high volumes of traffic on the See Tickets website
- Speed matters! Microsoft tested delays in search results from bing.com – delaying the speed results appeared by just 2 seconds lost them 4.3% of revenue. Hard proof that performance matters for the bottom line.
One thing you’ll notice is that these are BIG names, and these Performance Testing issues not only cause problems in terms of revenue, but also reputation! There’s probably hundreds more examples for smaller organisations experiencing the same problems.
A key theme from our webinar is that you cannot start Performance Testing too early! Early testing involvement is an indicator of good process and enables test teams to focus on ‘real’ issues (rather than obvious bugs and petty errors) such as making sure the system under test really withstands not just the anticipated load, but also higher traffic.
It is important to test throughout the whole V-Model, and not just follow, as Richard puts it, a ‘tick’ (i.e. don’t just integrate Performance Tests with the right hand side of the V-model). Performance Testing should start at the requirements stage – and even here it needs to be more than ‘it must be as fast as it was last time’.
Testing earlier lessens the ‘testing and the end squeeze’ and can even give us time to check whether the system still operates (and even potentially makes more money) with less resources! For example:
- Can we run green (using less hardware)?
- How does the system scale up – could we run servers in parallel? Or use cloud to scale servers up/down as load changes?
Ultimately, we get ‘better’, more effective tests by solving the big problems while it’s relatively easy to do so, and the risk of not hitting deadlines is noticeably reduced.
Overcoming Key Challenges – Our Recommendations
Testing too late is not the only challenge in Performance Testing. Others include poor requirements, unclear acceptance criteria, and an over reliance on (the wrong) test tools.
Modern technologies such as the web and mobile are also having an effect on Performance Testing. In the new web and app world, client side performance is a huge factor, as we’ve touched on in this blog many times before! It’s not just increasing variety of browsers, applications or devices, but also how they behave when on/off Wi-Fi, whether using a proxy…. The list goes on, and we must be sure to take this into account.
Sogeti’s top 10 recommendations for more effective Performance Testing are as follows:
1. Shift left – use the V-Model, not the ‘tick’
2. Define your Performance Test strategy in line with business goals, including what a ‘pass’ should look like, and document it!
3. Establish a Performance Budget
4. Agree measurable requirements and acceptance criteria – then measure, measure, measure, analyse and repost on what you find at every test phase
5. Keep requirements and success criteria in mind at all times
6. Foster team work and a ‘Performance Testing culture’
7. Use multiple techniques to identify risk areas, and highlight specific high risk areas i.e. high volume transactions, business importance and complexity…
8. Include different devices/browsers in the test if necessary
9. Reduce turnaround time for environments, and create re-usable Performance Testing tool scripts/consider SaaS tools
10. Keep up to date with new technologies and methods (blogs, conferences etc.)
Still in the dark about how to improve your performance testing efforts? Email the Sogeti team, or call us on 020 7014 8900 for a more detailed insight into increasing effectiveness.