Increasingly I feel that ‘risk’ is becoming one of the most over-used words in the English language. We are advised of the risk before drinking hot coffee or using a hot water tap. Financial advisors want to understand our ‘risk appetite’ before advising on our personal financial planning. And who isn’t now familiar with the Health & Safety ‘risk assessments’ that have become pre-cursors to even the most routine school trips or workplace activities? So many facets of our life are now seemingly governed by checking and double-checking our commit to risk.

Naturally it’s also a word I increasingly encounter in my own world of software testing. I frequently hear about project managers clutching the straw entitled ‘risk-based testing’ – a straw that is often clutched as their project faces missing deadlines.

Do we truly understand what risk is, and what it means in the context of our daily lives? And does the concept translate, unchanged, to the IT software testing world?

One of the benefits of prudent outsourcing of the test function is that this approach provides a transparent view of risk. That’s a bold statement, but allow me to outline the approach taken by the QA function of one well-known retailer I worked with in a previous role. For all of its IT, the test function was outsourced a single independent supplier. The function was set up to system test solution deliveries from multiple providers. The Test Outsourcer contract was constructed with financial penalties for any Priority One or Priority Two incidents occurring in live production operation during a three-month warranty period. Clearly this commercial construct required the test outsourcer to provide test plans, test coverage and test timetables commensurate with providing a sufficient level of quality to reduce the likelihood of attracting financial penalties. In essence the test outsourcer was now carrying the risk – a risk with real financial values attached.

Test projects typically follow a lifecycle that sees delivery teams delivering late but the business end dates remaining fixed. How can the test outsourcer react within such a lifecycle if they now have to take on extra risk, resulting in less time to test to the level of coverage required? The response must be a contractual one: the original agreement to sign up to zero Priority One and Priority Two incidents becomes impossible. The result? Concessions are proposed, allowing for example, a maximum of one Priority One incident and four Priority Two incidents during the warranty period.

So there we have it – a quantifiable assessment of risk that a business can properly assess using no more dark-art skills than accountancy.

Clearly for all but the most trivial IT solution, it is never prudent to test nothing. But nor is it practical to test everything. So the only remaining questions are: how much testing should we do, and what should be left out?  Intuitively the closer we are to the ‘test everything’ end of the spectrum, the less risk we will be taking; and conversely, the closer we are to the ‘test nothing’ end of the spectrum, the more risk we are taking. So the real question is not whether we should use risk-based testing, but how much risk are we prepared to take? Of course quantifying an objective, acceptable level of risk is extremely difficult. After all, how do you judge when the level of risk has gone too far?  This Test Outsourced approach gave the business a rational measurement of risk that enabled an informed and measured decision making process for IT deployments.

Phil Lupton AUTHOR:
Phil is a solutions director and manages the service delivery for a large outsourced test services project.

Posted in: Risk, Risk-based testing, Software testing      
Comments: 0
Tags: , , , , , ,