Questioning Automation
Improved efficiency, enhanced quality, faster delivery, and a better ROI – are the results of effective test automation. Indeed 88% of World Quality Report (WQR) respondents who are using automation have seen a significant reduction in test cycle times with automation, and 84% have benefited from cost reductions. So if automation has such potential, why does the WQR also reveal the proportion of automated test cases in the UK, standing at 42%, is 3% lower than the global average?
At Sogeti some of the most common questions we hear from our clients are: What are the main barriers to automation and how do we overcome them? What constitutes a higher quality delivery of automation? What are the dependencies for successful automation? How can we address technical debt in our regression suites?
Let’s Face it Automation isn’t Easy
One reason behind these uncertainties is that when adopting an automation strategy many businesses assume test automation, once set up, and means that they can test ad infinitum with little or no effort – as if on autopilot – but this simply isn’t the case. With the increasing speed and magnitude of changes in functionality automated scripts will require constant maintenance, otherwise they risk becoming increasingly redundant as the product transforms. Effective automation also requires a more agile way of thinking and this can therefore be a major business disruptor before you start to see the benefits. Not all testers have specialist automation or scripting skills, which is another major requirement for success. Developers are achieving continuous delivery through continuous integration, continuous deployment and the emergence of DevOps disciplines, but this speed of delivery can come at the cost of quality, and lead to reputational damage if your testers are not working in the same way and at the same speed as your developers.
Shift Left to Get it Right
Initially introducing automation into small projects and ensuring testers are trained in scripting will get you off to a good start. However the real game-changer is to adopt an approach to development which integrates testing. This not only shifts testing left, and therefore increases quality (or risk-reduction), but incorporates thinking about and creating test and automation strategies as an integral part of the development process from the very outset of the project. This test-first approach stabilises the underlying manual test strategy, gives clarity on what to test and when, and reveals what can be legitimately automated. Bugs can then be found and fixed in the script development stage as opposed to later on in the regression suite, reducing overall technical debt and ensuring a higher quality output. This requires a very high degree of collaboration and understanding between your test and development teams, with developers, end-users, and testers all contributing to the test automation strategy. The other crucial factor is to realise that automation is not “automation forever”; “autopilot” mode will not lead to automation success. As the software develops the automated tests will need to monitored, re-evaluated and changed to ensure automation remains effective.
Automation Webinar
So this is an overview of the right approach to automation but to really implement it effectively we need to take a deeper dive into the barriers, the solutions, and how to achieve Test-Driven and Behaviour-Driven Development, and how to alleviate technical debt. To help you achieve this we’re hosting a webinar on 28th July at 3.30pm BST entitled “Increasing the Efficiency, Productivity and Overall Quality of Your Test Automation”. If you are keen to get more value from your test automation strategy then you can register for this webinar here.


Posted in: Automation Testing, Behaviour Driven Development, Business Intelligence, Developers, DevOps, Digital strategy, IT strategy, Quality Assurance, Requirements, Research, Software Development, Test Driven Development, Testing and innovation, User Experience, World Quality Report      
Comments: 0
Tags: , , , , , , ,


Watching Ganesh’s presentation made me mindful of my early experiences of automation, trying to work out how to create automated tests using WinRunner. The vision my then manager was pushing me to deliver was one of “the big red button”. A shiny button one could press and the software on one computer would go and test the software on another computer, and report back that everything was good.

Looking back it seems there was a degree of optimism associated with primitive capture-replay automation tools, the sort of optimism that buys a salesman a Porsche or two.

All of which makes it feel all the more pertinent to be discussing the pitfalls of automation, as without a doubt some of that shiny optimism is still there, and I think that can only really be contributed to the fact that automation has so much to offer that it is very easily oversold.

There have been a great many changes in the automation tooling landscape since the days of WinRunner as automation has developed over roughly 12 years since my first introduction. Whilst there is perhaps more reason to be optimistic now than there was then, anyone attempting automation must genuinely still be mindful of the pitfalls outlined by Ganesh.

A reminder of the ten pitfalls to avoid when implementing automation

10. Lack of a scalable test execution model
9. Unavailability of automation test environment
8. Automation test team on a side track
7. Aiming for unrealistic automation test coverage
6. Insufficient automated test process
5. Inadequate automation test framework
4. Losing the big picture along the way
3. Evaluate before you buy
2. Lack of automation test management
1. Unrealistic stakeholder expectation

What I have referred to here as optimisim is covered by Ganesh’s number one pitfall of unrealistic stakeholder expectation. To automate is a project in itself and, as such, needs all the rigour and governance you would apply to developing, for example, your customer portal website. We know the realities of developing any software and how we are constrained by, for example the Prince2 triangle which trades-off cost, quality, and time. All too often time and cost are prioritised over quality.

A simple way to avoid this pitfall, and hopefully to avoid at least a few of the other pitfalls, is to focus on quality. It’s perhaps more fruitful to consider it not as increasing quality but as reducing risk, aligning to the approach taken to the other two corners of the triangle (minimising rather than maximising). Reducing risk is more likely to ensure the success of the automation project, giving a better chance of realising the benefits of automation.

This single change would hopefully also help avoid some of the other pitfalls: treating automation as a project in itself would lead to proper management and governance being implemented; as a project it should have defined goals, keeping track of the big picture of automation within the organisation; the governance would be in place to ensure a test process is followed.

Whilst I have focused on what I call “the optimism of automation”, or in Ganesh’s terminology “unrealistic stakeholder expectation”, each of the pitfalls have the potential to derail automation projects. And with increased pressure for increased agility and shorter delivery cycles, automation is the only effective way to reduce risk by providing comprehensive regression testing. Quantifying the risk of not doing automation well will help to plan to ensure success.



Posted in: Automation Testing, IT strategy, Managed Testing, project management, Software Development      
Comments: 0
Tags: , , , , , , , ,