These truths are about Test Automation in the context of the automation of functional tests. Its use in other areas of testing is governed by quite different truths. Now, Test Automation is a very useful toolset; but like many useful things it is prone to misuse. Today I want to highlight some factors that are often overlooked in both the decision to implement Test Automation and in the way it is used within projects.

Test Automation does not replace Manual Testing.

It only replaces manual regression testing. Test automation is code. Code contains bugs. Test Automation must be debugged. Therefore test automation requires manual testing. Once you have manually verified that the automation and the subject application are working, you can then confidently run the automation as part of a regression suite. But remember changes are not covered by regression and still have to be manually tested.

Test Automation is about Quality.

Test Automation produces less functionality for a given level of resourcing than pure manual testing.  Fairly self-evident that test automation needs more resources than manual but it is amazing how many people miss that. It does however, if done well, increase quality so results in better code being produced more slowly; the fundamental truth is that, for a given investment, test automation trades scope for quality.

Test Automation increases project cost

Test automation saves money on manual regression testing so should reduce costs. In practice people always minimised the amount of manual regression testing that was done in order to save money. The cost of developing and testing the automated tests will often be more than the money saved from not doing manual regression testing during the development project. The payoff from test automation comes in ease of maintenance and reduction in risk of regression defects not being detected before live once the product is in maintenance; but the cost of this is up front project costs.

Test Automation reduces the effectiveness of your testers.

Testers exist because people are very bad at testing their own developments. Turning your testers into automation developers makes them less effective testers. They start thinking more about how they will automate the tests rather than what tests are needed and then they miss bugs in their own code because they see it with developer eyes. You need the same separation between tester and coder in test automation that you need in other areas of code development.  Of course you can just have your testers write the test specifications and then get developers to write the code.

Test Automation doesn’t always pay back the investment

In order to repay the investment test automation has to replace the quantity of manual testing that would have cost more than the automation. If you have a well-built application with stable requirements you probably will not deliver value as the automation will just not be needed enough. If you have requirements that are dynamic, the changes to the test criteria will mean that the tests do not have a life long enough to repay the investment; remember every time an automated test is changed it has to be manually tested to verify that it works. So when planning your project you have to ask if test automation is appropriate.

Principal Consultant and Agile SME

Posted in: A testers viewpoint, Automation Testing, Opinion, Software testing, Test Automation, Test Methodologies, Testing and innovation, Working in software testing      
Comments: 0
Tags: , , , , , , , , , , , , , , , , , , , ,


I was recently in a meeting where question of the agility of a supplier came up. The discussion quickly reached an impasse as we struggled to reach an agreement on whether or not the supplier could actually be classified as ‘agile’. The reason for this dispute is, I feel, interesting enough to prompt this blog.

So what caused the disagreement? Well essentially it came down to different views on the ‘meaning’ of Agile.

I accepted the supplier as agile because during a previous meeting with them we had discussed working processes, and I had found their attitude and approach embraced the values in the Agile Manifesto. My colleague, on the other hand, would not accept them as agile because the working processes did not match those that he associated with agile; to him agility was defined by a set of processes, while to me agility was defined by the attitude of the people involved.

 This difference can be clearly seen in considering the question in my mind: ‘Could you have an agile waterfall project?’ To my colleague, the idea of an agile waterfall project is as much of an oxymoron as that of a solid liquid. To me there is no rational argument for regarding this as an oxymoron and I see no barrier to having a waterall project done in an agile way.

So what is my argument for viewing agile as attitudinal rather than procedural?  Looking first at the agile manifesto we see agility expressed as a set of values, and not as a set of processes, and the supporting twelve principals are expressed not in terms of prescriptive processes, but in the ways of behaving whilst executing a process. Across all of these values and principles, the only one that really seems incompatible with using the v-model is that of delivering working software frequently; and it seems a weak argument to argue that the v-model is incompatible with agile working on just these grounds.

It appears that the desire to define agility in terms of process has nothing to do with the principles of agile software development. The first value in the agile manifesto is that individuals and interactions are valued over processes and tools, and to define agility in terms of processes is therefore highly contradictory. To consider where this idea comes from let us look at all the right hand elements in the agile values – they are all focussed on removing uncertainty; or, in reality, creating the illusion of removing the uncertainty (borne out by the history of IT project failures). It seems that what sets agile apart is the willingness to not just live with uncertainty, but to turn this uncertainty into an asset. After all, an approach that anticipates uncertainty must be inherently flexible.

Now a team may be using a set of processes and tools associated with agile working (such as SCRUM, Kanban or TDD which are useful tools and processes), but following these processes and using these tools does not mean that you are working in an agile manner any more than playing a musical instrument means that you are playing in a creative manner. Using tools developed by the agile community with an attitude that seeks certainty will make them as inflexible and ineffective as the tools they were intended to replace.

Conversely using v-model processes and tools while applying an agile mind-set will expose in them unexpected flexibility. Working with the v-model does not require heavier documentation, tighter contracts, or more rigid plans than working in SCRUM, nor does it require you to value processes over people. Such things are the product of fear of uncertainty and the risks associated with it, and fear will stifle a SCRUM project just as readily as a v-model project.

When it comes down to the wire the benefits of agile software development come not from which tools and processes you use, but from the attitude of the people involved.

Keen to learn more about testing within an Agile approach? Register now for our webinar on 3rd July, where we will be discussing how testing should be regarded as an integral part of the Agile development process to support the project team in delivering on time, on quality and to budget. Find out more and register.

Principal Consultant and Agile SME

Posted in: Agile, Opinion, Webinars, Working in software testing      
Comments: 0
Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,


The IT sector has no shortage of methodologies, ranging from programming to programme management, and yet some projects still come up short. A vicious cycle then occurs, where IT professionals rearrange the deckchairs to create new variations on a theme, in the hopes that the correct formula will come along and deliver success – but it rarely happens.

Group testing can lead to cognitive biases.Unfortunately the idea that ‘if I use the right methodology my projects will succeed’ is nothing more than wishful thinking.

Projects do not always succeed or fail as a direct effect of the methodologies used; rather, success is measured by the people involved. People have inherent cognitive biases, which can lead to poor decision making. Some examples of this practice include:

– Anchoring: An action that occurs when an individual anchors on a particular piece of information while ignoring others, like focusing on the quantity of code produced while ignoring the code’s quality.

– Availability Cascades: Self-reinforcing cycles that take place when a group belief is repeated until all individuals agree it is true, such as reiterating the phrase ‘this project cannot be late’.

– Backfire Effect:  This occurs when negative evidence strengthens positive belief, like when an individual persists that targets can be met despite constantly falling behind schedule.

– Confirmation Bias: The tendency to look only for information that fortifies ones opinion, and ignore all contradicting evidence. An example of such practice would be making a decision based on a large number of passed non-critical tests, rather than the small number of failed critical tests.

– Contrast Effect:  The consequence that occurs when an event is given more importance than it merits, like when a single resolved critical defect is given more weight in decision making than a large number of outstanding defects.

The practices listed above are just the beginning of a longer alphabetical list that goes on and on, reaching the letter ‘Z’:

– Zero-Risk Bias: The trend that results when reducing small risks to zero is preferred over a significant reduction in a large risk. An example is putting effort into removing a 0.1% chance that a piece of information is accessible by an unauthorised person, rather than focusing on significantly reducing the risk (from 50 to 30%) when the same piece of information is inaccurate.

These biases lead to decision-making that is based on beliefs distorted by cognitive opinions, rather than objective information. Not only do these practices undermine the effectiveness of methodologies, but they also lead to neglect of methodologies when beliefs are challenged.

Those who have been in the IT industry for a significant amount of time have most likely worked in a project setting where the person in charge drifts into managing the project based on his or her beliefs, rather than reality.

So, what can be done about this? The answer lies in Quality Assurance (QA) – a concept that can be applied to both the output and the internal process of a project, by insisting on evidentiary validation of project decisions.

The QA team is required to maintain a certain level of independence from the project. In this role, a company such as Sogeti can add value by providing an independent external perspective on a project. Sogeti can also offer insight through its specialised staff – whose members possess the training and experience needed to expose and correct the effects of cognitive biases.

To avoid project failure, be sure to visit the Sogeti website and learn about the various methodologies that can help your business. To be in touch with one of our testing specialists, please email

Principal Consultant and Agile SME

Posted in: A testers viewpoint, Opinion, Software testing, Test Methodologies, Testing and innovation, Training, Working in software testing      
Comments: 0
Tags: , , , , , , , , ,


Consider two development teams with the same number and composition of developers, testers, project managers and business analysts. Each person has the same history of industry and technical experience as their counterpart in the opposite team, and has worked on projects of comparable complexity, quality assurance and project management standards.

Anyone following pure precepts of scientific management theory would expect both teams to achieve the same levels of productivity and quality over a given period. However, those with real world experience know that the above situation can, and regularly does, produce vastly different results between the two teams.

The problem is that scientific management employs scientific analysis to anticipate results, and – as has become understood over the last quarter century – such analysis does not produce meaningful predictions when applied to chaotic systems. Since each human brain is a chaotic system in itself, groups of people forming development teams are inherently chaotic. What this means from a management perspective is that the performance of such teams cannot be predicted based purely on the consistent factors such as those listed above.

Unfortunately for those who seek an ‘if X then Y’ style conditional solution, chaotic systems just don’t work that way. Instead, when evaluating the expected performance of a team, more appropriate factors can be derived from soft measures such as team cohesion, satisfaction with working environments, individual abilities, morale, and attitudes to work. Of course these are difficult to evaluate, and represent even higher risk when dealing with new people.

This is when a professional testing services organisation like Sogeti UK can de-risk the situation. When you really need to rely on a test team, a professional testing services organisation will provide teams that are trusted and known to be able to work together in an effective manner, effectively removing the risks associated with recruiting and building an untried team. Also, when the need is not for a team but for individuals to supplement existing teams, then the long-term relationships built up between professional testing services organisations and their clients ensure that account managers know the clients and the nature of their teams, and can select consultants who will rapidly achieve cohesion with those client teams.

Principal Consultant and Agile SME

Posted in: Managed Testing Services, Opinion, Performance testing, Software testing, Test Methodologies, Testing and innovation      
Comments: 0
Tags: , , , , , , ,


There is an old Taoist saying:

In the beginning there was the Tao;
When the Tao was lost then came the power;
When the power was lost then came religion;
When religion was lost then came ritual.

Of course you’ll wonder what this has to do with software testing. Well, I think that these four levels correspond to different levels of engagement within the test process.

Those who possess the Tao engage with each project as an element within the Tao, and seek to find ‘the flow’ that will deliver the objectives. They know that every project, in its own way, is trying to deliver quality and value and they endeavour to aid the flow of the project towards these objectives. The question they ask themselves is: “What does the project need me to do?” and they ask that question every day, as the answer can change every day.

Not everyone can feel the flow of a project and respond with the flexibility of those who have the Tao. Others have observed projects and seen techniques that work; they embrace these techniques and thus have the testing power that comes from these techniques. But they are wise enough to know that the techniques are limited and need to be moulded to the requirements of the project in hand. The question they ask when they come to a project is: “How can I adapt my techniques for this project?”

Then there are those who believe in the power of the techniques but do not understand the need for flexibility within their approach. They take those methods, array them in an ideal form and seek to impose this ideal form onto any project. When they come to a project, the question they ask is: “How can I make this project fit to my techniques?” This is where test techniques are taken as statements of faith and map to a belief that, if the procedures are followed, then success will come. Between the power and religion of testing, there is a major change in emphasis. Those who apply the Tao and the power have the attitude that they are servants of the project and that they should adapt to the needs of the project. Those who follow the religion see the observation of the techniques as the vital factor in project success.

Finally, there are those who learn the techniques but have little understanding of why they work, and without such understanding they cannot be flexible. When these techniques fail, they do not try to understand the reasons why, and instead seek safety in following the ritual of their techniques. For these testers the question they ask themselves is: “How can I prove that I followed best practice?”

So what question do you ask yourself when you come to a project?

And apologies to SAP as this was not about the SAP Test Acceleration and Optimization (SAP TAO) software which streamlines the creation and maintenance of ERP business process testing. But then maybe it was ….

Principal Consultant and Agile SME

Posted in: A testers viewpoint, Opinion, Software testing, Testing and innovation      
Comments: 0
Tags: , , , , , , , , ,