As public services are increasingly digital by default, shorter production lifecycles and the demand for quality and continuous testing put increasing pressure on testing organisations to create a set of metrics that enable better decision-making. So how do you maximise test metrics to get the best result for clients and end-users?

Why are test metrics so important?

The right metrics enable risk assessment, earlier bug detection, quality output and an insight into the effectiveness of the current test strategy, while holding developers, testers and 3rd party vendors accountable in a transparent way. While the Cabinet Office strategy of working with smaller suppliers helps to bring about innovation, and cost savings, this agility also creates an IT environment where change is constant. Effective test management and having a strong set of test metrics is more critical than ever in these multi-supplier environments where more risk is introduced into the IT delivery process. Comprehensive test metrics give granularity at project and service level, they creates a safety net and provide core values to return to when innovation or problems cause you to veer away from the bigger picture.

What should we measure and who decides?

At Sogeti we use our Live Dashboard to gauge the performance of the Development team and other stakeholders who affect the test process. KPI-aligned metrics also determine our own performance as the testing services provider. This last measure directly impacts our revenue, encouraging quality output and allowing us to offer clients the fairest fees. At Sogeti we use a comprehensive set of around 20-25 metrics which are broken down into KPIs, Service Measures, and Quality Measures. Examples of KPI-aligned metrics include defect leakage into production and testing the percentage of defects raised in error. Service and Quality metrics include test design and execution productivity, defect leakage into integration or UAT, defect turnaround time and reoccurrence respectively. The set of metrics we use are very similar in the Private and Public sectors. Metrics should be determined by the CIO, Chief Transformation Officer and other C level executives, the Vendor management groups, the test services provider and whoever owns the QA function on the client’s behalf.

How do you get the Project team to buy into the metrics and feedback in a productive way?

To get the client on board it should simply be a case of explaining the cost, time, quality, transparency and accountability benefits we’ve discussed. To engage those team members whose work is showcased by the metrics, it’s a case of reminding them that the purpose is not to name and shame, but rather to continuously develop their expertise, innovate and provide a better service. As an employee of a top tier test provider, testers should expect their performance to be measured and see it as an opportunity for growth and to collect evidence to prove the need for changes that will actually make their jobs easier. With Sogeti’s Live Dashboard, 3rd party vendors have no reason to query the process as we are putting ourselves under the microscope as well.

This year’s World Quality Report shows that Government is focused on improving engagement and Customer Experience (CX); how does this impact test metrics?

Currently the emphasis is on functional defects, usability and accessibility from the internal client’s perspective. Now that CX is a major driver, the test metrics and analysis will be more focused on the true end user; the individuals and businesses that interact with our clients. As customer expectations rise, the responsibility for quality and reputation management grows. To meet this, testers must take a walk in the shoes of the end user and ensure test metrics are dynamic and adapting to reflect the customer journey, feedback and changing requirements.

This article has just been published on Brite Innovation here:


Gary Moore AUTHOR:
Head of Delivery for Government Testing Services

Posted in: Developers, Digital, functional testing, Human Interaction Testing, Innovation, IT strategy, mobile testing, Quality Assurance, Research, Software Development, test data management, Test environment, Testing and innovation, Transformation, User Experience, World Quality Report      
Comments: 0
Tags: , , , , , , , , , , , , , , ,


Would you pay a ransom to hackers in the hope they would keep their word and refrain from carrying out a cyber attack on your business?
According to the recent Cloud Security Alliance (CSA) report, “The Cloud Balancing Act for IT: Between Promise and Peril”, 24.6% of companies would and 14% of those would pay more than $1m (£691,000). Interestingly, one of the main barriers to preventing hackers gaining access in the first place is the skills gap in Security and Security Testing expertise, particularly in Cloud environments.

Held to Ransom

So what’s the US government’s stance on these ransoms? Well, at a Cyber Security conference in Boston in October last year, Joseph Bonavolonta, the Assistant Special Agent in Charge of the FBI’s CYBER and Counterintelligence Program said “The ransomware (malware) is that good…we often advise people just to pay the ransom.”

On this side of the Pond, as indicated in the Cyber Essentials Scheme June 2014, the UK government’s position is not dissimilar. The FBI admission caused an outcry with Senators asking “Does Uncle Sam pay ransoms?” Many recent news articles have asked the same question of the UK Public Sector. Of course the government cannot be seen to be bending to extortion and paying hackers in the vague hope they’ll go way. The government already have several initiatives in place such as the National Cyber Security Programme 2010 (updated 2014), the Cyber Essentials Scheme and the innovative Cyber Security Test Range to assist both the Public and Private Sector. However, what we really need to do is close the skills gap and approach Cyber Security Testing in the right way to reduce the likelihood of Public Sector organisations having to make such decisions.

Chief Integration Officer

The combination of acquiring new skills and expanding existing roles, making new hires, and creating new roles will help to close the skills gap and ultimately ensure a more comprehensive test strategy with more effective security testing.

Testing integration services (including local, public, private and hybrid cloud) is a major challenge in Public Sector and although Public Sector adoption of cloud is currently lower than the cross-sector average due to security concerns, 27% of applications are cloud-based and 45% of testing takes place in the cloud. Given that Gartner predicts that by 2018 government agencies will be citing security as a reason to move to public cloud, rather than avoid it, addressing these issues now is of paramount importance.

Chief Information Officers who want to ensure their relevancy going forward, should start to take on the responsibilities of Chief Integration Officer. A primary area of focus will be building  security into solutions. This emphasis on cyber risk management, privacy and governance will address many of the concerns of the wider business, creating better relationships and helping to make security and security testing everyone’s responsibility.

Chief Digital Officer

47% of Public Sector organisations have already hired a Chief Digital Officer to oversee and bring together the full range of digital strategies, drive digital transformation and manage change. A further 21% are planning to hire a CDO in 2016. This new role will also be fundamental to transforming the security and testing strategies and aligning them with the wider business goals.


In terms of test strategy itself, failures, attacks and breaches are most likely to occur at the application layer, so continuous testing via automation is a good approach. These methods can often be most easily and cost effectively facilitated via an external Testing Centre of Excellence, so decision-makers should be considering this option to bring the Public Sector into line with other more mature sectors.

All statistics not attributed to the CSA report are from the World Quality Report 2015-6. You can download you free copy of the Report here:


Gary Moore AUTHOR:
Head of Delivery for Government Testing Services

Posted in: Automation Testing, Cloud, Digital, Digital strategy, Innovation, IT strategy, Opinion, Quality Assurance, Security, Software Development, Software testing, test framework, Testing and innovation, Transformation, World Quality Report      
Comments: 0
Tags: , , , , , , , , , , , , ,


Testing The Security of Things in the Public Sector

Innovation Vs. Risk

Just 2 months before Ed Vaizey announced the new interdisciplinary Research Hub for the UK government’s £40m, 3-year IoT UK initiative, EU policymakers were at the Future of the Internet of Things Conference in Brussels, discussing the age old battle of innovation versus risk.

The debate centred around 4 main topics: Security, Privacy, Standardisation and Legislation. When you consider that a recent survey by PricewaterhouseCoopers estimated that the average cost of a single security breach is £1.46-£3.14 million and the fact that our World Quality Report highlights that customer experience and reputational management are currently top of the CIO agenda, it’s not surprising that these were the hot topics in discussion.

With Gartner predicting that 6.4 billion connected things will be in use worldwide this year, up 30% from last year, IoT Security Testing should be high on everyone’s agenda and none more so than the Public Sector. This view is compounded by the Chief Scientific Advisor, Mark Walport’s Report: The Internet of Things, making the most of the Second Digital Revolution, where he says, “Departments should recognise that Internet of Things applications will be shaped by disruptive small enterprises, as well as by large companies. In order to fulfil their potential, these projects will require a culture of innovation, testing, learning and scaling.”

To enable accurate threat modelling and security testing it’s essential to consider all the avenues potentially open to motivated and capable threat actors, hackers and Cyber Criminals, such as access by exploitation of the universal plug and play protocol; stealing information and identity by exploiting default passwords; causing physical harm or loss of life by, for example, interfering with healthcare devices and information; and affecting the economy by rendering IoT devices inoperable or interfering with business transactions.

Shift Left for Security

Firstly a shift left approach to security and security testing is a necessity, enabling early anticipation of possible threats, testing early in the development lifecycle and at every stage to detect bugs, back doors and weaknesses. The Public Sector is becoming increasingly aware of this need with Sir Mark Walport advocating that the scale of IoT connectivity demands a new approach to security, bringing it to the forefront of decision-making.

As to how we test the security of the IoT in the Public Sector here are some essentials to consider:

  • Production hardware schematic review and verification to ensure that any previous test functionality that could be exploited for research purposes has been removed.
  • Base Platform Analysis to determine whether the required security properties, features and configuration have been implemented.
  • Network Traffic Analysis to ensure that no data can be intercepted or modified or has been left unencrypted.
  • Interface Security and Negative testing to ensure all functional security requirements are operational.
  • Verification and Negative testing of functional security requirements, design and architecture.
  • Security focused static and dynamic code reviews should be performed, with particular focus on the most sensitive or security impacting components.
  • End-to-end security testing, product or code assisted penetration testing.

Other Considerations

Even if a shift left approach to IoT testing is implemented there are still other barriers to testing that we need to address such as: an industry-wide lack of IoT-specific testing skills; difficulty accessing devices at reasonable times, replication being impractical and expensive, multi-party vendors and ownership of components, testing the right thing at the right time and properly determining responsibility and accountability. These are all issues that the government and their test partners are going to have to address in order to create a successful Security Testing strategy.

For a more in depth look at the elements of security testing take a look at Sogeti’sSecurity Testing Services and our Cyber Security Services.


Gary Moore AUTHOR:
Head of Delivery for Government Testing Services

Posted in: A testers viewpoint, End to end testing, Innovation, Internet of Things, IT Security, IT strategy, Research, Security, Software testing, Testing and innovation      
Comments: 0
Tags: , , , , , , , , , , , , , ,


The Buck Stops with Effective Quality Assurance

We only have to compare last year’s World Quality Report (WQR) to this year’s, to see that economic improvement has caused an interesting shift in Public Sector testing priorities. The 2014/15 report highlighted an overriding need to reduce costs whereas our latest report for 2015/16 places customer experience (82%) and security (79%) as the two top strategic priorities for IT this coming year.

This change in emphasis is highly visible when considering examples of current Government investment. There are many automation and self-service projects focused on improving customer experience (CX) at the same time as cutting costs, and the Cyber Security Capacity Building Programme is funding projects to support huge economic and social value from secure cyberspace.

Modernisation and digital transformation programmes are without a doubt enabling the Public Sector to engage more closely with citizens and businesses. However the QA and testing function needs to ramp up to bring the growing number of new systems and applications quickly and securely to market. In fact 39% of IT Executives in the Public Sector have confirmed that they find the implementation and testing of multi-channel applications very challenging. Indeed a lack of test time is one of the biggest challenges with mobile and multi-channel applications (46%). In addition, when looking at security it is the Public Sector that is performing the least amount of systematic security testing at 57%.

Cost reduction and industrialisation still remain at the centre of the Public Sector agenda, but the latest report shows the budget allocated to Testing has actually increased from 25% to 33% of total IT spend. If the proportion of spend allocated to ensuring application quality and performance continues to rise at this rate, then it will reach 40% by 2018. This underlines the increasing strategic importance of this function.

A Cheaper, More Flexible Managed Test Service

Within this cost-driven, yet customer focused testing landscape we understand the importance of adopting new techniques and new methods to enable Public Sector leaders to deliver on complex modernisation programmes, which are made even more complex due to the nature of operating within multi-tower, multi-supplier environments.

The reports’ findings, coupled with the recent requirements of one of our biggest Public Sector clients and the wider needs of the sector to deploy and modify applications quickly and cost effectively, were the catalyst for the creation of this new Managed Test Service (MTS), delivered by Sogeti and the Capgemini Group. It provides:

  • An assessment that benchmarks where your organisation is today and the creation of a dynamic and flexible solution that seeks to continually improve your QA and testing and meet rapidly changing goals.
  • Pre-agreed costs savings built into the programme providing a 25-40% reduction in overall testing costs.
  • Test Management that supports continuous deployment in traditional, Agile and DevOps environments resulting in a minimum 15% reduction in testing time and faster time to delivery.
  • An holistic approach with new ways of working, including using optimised test processes, shift-left techniques to find bugs earlier and tools to automate and virtualise test execution.
  • Cybersecurity Testing to protect reputation and revenue and combat the fact that the Public Sector performs the least amount of systematic security testing out of all the main industry sectors.
  • Test data provisioning with a high level of synchronisation across a wide variety of integrated apps.
  • Functional and non-functional testing with a high level of automation and less than 0.9% defect leakage.
  • Enterprise Release Management with flexible governance principles, processes and tools for a smooth transition from IT change and inception to go-live.
  • User acceptance and business process testing.
  • Business-aligned Regression Testing to minimise risk.
  • Test Assurance processes that enable you to work towards KPIs and best in class standards, whilst simplifying the complexity of the multivendor landscape.
  • A managed cloud-hosted portal to enable organisations to consume single services while delivering real-time updates to enable you to meet timescales and self-select and schedule essential services on-demand. The majority of services are also available to procure via G-Cloud.

The key to success with MTS is that it doesn’t only address the functional dynamics of a requirement; it focuses on the entire customer journey and end-user satisfaction with an emphasis on usability, compatibility, performance, security and overall citizen and business CX.

Bringing Agility to Accountability to your MTS

With 46% of the Public Sector using Agile testing and 39% adopting Test Driven and Behaviour Driven Development, organisations will be glad to know that our approach to MTS interacts with both agile development teams and legacy project teams to identify the right tests at the right time for a flexible, scalable, efficient and collaborative service. The MTS promotes automated regression testing and continuous integration, creating automated scripts that can be run during the development team Sprint process. This enables a clear view of the impact of a release across the entire estate, so that developers and testers can focus on new functionality while the integrity of the overall product can be validated. This saves time, minimises risk and alleviates the potential for reputational damage. In this way, the team delivering Sogeti’s MTS acts as a binding agent between multiple towers and an independent broker between Development and Operations.

Building Your In-House Capability

The rapid speed of Digital Transformation means that finding the right experts with the right testing skills can be a real challenge. We’ve seen some interesting trends developing, with the Public Sector recruiting from sectors with greater digital maturity such as Telecoms, coupled with a rise in specialist recruitment with 70% employing analytics experts and 48% taking on mobile testing professionals. Two other interesting developments are that the role of Chief Digital Officer is growing exponentially with 47% of Public Sector bodies already employing one and a further 21% planning on it in 2016 and the role of Chief Information Officer is morphing into that of Chief Integration Officer.

A major component of the MTS is integration testing, with the shift left methodology meaning that more bugs are found prior to the integration testing stage, so at the integration stage it’s easier to review the outcome for a multichannel experience, all of which supports the CIOs in their new role. Furthermore, one of the overriding benefits of Sogeti’s MTS is that as our transition teams shadow your existing work and build new operating processes using cutting edge testing tools, we also introduce more innovative ways of working, and transfer skills to build your own in-house capability, helping you to close the skills gap.

Scalable, On-Demand Services

Our MTS for Public Sector clients responds directly to the concerns raised by the Public Sector IT leaders who participated in the World Quality Report and the particular requirements of our clients in this sector. We offer our client’s best in class tools, highly experienced test professionals with appropriate security clearance and the expertise to optimise your testing assets. As a result, this Managed Test Service helps to improve product quality and deliver a faster time to market. The services are flexible, scalable up and down and available for self-selection on demand to ensure that our Public Sector clients can collaborate with us to create a truly bespoke MTS that meets specific requirements.

As one of our client’s CIO’s puts it, the service provides you withbetter access to the increasingly scarce IT skills that support every part of our business and our customers, improving the overall performance of our business.” In this way our MTS team can offer you greater certainty, quality, efficiency, speed and quantifiable results in times of transformation and change, with the understanding of the need to step back when certain services are not required, but always be on hand for when they are in demand again.

All statistics are taken from the World Quality Report 2015-16. You can download your free copy here: We are also hosting a webinar at 10am GMT on 9th February to share the results and the recommendations from the report:


Gary Moore AUTHOR:
Head of Delivery for Government Testing Services

Posted in: Behaviour Driven Development, DevOps, Digital strategy, functional testing, Infrastructure, Innovation, IT strategy, mobile testing, MTS, Omnichannel, Quality Assurance, Security, Transformation, User Experience, User Interface, World Quality Report      
Comments: 0
Tags: , , , , , , , , , , , ,


Thanks to Kevin Bacon appearing on your TV screen in every ad break for the past few months, you’ve heard all about the benefits of 4G but what exactly does it mean? And why should you bother to upgrade?

4G is a telecommunications term which describes the fourth generation of mobile phone communication standards. It’s already had a huge and exciting impact by delivering super-fast broadband web access, smartphone and tablet performance, mobile web access, IP (internet protocol) and telephony services. We can also expect it to significantly improve the quality of gaming, high definition mobile television, and video conferencing, and even enhance the future of TV and 3D television.

The 4G upgrade is being rolled out across the world, and Britain’s offering was launched at the start of October 2012. You may have heard 4G being referred to as ‘LTE’ or Long Term Evolution – it’s simply another name for the cutting-edge technology that’s already making substantial improvements in the way we interact with our devices and gadgets, making connections both faster and more reliable. This means that our websites will load faster and we will be able to stream large files like videos and podcasts without the annoying buffering delays we’ve become accustomed too.

Just over 55% of the UK will have access to 4G coverage by the end of June 2013, but just how much of an improvement will we really see and at what cost? The change so far has been dramatic, with services about five times faster than the existing 3G technology. In theory this means download speeds as fast as 100Mbps – although in real life it’s unlikely we’ll actually able to quite reach this rate. Nevertheless, it’s a treat!

Most of us will probably experience the mobile phone and smartphone benefits first. Sadly existing 3G handsets won’t work on 4G networks – a compatible devise is required, so unless you’ve upgraded recently it’s unlikely you’ll be able to reap the benefits. For those looking to jump on the 4G bandwagon, you can currently choose from handsets such as the iPhone 5 and the special 4G-enabled Samsung Galaxy S3 LTE, as well as some Windows Phone devices.

A mini-revolution is happening that will improve your communications no end. And now with the latest Ofcom auction complete – Hutchison 3G UK; Niche Spectrum Ventures, a BT subsidiary; Telefonica (O2); and Vodafone are set to join EE (Everything Everywhere) with their 4G offerings. This means that superfast will soon be super available and at a super price!

The industry has already begun to learn that customer experience is an important part of the revolution. For 4G customers, it’s not just about having a new gadget and the ability to receive information at a faster pace, but also the experience from initial engagement with the provider through the multitude of channels available to the customer; combined with information gathering and questions to the provider; a low pain approach when signing up to 4G; receipt of the product; and finally to enjoyment of the product and a great aftercare service.

All of these areas need to be addressed and tested by the provider, and the level of experience measured to ensure it finally validates the reasons for customers to make the move to 4G.

It’s no surprise that IT systems in a 4G world will need to support multiple business and operating models spanning multiparty services, which will enable customers to have greater control over services consumed and deal with multiple types of devices. In order to ensure that these new 4G operators make the most of the 4G transition, they will need to remove any legacy IT complexities and take a long-term view, creating systems that can thrive on partnerships.

At Sogeti, we have first-hand experience of effectively managing the transition and successful deployment of 4G technology so while we wish the latest winning bidders a smooth and profitable roll-out we are keen to get involved and share our insights and experiences.

After all, who wouldn’t want to be a part of the 4G revolution?

Gary Moore AUTHOR:
Head of Delivery for Government Testing Services

Posted in: 4G, mobile applications, mobile testing, Mobility, Testing and innovation      
Comments: 0
Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,