Skip to main content
 
nz

  • Increase Speed to Market
    Deliver quality quicker by optimising your delivery pipeline, removing bottlenecks, getting faster feedback from customers and iterating quickly.

  • Enhance Customer Experience
    Delight your customers in every digital interaction by optimising system quality and performance to provide a smooth, speedy and seamless user experience.

  • Maximise Your Investment
    Realise a positive ROI sooner and maximise your investment by focusing your energy on high-value features, reducing waste, and finding and fixing defects early.
  • The Wellington City Council (WCC) wanted to deliver quality outcomes without breaking the bank. Find out how Planit’s fast and flexible resources helped WCC achieve this goal.

this is a test Who We Are Landing Page


INSIGHTS / Articles

Improving Software Quality and Reducing Cost through Quality Engineering

 10 Nov 2022 
QE team shifting quality to the left and right QE team shifting quality to the left and right
QE team shifting quality to the left and right
INSIGHTS / Articles

Improving Software Quality and Reducing Cost through Quality Engineering

 10 Nov 2022 

The terms quality engineering (QE), quality assurance (QA), and testing are often used interchangeably within the IT industry. Whilst there is a direct relationship between the terms, they are three distinct concepts, each offering an increasing range of benefits.

The aim of this paper is to explore the relationship between them, and to clarify:

  • How transitioning from testing to QE can uplift the quality of your systems,
  • How QE can reduce software development and delivery costs,
  • How QE can accelerate delivery of systems to production, and
  • How testers that transition from testing to QE can become even more valuable to their teams, their companies, and their industry.

What are QE, QA, and Testing?

Let’s start with clarifying the relationship between QE, QA, and testing.

Testing is the process of evaluating a system against its requirements. It is used to determine whether requirements have been met, to detect defects, and to improve confidence in release readiness. It includes processes for test planning, test design, test execution, and reporting.

Quality assurance is the process of preventing and detecting defects to ensure a system meets its requirements and to review and improve lifecycle processes. QA includes testing (defect detection), and a range of defect prevention and process improvement activities.

Quality engineering is the process of designing a quality-optimised development lifecycle that builds quality into each system, and that determines whether quality is present, as early and frequently as possible. QE includes and goes well beyond QA and testing, incorporating a range of requirement, design, architecture, development, test, release management, operations management, production management, and maintenance activities that shift quality to the left and to the right, across the full software lifecycle. QE uplifts quality across the lifecycle, and utilises a range of tools, techniques and metrics to optimise and continually improve the process.

Figure 1: Relationship between QE, QA, and Testing

Techniques for QE, QA, and Testing

Another way to illustrate the differences is to consider the practices that can be utilised within QE, QA, and testing. As this diagram shows, testing is one of many activities that can be applied to improve software quality.

Figure 2: Relationship between QE, QA, and Testing

Improving Quality

How does transitioning from testing to QE benefit our industry? Put simply, you can’t test quality into a system – you can only build it in!

If companies focus their quality budget on testing after requirements have been written, and as their only mechanism for improving software quality after development has commenced as their only mechanism for improving software quality:

  • They miss the opportunity to prevent defects from being introduced in the first place, such as requirements and design/architecture defects
  • Common requirement issues occur, such as non-functional requirements that are untestable and unmeasurable, unspecified API requirements, unspecified end-to-end business workflows, and missing requirements, leading to significant system deficiencies
  • They get stuck in a costly (and avoidable!) rework cycle of: delivering code to test > testing > finding defects > fixing defects > retesting > finding more defects
  • They can sometimes place blame for poor quality and late delivery on testers, when the actual cause is defects that were introduced and not detected in upstream phases
  • Most of a tester’s investment in quality occurs too late in the lifecycle, with little ability to influence quality
  • As supported in the accompanying industry statistics, it is one of the slowest, most expensive, and least successful ways to build software.

In short, it places too much emphasis on testing, when software quality improvement comes from creating better quality requirements, design, architecture, code, configuration, release management, and production management practices.

Through adoption of QE practices, our investment in quality shifts to the left and to the right across the full lifecycle of each system. Software development teams can “get it right first time” by engineering products that meet their requirements faster and with less cost. The goal is to:

  • Ensure teams specify requirements that are correct, consistent, testable, measurable, and with far fewer gaps and defects
  • Design, architect, code, and configure systems that meet requirements, with reviews and quality coaching to avoid defects
  • Test as early and as frequently as possible, with sound coverage of requirements and quality risks, and with traceable evidence and automation wherever valuable
  • Run product quality risk assessments to identify the ways in which a system might fail in production, so that relevant teams can specify, design, and test systems to avoid such defects
  • Evaluate requirements, designs, and systems against the business case, to ensure desired business right value is realised
  • Review and test the go-live process (e.g., via dress rehearsals, dry runs, and reviews of go-live plans)
  • Monitor systems in production, with automated alerting and controls in place to prevent failure
  • Gather feedback regularly from end-users in production
  • When using external software development vendors, ensure their contracts require early testing and QE activities, including unit, integration, and system testing, along with inspections, walkthroughs, and/or reviews, to ensure defects are detected as early as possible
  • Utilise tools, technology, automation, and metrics to optimise the lifecycle

QE distributes the process and the responsibility for quality across the full lifecycle, truly making quality everyone’s job. QE It is also about planning for and evaluating quality at every step of the lifecycle.

QE includes testers, business analysts, developers, architects, product owners, project managers, release, operations, the business, and end-users in the process of evaluating quality throughout the lifecycle. In doing so, it optimises feedback loops, reduces rework, reduces development costs, and accelerates product delivery.

Changing Roles

A critically important aspect of shifting left is engaging quality-minded employees like testers early in the lifecycle, including in requirement reviews, design, and architecture reviews, and code reviews; quality coaching for the team; product quality risk assessments; and earlier planning and strategising of QE activities. Engagement should be early and upfront, while requirements are still being written, and ideally while business cases and vendor contracts are still being written. This enables an up-front investment and improvement in quality, and ensures the budget, schedule, resources, and delivery strategies include QE practices that allow the organisation to build quality in and achieve their desired quality outcomes.

It is also important to adopt shift-right activities for QE, such as dress rehearsals, dry runs, release readiness reviews, and the uplift of release and operations management and maintenance processes.

Testers possess inquisitive, questioning minds. They spend their working lives considering how systems might fail, and working backwards from that, to understand how software and artefacts can be reviewed and tested to improve quality. They also often have skills in writing requirements, doing system design and coding, and are often skilled at writing documentation.

Testers are also skilled at discussing quality with stakeholders from across the full lifecycle, from business analysts to architects, developers, product owners, project managers, business representatives, operations and beyond. They often already have a holistic quality mindset. Testers are therefore well-placed to transition into roles in quality engineering, that support and enable the enhancement of software quality across the full lifecycle.

Not only does transitioning from testing to QE support each organisation in achieving its software quality goals, it also enriches the skillset of each tester, which can be very motivational and uplifting. This makes each tester more valuable to their teams, organisation, and industry.

Changing Mindsets and Language

Transitioning from testing to QE requires a change in the language and the mindset of our IT industry.

Currently, when a new system is built, it is common for testers to be asked to write a test plan and strategy that describes how they will test the system. Such artefacts usually have very limited coverage on how the quality of requirements, design, or architecture will be reviewed or improved. Typically, there will often only be a very brief mention of unit testing being the responsibility of the developers, with little to no responsibility for quality or testing spelled out for any other phase or role across the lifecycle.

In today’s fast-moving digital world that embraces multiple platforms and Software-as-a-Service (SaaS) products, testers are often brought in to plan and strategise the testing after most requirements have been written, after a new platform has already been purchased, and just prior to configuration and customisation. Even in Agile environments, by the time testers join the team, it is common for much of the product backlog to already be built into the chosen system, and for the product to have already been purchased. By then it is too late to identify and resolve early quality issues in the requirements, design, and proposed solution, shifting the investment in quality to too late in the lifecycle.

A much more optimal way to build software is to assign quality engineers to facilitate an early whole-team approach to strategising and planning quality and testing across the lifecycle, by creating a “product quality plan” (rather than a test plan). In addition, the organisation itself should adopt a “quality policy” (rather than a test policy) and an “organisational quality strategy” (rather than a test strategy). This enables the full team to take responsibility for the quality and testing of each system, and allows everyone to actively contribute to the discussion on how to ensure quality is built into each product.

Throughout our industry, we should create artefacts that focus on quality, rather than testing alone.

Figure 3: Changing mindset and language

Rationalisation

Key drivers for transitioning from testing to QE are the time, cost, and risks saved, and the improvement in software quality that can be achieved. A range of metrics illustrate this.

The Standish Group have been reporting on project success and failure rates since 1994, for over 50,000 projects in total, and an average of 5,000 per year. Throughout this time, the number of projects that have succeeded (on time, on budget, meeting requirements), been challenged (over budget, over schedule, and/or not delivering required features), and failed outright have not changed significantly.

Figure 4: Chaos Report Project Resolution Rates (Standish Group, 1994 to 2020)

Two of the top reasons reported by the Standish Group for project failure are incomplete requirements and a lack of user involvement in IT projects.

A separate review of over 5,400 projects by McKinsey and Oxford University found that within their study:

  • Half of all large-scale IT projects (> US$15 mil) had “massive” budget overruns, running on average 45% over budget, 7% over time, and delivering 56% less value than they had planned – with a combined total overrun cost of US$66 billion
  • The longer the IT project, the more likely it was to run over time and over budget, with each additional year of project duration increasing cost overruns by 15%
  • 17% of IT projects went so badly, they threatened the company’s very existence

According to the Project Management Institute, 39% of companies that experience project failure cite inaccurate requirement gathering as primary cause of failure. In addition, a recent study by the Consortium for Information and Software Quality (CISQ) found that in 2020, the total cost of poor quality in the USA alone was $2.08 trillion, on top of another US$1.31 trillion in technical debt residing in severe defects yet to be corrected.

Add to this the oft-quoted statistic that a defect found in production is 30 times (and up to 100+ times) more expensive to identify and fix than a defect found in the requirements phase. The goal of QE is to invest more in finding defects much earlier in the lifecycle, so that the overall cost of quality and delivery is far less overall.

Figure 5: Cost of defect repair (NIST), with the comparative goal of reducing the overall cost of quality via QE

The Economics of Quality

Another way to look at the economics of quality is to consider the effectiveness of early lifecycle quality improvement practices such as requirement, design, and architecture inspections, compared to later-stage approaches such as testing.

According to Capers Jones, later-stage approaches like testing are effective at identifying and supporting the removal of around 30% to 40% of defects respectively. Compare this to early-stage approaches like requirement, design, and architecture inspections and pair programming, which have defect removal efficiencies of 75% to 87%.

This doesn’t mean that all we need to achieve quality are early-stage QE approaches. What we do need are a range of QE approaches across the full software development and delivery lifecycle, that will help reduce defects, reduce, and rework, and accelerate delivery to production.

Figure 6: Defect Removal Efficiency of a variety of software engineering best practices (Capers Jones)

When Should Company’s Start Utilising QE Practices?

Engineering high-quality systems starts well before software product selection, design, development and testing starts. Ideally, it will begin at the ideation stage, when the business case and requirements are being specified, well before software product selection, design, development and testing starts.

As soon as the requirements start being specified, a quality engineer can be used to facilitate whole-team discussions on:

  • What quality requirements the system has
  • What product quality risks exist that could cause the system to fail in production or that could cause the project to fail
  • What the material impact of each failure could be (e.g., cost, reputation, regulatory, human impact)
  • What QE practices could be used across the lifecycle, to cover the requirements and to reduce the risks

Balance the wish list against the available budget, schedule, and resourcing, by discussing the risk of not implementing each QE practice. Any time a recommended QE practice is deemed not affordable, ask your development and/or operations teams, steering committee, and/or business and executives what the cost of not addressing each risk is. Consider the potential impact each risk could have on your company, employees, customers, end-users, shareholders, industry, Government, and the general public.

Also consider the potential human, business, reputational, and cascading impacts. Ensure risks and their treatments are reviewed by senior stakeholders, managers, and executives in your organisation, and check whether they are willing to accept any risk that is not satisfactorily treated/mitigated.

Rather than saying, “cost, schedule, or quality – only pick two”, companies should seek a balance between all three, with the material risk and impact of not delivering quality being the deciding factor on how to achieve that balance.

Summary

You can’t test quality into a system – you can only build it in. To that end, there are a range of QE practices you can utilise across the lifecycle to improve the quality of requirements, design, architecture, code, testing, release, and operational management of software systems. These can be utilised via a “whole-team” approach to quality.

QE practices can determine whether quality is present earlier and more frequently across the lifecycle, and ensure that quality truly does becomes everyone’s responsibility. Therefore, investing earlier in system quality will not only help to reduces the overall cost of delivery, and it will also enables systems to be released to production sooner, with much greater confidence, and with far fewer production defects and failures.

 

 

Tafline Ramos

Practice Director - Quality Engineering and Assurance

Deliver Quality Quicker

In today’s competitive landscape, organisations expect to deliver more ambitious technical outcomes at improved efficiency. We can help you achieve these goals by embedding quality throughout the lifecycle, optimising your delivery to improve outcomes, accelerate speed, and decrease cost.
 
Find out how we can help you mature your quality engineering practices to consistently achieve better results with greater efficiency.

 

Find out more

Get updates

Get the latest articles, reports, and job alerts.