A well-established method for measuring the quantitative value and efficiency of testing is called Cost of Quality (or, sometimes, cost of poor quality). Cost of quality involves classifying project or operational costs into four categories:
- Costs of Prevention e.g. training developers to write more maintainable or secure code
- Costs of Detection e.g. writing session sheets, configuring test environments, and reviewing user stories
- Costs of Internal Failure e.g. fixing defects detected during testing or reviews, prior to delivery
- Costs of External Failure e.g. support costs associated with defective software delivered to customers
A portion of the testing budget is a cost of detection (i.e., money that would be spent even if testers find no defects, such as money spent developing tests), while the remainder is a cost of internal failure (i.e., the actual cost associated with the found defects). The total costs of detection and internal failure are typically well below the costs of external failure, proving the value of testing and justifying spend. By determining the costs in these four categories, test managers and test leads can create a convincing business case for testing.
Both the Agile Manifesto and 12 Principles closely align to the first three classifications above through making quality the whole team’s responsibility, starting at the project’s inception.
In Agile, the principles of “just enough” and “at the right time” are key and certainly apply to testing. However, these principles can cause issues for many teams, as the challenge of aligning testing and completing the appropriate amount of testing within the timebox can result in inadequate coverage or poor quality. So how can you address this with your teams?
First and foremost there are a couple of fundamental concepts to understand:
- Quality is the whole teams's responsibility
- Quality should be built in and not tested out
There are some great techniques to help with this.To start the process of building quality in, requirements need to be well written and include the right level of detail, which can be achieved by using techniques such as Bill Wake’s INVEST acronym (Independent, Negotiable, Valuable, Estimable, Small, Testable). It is also important to use a carefully crafted definition of readiness to ensure that user stories contain good attributes such as being unambiguous and measureable.
By following developer checklists such as contained within the Pragmatic Programmer (Andrew Hunt and David Thomas) they can ensure code is written in an optimised way to make it testable. Unit testing is a critical task for a developer following this checklist or if following methods such as Extreme Programming, with no code being checked unless it has already been unit tested. Taking this a step further, for teams practising Test Driven Development (TDD) or one of its many derivatives, tests should be created and understood before a single line of code is written; potentially automated.
You could, by now, be thinking “maybe we don’t need professional testers on the team”. Even if you have some well written user stories that are completely understood by the developers and they have done some testing, this is only part of the equation. Testers have a very different mindset than the rest of the team. Foremost they provide confidence in the product through an independent viewpoint. This is often spoken about as a secondary activity of testers with many people seeing their main (or sole) purpose as finding defects. I believe that this should be reframed as testers should be working with the team to move quality injection further to the left of the delivery cycle.
Testing is a risk mitigation activity.
That is not to say that testing can mitigate all risks, however we need to ensure that all test coverage relates back to appropriate risk categorisation by product delivery. This risk identification should be completed at the Release Planning meeting, but can be updated as more is discovered about the product. At this time the Test Strategy needs to be discussed and the salient points captured.
Aligning to the Test Strategy would be the Test Plan, which may only be one page per Iteration. The plan would identify who would be doing what testing, with the entire Agile team being involved. Normally developers focus on confirming that the code does what is supposed to do, whilst Business Analysts (and the Business) look to see if the system(s) support their processes. Testers add value by providing an alternative view, finding the negative scenarios or “unhappy path”, checking if it does not do what it is not supposed to do while also looking at the less common scenarios and “what if” cases.
Additionally, testers ensure that regression testing is factored into each story and across the iterations. Here the skilled agile tester can add real value and significantly reduce the cost of quality. No longer is there a pack of, for example, 500 regression tests which are run one week before “go live”, rather regression testing is done daily in Agile projects (or as often as we get new builds).
It may seem like the significant emphasis on regression testing would increase costs, but this is not the case.
By introducing quicker, more efficient feedback loops for finding defects, teams are able to reduce costs. And by collaborating closely with developers, testers can be smart about what tests are needed, only running a handful of tests, reducing the workload and costs.
A cautionary note: there are arguments that, in Agile teams, a significant amount of testing is, and should be, automated. If these scripts are well structured and in a solid framework then teams can readily run them each time there is a build, particularly if this is overnight so as not to cause an outage, bringing with it some obvious advantages. What is often not considered is that while running sweeping regression scripts may not require time or effort, analysing the results will. If there is insufficient time to analyse all the results, running the full suite of tests is potentially of little benefit and therefore a waste. You could instead schedule this whole pack to run on a slightly less frequent periodic basis.
Another area where testers add value when considering the cost of quality is around non-functional attributes which drive the different types of testing. Typically non-functional requirements are poorly defined, if at all. Here the tester can be of value by highlighting these requirements and making them testable through static reviews… Say goodbye to words like “easily” or “quickly”!
The non-functional requirements should be separate stories or acceptance criteria. For a checklist of quality attributes see ISO25010:2011 replacing ISO 9126.
Often it is not whether the functionality works exactly as specified but rather whether you can access it at all. We all know of cases where a new promotion has been launched or tickets to a popular band go on sale and, due to performance issues, people cannot access the product. Another example is the increasing instances of companies being sued because they do not meet W3C accessibility standards. Both of these examples have massive impact on brand image and result in loss of sales, mounting potentially into the millions of dollars. When you consider this against some functional issue where it may be annoying but you have a workaround, you can easily see where non-functional testing adds value.
Whilst the scenarios outlined above are also true of traditional projects, in Agile this is tackled far earlier, as each iteration will usually include some non-functional delivery and test ensuring that these issues are highlighted and fixed. This is done to break up the components as you would with functionality and deliver parts throughout the iteration, relating to the stories you have committed to for that iteration. As each component is delivered the appropriate type of testing is applied to provide non-functional coverage. As a simple example, in a process that is specified to take 5 seconds in total and is made up of three components, if you test component 1 as the first deliverable and it takes 5 seconds to process, then you have earlier warning that the whole process is not going to work as specified and can start work on optimising it.
Session Based Testing (SBT), coupled with exploratory testing techniques, is a very powerful skill that testers bring to the team, and holds a wealth of advantages for test preparation. Using session sheets can significantly reduce both the amount of documentation as well as preparation time. In doing so, the level of analysis remains the same as the tester needs to think about what needs to be tested, and the tests still need to be designed, however efficiencies are gained in capturing this information whilst maintaining scope/level of coverage. This is not to say that there is no planning involved, as the tests should still be planned prior to execution within your session. In fact you may achieve a greater level of coverage overall as more time can be spent in execution. This may manifest itself as greater testing of a particular user story or in the testing of more stories in total.
In addition to testing, testers are able to coach the team in quality assurance and create a culture within the team where quality-focused activities become second nature. The argument for building quality in not testing it out is clear; Doing so can result in huge savings in time and money, as user stories are made testable upfront, peer reviews find defects earlier and risk is mitigated through ensuring requirements are well written and testable. In fact, fixing a defect at the requirements phase is 1000x more efficient that fixing at acceptance testing.
In conclusion, there are plenty of ways that testing can contribute positively to the cost of quality. Agile certainly supports the “shift left” philosophy whereby quality activities commence at the beginning of the entire process, providing further cost savings through early testing. Test professionals also bring significant value by providing a different view from other members of the team, uncovering many defects that would likely go undiscovered and providing confidence by using the appropriate testing techniques and test types to ensure test coverage and efficiently and effectively mitigate risk.