In an earlier article, we looked at the test policy and strategy of applying TMMi to Agile projects and organisations. Here we will look at the remaining steps.
Process Area 2.2 Test Planning
The purpose of Test Planning is to define a test approach based on the identified risks and the defined test strategy, and to establish and maintain well-founded plans for performing and managing the testing activities. Note that the key to successful test planning is in upfront thinking (“the activity”), not in defining the associated test plan (“the document”).
For Agile lifecycles, two kinds of planning typically occur: release planning and iteration planning. The Test Planning process area at TMMi level 2 focuses on the testing-related activities at both release and iteration planning.
SG1 Perform Risk Assessment
Exhaustive testing is impossible, and choices always need to be made and priorities have to be set. This TMMi goal is therefore also applicable for Agile projects.
For Agile projects, a high-level product risk assessment can be performed based on a product vision document and/or at release planning. Also for each iteration, a more detailed product risk session should be performed based on the user stories or other requirements as part of the iteration planning session.
SG2 Establish a Test Approach
The test approach that is defined to mitigate the risks can cover, for example, reviewing of user stories and acceptance criteria, testing effort proportional to the level of risk, and the selection of appropriate test technique(s) based on the level and type of risk. Often the test approach is held or displayed on the team/project wiki.
Within Agile software development, testing is an integral part of the team process and an almost continuous activity. Consequently, there is no need for a specific checklist or gateway to determine if testing can, or cannot, be started. This also applies for a component going within an Agile team from one test level (e.g. unit testing) to another (e.g. acceptance testing).
Testing exit criteria (specific practice 2.4) are part of the Definition of Done (DoD). It is important that the DoD has specific test-related criteria for test coverage and product quality (defects).
SG3 Establish Test Estimates
For Agile teams, estimates for testing will be made during the iteration planning as part of an overall team estimate rather than separate ones for test and development. Typically, in Agile projects, the estimation is focused on size using story points. The TMMi specific goal and its specific practices are therefore applicable with the exception of specific practice 3.2.
SG4 Develop a Test Plan
Again, this goes back to the comment that test planning is about upfront thinking (“the activity”) and not about defining the associated test plan (“the document”). In Agile, some additional planning activities, as defined by the TMMi specific goal, need to be performed during iteration planning. The result of these activities will not be documented in a test plan, but they could be reflected on the task board.
A test plan is developed as part of the iteration plan. The result of the discussion that takes place in the context of test planning are, however likely, to be captured in a lightweight form such as a mind-map.
SG5 Obtain commitment to the Test Plan
Within Agile, the process to develop and establish a test approach and plan should be a team-based exercise lead by a test professional on the team. Product-quality is a team responsibility. As such, provided the team follows the correct process, commitment to the (test) approach and (test) plan is already an implicit result of iteration planning as a team-effort.
The reconciling of work and resource levels is not a necessary activity, as Agile teams should remain stable. The team itself commits to the scope for each iteration. The specific practice 3.2 Reconcile work and resource levels is therefore a practice that is not relevant in an Agile context. Daily stand-up’s will be used to address any issues and allocate appropriate resources immediately, or remove a deliverable from an iteration to be reconciled in future iteration/release planning.
Process Area 2.3 Test Monitoring and Control
The purpose of Test Monitoring and Control is to provide an understanding of test progress and product quality so that appropriate corrective actions can be taken when test progress deviates significantly from plan, and product quality deviates significantly from expectations.
Following the Agile manifesto and its accompanying principles, there are some things that are fundamentally different for monitoring and control in Agile projects compared to traditional projects.
From a test monitoring and control perspective, this means we are not plan-driven, but constantly reviewing our progress and results from testing, and adapt our plan and approach as appropriate. As a result, testers do not report to a test manager as with traditional projects, but to the team.
SG1 Monitor Test Progress Against Plan
Testers in Agile teams utilise various methods to monitor and record test progress e.g., progression of test tasks and stories on the Agile task board and burndown charts. These can then be communicated to the rest of the team using media such as wiki dashboards, as well as verbally during stand-up meetings. The whole team reviews the status of the task board regularly, often during the daily stand-up meetings, to ensure tasks are moving across the board at an acceptable rate.
The DoD serves as exit criteria against which progress is being measured. The DoD should also be related to testing activities, and shows all criteria that need to be satisfied before testing of a user story can be called ‘Done’.
The milestone review within Agile is at the completion of an iteration. The accomplishments of testing, for example against the DoD, will be part of the iteration review. Demos are organised with stakeholders to discuss the business value and quality of the product being delivered.
SG2 Monitor Product Quality Against Plan and Expectations
For product quality monitoring, largely the same mechanisms are used as for progress monitoring (see SG1 above). In Agile, for product risk monitoring, the focus lies on a review of list of product risks in regular meetings rather than the review of any detailed documentation of risks.
SG3 Manage Corrective Actions to Closure
Agile teams would very quickly notice issues, such as deviations from expectations, in a burn-down chart and/or lack of progression of (test) tasks and stories on the Agile task board. Managing corrective actions in Agile projects is primarily a responsibility of the self-organising team. The team can define and implement appropriate corrective actions, or escalate any issues as ‘impediments’.
Process Area 2.4 Test Design and Execution
The purpose of Test Design and Execution is to improve the test process capability during test design and execution by establishing test design specification, using test design techniques, performing a structured test execution process and managing test incidents to closure.
Although the underlying objective (“mitigate the risks and test the software”) is the same for an Agile project, the approach taken on how to test is typically very different. In Agile, flexibility and being able to respond to change are important starting points. Also, test analysis and design, test implementation and execution are not subsequent test phases, but rather are performed in parallel, overlapping and iteratively.
The level of detail of test documentation established is another key difference. Typically, more often experience and defect-based techniques are used in Agile projects. A final major difference is the level of the regression risk, which calls for more regression testing. Ideally, regression testing is highly automated.
SG1 Perform Test Analysis and Design using Test Design Techniques
In Agile, test analysis and design, and test execution are mutually supporting activities that typically run in parallel throughout an iteration. Testers are part of a team that collaboratively creates and refines user stories. Exploratory testing is widely used, where acceptance criteria are proven through the use of test ideas (part of test charters).
For automated unit testing, an approach like Test-Driven Development can be considered. For higher test levels, Behaviour-Driven Development (BDD) is a popular Agile development approach that also supports testing. Both BDD and ATTD are also highly related to test automation.
The prioritisation of tests follows the prioritisation of the user story they are covering. The prioritisation of the user stories is based on business value, where highest priority relates to highest business value. It’s important that tests are established to cover functional and non-functional risks, but also, especially in Agile, specifically to cover the regression risk.
Once these tests are established, the tests themselves often become the detail agreed-to as part of the requirements. Using this approach may suffice to achieve the intent of the TMMi specific practice on requirements management related to horizontal traceability.
SG2 Perform Test Implementation
Test implementation is about getting everything in place that is needed to start the execution of tests. In an Agile environment, detailed test procedures are not common practice. Working in a knowledgeable team, tests will most likely be documented at a much higher level of abstraction. This will suffice for those executing the tests, since it is expected that the team will have the required level of domain and technical knowledge to perform the tests.
Continuous integration is a key practice for Agile projects. Testing is a parallel and fully integrated activity to be performed by the team. It is not an independent separate activity or phase. The specific practice, SP2.3 Specify intake test procedure, therefore becomes irrelevant within an Agile team. In Agile environments, no specific test execution schedule will be created, and therefore specific practice 2.4 Develop test execution schedule is not applicable.
SG3 Perform Test Execution
Often no detailed test procedures and test logs are produced. The tests, especially the regression tests, should, as much as possible and feasible, be automated. In an Agile project, software is delivered, as a minimum, on a daily basis, and testing is an integral part of the Agile development process, whereby work is performed in mutual cooperation to deliver a quality product. Therefore, a specific intake test, as defined in specific practice 3.1 Perform intake test, is not applicable in a pure Agile environment.
The incidents that are found during testing may be logged and reported by the team. There is typically a discussion within Agile projects whether all incidents found should indeed be logged.
Particularly in the case where the team is co-located, testers need to talk with developers and incidents/defects that can be fixed immediately and included within the next build may not need to be logged, since they just need be fixed. Some teams only log incidents that escape iterations, some log it if it can’t be fixed today, and some only log high priority incidents.
If not, all incidents found are logged, criteria must be available to determine which incidents should be logged and which ones shouldn’t. Remember, the intent is to get the defect correctly fixed, not to log an incident.
SG4 Manage Test Incidents to Closure
In principle, managing incidents in Agile is simple. In cases where the incident is logged on the Agile task board, it is visualised as a defect blocking a story and its tasks from getting completed. The standard Agile way of working will be applied as with any other impediment or task that is blocking progress.
In case where a decision is made to defer the incident found to another iteration, they become merely another desired option or change for the product. Add them to the backlog and prioritise accordingly.
Process Area 2.5 Test Environment
The purpose of Test Environment is to establish and maintain an adequate environment, including test data, in which it is possible to execute the tests in a manageable and repeatable way. With the process area Test Environment, an adequate test environment is established.
Of course, using the Agile lifecycle, an adequate test environment is indispensable. Because of the short timeboxes of an iteration, the test environments need to be highly stable and available. Problems in the test environment will always have an immediate impact on the progress and results of the iteration.
SG1 Develop Test Environment Requirements
Specification of test environment requirements is performed early in the project. Early requirements specification has the advantage of providing more time to acquire and/or develop the required test environment and components, such as service virtualisation, simulators, stubs or drivers. In some Agile development projects, an initial iteration (iteration 0) is used, where the elicitation and specification of test environment requirements is performed.
SG2 Perform Test Environment Implementation
Sometimes the implementation or part if it takes place in the initial iteration. In this way, one tries to achieve what the first software development iterations already have with a (partly) working test environment.
SG3 Manage and Control Test Environments
Management and control of the test environment will take place throughout the Agile project. As a main conclusion for the process area Test Environment, the specific goals and practices are all applicable and do not change in essence. The only change is their timing in the lifecycle.
Whilst the existing TMMi methodology and assessment can equally be applied to Agile projects and organisations as is, “TMMi in the Agile World” explains how to do this using Agile terminology and the ways to apply TMMi within Agile practices.
If you want help to optimise your projects and assess their maturity against industry standards, then contact us. In the fast pace of today’s world, can you afford not to?
TMMi in the Agile World, https://www.tmmi.org/tmmi-documents/