Can Agile help with TMMi competency? This is a question I will be answering when looking at how Agile methodologies/frameworks align with gaining TMMi Level 4 accreditation.

This article is an overview of how to achieve TMMi Level 4 accreditation. For more in-depth information about Level 4, check out the full document on the TMMi foundation website.

If you have not had the opportunity to read the previous articles describing Levels 2 (PART I: Test Policy and Strategy, PART II: Test Planning, Monitoring and Beyond) and 3 (PART I: Test Planning, Monitoring and Beyond, PART II: Non-Functional and Peer Reviews), I encourage you to check them out to understand the TMMi path towards Level 4.

TMMi Level 4 Measured

It is important to remember there is a natural break between TMMi Level 3 and TMMi Levels 4 and 5. There also continues to be significant controversy over the value of moving an organisation to TMMi Level 4 and 5.

Agile organisations are recommended to critically choose and only do the practices at TMMi Levels 4 and 5 that matter and have added value. Whilst TMMi is comprehensive, to be successful, organisations must identify the key testing practices and improvements requiring focus.

Hereafter information will be provided regarding TMMi Level 4 and how one can gain the values from these higher TMMi Levels, and its practices by using them less formally with an Agile approach.  In my opinion, Agile is far more disciplined than traditional approaches, as the timelines are condensed and the key is to eliminate waste.

Firstly, somewhat contrasting with what has been stated, one may also consider already selectively using TMMi Level 4 practices with Agile techniques to address key business objectives while trying to achieve TMMi Level 2 or 3.

You do not need to wait and you shouldn’t. Be flexible and do not get stuck on TMMi Level 2 and 3 testing practices. As such, use TMMi in a more continuous mode to some extent, and don’t use the maturity levels too strictly.

Process Area 4.1 Test Measurement

The purpose of the Test Measurement is to identify, collect, analyse and apply measurements to support an organisation in objectively evaluating the effectiveness and efficiency of the test process, the productivity of its testing staff, the resulting product quality, and the results of test improvement. As such, the test organisation will develop and sustain a test measurement capability that is used to support management information needs.

Agile supports getting commitment of those who must perform and accurately measure work in projects. However, Agile approaches don’t tend to provide much support when it comes to stepping back to ensure the right measures are used across the organisation.

Agile team autonomy, self-organising, and managed teams seem to go against organisational alignment. However, it is important to decide which boundaries must be kept across all teams, and where teams can go their own way.

SG 1 Align Test Measurement and Analysis Activities

A measurement program should be highly focused and only things that matter should be collected. Start by defining clear objectives for the measurement program and revisit those on a regular basis.
Next, discuss with stakeholders which test metrics will truly support these objectives. Based on this brainstorm, a limited set of core metrics will be defined.

Specific context-relevant measures are needed based on business needs. The purpose of measurements is to guide decisions. Use small empowered teams to derive meaningful measures, and then review and refine them in short cycles.

With a focused test measurement program starting from added (business) value, the TMMi Test Measurement process area is totally in line with the Agile principle of simplicity. Ensure there is a lean test measurement and data collection strategy within the organisation.

It’s worth pointing out that one can evolve a test measurement program by starting with just a few focused measurement objectives and a few key measures, and expand or modify them later as business needs evolve and change.

Note that within Agile the focus will be a more team and systems-based thinking. This may result in a corresponding broadening of some of the metrics to the team and overall system, rather than being confined solely to the specifics of testing itself. This may well lead to more challenges at the measurement analysis phase.

Of course, the metrics that will be defined, and for whom data will be collected, will depend on the measurement objectives, but some examples of typical testing metrics for Agile teams are:

• Defect cycle time - Agile teams should strive to fix defects as quickly as possible. In fact, one of the main aims of the collaborative Agile approach is to fix defects quicker so that software is released sooner.
• Defect spill-over (number of defects deferred to a future release) - Agile teams aim to produce working software with each iteration. Defect spill over measures the defects that don't get fixed during a given iteration or Sprint by simply counting the defects remaining at the end of each Sprint or iteration. This is, in some Agile organisations, referred to as technical debt.
• Number of defects found by people outside of the team.
• Number of customer support requests.
• Percentage of automated test coverage.

SG 2 Provide Test Measurement Result

Once the relevant measures are defined based on business needs, the specific practices within this specific goal are typically all applicable in largely the same way as within a traditional organisation. The only difference being that this will be done with a different mindset to keep the practices as lean as possible.

The necessary test measurement data is collected and checked for completeness and integrity. Subsequently the data is analysed as planned and the results are communicated to all relevant stakeholders.

Process Area 4.2 Product Quality Evaluation

The purpose of the Product Quality Evaluation is to develop a quantitative understanding of the quality of products, and thereby support the achievements of specific projects’ product quality goals.

SG 1 Measurable Project Goals for Product Quality and their Priorities are Established

In essence, there is no real difference to identifying and setting measurable prioritised goals for product quality in an Agile context compared to doing this in a traditional environment. The overall objective is to contribute to satisfying the needs and desires of customers and end users for quality products.

There are multiple options for doing this in an Agile context. Some use Definition-of-Done (DoD) to define measurable goals for product quality, others use features, epics or user stories with acceptance criteria to make them measurable.

The choice (way of working) depends, amongst other things, on the nature of the product quality goal. Refer to ISO25010 for an overview of quality attributes.

DoD is a comprehensive checklist of necessary team activities and/or work products that ensure that only truly done user stories are delivered, not only in terms of functionality, but in terms of product quality as well. The advantage of stating product quality goals within the DoD is that it keeps them visible to, and therefore actionable by, the entire team.

There is also greater collaboration with the end customer in Agile, so there is likely to be a better understanding of what quality means to the customer rather than an assumption of what it is.

SG 2 Actual Progress towards Achieving the Project’s Product Quality Goals is Quantified and Managed

Once the quantitative product quality goals have been set, they will be monitored, controlled and possibly adjusted when necessary. The same methods and techniques can be used for tracking and managing progress towards achieving the product quality goals as already explained and described in the process area 2.3 Test Monitoring and Control at TMMi Level 2.

Progress will be measured and discussed daily through the daily standup meeting reporting on progress, quality and risk. In addition, Product Quality Evaluation is typically also supported by the Test Measurement process area, which provides a measurement infrastructure.

Process Area 4.3 Advanced Reviews

The purpose of the Advanced Reviews, building on the practices of the TMMi Level 3 process area Peer Reviews, is to measure product quality early in the lifecycle, and to enhance the test strategy and test approach by aligning peer reviews (static testing) with dynamic testing.

SG 1 Coordinate the Peer Review Approach with Dynamic Test Approach

In an Agile environment, the approach to ensuring product quality should be one in which all testing related activities, both static and dynamic, are organised as one coherent and coordinated approach. The Agile team is expected to take the lead in establishing a coherent and effective integrated approach. In Agile, there is in fact encouragement to find defects early (shift left), so static testing is critical.

SG 2 Measure Product Quality Early in the Lifecycle by Means of Peer Reviews

At TMMi Level 4, the organisation sets quantitative goals for software products and related work products. Peer reviews play an essential role in achieving these goals.

Whereas at TMMi Level 3 peer reviews are mainly performed to find defects, the emphasis is now on measuring product/document quality to control product quality early in the lifecycle. Review practices are enhanced to include steps such as sampling, applying exit criteria, and prescribing rules.

This is clearly a practice that will fit with a high maturity Agile organisation. Another example would be to use the INVEST rules [Wake] for user stories as a Definition of Ready (DoR), and to measure the quality of the user stories. Having a DoR means that stories must be immediately actionable.

SG 3 Adjust the Test Approach Based on Review Results Early in the Lifecycle

When peer review results and dynamic testing are coordinated, early review results and data can influence product risks and the test approach. As already stated at specific goal 1 of this process area, in Agile, quality is a team effort and the results of verification and validation-based activities will be discussed at team meetings.

Therefore, the principle of using early testing results to influence successive testing activities has an almost perfect match with the way-of-working within Agile teams and projects.

To summarise

Agile practices can support the attainment of TMMi Level 4 and, in fact, a lot of the extra discipline required to be successful are definite enablers. Whilst measurement can seem less crucial in Agile projects, it is the fact that critical success factors are being measured to ensure customer value and satisfaction.

If you want help in optimising your organisation and projects, assessing their maturity against industry standards or get a baseline then roadmap to improve, then contact us.