Skip to main content
 
in

  • Increase Speed to Market
    Deliver quality quicker by optimising your delivery pipeline, removing bottlenecks, getting faster feedback from customers and iterating quickly.

  • Enhance Customer Experience
    Delight your customers in every digital interaction by optimising system quality and performance to provide a smooth, speedy and seamless user experience.

  • Maximise Your Investment
    Realise a positive ROI sooner and maximise your investment by focusing your energy on high-value features, reducing waste, and finding and fixing defects early.
  • The Wellington City Council (WCC) wanted to deliver quality outcomes without breaking the bank. Find out how Planit’s fast and flexible resources helped WCC achieve this goal.

this is a test Who We Are Landing Page


INSIGHTS / Articles

Quality Products: More with Less

 2 Mar 2023 
Quality Products: More with Less Quality Products: More with Less
Quality Products: More with Less
INSIGHTS / Articles

Quality Products: More with Less

 2 Mar 2023 

Consumerisation, and an increasing reliance on digital channels, has created pressure for organisations to deliver faster or be first to market with their product to remain competitive. However, this can sometimes come at the cost of software quality meeting the customers’ demands for excellence.

This rushed approach can provide some perceived, short-term benefits. But in the digital age that we’re living in, this can end up costing more in the long run, as customers will switch as soon as they have a bad experience. And, since customer loyalty can sometimes be fleeting these days, a customer that leaves may often not come back.

To attain enterprise objectives, organisations face numerous challenges, including pressure to release at greater velocity, maintaining increasingly complex systems, meeting resilience, reliability, and security benchmarks, and keeping pace with a new digital world that is rapidly accelerating. So how will you know if your apps and software will meet customer expectations for quality?

Quicker delivery demanded

Reading the recently published Smartbear report, “State of Software Quality | Testing Latest Trends and Insights for 2022”, there were a couple of statistics that I found interesting and aligned with what I am seeing too. All the illustrations that I have used in this paper are taken from this report.

Whilst the demand from C-level Executives may be to deploy quicker, it is looking like the current trend is to slow down, as displayed in the graph below, where weekly, monthly, and quarterly releases have all increased.

The reduction in quality or, simply put, the increase in production defects, both functional and non-functional, may not be the only contributors to this slow down. However, this is what I am going to explore in the rest of this article, as well as some of the actions we can take to reverse this trend.

Understanding your customer

As I stated in the introduction, it is important that you understand what your customer expectations are. Do you have several stakeholders that may have conflicting expectations? If so, how do you decide which takes priority? Or even more basic than that, who are your customers? Or do you want to change the profile of your customer or market to a wider customer segment?

There are a couple of suggestions here that I would make:

Personas

A persona is an invented character who represents a group of customers or stakeholders, and provides the development team, as well as other interested stakeholders, with a person to focus on as a client for the product. It is normally more effective and efficient to define what might benefit a specific person rather than to have an abstract group or market segment in mind.

Personas are about more than user experience (UX). They address value propositions, customer expectations, and user satisfaction. Personas can help to ensure that the development team considers the reason for the product rather than just the technical challenges of building it.

Typical personas will have details like name, gender, age, family, income, home town, workplace, education, qualifications, attitudes, behaviours, personal preferences, expectations, and more. They will even have a face so that you can easily visualise them.

Customer Journeys

Once you understand your personas, you need to understand how they are going to interaction with your systems or apps.

The customer journey is the complete sum of experiences that customers go through when interacting with your company and brand. Instead of looking at just a part of a transaction or experience, the customer journey documents the full experience of being a customer.

The four elements of the customer journey are:

  • Audience engagement.
  • Leads converting into customers.
  • Nurture the customers.
  • Fulfill the customer expectations.

These are only two techniques you could use, but once you have a better understanding of your customer, these elements can help you match your releases to their expectations. This drives the product risk, and therefore, the severity you give to both testing and defect fixing to understand when is enough or when to stop. It also ensures that you are not developing software that is of no or little use to them, which will slow down your release cadence without adding a value.

Risk vs reward

As stated above, you need to understand your product risk, and this should be your key driver for writing requirements, design, and development, as well as for your testing. Do not confuse product risk with project risk (e.g. environment ready on time, team have the required skills, requirements not signed off in time, delivery on development not on time, etc).

Start with your highest risk product features and work in small cycles of the delivery. Do not wait until “all” your requirements have been written before starting design, build and test. Adopt an Agile way of working across the whole team with elevated levels of collaboration so that the team are all working to the same release goals, in small increments linked to your product risk, Epics, and User stories.

Any assets you create which you do not consume soon after they are created are waste. They become out of date, not maintained, and in some cases, not even used. Have you ever got to close to the end of a delivery and the stakeholders start descoping? Requirements, that had taken days, weeks, or even months to gather and document, including large numbers of business people, are now not deemed as important.

Prioritise requirements/user stories

For this challenge, we prioritised our user stories and features, so we were clear about which ones we would work on in the next iteration. We agree upon this at our iteration planning meeting and then link our tests and defects to these with labels in our tool.

Less tests, better quality

We all know that it is impossible, cost prohibitive, and does not support the “need for speed” demanded today to test every combination or even all tests before a release can be made. This is even more the case when you are working in Agile teams, where the release cycle may be two weeks, one week, a day, or even hours.

When I am conducting quality/test audits, or even as part of Test Process Optimisation reviews with customers, I will typically see at least 20% of written tests that are not even executed or give no value as they are just duplicate tests. This is waste and one thing that we can easily avoid.

If you only write the tests just in time and as you are about to consume them, this is an immediate saving of 20% of your test creation effort/cost. On the flip side, I often see only “happy path” tests with no edge cases/negative scenarios. Whilst the latter can grow your test set exponentially, we do need to include some - end users do not always do what you expect them to do!

We need to look at smart tests that are based on mission critical features or paths through your solution. We also need to be moving from test coverage to application coverage, whether that be a system or end-to-end journeys. Focus the testing on the high risk areas and do testing in depth here with breadth testing elsewhere.

Pathfinders

One of the tactics that we used on our team was something that we called “Pathfinders”. As the name suggests, we wanted to find a path through the system under test to get the associated tests passed.

The situation that we found ourselves in was that we had hundreds and hundreds of open defects, so much that we could not see the “wood from the trees”. The testers and the stakeholders were getting frustrated as we were running hundreds of tests and not getting them passed.

We would get defect fixes back, retest the defect, and if it passed, re-execute the linked test. We would get to the next step or a couple of steps further and then it would fail again.

So, we were doing lots of work, but the daily metrics did not show that we had made any progress as the tests were still marked as failed. The defects were getting closed, but we were not able to give confidence that the feature/path was ready to be released.

Anyone recognise this situation? I am sure we’ve all been in it at least once. Therefore, we had to do something differently and adopt a different approach.

First, we looked at which were the most important defects to fix. These were either defects linked to high risk product areas/features or those that were blocking the most tests. These were then prioritised with the Product Owner and grouped to see which defects could be related.

We then setup “war rooms” with the whole team that could fix and retest the defects/tests. This was done in real-time with the testers demoing the defects and the developers fixing or at least identifying the root causes. The defects were fixed and re-executed on the call so that we could check that the whole path in the tests passed rather than just the one defect.

Later, we created squads which were given features to complete employing this tactic. It allowed us to provide better metrics, which then allowed the team to make decisions about when we were confident about a feature, and when we could release the next product/feature to the customers.

They cannot afford to release products that do not have the highest quality and meet customer expectations.

The ultimate result of taking this approach was that the team was focused, and we brought down average defect turnaround rates from over 15 days to less than 5 days, thereby speeding up our features releases.

Automation

Another way to speed up releases is to automate your tests- Not just the regression tests, but progression tests and any other repetitive tests, such as smoke tests for environment stability or checks after a new release of code has been communicated to testers as being ready for tests to start.

Note it is important to not just smoke test the change, but also any parts of the system that may have been impacted by that change. This is why it is critical that you work with the developers to ensure that they document root causes and detail the fix. These two should be driving the tests that you run for each defect fix as well as the originating test that found the defect, plus any that you may have blocked due to the defect.

We need to be mindful that we write adaptive tests that can be used for progression testing, defect fixes, regression, and be used, in conjunction with pairing, by the quality automation engineers. Test designers are key, not the ability to write coded automated scripts. Of course, we need engineers that can script, although with modern automation tools and AI/ML, we are getting closer and closer to no code tests.

Interestingly, in the Smartbear report, testers are still spending most of their time in manual testing. Therefore, there are plenty of opportunities to move these manually run tests to automated tests, where there is a return on investment of course. It is unlikely that it is even possible or economic to automate all tests.

This second graph also highlights opportunities for automation, with a third of respondents automating only up to 25% of their tests, and 7% still only doing manual testing.

It is unclear from the above if these percentages are based on total potential tests, or only tests created by the testers that could be automated.

Again, the next graph for the first six roles, there has been an increase in time spent testing. It is unclear as to whether this is the total time in testing i.e., including test preparation, test planning, defect management, etc. or just in test execution. It is good to see these increases, and it underlines the importance being placed on testing by organisations. They cannot afford to release products that do not have the highest quality and met customer expectations.

The most significant figures for me are those of the developer, namely the time they spend testing has decreased from 50% down to 40%. Given that everyone is talking about “shift left” strategies and how delivery can be accelerated, this does not seem to reflect this.

We all know that, according to the cost escalation model, that the earlier in the development cycle that defects are found, that easier it is the fix and, of course, that it is also cheaper to fix. I would therefore expect developer testing to be increased and not the reverse trend.

Plus, given the benefits above related to automation, I would be expecting developers to be automating their tests. In fact, approaches like Test Driven Design/Development (TDD), which starts with the tests being written before any code is written, relies on automating tests. This step is important, as it means that the developer needs to understand how verify that the code works as intended before beginning.

These tests should be automated. If written well, then some of these may be re-used by testers, again speeding up our delivery. Any re-use will provide us some savings and, therefore, reduce the time and cost of testing, reducing the overall cost of quality.

Finally, this last graph shows the biggest challenges to automation as reported by participants in the survey. Up since last year, from 15% to 20%, this biggest challenge is now the frequency of application changes.

This is remarkably interesting given that most organisations are adopting Agile ways of working, which mean that applications/systems/features are going to be changing frequently, even hourly. This is definitely a challenge that teams are going to have to overcome.

One solution I believe can be used is linked back to the design of tests and its critical importance as a skill. Well-crafted tests are modular and therefore less brittle and require less maintenance. The individual module of the test is updated once, and then for all the tests that use this, they are updated too (think login as an example). A lot of the modern automation tools help with this.

Another interesting observation here is the huge reduction in the highest challenge last year. which was “not enough time to test”, decreasing from 37% to 15%. It is hard to reconcile the reason behind this from the report, but my assumption is that automation has been acknowledged, as a must have in product delivery, so teams are being given enough time to automate.

Maybe it is also due to the maturity of automation, in teams setting stakeholder expectations. In other words, it is not a “silver bullet,” nor does it instantly speed up delivery, as the automation scripts take time to create well. The benefits come in removing repetitive tasks, running tests in smaller windows particularly where they run in parallel or running tests out of hours. This frees up testers to do more exploratory or visual tests where they provide the maximum value in improving quality.

Summary

There are many approaches that we, as quality professionals, can take within both testing itself, and wider supporting and adopting an advisor role across our teams. We can introduce new approaches to deal with the situations in which the teams find themselves to accelerate delivery without compromising quality. Maybe we are even increasing quality, through finding the defects earlier so they do not have a compounding effect.

I challenge you to be creative to find those approaches that help you to do more with less and eliminate the waste instead of contributing to it.

Leanne Howard

Business Agility Practice Director

Embed Quality in your Agile Delivery

No matter where you are on your Agile transformation journey, we offer a broad range of consultation, assessment, training, and coaching services that enable your teams to deliver better outcomes faster, embed quality in every iteration, and adopt 'shift left' practices.
 
Find out how our Agile Quality experts can help you understand your Agile maturity and fast-track your journey to Agile success with quality embedded throughout. .

 

Find out more

Get updates

Get the latest articles, reports, and job alerts.