Skip to main content
 
au

  • Increase Speed to Market
    Deliver quality quicker by optimising your delivery pipeline, removing bottlenecks, getting faster feedback from customers and iterating quickly.

  • Enhance Customer Experience
    Delight your customers in every digital interaction by optimising system quality and performance to provide a smooth, speedy and seamless user experience.

  • Maximise Your Investment
    Realise a positive ROI sooner and maximise your investment by focusing your energy on high-value features, reducing waste, and finding and fixing defects early.
  • The Wellington City Council (WCC) wanted to deliver quality outcomes without breaking the bank. Find out how Planit’s fast and flexible resources helped WCC achieve this goal.

this is a test Who We Are Landing Page


INSIGHTS / Case Studies

Better Outcomes from Better Testing

 10 Mar 2022 
Better Outcomes from Better Testing Better Outcomes from Better Testing
Better Outcomes from Better Testing
INSIGHTS / Case Studies

Better Outcomes from Better Testing

 10 Mar 2022 

How this telematics company uplifted quality by improving quality engineering practices.

Problem

This company’s innovative telematics solutions can be found in many of today’s popular cars. Through these intelligent devices, car manufacturers, fleet managers and automotive companies use smart analytics to monitor and improve their products and services.

Although their telematics solutions were already popular with their customers, the company was keen to improve quality further and unlock greater efficiencies in their operations, by optimising their testing practices.

A key challenge was that their delivery teams were building various products without a consolidated approach to testing. Also, while they were doing a great job in implementing functional requirements, they struggled with specifying and testing non-functional requirements.

Defects were often found late in the lifecycle, slowing delivery. Teams were also dissatisfied with their test management tool, as it was difficult to track traceability and requirement coverage. They suspected that these challenges may have caused some of their apps to receive lower than expected scores in the Apple and Google Play stores.

To demonstrate their ability to consistently deliver quality products and services, the telematics company had already achieved ISO 9001 accreditation for their product development. Uplifting quality was crucial to maintaining accreditation to this global standard.

The telematics company had also adopted Agile to achieve better quality at a faster pace. However, they found they were not enjoying as many benefits from agile as they had hoped.

The telematics company decided that a review of their quality and testing practices was required to understand what their teams could do to uplift their lifecycle and deliver higher quality products to their customers. They knew their teams and products would benefit from better quality engineering and testing practices - they just needed someone with the right level of knowledge and experience to put them on the right path.

Some of the questions the telematics company was seeking answers to were:

  • “How can non-functional quality-related requirements be captured earlier?”
  • “How and when do quality requirements get built into each system?”
  • “How do our teams know what's the best approach to testing each product?”
  • "What can they do to get better quality requirements in, and better system quality out, at an accelerated pace?"

Solution

Satisfied with the quality of testing Planit was already providing them, the telematics company enquired requested our support in evaluating and improving their testing and quality practices. With a long history and reputation of delivering quality engineering services and leadership, we stepped up to evaluate the testing practices of the telematics company’s delivery teams and implemented a range of improvements throughout the organisation.

We started with a three-week test practice review (TPR) and identified an initial set of 22 recommended improvements. This was followed by a six-month engagement to implement each improvement, including a range of best practices within each delivery team and throughout the lifecycle.

As stated in the problem statement, one of the key challenges was that delivery teams were conducting non-functional testing late in the lifecycle, which introduced significant security, performance, usability and resiliency/availability risks. Finding those defects late in the lifecycle was proving to be a costly exercise that posed a significant business risk.

The lack of non-functional requirements also meant that test plans, strategies and estimates for in-flight projects were insufficient, focusing predominantly on functional testing, and therefore lacking realistic timelines and budgets. In addition, stakeholders were often not informed of the test approach that was being taken for each product, which made it difficult for them to trust that the right testing was taking place.

To resolve this, we introduced quality risk assessments that enabled teams to rapidly identify quality risks and corresponding non-functional requirements much earlier in the lifecycle. By providing training and mentoring on those practices, we enabled the telematics company to fill the gap in their testing, thereby reducing risk for their business and their customers.

This also allowed them to socialise and plan a much more complete approach to testing with their customers earlier in the lifecycle. This was further supported by defining a standard set of non-functional requirements, and likely non-functional testing costs, as a starting point for all future products.

We also introduced lightweight test plan and strategy templates that allowed teams to design a test approach covering both functional and non-functional testing in a short timeframe. We supported this with training and mentoring in test management practices to uplift the oversight and governance of testing across the lifecycle. Estimated cost savings were in the vicinity of $10,000 per year.

We also introduced the use of security testing tools, to enable much earlier security and penetration testing, reducing the number of security vulnerabilities and reducing the overall cost of quality. This also allowed later phases of outsourced penetration testing to focus on more complex security risks, providing greater return-on-investment. We supported this with security testing training for testers, developers, project managers and delivery leads to bolster their ability to secure their systems much earlier in the lifecycle.

Similarly, to support their teams in conducting the earlier performance testing, we provided their delivery teams with training in this type of testing and showed them how they could introduce it much earlier in their lifecycle.

There was also little evidence of previous cycles of disaster/recovery, availability and resiliency testing. We assisted their infrastructure team with developing a test plan for this testing along with high-level test cases, with a recommendation that their team record and report test outcomes to ensure evidence was available for future customers.

We also supported the telematics company with introducing usability testing for their apps, and with developing guidelines for usability testing, which allowed them to identify usability issues much earlier in the lifecycle.

We also supported the telematics company with uplifting their User Acceptance Testing and supported their customers in gaining a better understanding what is expected of them during testing, and vice versa.

The telematics company was also challenged by their existing test management tool, which was integrated with Jira, but could not provide a traceability report identifying which requirements had passed or failed testing across a release. We assisted with developing a business case and pilot that proved that Tricentis qTest would provide with significant cost savings of over $32,000 per year. The tool was subsequently adopted by the organisation.

For in-flight projects, we conducted a test automation review to enhance tests that had been developed in Postman for one set of APIs, and we assisted with the introduction of SoapUI suite for testing another system’s APIs. Recommendations for extending test coverage were offered to each team, with positive improvements being made in the automation approach for each project. 

We also noted that testers and developers had a different understanding of unit testing, with neither being fully aware of each other's viewpoint, nor having a full appreciation of the full suite of early tests could be carried out. To resolve this, we provided training sessions for developers, testers and BAs on black-box and white-box test case design techniques to support more cost-effective testing much earlier in the lifecycle.

A test documentation review was carried out for several projects to provide practical ideas on how to improve and streamline test documentation while still maintaining high quality and continuing to meet ISO 9001 accreditation requirements. We created a variety of lightweight templates to support testing and to provide greater visibility of test plans, strategies, test progress and outcomes.

We also supported the telematics company by facilitating a corporate risk assessment with their executive team to identify a comprehensive set of risks that the organisation could face, allowing them to identify treatment plans for the reduction of each risk.

We also assisted the organisation in understanding what quality means to them by running a survey with staff throughout the organisation, resulting in the creation of colourful “word clouds” that could be showcased on office walls, to remind staff of the importance of testing and quality to them and their customers.

Each improvement was captured on a Testing Services Wiki, including processes, templates, and training materials. This allowed existing testers to utilise the practices for their products, and supported when new testers joined the organisation. Training and mentoring were provided throughout the engagement, to support teams with embedding each improvement into their Agile lifecycle.

We also implemented improvements to several sections of the company’s Business Manual to better support new staff joining the organisation and assist them with maintaining their ISO 9001 accreditation.

Outcome

We implemented a total of 23 improvements across the telematics company delivery teams. Each improvement assisted the telematics company in fulfilling its goal of uplifting system quality and accelerating delivery on a variety of in-flight projects.

By adopting our best practices, delivery teams improved their:

  • Understanding of product quality risk, leading to much earlier and more effective non-functional testing, including performance, usability, and security testing, which improved software quality on several projects.
  • Approach to performance, resiliency/availability, and security testing, strengthening their understanding of these approaches and enhancing the company's ability to satisfy ISO 9001 requirements.
  • Test case traceability and test documentation timelines, following the introduction of the qTest test management tool, security testing tools, improved test automation approaches and improved test planning templates. These improvements alone helped save the company over $32,000 per year.
  • Visibility of test plans, strategies, estimates, progress, and results.

Other key improvements following our engagement included better:

  • Test planning on projects from industry leading test management practices.
  • Understanding by project managers and the PMO of what is typically required in non-functional testing with the help of standard non-functional requirements and typical high-level estimates.
  • Staff understanding of the company’s ISO 9001 accreditation via an uplift to the Business Manual.
  • Executive management understanding of corporate risks relevant to their organisation via the introduction of a new corporate risk identification process.
  • Motivation of staff throughout the company by identifying what quality means to them and their customers via brainstorming sessions and resulting word clouds.

Each improvement was supported by workshops, training, mentoring, templates, and documentation reviews with staff responsible for testing and quality, including testers, developers, BAs, delivery leads, project managers, business development managers and executives. These enhancements were retained in a Testing Practices Wiki, including processes, templates, and training materials for each improvement to ensure each practice would continue to benefit future teams’ products and customers.

Key Outcomes:
  • Uplifted system quality and accelerated delivery across 23 improvement areas.
  • Earlier and more thorough understanding of product quality risks and earlier non-functional testing.
  • Improved test management and security testing tools, with cost savings of over $32,000 per year.
  • More thorough test case traceability and faster reporting.
  • Better visibility of test plans, strategies, estimates, progress, and results.
  • Consolidated approach to testing across the company.
  • Improved ability to meet ISO 9001 requirements.


Delivered:
  • Quality Engineering
  • Training
  • Mentoring
  • Staff Augmentation

Tools:
  • Tricentis qTest
  • SoapUI
  • Postman
  • Atlassian Jira
Download Full Case Study

Deliver Quality Quicker

In today’s competitive landscape, organisations expect to deliver more ambitious technical outcomes at improved efficiency. We can help you achieve these goals by embedding quality throughout the lifecycle, optimising your delivery to improve outcomes, accelerate speed, and decrease cost.
 
Find out how we can help you mature your quality engineering practices to consistently achieve better results with greater efficiency.

 

Find out more