“ASB needed to deliver more frequent releases of our core online products, reducing time to market. We needed to maintain high levels of quality and ASB's reputation as New Zealand's best online bank whilst completing regression testing more quickly and more cost effectively. These needs were met by Tricentis and Planit, who worked with us to implement a new test automation solution leveraging TOSCA.Together we delivered test automation for Fastnet Classic following a risk based approach, far exceeding our target of a 50% reduction in regression testing time. Our expectations of the tool (TOSCA) were exceeded, and we were very impressed by the commitment shown by Tricentis and Planit throughout the initiative.” - says Raewyn Ludlam, Head of Test Services at ASB
ASB, New Zealand's first savings bank established in 1847, has now been in business for more than 160 years, and more than a million personal, business and rural customers choose to bank with it . In 1997 ASB introduced New Zealand's first internet banking site, FastNet Classic for its customers to enjoy the convenience of managing their accounts online, and ever since has won numerous awards for its innovations, most recently the inaugural Canstar award as “New Zealand's best online bank 2012”.
ASB is transforming its software development capability towards iterative software development best practices, in order to maintain its competitive edge as the innovator in the New Zealand market.
With ASB's moves towards iterative software development, the demand for regression testing had increased significantly. Manual regression testing had over time become a major scheduling and cost challenge. Previous attempts at test automation did not deliver the desired results. Test suites implemented with script-based technologies and test automation frameworks required high maintenance overheads, were complicated to use and, therefore, ASB resorted back to costly manual regression testing.
In order to address the increasing time and cost pressure ASB was looking for a cost effective and scalable test automation solution. A scalable solution that would allow ASB to engage its functional testers who do not possess programming skills in the creation and maintenance of automated test cases, thus, breaking down the traditional silos between the manual testing team and the automated testing team. Further, ASB required the solution to provide reusable and maintainable assets.
“Our TOSCA regression suite is highly maintainable, because of TOSCA’s model-based approach to test automation and its dynamic steering/functions.” - says Stephan Gianonatti, Manager for Test Automation & Performance
After performing a comprehensive live hands-on evaluation of TOSCA on various ASB systems (including two HTML portal solutions, a native WIN32 application and SAP) ASB selected TOSCA’s model-based testing approach over its in-house develop test automation framework, because of TOSCA’s ease of use, maintainability2, and last but not least, its significantly lower total cost of ownership. Model-based testing does not require the time-consuming and costly development and maintenance of test automation frameworks.
A team of four was established and tasked with automation of the core regression suite for FastNet Classic, supported by subject matter experts in FastNet Classic and online banking.
After initial training, the team of four used an incremental approach to test automation, applying a risk based test management approach, focusing on high business and technology risks in the early iterations, demonstrating weekly tangible results to the major stakeholders of FastNet Classic.
In less than five months ASB automated the core regression suite for FastNet Classic, comprising of 1,100 test cases, 11,968 test actions and 92 end-to-end business process scenarios (including multiple day scenarios) covering 1,797 verification points.
Execution effort of the core regression suite is down by 95% from over 300 hours to less than 12 hours, including error log evaluation.
“Defects are being detected much earlier than would have occurred with manual testing, owing primarily to the fact that the full regression suite, including integration testing, system testing and end-to-end business process testing can now be executed on-demand in less than 12 hours.”- says Stephan.
On demand staffing – ASB has broken the mould and is now engaging test analysts and business analysts alongside with their test automation team to develop automated test scenarios.
“Test automation has been an important strategic initiative for technology within ASB and was strongly supported by Russell Jones, Chief Operations Officer for ASB, and was included in the business plan for ASB Enterprise Services. The success of our first initiative utilising TOSCA has positioned us well to continue test automation across other key applications. The structured approach (business scenarios, test cases, and modules) brought together business and technical SMEs, testers and automation specialists. We delivered within budget for 1400 test cases as planned. Daily stand-ups allowed issues and challenges to be addressed collaboratively and promptly. Lessons have been learnt on the journey, and we are delighted with the results.”- says Raewyn Ludlam, Head of Test Services
2The separation of concerns, encapsulation and polymorphism that comes with model-based testing eliminates maintenance issues inherent in script-based test automation.