The corporate services provider needed an external testing partner to assess the performance of the platform, as well as replicate the payroll issue. They selected Planit for the role due to our demonstrable track record and reputation with performance testing, as well as our large portfolio of supporting services that they could draw upon if needed.
After thoroughly analysing the in-scope systems, we concluded that a repeatable, production-like test framework leveraging an open source load test tool, such as Apache Jmeter, should be used to benchmark their current performance. This framework could also be used to test subsequent releases, recreate the payroll file processing issue, and confirm its impact on customers.
A challenge was that the company had limited performance testing experience or results for us to draw upon. This required us to work closely in partnership with them to fully understand the project’s performance requirements, business risks, and technical risks, and then develop a performance testing framework that would meet the key objectives as well as provide a solid base on which to build upon for subsequent releases.
To recreate the existing production issue, we began by carrying out an extensive analysis of the Internet Information Services (IIS) logs in conjunction with client workshops and risk assessment sessions. This enabled us to build a representative workload profile covering more than 30 user journeys across two key business critical applications running on the platform.
The resultant tests were configured to run for ten hours covering 90% of the daily traffic. In addition to closely replicating daily production traffic through the system, these extended load tests were crucial to correctly seed data for the batch performance testing, ensuring that the batch sizes and performance metrics were also comparable to production.
Extensive analysis of available log data was carried out post-execution to verify that the results observed in the test environment mirrored those seen in production. This verification provided confidence that the results observed could be used as a valid benchmark for subsequent performance tests.
Aside from the overall technical complexity and scope of functionality that needed to be accounted for, the main challenge of this project was the sheer amount of data involved. The testing had to take into account payroll processing across multiple customers and thousands of employees.
A well-defined approach to data management was critical to the successful completion of the project. Production logs had to be carefully analysed to identify the correct data, and a robust database backup/restore process was required to enable repeatable tests.
In light of their limited application performance management (APM) or analytics tools, our performance consultants worked diligently to guide the corporate services provider through the workload modelling process and verify that the test coverage was fit for purpose at each key milestone.
Database log analysis was carried out throughout the test design and execution phases to continually validate and compare it with production. The outcome of this analysis was fed back into the preparation activities to ensure that the test coverage and scenario designs were accurate.