ABSTRACT: This paper attempts to dispel some common misconceptions regarding the roles of Testing and Software Quality Assurance (SQA) by referring to existing industry information and extensive professional experience. It provides insights into the differences between these terms, while also outlining challenges and providing recommendations for successful SQA teams.
The terms Testing and Software Quality Assurance (SQA) are often used almost interchangeably in the IT industry with testing professionals often classed as quality assurance professionals. So, are these terms the same? Although both have quality as their overall objective, a fundamental difference between the two is that testing is performed after a product has been built, or in the case of static testing, after a document has been written. In contrast, quality assurance consists of activities to ensure that quality will be built into the product. To appreciate the differences between Testing and SQA further, it is important to first understand the closely related concepts, Quality Control (QC) and Quality Assurance (QA).
Quality control
ISO9000 defines Quality Control as a part of Quality Management focussed on fulfilling quality requirements.
Quality Control is a set of activities designed to evaluate whether a developed product (project document, developed system etc.) meets customer requirements. It ensures that delivered products are checked for quality and determines how well it is built. Its focus is to find defects and to ensure that they are corrected.
QC is the responsibility of the project team.
Testing forms an integral part of Quality Control, as displayed in Figure 1. However, not all QC activities are testing activities. Code inspections, technical reviews and stage gates are other examples of Quality Control activities.
FIGURE 1
Quality assurance
ISO9000 defines Quality Assurance as a part of quality management focused on providing confidence that quality requirements will be fulfilled.
The goal of QA is to provide assurance that a product will meet the customer’s quality expectations. It consists of activities designed to ensure that quality will be built into the product. These activities usually precede development of the product and continue while the development is in progress.
It is a QA responsibility to develop and implement processes and standards to improve the development life cycle and to make sure that these procedures are followed. The focus of QA is defect prevention, processes and continual improvement of these processes.
While QA is a proactive activity, QC is reactive.
Examples of QA activities include establishing standards and processes, quality audits, selection of tools and training.
The relationship between QA and QC is depicted in Figure 2 below.
FIGURE 2
How do test and QA team responsibilities differ
Test teams perform test basis document reviews, test planning, test analysis and design, test validation and verification, and test reporting through the different test levels.
In contrast, SQA teams perform the following functions:
- Implement organisational quality policies, standards and processes
- Assist projects with preparing software quality assurance or project quality plans
- Assure that project processes conform to quality plans
- Conduct regular audits of project products and processes, and present regular assessments to senior management
- Escalate situations where there are deviations from guidelines or standards
- Assure that:
- Independent reviews are conducted
- Change control procedures for projects are in place
- Configuration management procedures for projects are in place
- Procedures are in place for identification and management of Risks
- There are retrospectives or lessons learned processes planned and conducted
- Provide assurance through the system development life cycle
- Conduct continuous improvements to QA process and guidelines based on lessons learned
Although these attributes are labelled as QA team responsibilities, it should be noted that this does not mean that the QA team will develop these artefacts, but rather assure they are produced in a manner that is ‘fit for purpose’.
As part of their assurance role, QA teams may also review requirement traceability matrices, project document samples, software configuration management activities, design, code etc.
How do test and SQA planning and documentation differ?
Test teams prepare test strategies and plans based on test basis documents such as business requirements and solution design documents. These test planning documents cover test processes across the various planned test levels including details such as the test approach, entry and exit criteria between levels, detailed test schedules, environment requirements, defect management, test management and reporting.
In contrast, software quality assurance or quality plans deal with a broader set of activities across the life cycle. This ties in with project management methodologies such as Prince2 that encourage the use of project quality plans and quality logs that are developed early in the life cycle, when initiating a project.
A typical Project Quality Plan includes customer quality expectations, acceptance criteria, quality responsibilities, planned quality control and audit processes, stage gates, configuration management plans and change management procedures. Quality plans for projects use the organisation’s quality policies, standards or guidelines as a basis, where they exist.
Monitoring of the Project Quality Plan during the project is done by continually updating results of planned quality activities in the Quality Log.
There is an overlap between Risk Management and Quality Management and therefore, the Risk Register can play an important input into the preparation of Quality Plans.
Who should perform the QA function in an organisation
The need for a software quality assurance team grows with the organisation’s size and its level of quality policies. Where such a team is required, it is essential that the QA function remains independent from project and operational teams. Their reporting line, however, must provide them with strong support, as and when required.
Project management methodologies, for example, Prince2 are consistent with this aspect of retaining independence of the quality assurance function. Prince2 recognises three interests that must be represented on the Project Board at all times – Business, User and Supplier, and goes on to recommend project assurance roles that align to each of these interests. Importantly, Prince 2 recognises that all of these project assurance roles must be independent of the project manager. Project Board members may use members of the independent QA team to monitor quality aspects of their individual interest.
Some organisations have the QA function embedded within their enterprise Project Management Office (PMO) departments. This meets the independence criteria, however, organisations following this model must ensure that this team has trained and/or specialised quality assurance analysts.
Observations and recommendations for successful SQA teams
While assuming responsibility for software quality assurance, I experienced a number of challenges. Some of these challenges are outlined below together with learnings and recommendations for organisations considering creating a SQA team.
Independence
As outlined above, to be successful, SQA teams need to be independent from project and delivery teams. This provides the team with the ability to conduct objective assessment of projects.
A question that arises is whether the Testing and SQA function should reside in the same team. Although this may work well in smaller organisations, in my experience, this has the disadvantage of creating a possible conflict of interest when monitoring testing activities. It also tends to emphasise the misconception that testing and SQA are essentially the same.
An option to resolve this is to embed the SQA function within the enterprise PMO, or depending upon the size and quality policies of the organisation, have a separate team reporting to a senior manager who is responsible for the function.
Project team relationships
If quality assurance analysts are too process-oriented, insisting on processes or documentation that may not add much value, it could strain relationships with project managers. For example, the level of discipline imposed on smaller and lower risk projects would be different from that for complex, higher risk ones. There may also be differences based on the methodology used, for example, Waterfall or Agile.
SQA teams would find it much easier to work with project teams if they keep in mind the ‘fit for purpose’ principle. Providing guidance and assistance to project teams, forms a basis for maintaining good relationships, which is an important aspect of successful QA teams.
Senior management support
I have managed a SQA team that was strongly supported by senior management and also one that had only lukewarm support. There was a world of difference between the two experiences. With the latter, for example, escalations were not dealt with in time and problems were carried through the life cycle leading to production problems and higher than anticipated costs. In such instances, it is essential that SQA managers have open discussions with the leadership team.
SQA teams can only ensure adherence to processes and organisational policies if there is strong and consistent leadership team support.
Employing the right people
Another ingredient for successful SQA teams is employing the right staff. People with experience in the system development life cycle or software engineering make good candidates for QA roles. Some training in ISO and/or CMMI principles would supplement knowledge of someone with a prior development background and keen interest in quality principles.
Checklists
Standard checklists are a useful mechanism to conduct reviews or audits of projects, particularly if they are developed in line with the phases of the development life cycle. For example, in the Design phase a checklist question could be ‘Is there traceability between design and requirement elements?’
To avoid frustration from project managers, I learnt that it is important to ensure early stakeholder engagement to gain their feedback when a project is initiated, and when changes are proposed to checklists.
Communication and reporting
Although, regular reporting to senior management is important, developing the right templates and metrics that provide the senior managers with what they require ensures that these reports are given due consideration. This is best accomplished by conducting meetings with relevant senior management providing them with options and obtaining their feedback.
SQA teams need to continually get approval for changes to quality processes and standards and ensure effective communication with stakeholders.
Continuous improvement
Lessons learned from projects provide a SQA team with a basis for evaluating its quality processes and guidelines, and incorporating continual improvements. We learned several lessons that required making changes to our procedures. This included developing checklists, flexibility, maintaining good stakeholder relationships and making improvements to our regular management reports.
Since continuous improvements may also require changes to the System Development Methodology, it is recommended that a SQA team maintains an IT department’s development methodology.
Benefits
A successful SQA team can add considerable value for an organisation. Some of these benefits include,
- Higher quality products
- Consistency in processes used for delivery
- Continued improvement of organisation processes
- Lower overall costs of delivery
- Increased application support documentation
Disadvantages
- Upfront costs in staffing of quality assurance analysts
- Increased processes that could generate frustration in some staff
Conclusions
Comparing the differences in activities and responsibilities between Quality Control and Quality Assurance provides us with a good appreciation of these different terms. QC validates that a specific deliverable meets standards and specifications. In contrast, QA is a broad function covering proactive planning and monitoring throughout the development life cycle. Testing on the other hand, forms an integral part of QC. For an organisation to effectively implement Quality Management processes, these streams must work in tandem. It is recommended that organisations carefully consider the various possible challenges and have appropriate plans in place prior to embarking on quality management processes.
Download Whitepaper