ABSTRACT: This paper looks to provide those who are new to web services testing with guidance on the best practices and approach to testing. It also highlights some of the key challenges that may be encountered.
Web services have become increasingly popular and a larger number of companies have moved to solutions based on Service Oriented Architecture and Web Services. This has been apparent from my own experience, where I have worked on the integration testing of web service with large organisations with well-established processes, through to small vendors who have adopted web services technology for the first time and have little to no established web service testing processes.
Due to this increased popularity, companies are increasing the level of test effort in this area. It is increasingly likely that many testers will find that they will work in this area during their career. In this paper, I look to provide guidance to those new to web service testing by introducing the concepts of web service technology and architecture; understanding the challenges that are likely to be encountered by testers; and suggesting the best approach for testing.
What are Web Services?
In order to successfully test web services, it is essential that the tester gains a good understanding of the underlying technology. This understanding will assist in identifying the key challenges and devising a suitable test approach. Here I attempt to describe the basic concepts of Web Service and firstly look at the most common underlying architectural called Service Oriented Architecture.
Web services are based on an architectural style called Service Orientated Architecture (SOA) that delivers functionality as services that can be consumed by any number of clients. The services role is to respond to client requests with some specific information based on a business need. As an example, when you search for insurance quote online via the broker’s web site, the web server will talk to a series of insurers web services distributed over the internet. These will respond with a quote based on the information passed in the request. The broker’s web server will take the returned information and present this in a suitable format for the customer on its website.
In SOA, services providers can publicise their existence via the service registry to allow other clients to find and communicate with it in an open and public format. This communication takes the form of a well-defined and strictly enforced contract. Clients can be web servers or even other web service providers. Service consumers can call other services and aggregate the results of the services response, and publishing them as another service. This results in a distributed architecture where service providers are loosely coupled with service consumers.
In summary, within a SOA, services exhibit the following behavior:
- A service is specified in the business context and in terms of the contracts between the provider of that functionality and its consumer
- Services are re-usable components with distinct business or technical functionality
- Services are distributed and loosely coupled with the dependent service consumer
- A service has no User Interface and is called by other systems
- A service may call other services to assemble a complete business process which is known as the orchestration of services
Essentially with SOA, instead of viewing the architecture as independent systems, it can be considered as black box services that deliver business functionality that can be reused without making changes.
The concepts behind SOA are not new, but earlier attempts relied on proprietary interfaces where all parties relied on developing custom built communications. This issue has been addressed by using standards like XML through SOAP. This standard has provided a platform for building specifications for enterprise services. Web services have evolved as the most common standard for the interactions and exchange of information between different systems.
Web Services Explained
Web Services are based on a set of open communication standards to enable the exchange of data between the service provider and service consumer. This standardisation enables communication between client/server applications programmed in different languages and running on different operating systems. Web Services communication standards use XML for representing data, XSD schema defining the format, Simple Object Access Protocol (SOAP) to enable data exchange and Web Serviced Description Language (WSDL) to describe the functionality of the service.
So What Does this Mean?
- XML – Extensible Markup Language (XML) is a mark-up language used to define the data in a service request and response message as a structured, plain text format. This is both human and machine readable and enables data to be read by different incompatible applications. It is defined in the XML 1.0 specification produced by the World Wide Web Consortium (W3C).Characters in an XML files consist of markup and content. A tag in XML is an example of markup and starts and ends in <>. There are three types of tags consisting of start-tags < Element>, end-tagsand empty-element tags. Start tags must end with a matching end tag. XML documents form a tree structure that starts at “the root” and branches to “the leaves”. Any characters contained between and start and end tag is the content. For example:
<heading>Web Services Testing</heading>
- XSD – is an XML schema language and defines a set of rules for the data type, format and structure of the XML document to be valid. An example of a XSD schema for the above XML:
<xs:element name=”heading” type=”xs:string”/>
<xs:element name=”author” type=”xs:string”/>
- SOAP – Simple Object Access Protocol for data exchange and encapsulates the XML documents. It has an optional header containing information about the request/response message and a mandatory body containing the payload of the message.
- WSDL – Web Service Definition Language is an XML-based language used to define Web Services and describe how to access them and is often used in combination with SOAP and XML Schema to provide web services over the internet. WSDL comprise of a number of elements. The <portType> element that describes the web service, the operations that can be performed and the messages that are involved. The <message> element defines the data elements of an operation. The <types> element defines the data types that are used by the web service.
Why Perform Web Service Testing?
Testing at the web service layer can offer a number of advantages. Isolating the services testing will remove the dependency on the front end system being fully developed in order to start testing. This facilities early testing and can reduce costs by identifying defects earlier in the SDLC cycle. It also allows the service to be tested only once irrespective of how many systems consume the service. It can also help reduce the number of issues with the system integration testing and reduce the need to test each consuming system individually. Service testing can emulate the end-to-end business process prior to testing it during UAT. It also allows the ability to perform negative scenario testing to ensure that exception cases are managed correctly. This may be difficult to perform as part of the UAT testing.
Those new to testing services may not have previously encountered some of the challenges posed by testing web services. They may have come from a background where the testing was conducted through user friendly user interfaces with little knowledge of the underlying interactions with other systems. Web services on the other hand are consumed by other servers and have no user interface to test with. Services may be delivered earlier in the SDLC and need to be tested prior to the availability of the consuming client. It’s clear that some level of tooling will be required to test.
Web services are loosely coupled with service consumers and may not control the clients that access them. It is critical that the service implement the correct level of security and authentication. Services also need to exhibit correct behaviour when the client request is not as expected. Security testing is needed and the testing of negative scenarios is imperative. They may service multiple clients concurrently and this may pose additional performance and scalability considerations.
Testing needs to ensure the proper behaviour no matter what application uses them and no matter how often it is accessed. All these challenges will require testers to learn new technologies and tools.
What to Test?
Functionality testing comprises of verifying conformance of the service behaviour against the given service design. It is advisable to isolate the service and consider it as a black box from a functional test perspective. The following items provide a guide to the type of testing to be performed:
Field level validation – validations for each data field to confirm behaviour against the service design. The testing is conducted for each service in isolation by manipulating the XML message to exercise both positive and negative conditions. In particular, the boundary values and error checking are important factors for this testing. Field level validation tests include:
- The testing of mandatory and optional fields
- Testing to verify the format and value of data fields such date, amounts, alpha-numeric, numeric and special characters
- Testing to verify that correct value pair are supported by the service
- Testing to verify enumeration where all values within a complete list is supported in that field
It is worth adopting techniques such as Boundary Value Analysis and Equivalence Partitioning for this testing.
- Integration testing – this testing focuses on the interactions between the services and interfaces to form a complete business transaction.Integration tests can include:
- Orchestration / Choreography testing to ensure that services follow the correct workflow order
- Message routing through the system including content based routing, data field mapping and transformation
- System integration testing where the interactions between systems and interfaces are implemented as workflow
Given that web services are designed to be re-used and consumed by multiple clients, it is critical to understand the performance characteristics to enable the scalability to be determined. However, prior to embarking on performance testing, the business needs to define clear performance goals.
These goals should consider the following questions:
- What is the performance test scope? What services, interfaces and systems are in scope?
- What is the expected number of concurrent clients that will access the services (both peak and nominal)?
- What is the payload of the incoming requests?
- What is the service workload mix i.e. % mix of the different services?
- What does the target hardware and network systems look like?
These are but a few of the many questions that need to be asked when considering performance testing. These goals will differ depending on the types of services and systems being tested, however, they should also define some of the following:
- Concurrency – testing the simultaneous access of the service by multiple clients to replicate production
- Throughput – the maximum transaction rate that the service can support
- Payload – the size of the incoming requests beyond which the service behaves in unexpected ways
- Response time – the time it takes for the system to respond to a request
- Utilisation – the resources used by the system under load conditions
The scope and requirements of performance testing will determine the test types, however performance testing typically comprises of:
- Load tests – The primary goal of load testing is to evaluate whether the system can handle a specified quantity of load. This can be defined in terms of a number on concurrent clients calling a specific number of transactions within a set duration
- Stress test – Testing to evaluate the system at or beyond the limits of its specified requirements. This kind of testing will measure the frequency of errors or system crashes to determine whether the system performs adequately
- Endurance tests– Evaluates the system’s ability to sustain constant work load over a period of time. Endurance test help detect any bottlenecks, memory leaks, performance degradation and component limitations. It essentially involves applying a significant load to a system for an extended, significant period of time. The goal is to discover how the system behaves under sustained use
- Spike tests – Testing to evaluate the system behavior when a sudden increase in load is applied. The goal is to determine whether performance will suffer, the system will fail, or it will be able to handle dramatic changes in load
Given that web services are loosely coupled with the service consumer, the service provider may not control the clients that access it. Security testing is critical to ensure that information is not used by unauthorised clients for malicious purpose.
The security testing of web services should consider:
- Authentication – The process of assuring that the request actually originated from an authorised source. In addition to authenticate the source, the service provider may need to prove the message origin to other consumers
- Authorisation – This provides assurance that only authorised requesters are allowed to access the service. This goes hand in hand with authentication to ensure that malicious parties cannot mimic a valid client
- Penetration – A Penetration Test simulates an attack by a malicious party. This testing attempts to find and exploit vulnerabilities to determine what information and access is able to be gained. This is designed to mimic the actions of an attacker exploiting weaknesses in network security without the usual risks
- Protocol / encryption standards testing – this provides assurances that the service transaction are encrypted using the defined encryption techniques. Secure encryption standards should prevent attempts to decrypt traffic, known as encryption attacks
It is apparent throughout this paper that tools are critical to enable comprehensive testing of web services. In fact without appropriate tools then much of the functional, performance and security testing could not be conducted.
There are a number of open source and commercially available tools that offer various benefits. While open source tools provide a cheap option they may not provide the capabilities for all the testing required. While they are often suitable for manual functional testing, they may not offer the capabilities for automation, performance and security testing. There are commercial tools, on the other hand, that are designed for these specific tasks. These of course come at a significant cost.
Prior to choosing tools, it is advisable for an organisation to adopt a long term test tool strategy from the onset. The correct strategy should be put in place and consider factors such as the tool requirements, existing processes, benefits, costs, expertise and training. Consider carrying out a proof of concept to assess the potential benefits. If the correct planning and processes are not put in place initially, then there is a likelihood wrong tool is implemented which can be a costly mistake.
Software solutions based on web services are becoming increasingly popular due to the benefits it offers to organisations. The alignment of business processes with IT has undeniable appeal. As more organisations adopt this technology the resourcing of testing in this area will increase. Testers are increasingly likely to find themselves working in this area.
As outlined in this paper, this technology offers a number of challenges to those new to testing web services. The technical nature of the communication protocols, the heavy reliance on tools and lack of user-friendly interfaces means that testers will need to learn new technologies and tools. However, for those with the characteristics of a good tester, an analytical mind with an understanding of test processes and methodologies, should find themselves able to quickly develop the correct skills.
This paper has introduced some of the basic concepts of web services testing. It will hopefully give those new to this area some useful ideas and guidance on how to best to approach web services testing.