Automated Regression Testing can help us ensure the quality of our growing applications. There are many open source software testing tools in the market today and identifying the right tool for your need is challenging. Some people build their own test automation framework however the time spent in building your own framework can be significantly reduced by using existing open-source tools and libraries. These free and ready-made frameworks may be able to meet your requirements without having to write code and often provide a better outcome.
One of the most popular open-source test frameworks that you can get today is the Robot Framework. It is a Python-based solution that uses a keyword-driven approach to make tests readable and makes construction of test suites easier. It is widely used for testing different mobile devices, embedded systems, software systems and protocols via GUI, APIs and other interfaces.
Robot Framework has a modular architecture as shown in the illustration below.
Figure 1 - Robot Framework modular architecture diagram
This makes it reusable, easy to expand, and painless to maintain. You will see more information below as to how this modular structure makes it easier for the tester to use and develop testing tools out of this framework.
This guide is current at the time of publishing. You can check http://robotframework.org/#documentation for additional information and future updates. Sample code is intended to be simple and may not follow best practice.
For testing web-based application using Robot Framework, you need to have the following installed in our system:
- Python 2
- Robot Framework
- Selenium Library
- Source Code Editor or IDE
To get and install all of the above requirements, follow the steps below.
Step 1: Download and Install Python 2
- Download the latest release of Python 2 from python website.
- From Downloads folder, click the python-2.7.13.msi* and completely install the application.
- Ensure that C:\Python27\ and C:\Python27\Scripts are included in the Environment Variable Path. This enables python to be executed in command window. To open the Environment Variables window:
- Click on Start
- Right click on Computer
- Click on Properties
- On the left side of the System window, click on Advanced system settings link
- On System Properties pop up window, click on Environment Variables button
Figure 2 - System Variable Path
*python-2.7.13 is the latest release of Python version 2 as of this writing.
Step 2: Download and Install Python 2
- Open command prompt and point the directory to C:\Python27\Scripts.
- Enter pip install robotframework
- Ensure that robot libraries are added: C:\Python27\Lib\site-packages\robot
Step 3: Install Selenium2Library
- Open command prompt and point the directory to C:\Python27\Scripts.
- Enter pip install robotframework-selenium2library
- Ensure that Selenium libraries are added: C:\Python27\Lib\site-packages\Selenium2Library
Step 4: Install a Source Code Editor or IDE
Source code editor is designed specifically for editing source code of computer programs. It could be stand alone or it may be built into an integrated development environment.
You can install Notepad++ as your source code editor.
- Download the installer from the Notepad++ website
- Navigate to the local folder where the installer was saved
- Run and install the application
There are several options for IDE software; PyCharm, RIDE, Eclipse, Visual Studio Code, etc. The automation code examples of this document were created in Notepad++, but if you preferred IDE, PyCharm appeared to have the best support for python development. The full install of PyCharm is licenced, however there is a free community version.
Learn the Framework
It will be easier to understand the framework if the user has basic programming skills and has knowledge of the Python programming language. Running real-life examples from the demo projects available online can help new users to quickly grasp the concept of this framework.
A keyword-driven test is a sequence of actions based on the list of keywords in the test data that simulate a real user’s actions with the system under test.
Test Automation Code
Test Automation Code is defined in files using human-readable syntax. The plain text format is easy to edit using any text editor as shown in the example below. They also work well with version control thus it has become the most used data format with Robot Framework. The sample codes shown in this document are intended to be simple and may not be following the best practice in coding. You may visit Style Guide for Python Code found in python website to know more about coding python.
Figure 3 – Sample robot test file (resource.robot)
The screenshot shown below is a plain example of a logically named directory with several robot files which hold the test cases about logging in to the system under test. The file extension of the test script files used is .robot and these files contain test cases that can be grouped together to create a test suite. Placing these files into logically named directories will make a nested structure of test suites. Some full blown test automation suites would have base folders such as “Tests” or “Models” containing “pages” & “dialogs”. Base folder such as “Implementation” or “Libraries” that contains additionally developed python files.
Figure 4 – Directory with Robot files
The login_tests directory becomes a test suite which you can use to run your test automation. The command to run this test suite is shown in the Test Execution section below. It will run all the test cases inside each robot file.
As stated earlier, Robot Framework is a keyword-driven test automation framework thus its test automation code syntax is based on keywords. Keywords and variable declarations can be saved in a resource file, which can be used by various test suites. Reusing this resource with other tests can help avoid duplicated effort thus saving time and later maintenance. An example of a robot file with a test case that used a keyword from the resource file is shown below.
Figure 5 – Robot File with Test Case (01__valid_login.robot)
You can create new keywords by using pre-existing keywords from the libraries after you install the framework. See the use of the Open Browser To Login Page keyword in the resource.robot file shown below. The Open Browser keyword in line 25 is from a pre-existing Selenium 2 library of keywords.
Figure 6 – Keyword created from pre-existing keyword
The screenshot below is _browswermanagement.py, a file coded in python that was created after installing Selenium2Library and found at C:\Python27\Lib\site-packages\Selenium2Library\keywords folder. It is a library that holds pre-existing keywords for automating browser related actions such as opening and closing the browser.
Figure 7 – Keywords for Browser Actions
There are ready made test libraries for the framework which you can use to build your automated testing suite. There are standard libraries that are packaged in with the framework such as Builtin that contain generic keywords, OperatingSystem with keywords that enable you to perform tasks such as creating and removing directories, Screenshot that provides keywords to capture screenshots of the desktop, etc. There are also many separately developed external libraries that extend the testing capability of the framework by providing keywords which you can install based on your needs. Selenium2Library is one of them. With this, you can make scripts that won’t take too much time and effort. Aside from that, by creating scripts out of the ready-made keywords, you will learn more about the tool’s functionality which will help you design faster and better scripts. You can also easily create our own library written in Python or Java if you desire.
Setups and Teardowns
Using Setups and Teardowns is best practice as it adds a level of control to your test by putting your environment into a known state before and after your tests. The Setups and Teardowns affect the test execution flow. They can be used in test suite, test case, and user keyword levels.
If a test suite has Suite Setup, it is executed before its tests and child suites. The Suite Setup is often used for setting up test environment for the test suite. If it fails, the tests, setups, and teardowns within its child suites are not executed and all the test cases and child suites are marked failed.
If a test case has Test Setup, it is executed before the keywords. The Test Setup is often used for setting up the environment for that particular test case. If it fails, the keywords are not executed.
If the test suite has Suite Teardown, it is executed after all its test cases and child suites. The Suite Teardown is executed regardless of the test status and even if the matching Suite Setup fails. If the Suite Teardown fails, all the tests within the test suite are marked as failed in the generated reports and logs. All the keywords used in the teardown are executed even if some of them failed that is why teardown is often used for cleaning up the test environment after the execution because it ensures that all tasks are done.
Likewise, the Test Teardown is commonly used for cleaning up activities. It is fully executed after the test case has been executed even if some keywords have failed and even if Test Setup has failed.
User keywords cannot have setups but they may have teardowns that can be defined using [Teardown] setting as illustrated below.
Figure 8 – Keyword teardown in a test case
The keyword teardown works similar to test case teardown. All steps of the teardown are executed even if one of them fails. However, a failure in keyword teardown will fail the test case and subsequent test steps are not run. The name of the keyword to be executed as a teardown can also be a variable.
Test Execution is started, whether via the command line or through IDE’s. Robot Framework test cases are created in files and directories, and they are executed by giving the path to the file or directory to the selected runner script. The path can be absolute or, more commonly, relative to the directory from where tests are executed. The given file or directory creates the top-level test suite, which inherits its name.
Sequence of Execution
Test cases in a test suite are executed in the same order as they are defined in the test case file. Test suites inside a higher level test suite are executed in case-insensitive alphabetical order based on the file or directory name. If multiple files and/or directories are given from the command line, they are executed in the order they are given.
If there is a need to use a certain test suite execution order inside a directory, it is possible to add prefixes such as 01 and 02 into file and directory names. Such prefixes are not included in the generated test suite name if they are separated from the base name of the suite with two underscores:
Figure 8 – Test Files with pre-fixes to control the execution order
When execution is started, the framework first parses the test data in the robot test file. It then makes use of the keywords provided by the test libraries to interact with the system under test. Libraries can communicate with the system either directly or using other test tools as drivers such as Web Driver leveraged in Selenium2Library. This library runs tests in a real browser instance that should work in modern browsers.
Test cases are always executed within a test suite. A test suite created from a test case file has tests directly, whereas suites created from directories have child test suites which either have tests or their own child suites. The screenshot below is an example of running a test suite created from directory.
Figure 10 - Running test suite created from a directory
The screenshot below shows a running test suite created from a test case file.
Figure 11 - Running test suite created from a test case file
You will see the progress of the test in the command window during test execution. You can also see the result of each test case right after it’s executed as shown in the screenshot below.
Figure 12 – Test Execution in Progress
By default all the tests in an executed suite are run, but it is possible to select tests using options --test, --suite, --include and --exclude. Suites containing no tests are ignored. The execution starts from the top-level test suite. If the suite has tests they are executed one-by-one, and if it has suites they are executed recursively in depth-first order. When an individual test case is executed, the keywords it contains are run in a sequence. Normally the execution of the current test ends if any of the keywords fails, but it is also possible to continue after failures by using any of following keywords found in BuiltIn library:
- Run Keyword And Ignore Error – Runs the given keyword with the given arguments and ignores possible error.
- Run Keyword And Expect Error – Runs the keyword and checks that the expected error occurred. The expected error must be given in the same format as in Robot Framework reports.
- Run Keyword And Continue On Failure – Runs the keyword and continues execution even if a failure occurs.
Errors caused by invalid syntax, timeouts, or fatal exceptions are not caught by these keywords. Otherwise, the keyword itself never fails. Variable errors are caught by these keywords.
Library keywords report failures using exceptions, and it is possible to use special exceptions to tell the core framework that execution can continue regardless the failure. The way to signal this from test libraries is adding a special ROBOT_CONTINUE_ON_FAILURE attribute with True value to the exception used to communicate the failure. This is illustrated by the Python code below.
Figure 13 – Python code to continue on failure
The screenshot below is empty_login.robot file, an example of suite created from a test case file.
Figure 14 – Suite created from a test case file (03__empty_login.robot)
After test execution, Robot Framework automatically generates the test report, log, and output files. These files provide an extensive look into what your system did during test execution.
HTML Report File
The generated report.html provides you with an overview of the test execution. Summary Information shows the overall status, Pass/Fail ratios, and elapsed time of the test execution. Test Statistics shows the same information for each test suite, and Test Details allows you to drill down to test cases in a test suite. When there is a failure the background colour of the test report is red. Sample reports are shown in the screenshots below.
Figure 15 - HTML Report for Passed Test
Figure 16 – HTML Report for Failed Test
HTML Log File
The generated log.html is a more detailed log file that provides more information on each executed keyword. Log file is needed when test results are to be investigated in details because it allows you to drill down on the specific part of the test in case of failure as shown in the screenshot below.
Figure 17 – HTML log with warning and errors
Output XML File
The generated output.xml file contains all the information about test execution. It can be opened in any text editor or Internet browser as shown in the screenshot below.
Figure 18 – Output.XML
There are supporting tools that you can use to help design, build, and run your test automation suite. Robot Framework has built-in tools for reporting, documentation and cleaning up test data files.
- Rebot – this is a built-in tool for generating logs and reports based on XML outputs and for combining multiple outputs together
- Testdoc – this is a built-in tool that generates high level HTML documentation based on Robot Framework test cases
- Libdoc – this is a built-in tools for generating keyword documentation for test libraries and resource files
- Tidy – this is a built-in tool for cleaning up and changing format of Robot Framework test data files
There are individually developed tools for editing test data, tools for running Robot Framework tests, and tools for collecting and publishing Robot Framework test results.
Other Supporting Tools
- RIDE – is a standalone Robot Framework test data editor
- Editor Plugins – there are several plugins that would provide support for editing Robot Framework tests cases using your IDE of choice such as Atom, Bracket, Eclipse, Gedit, etc.
- Build Plugins – there are available plugins that would allow other build servers or Continuous Integration Tools such as Jenkins and Maven to run and publish Robot Framework test results
Robot Framework is available to everyone to download and use, featuring fast and easy installation and setup, as well as non-complicated methods of writing automated test cases. The tool automatically generates test reports and logs that are viewable on web pages, and the amount of readily available libraries significantly increases its testing capabilities. Support from users all over the world makes the system comfortable to operate and explore, whilst the maximum ability of the tool derived from this framework can be achieved with no tooling costs. With a range of benefits on offer, it becomes clear why Robot Framework is one of the best automated testing frameworks for regression testing.
- ROBOT FRAMEWORK Generic test automation framework for acceptance testing and ATDD, http://robotframework.org/
- Python Software Foundation, https://www.python.org/
- RobotDemo, https://bitbucket.org/robotframework/robotdemo
- WebDemo, https://bitbucket.org/robotframework/webdemo
- Selenium2Library, http://robotframework.org/Selenium2Library/Selenium2Library.html
- Robot Framework Documentation, http://robotframework.org/robotframework
- Robot Framework User Guide, http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html
- Installing and Configuring PyCharm, https://git.planittesting.com/automation/robot-framework-projects/wikis/installation
- Notepad++ website, https://notepad-plus-plus.org/
- Style Guide for Python Code, https://www.python.org/dev/peps/pep-0008/