How to Use Python for Automation Testing: Building a QA Framework from Scratch

Test automation is a great way to save time for quality assurance (QA) engineers while ensuring the delivery of a high-quality product. In our projects, we often use Python and Python-based tools for this task. Learn how to use Python for automation testing and explore ways to develop a test strategy and build test automation process for a project with web and mobile applications based on a practical example.

This article will be useful for project managers and QA leaders looking for ways to enhance their team’s QA efforts.

Using Python for test automation

Python is one of the most popular programming languages and has a vast variety of applications, from artificial intelligence to data analysis and quality assurance (QA). Python’s flexibility and readability, numerous third-party libraries, and community support allow developers and QA engineers to quickly write code and resolve issues they encounter. 

Let’s take a look at reasons to implement Python-based test automation:

Smooth learning curve. Automation engineers often don’t have strong development skills, so learning a new programming language can be time-consuming for them. Python is human-readable, has simple syntax, and is well-explained in support documentation and various guidelines. Even if a QA team doesn’t have programming skills, they can quickly master test automation with Python.

Available libraries, frameworks, and plugins. Lots of popular QA tools are written in Python or support this language. With these tools, it’s possible to write any functional test, including unit, integration, and end-to-end tests. Python-based test automation tools allow QA engineers to run their tests on various platforms, extend testing capabilities, integrate with new platforms, and more. Such a variety of available tools helps QA specialists find exactly what they need and save time on manual work.

Powerful command-line interface (CLI). Python’s CLI has many features that simplify automation workflows. For example, a framework can launch the console to search for tests, run them, and collect results. Built-in features like these simplify and speed up automated test management, allowing the QA team to focus on testing and save a lot of time and effort on researching, deploying, and troubleshooting third-party tools.

Intuitive integrated development environments (IDEs). An IDE is essential for automating manual test cases. There are many Python-based IDEs that can turn user actions into test scripts, translate these scripts to various languages, and automate many testing activities. Most Python IDEs provide QA engineers with an easy-to-understand UI, numerous built-in tools, and templates that simplify test automation. Engineers can quickly study an IDE of their choice and use it to the fullest.

Testing versatility and portability. Python scripts can be written once and executed on different operating systems without the need to recompile them. QA engineers can also use the same tests and automation scripts for various projects, reducing the time required for QA activities.

Because of these factors, automated testing with Python is essential for any Apriorit project. It helps our QA engineers avoid a great deal of manual work and focus their efforts on high-priority tasks while making sure their tests run as intended.

Whenever we decide to implement test automation with Python, we start by creating an automation strategy. Let’s find out what it is and why it’s important.

Preparing the automation testing strategy

For this article, we examine one of our recent projects that required automated QA: an antivirus solution with versions for Windows and Android, as well as a website for selling it. We needed to test all of these products.

To implement automation testing using Python, we start by preparing a testing strategy. This document contains a high-level description of the project’s automation testing processes. 

We base our strategy on the requirements of the International Software Testing Qualifications Board (ISTQB). 

Let’s take a look at common items to include in such a strategy.

Testing objects, or the parts of the project we plan to prepare automation tests for, are applications for Windows and Android, as well as the website.

Our testing environment consists of a Samsung Galaxy S21 and Samsung Galaxy Tab S7 for the Android app, as well as an endpoint with Windows 11 and a virtual machine with Windows 8.1 for the desktop application.

We’ll automate the following testing levels:

• Component testing
• Integration testing
• System testing
• Acceptance testing

We’ll also automate these testing types:

• Unit testing of backend components
• Regression testing for the Android app, Windows app, and website
• API regression testing
• Static analysis
• Web application performance testing

Our testing approach for automated activities includes the following high-level principles:

• Automation tests are executed inside the prepared environment
• We can start automation tests using the CI/CD system
• Unit tests and static analysis on new project builds are executed automatically 
• Feature regression tests run according to results of an impact analysis
• Tests from automated acceptance checklists are executed on release builds

The testing strategy also describes key testing activities for automated QA engineers. Here’s what they do in our project:

• Set up an automation testing environment
• Create new automation tests
• Configure a CI/CD system to work with automation tests
• Run automation tests as part of testing new builds
• Update the automation testing coverage map
• Localize manual bugs found by automation tests
• Update automation tests as the project changes
• Report results of automation testing to the team

Note that this strategy describes only the basic activities of QA engineers. As the project changes, this list may also change.

Exit сriteria describe what we need to achieve to release a high-quality product. We set the following quality criteria for our products:

• All scheduled automation testing activities are completed
• Issues of failed autotests are successfully localized
• No medium or higher priority issues are found
• The results of static code analysis correspond to the required level of code quality
• The results of load testing correspond to the required level of performance

For this project, we use the following automation testing tools:

• Python 
• Selenium (for website testing)
• Robot Framework (for Windows application testing)
• Appium (for mobile app testing)
• SonarQube
• GitHub
• Jenkins

Since we have several testing objects and environments, we need to take care of testing artifacts. In particular, we need to manage:

• Automation test files, Python scripts, and CI\CD pipelines in Apriorit’s GitHub repository
• Coverage diagrams and other types of QA documentation in Confluence
• Testing report files generated by Jenkins

The Testing reports section of the strategy describes which automation testing reports we need to generate and share with the project team. For this project, we use Jenkins to generate the testing report which contains screenshots of testing steps. Jenkins stores the last ten reports and automatically sends new ones via email to the test manager and project manager.

We should also assess the risks of automation testing and describe action plans for each of them:

Once the testing strategy is completed, we can start implementing automated software testing with Python.

Working with Python test automation frameworks

A QA automation framework creates a set of rules for writing test cases and executing a testing strategy. It usually provides one or several tools for test automation and defines automation rules and procedures. It’s possible to have several testing frameworks in one project depending on the complexity and diversity of testing environments.

There are dozens of frameworks for this purpose, but at Apriorit we often rely on Selenium, Robot Framework, and Appium for our projects. These tools are free and provide all the features needed to test a product. Since our product contains a website and mobile and desktop applications, we need all three of these frameworks.

Let’s examine how these frameworks work and when to use them.

Selenium

Selenium is a well-known open-source framework mainly used to automate interactions with web elements. It’s so popular that it’s almost a standard tool for web app testing. There are also a lot of other frameworks based on Selenium. Here’s why it’s so common:

• Multi-language support. Selenium supports Python, Java, C#, JavaScript, Ruby, and PHP. In our experience, Python is the best option for web automation with Selenium.
• Large and supportive community. Since many QA engineers use Selenium daily, it’s easy to find answers to tricky questions. Also, Selenium developers provide detailed documentation for their framework and online support.

However, there are several downsides to working with Selenium:

• Need for programming experience. To read and develop tests, a QA engineer has to know a programming language and have basic development skills.
• Low speed of test development. A lot of testing activities require writing code, which is slower compared to other tools. 
• No support for mobile or desktop projects. Since Selenium only works with web-based products, you need to combine it with other frameworks when testing cross-platform projects.

Let’s take a look at the Python script for automation testing of website authorization with Selenium:

import unittest  # Import Python module to work with test cases
from selenium import webdriver  # Import Selenium WebDriver features to work with modern browsers
from selenium.webdriver.common.keys import Keys  # Import module which allows for working with keyboard buttons
from selenium.webdriver.common.by import By  # Import module which allows for locating elements on the page

class AntivirusLogin(unittest.TestCase):  # Define test case name
    def setUp(self):  # Describes what should be done before executing test cases. 
        self.driver = webdriver.Chrome()  # Create an instance of Chrome WebDriver

    def test_valid_login(self):  # Describes test case steps
        driver = self.driver  # Create a local reference to the driver object created in the setUp method
        driver.get("https://project-antivirus.com")  # Navigate to the tested website
        self.assertIn("Project-Antivirus", driver.title)  # Verify that the opened page has the required title
        username_field = driver.find_element(By.NAME, "username")  # Find the username field on the page by its HTML element name
        username_field.send_keys("[email protected]")  # Write the username to the username field
        password_field = driver.find_element(By.XPATH, "//input[@type='password']")  # Find the password field on the page by its XPath
        password_field.send_keys("24Dqe*}M")  # Write the password to the password field
        sign_in_button = driver.find_element(By.NAME, "sign_in")  # Find the Sign In button on the page by its HTML element name
        sign_in_button.click()  # Click the button
        displayed_username = driver.find_element(By.CLASS_NAME, "full_name")  # Find the displayed username after the sign in
        self.assertIn("Automation QA", displayed_username)  # Verify that the displayed username is the expected one

    def tearDown(self):  # Describes what should be done after test cases have been executed
        self.driver.close()  # Close the browser

if __name__ == "__main__":  # Required to run the test suite
    unittest.main()

Robot Framework

Robot Framework is a keyword-driven framework designed for automated testing of UI elements, mobile apps, desktop software, APIs, databases, etc. Written in Python, this framework allows for writing tests with ready-made keywords. 

The functionality of Robot Framework can be extended with additional libraries including SeleniumLibrary, AutoItLibrary, and AppiumLibrary. You can implement additional libraries with Python, Java, and other languages. You can also create your own keywords and connect libraries with your custom Python functions.

Here are the key benefits of using this framework:

• Human-readable tests. Writing and reading tests doesn’t require a technical background. A QA engineer can use human-readable keywords to write a test and read its results.
• Easy to learn. To work with this framework, engineers don’t need to learn programming languages and have development experience.
• Lots of additional libraries. By adding third-party libraries and writing custom ones, it’s possible to significantly expand the framework’s native functionality.
• Multiplatform. Robot Framework helps create tests for many platforms, which makes it suitable for cross-testing projects.
• Generation of default reports. Robot Framework automatically generates visualized testing reports after test execution.

We also need to consider the following disadvantages of working with Robot Framework:

• Issues with support for additional libraries. Robot Framework libraries are free and developed by community enthusiasts. That’s why getting library updates, fixes, and answers to questions can take more time compared to getting the same from a professional development company.
• Limitations of the special IDE (RIDE) for Robot Framework. At the moment, RIDE has fewer features than other popular IDEs like PyCharm or Visual Studio Code.

Let’s take a look at an example of Robot Framework code for testing activation of the Windows antivirus app:

*** Settings ***  # Describes settings of the test suite
Documentation   A test suite for activation of Windows Antivirus app. # Test suite description
Resource            keywords.resource  # Connects the keywords.resource file to use its keywords.
Suite Setup         Install and Run Antivirus  # Suite setup describes what should be done before test cases are executed. In our case, it will install the antivirus software and run it.
Suite Teardown   Close the Antivirus Window  # Describes what should be done when test suite execution is completed. In our case, it will close the window with the antivirus app.

*** Keywords ***    # Describes the custom keywords which will be used in test cases
    Verify Failed Activation  # Create a keyword that verifies the failed activation action. Simply call it and add necessary arguments.
    [Arguments]  ${activation_key}=${None}
   # Describes arguments which could be passed to the keyword
	Enter Text And Submit  ${activation_key_field}  ${activation_key}  # Enter the activation key in the required field and submit
    Element Should Contain  ${message_field}  Wrong activation key  # This keyword verifies that the error message appears

*** Test Cases ***  # Describes test cases
Successful Activation with Valid Key  # Test case title
   Enter Text And Submit  ${activation_key_field}  my_valid_key
    Element Should Contain  ${message_field}  Activation successful  # Verifies that a successful activation message appears

Failed Activation with Wrong Key
   Verify Failed Activation  activation_key=invalid_key  # Call the keyword created above

Failed Activation with Empty Key
   Verify Failed Activation  activation_key=${None}

Appium

Appium is a powerful open-source tool for automation testing of mobile applications that supports a wide variety of real and virtual devices. This Node.js program allows QA engineers to use any popular programming language to write their tests.

Appium provides a wide set of test automation features, as well as options for adding custom functionality. You can add custom code to Appium via a third-party driver you develop or get from other users. 

Key benefits of using Appium are:

• Support for various platforms. Appium can handle testing on both Android and iOS devices. Cross-platform support saves QA engineers time on rewriting a test case for another platform as well as efforts on searching for and supporting one more testing tool.
• Support for real and virtual devices. Appium can create configurable virtual devices and add them to the testing pool alongside real ones. This way, a QA engineer can configure the required testing environment if they don’t have the necessary real devices.

However, this tool also has some limitations:

• Difficult environment configuration. It’s only possible to configure Appium tests on PC, not on mobile or virtual hardware. A QA engineer also needs experience with Android Studio to test Android applications.
• Support for mobile apps only. Appium can’t create tests for web and desktop applications.

Let’s take a look at an example of Appium code for testing the settings button of the Android application:

import unittest  # Import Python module for work with test cases
from appium import webdriver  # Import Appium WebDriver features to work with mobile applications
from appium.webdriver.common.appiumby import AppiumBy  # Import module which allows for locating elements in applications

capabilities = dict(  # Describes parameters to execute the test on the tested device
    platformName = 'Android',
    platformVersion = ‘10’,
    automationName = 'uiautomator2',
    deviceName = 'Test_S10',
    app = ‘/path/to/antivirus.app’
)

appium_server_url = 'http://localhost:4723'  # Appium server location

   class TestSettings(unittest.TestCase):  # Define the test case name

       def setUp(self) -> None:  # Describes what should be done before test cases are executed 
            self.driver = webdriver.Remote(appium_server_url, capabilities)  # Create an instance of Appium Driver

        def tearDown(self) -> None:  # Describes what should be done after test cases have been executed
          if self.driver:
                 self.driver.quit()  # Close the Appium session

      def test_find_settings(self) -> None:
settings = self.driver.find_element(by=AppiumBy.XPATH, value='//*[@text="Settings"]')  # Find the settings button by its XPath
settings.click()  # Click the button from the previous step
  
if __name__ == '__main__':  # Required to run the test suite
      unittest.main()

You can use these frameworks as standalone tools or combine them into one system. After you choose and configure the required frameworks, you’ll also need to integrate them into your project’s continuous integration (CI) and continuous delivery (CD) system.

Generating test result reports in the CI/CD system

CI/CD systems help to automate the parts of software development related to building, testing, and deploying. You can use such systems to execute remote automation tests and configure testing environments. Popular CI/CD systems include Jenkins, Bamboo, and GitLab CI/CD.

We prepare CI/CD pipelines and store them in the version control system. This gives us the ability to quickly configure our jobs in any new environment without wasting time.

A CI/CD system helps us generate testing reports according to our needs. We can configure the format and contents of the report, as well as when to generate it.

In the project we mentioned earlier, we used Jenkins because it:

• Is open-source software
• Provides numerous configurations
• Allows for integration with other services
• Offers lots of updates for plugins and Jenkins itself
• Is supported by a large community

Here’s what a test result report looks like in Jenkins:

We can see the results of previous test executions in the pipeline execution history, which allows us to track if there are any recurring issues. Here’s an example of the execution history for our project:

When all tests are executed, Jenkins sends test results to those who need them via email and messengers. Jenkins can send such reports on a specified schedule or after particular events. The message with the automation test execution report looks like this:

With periodic reports, you can be on top of all automated test runs. Your QA engineers won’t need to spend their time handling these tests manually and will only need to review reports and investigate failed tests and suspicious results. 

Conclusion

Python provides QA engineers with lots of tools for quick and efficient test automation. To introduce automation to a new project, engineers have to develop a strategy, choose a framework to work with, and integrate it into the project’s CI/CD system.

The trick to automating your tests in a reliable manner is to choose the right tools and write relevant test cases. At Apriorit, we always make sure to cover key product functionality with tests and automate as many of them as possible. This approach helps us continually deliver quality products and use our QA engineers’ time efficiently.

Reach out to see how we can help you automate QA activities on your project!