Many articles and books are available at the present moment. They are about the way how to raise the automation process from the beginning, what and how to automate, and what tool it is better to choose.
I will share with you my experience in auto-testing. The project specificity plays an important role, but I would like to find something in common for all. Also, I want to talk about the way that you really need to follow in order to get a good result.
In this article, you can find the description of the software life-cycle, principles and requirements that are useful to observe in any automation process, the examples and advices in making important decisions in the process planning.
1. Environment for the auto – tests performing
2. The life-cycle of auto-tests
3. The main principles of the projecting
4. The requirements for auto-tests
It is perfect, when managers formulate the detailed plan for the automation process and at first define the tool and its features and then hire the specialist, who knows how to use this tool. In reality, the situations can be very different and all these steps can be performed in another order. So, it is good if a person has higher technical education, is interested in programming and already has the good experience in software development, and also has some experience in the administration.
Before you use some tool, make sure that it conforms to your expectations and performs all necessary functions, for example, that it identifies the necessary GUI elements.
The tools for the automation can have the client-server architecture (for example, TestComplete, Microsoft Test Manager). In this case, ready script will be executed only by the installed client. You should take this fact into consideration before you implement such tool, because it is necessary to install clients on every PC, where the scripts execute.
Other tools can compile the executable files (for example, AutoIT). This fact simplifies using of such scripts. But the distributed system, which can run scripts on the defined PC, controls the execution process and captures the final information about the task and system status, is absent in this case. So, we have to organize such scripts manually.
Also, it is necessary to consider the risks, which can appear as a result of not thorough knowledge of the tool. For example, it is inconvenient to test the program installation that needs a reboot in Microsoft Test Manager, because it is not provided that the system reboots during the test execution. In this case, it is convenient to use AutoIT.
The process of the automated tests development and the process of any software development are the same. So, it is necessary to think about the auto-tests seriously and thoroughly.
All the models of the software life-cycle fit here. The iteration model will be the most convenient model (see Figure 1), because the tests will be created step by step and a small scheme little by little will become a big scheme. Auto-tests will pass all the development stages.
Figure 1. Stages of the auto-tests development
You can see the generalized structure of the testing automation system on the Figure 2. Such information is created and stored there:
- The test set that is sufficient for the covering tested application according to the selected test criterion.
- Results of the test set running fixed in the Log file. The log file contains traces (“protocols”). Traces are the sequences of certain events (values of certain variables or variables packages) that are implemented in the test run and the implementation points of these events.
- Test cycle statistic that contains:
1) The results of running every test from the test set and their comparison with the reference values;
2) The facts that were the reason for making a decision about the continuation or finishing testing;
3) The cover criterion and its meeting degree that was reached in the test cycle.
The results of every run is a list of problems (mistakes and defects), which is saved to the so-called “project development base”.
Then, the fix work takes place. Every detected problem is identified, attributed to the corresponding module and developer, prioritized and tracked. This guarantees the problem solution (problem fix or referring to the list of the known problems, whose solution is delayed for some reasons) in the future builds. The version, which is fixed and built for testing, goes to the next testing cycle. The cycle repeats until the needed quality of program complex is achieved. In this iterative process, the tools of the testing automation provide the quick control of the fix results and checking the quality level reached in the product.
Figure 2. The generalized structure of the testing automation system
Intensity of the bug detection per unit of cost and reliability depend on the testing time and the SQA (Figure 3A). The more man-hours are put into the testing process, the fewer bugs remain undetected in the product. However, the perfection in the industrial programming has its limits. First of all, these limits are related to the costs on the program product creation and to an excess of quality, which is not needed for the customer. Finding optimum is a very important task for the tester and project manager.
In other words, the more the product is tested, the more errors are found and fixed and the product becomes more secure.
Figure 3. Testing costs
It is necessary to combine different methods and strategies of the debugging and testing to ensure the planned software quality when the costs are limited (Figure 3). This can be reached using the process of the software quality management.
The general scheme of the automatic testing system looks as follows:
Figure 4. The general scheme of the auto-test system
The auto-test system should have the following roles:
Server – an external data source, the storage for versions and results, the controller of the testing process.
Auto-test Manager – an application that distributes the tasks to the test agents. It can be located on the working PC of the tester.
Test Machine – a machine, which the test is performed on by a test agent.
The design philosophy can help you to develop easy to use and to manage auto-tests. These principles are as follows (all examples are given in the AutoIt language):
Auto-tests should be created for the specific purpose, which should provide a benefit to the project. For example, the benefit can be the time saved on regression testing, implementation of the tests, which are impossible or difficult to do manually (for example, load, performance testing, stress testing), etc.
Effectiveness is appreciable when you can get more results for the same time intervals using the automatic tests.
Auto-testing system must be stable and must respond to any failure or error. If one test from the test set finished with the error result, it must not prevent the execution of other tests that do not depend on it. So, we get the conclusion: the tests sets must be independent, if it is possible.
There are two advices how to make the scripts stable:
- Use the time-outs
- Global time-out for the whole script: it is used when local checks do not work or tested program hangs up. The script checks the value of the global time-out from time to time. If it reaches the critical value the script cancels its work with the error code and executes the necessary functions (for example, recording to the log, saving necessary information, and the system cleanup from the temporary files/registry keys, created by the script).
- Local time-out for such functions as
WinWaitActive(functions that receive the time-out as a parameter) means that function cancels the window waiting, completes with the error code and continues script execution. Then, the returned value is checked. If there is some error, the record is saved to the log. Then, script completes or performs something that does not depend on this window depending on the context.
- Catch the unexpected windows
Surely, every project has a set of windows that can appear or not appear depending on the settings of the project or the environment, in which it executes.
But if the window opens (for example, the message that the flash-player is not installed on your PC), it disables further work. You can use the
AdlibRegister(function1, timeout) and
AdlibUnRegister() functions in this case. If you write
Adlibregister(function1, timeout) in the beginning of the script the function “function1” will be performed parallel with the main work of the script every “timeout” milliseconds. . For example, you can get out the blocked window using the 2 seconds time-out, if you provide this in the function1 function (if the flash-player window is active, click OK).
In case a blocking window opens, you should set the time-outs on every step. Making the system screenshot and saving it with the same time stamp as the record in the log file is available in this case and in other cases.
Thereby, the list of the known predictable windows, which have an influence on the testing process and can be closed, will be in your script. Also, the list can contain the critical messages. For example, the message with the Installation failed text appears during the project installation. This obviously means that the script work must be interrupted and the script must inform the control system.
The report is a result of the auto-test performing. We can say just if the test is test passed or not. But if, for example, the tested program hangs up in the middle of the script performing we will get the negative result (or we will not get the result at all). That’s why it is necessary to record all actions or at least the main ones. So, we have two levels of the result detail: general and detailed. We can provide the both forms and choose them by the special parameter value.
The report must contain:
-the progress of the test;
-the information about the error, if it exists;
-the result of the script execution (passed or failed);
-the result explanation (for example, none files were founded: the test is failed OR all required files exist: test is passed).
Besides the report, you can save the necessary information such as:
- System screenshot at the error detection moment;
- Part of the Event log, which was created during the script execution;
- The list of the services which are working in the system;
- Presence or absence of the files and registry keys.
Auto-tests must be easy to correct because projects as a rule constantly develop and change. Independence from the platform, parameterization, manageability, and modularity provide this. You can find the description of these requirements in the following section.
All input data must be stored in the external data source, in the corresponding network file. To provide the flexibility of the input data, it is enough to send the absolute parameter to the script. This parameter is an address of one of the key files. It is possible to find all the necessary information relative to this address. It means that the changing values in the source data will not cause code changing (Figure 5).
Figure 5. The test flexibility of the input data
Also, it is possible to distinguish the flexibility according to the different properties of the tests, such as:
- Flexibility of the platform (cross-platform);
- Flexibility of the input data (parameterization);
- Flexibility of the code changing (modularity, manageability).
3.5 Easy Configuration
1. Minimize pre-conditions
Every test has its pre-conditions that must be performed before running.
Here is a bad example, where it is necessary to perform a lot of actions that could be atomized:
- install the tested project;
- make a number of settings;
- run the console;
- click OK, when the message appears;
- activate the first element of the menu tree.
The easier way is just to install the necessary program and run the script that will prepare the environment itself. Quite everything can be atomized in this example. Installation is a separate type of the testing. There can be something specific that takes a lot of time in the settings. The console langing and OK clicking are very easy actions. And the requirement such as activate the first element of the menu tree can be performed by default. So, only some settings remain in the list. Last of the settings or all settings also can be configured using the script.
Thus, we waste time in the unification of all these actions in one script once, and it saves time for everyone, who will use the scripts after that.
2. Don’t copy files. Use the external data source.
All data that is used and created during the test execution (scripts, configuration files, reports) should be stored on a network source. It means that there is one PC that does not take part in the testing process, but has the shared folder with the necessary information or the database. Tests run and perform on the network. So, it is unnecessary to copy scripts and data to every PC, where the tests perform. Also, we can save the results and reports in the one convenient place.
All tests can be performed directly from the network. All results can be stored in the network. Let’s imagine that your test environment contains a big number of virtual machines and you roll back to the “clear” snapshots. At first, you will waste additional time for copying and controlling files versions. Secondly, if the most data is static, you will relocate a lot of information risking to lose or deform it.
3. The test must work independently
All decisions, which must be made by user, during test performing (for example, to select a menu item, to enter private data or passwords) should be saved to the configuration file. Then, the script will get this information, when it is necessary (Figure 6).
It is unacceptable for the test to require user actions during its execution because this fact conflicts with the automation, which is our main goal.
The situation, when every PC has its login, password, and options, may appear in our example with a big number of virtual machines. In this case, it is simpler to store such data on every PC at the same address. It is just the simplest and the most obvious solution but not the optimal one.
Figure 6. Reading the parameter from the ini-file and choosing the corresponding option in the window.
You should follow the generally accepted requirements if you develop scripts. This will help you to use scripts easily, make changes in them, and increase the effectiveness of your work.
4.1 Error handling
We are talking about errors and not about bugs, about the main program errors and not about the script errors.
The error handling is one of the testing atomization goals. It means that we know beforehand what error can appear, because we check concrete test-cases. But the unexpected errors, which must be found and described in the report, can appear too. This request has something in common with the stability request, because all the methods, which provide the stability, check the system behavior with the expected result, i.e. handle any error.
The error handling and errors depend on the tool that we use. For example, you should just open the form and click the button using the script. If you use AutoIt, it is possible that such situation appears. While the form contains including the button is loading, the script will have already sent the event of the button click and this event will be senseless. Then, the required button was absent from the script viewpoint. If you use Microsoft Coded UI, then the script will be waiting until the platform is loaded, and then it will send the request on the button click. It is not done manually, it is the CodedUI feature. If you use the AutoIt, you can do the following: detect the source of the problem (whether it depends on the system or on the hardware) and set a small delay (for example,
Sleep(10)) in this place.
The way how to deal the errors are as follows:
Error 1: the unexpected window opens or the expected window has unexpected look.
Decision: use the time-outs. Use the local time-outs for the specific functions. Use the global time-out for the whole script. If the behavior is predictable, then you should add the processing of the situation to the function1 function. The
AdlibRegister(fuction1, timeout) function will activate the function1 function regularly.
Error 2: the actual result does not correspond to the expected result.
Decision: this is a bug (for the script). It is necessary to cancel the script and report about the error. You should look into the problem, decide what was the reason of such a mismatch and whether it is really a bug (if you are testing the new version, it is possible that the result has already changed).
The essence of the requirement has already been revealed in the other paragraphs (see the Flexibility principle). Scripts should have the handy structure, consist from the functions, follow the coding standards, and use the accepted methods. If a great number of scripts is developed using the same scheme, then you can quickly orientate in the first-seen code and change it quickly if necessary.
The code structure should be intuitively understandable if possible. Variables and functions should be named simply and their names should correspond to their purpose.
The Modularity means that the auto-test code should consist of a big number of parts. For example, it should consist of functions and separate scripts.
Program code often is divided into some files, each of which is compiled separately. Such modularity helps you to decrease the recompilation time (recompilation is needed when the changes are made). Also, modularity simplifies the group development.
The modularity is a possibility to make the replacement of the separate components (such as dll libraries) in the final program product. This replacement does not demand the project rebuilding. The module presence allows you to use them in the different test sets.
You have the following advantages, if you use the modularity:
- Nonredundancy (the absence of the repeating code);
- Re–Usability (the ability to use the program components for different purposes repeatedly).
The simple code example without modules is shown on the Figure 7. The same code with function is shown on the Figure 8. Program became easier, because it is simple to trace the main actions sequence. If you relocate the function to the library or to the separate script, you will be able to use it in the other auto-tests.
Figure 7. Example of the code redundancy
Figure 8. Example with the function
You can perform testing with the different detail level. So, some flexibility must be present. You can shorten or extend the testing area changing the parameters value. Use the parameters for the paths, file names, constant values, element names, etc.
Make the auto-tests independent from the platform if possible. It does not mean, that the test will work equally everywhere. You should foresee the environment features and take them into your consideration by the elementary checks in the beginning of the script execution.
Make auto-tests independent from such things:
- system version and updating;
- 3rd party software;
- browser (if it is a web-project);
- weather conditions (it should not be that it works today, but it does not work tomorrow);
- other specific factors.
For example, you should check the file presence after the project installation. There is the
FileExists() function specially for this case. To make this function working on the systems with different capacities it is necessary to compile two executable files (for x86 and for x64 correspondingly). You can use the third script, which runs one or another test in accordance with the capacity (see Figure 9), in this case.
Figure 9. The system capacity checking
Logging is the most important part of the reporting. The log file must contain the information about the failure or success.
The log file script from the previous example (checking the file presence) is shown below.
The following information is shown if the test completes successfully:
Such information is shown if the test is failed:
4.8 The system cleaning after the auto-tests execution.
The system cleaning after the auto-tests execution is the mandatory requirement. It is obligatory because you can put the temporary data to the registry (for example, you can divide the process into the phases and write the phase number to the registry, save the read-in configuration file path), activate the autologon and auto-loading for the script, etc. All these actions can prevent the next tests executions. Especially if it was long ago and you forgot what changes had been made.
You should do a lot of things before the automatic testing system implementation such as defining the detailed plan of actions, creating the necessary documentation, auto-test developing and testing. The process of the automatic system development is similar to the process of the software development. That’s why auto-tests should pass all the software life-cycle stages.
The shell, which the scripts will be developed in, has great importance. You should study all advantages and disadvantages of the used tools to avoid the problems and wasting time on the solutions search in future.
You should clarify the common code structure and its interaction with the testing control system before the scripts development; clarify the rules for the code composing; define the necessary principles and requirements for each test.
In your development process, you should have the phase of review for both the code and the general system of the tests management.
Don’t forget to analyze the results received using the automatic testing system. It can be that this system only increases the budget and the term of the project delivery.
Read also about AutoIT Code (standards and best practices)
The Methods of the testing automation course.
Lecture 8. Testing automation.
Continue reading our software QA tips: Estimation techniques in software testing.
Leanr how to improve your processes by introducing impact analysis in software testing.