Performing Regression Test of windPRO

Introduction

What is Regression Testing?

'Regression testing is a type of software testing that seeks to uncover new software bugs, or regressions, in existing functional and non-functional areas of a system after changes such as enhancements, patches or configuration changes, have been made to them.'

(Source: http://en.wikipedia.org/wiki/Regression_testing)

Regression Testing is done all the time by the windPRO software development team. Whenever some feature has been added or changes, it has to be ensured that these changes do not influence the rest of windPRO in any negative way. Performing this kind of testing manually can be quite an extensive job. Over the years a number of procedures and tools have been designed to help the process. The purpose of this document is to describe how windPRO end-users can benefit from one of these tools.

Target audience:

This guide is aimed at people with extensive knowledge about both windPRO and general Windows OS setup.

Regression Testing as a tool for end-users

The windPRO Calculation Tester is a small tool delivered together with a standard windPRO installation (from version 3.1). It allows anybody to design a windPRO project and use this project to perform regression testing of windPRO.

The basic flow consists of four steps:

• Step I: Design a simplified project with some data and calculations typical for your normal work-flow
• Step II: Run the project through the calculation tool and save the results for future reference
• Step III: Whenever required re-run the calculations and compare the results to the saved ones
• Step IV: Evaluate any changes in the results.
  • What is causing the change (windPRO or external setup)?
  • Is the change a software bug or an improvement?

This can be beneficial on several occasions:

• After installing windPRO upgrades (New version or Service Pack)
• After changing the setup of the windPRO environment
  • New PC / OS
  • Installing Windows Updates
  • Updating firewall / antivirus
  • Rearranging the organization of windPRO data

Guide

Step I - Designing a project

The creation of the project is done using standard windPRO. Any windPRO project should in principle be usable for a regression test, but for efficiency here is a few guidelines:

• Setup the most simple possible project that captures the calculation setup you want to test
• Limit the amount data in order to have quick calculations and hence quick feedback
• Create new dedicated projects for regression test and keep all data used in the project folder (and sub-folders)
  • Avoid references to any data on network drives and any data used in other projects
• Remember that windPRO is the subject of the test. The project in itself might not be very interesting
• Avoid using confidential data. If any issues turns up, it is very handy to be able to share the projects with other windPRO users or EMD

Step II - The first run

Ensure that any normal windPRO sessions have been closed and that all project data have been saved.

Start the Calculation Tester:

The Calculation Tester is a simple one-window program:

It contains a simple setup section:

Program Folder: The path to the installed windPRO version. Typical C:\Program Files (x86)\EMD\windPRO3.1

Project Folder: The path to the project(s) to be used. Any folder can be used. If your select a folder containing several projects, it is possible to control which projects are used in the section Projects on the right. For simplicity the rest of this guide will assume that only one project have been selected.

Action Log Folder: The path where the Calculation Tester should save log files regarding test execution.

After checking the setup section and ensuring the correct project have been selected, click Run.

You will now see windPRO startup and load the selected project. One by one each calculation inside the project will be re-calculated and selected results will be written to a text-file inside a "results" folder. Certain types of calculations might also output shape files. (e.g. Shadow / Decibel)

Do not interact with windPRO while the project is running the calculations. After all calculations have been completed windPRO will close down again. You will see some status messages in the results section.

After windPRO have closed down the Calculation tester will look for a "Base" folder and compare to the "Results" folder, file by file. On this first run the Base folder will not exist and all you will see is messages like "Not found in base results: XXXXX".

Manually inspect the files in the "Results" folder. The content is roughly identical to selected Result-to-file exports from a normal windPRO session.

Rename the folder "Results" to "Base". Now these results will be considered as the correct results. At this point you should make sure to create a backup of the project and the matching Results folder.

Notice: Some settings are stored inside windPRO and not inside the project. E.g. the selection of WAsP version (9/10/11) is a windPRO setting and the it is not stored in the project itself.

To ensure the Calculation Tester runs with the correct WAsP version, you will need to create a small text file named "RegOptions.txt" and place it in the project folder.

The file should contain a single line:

WAsP/WAsPVersion=X

where X can be:

8= WAsP 12, 7 = WAsP 11, 6 = WAsP 10.2, 4 = WAsP 6-9

The file can also be use to control the language in which windPRO will startup.

Language/LocalizerExt=X

where X can be:

de = german, dan = danish, en = english

Step III - Rerunning the calculations

Rerunning the calculations in a project is straight forward. Simply re-open the Calculation Tester, if required change the path to the windPRO version used for calculation, select the project and click Run.

When the calculations are done, the "Base" and the "Results" folder will be compared and a simple Ok or Different will be listed for each file.

Step IV - Evaluating any changes

If all tests return the same results as the original run, there is no need for further actions. It has been tested that windPRO is working in the same way and is giving the same results as before. At least within the scope of the defined calculation. It is important to emphasize that this is a testing tool. It has been tested that the specified project/calculation is behaving as expected, it is not general validation of windPRO.

Evaluating any possible changes in the results will often be the most challenging part of running regression tests.

• Double check that the input to the calculation have not be altered in any way.
  • The first file in the "Results" folder will always be a file called "FilesInProject.txt". It contains a simple list of the external files used in the project. The file also contains the time stamp and the size of the files. It can be very help when ensuring that files have not been updated/overwritten.
• Having a good comparison tool is very important. The windPRO Tester tool do not have this function built-in. Several external tools are available https://en.wikipedia.org/wiki/Comparison_of_file_comparison_tools
• Changes in the results might stem from bug fixes or changes in windPRO. Always consult the list of changes for a given Version/Service Pack to try to find an explanation for any change.
• When a change in the results have been verified it should be copied to the base folder and saved together with the project.