Regression Tests Scripts
Scripts for regression test are all Python scripts that are located in the first level subdirectories of the doc directory within core Orange distribution and distribution of add-on modules. For core Orange, these subdirectories are reference, modules and ofb. For add-ons, the only such subdirectory is modules. Any exceptions for Python scritps in these directories that are not intended to be regression scripts should be listed in the file exclude-from-regression.txt that resides in this directory, one file name per line. The line should include only the name of the file with its extension.
When writing the test scripts, consider that:
- test scripts should be short (typically, up to several tens of lines of code,
- they should run fast (consider having hundreeds of scripts, each running for 10 minutes), runtime should typically be few seconds at most,
- they should not rely on any external or any locally available internal resources that do not come up with installation (e.g., include all necessary data sets in the same directory, but make sure they are small).
Running Regression Test Scripts
The environment for script-based regression testing is on SVN in the module testing, directory regressionTests. You can install this environment anywhere: it will use orngEnviron to figure our where is your current installation of Orange and add-ons.
The module contains a script xtest.py which executes all regression scripts (see above) and checks whether they finish without the error, whether they give the same result if they are run for multiple times, and whether that result is the same as the expected output of the script. The files with the expect output for each script also come with regressionTests module within the subdirectory results.
Following are few examples that illustrate how to run regression tests:
Running xtest.py
xtest.py takes several arguments: a command and options, added by an optional names of strings that should appear in the names of regression files to be tested.
- update
- Command
updatetells xtest.py to run the scripts which failed the last time. This is the default command. It assumes that you have run thetestcommand already; it will not do anything if you run it immediately after a fresh installation from SVN. - test
- Regression testing using all available scripts.
- report
- Prints out the report.
- errors
- Similar to report, except that it only prints out the failed tests.
- help
- Prints out the command line arguments.
Here are the options:
- --module [orange|obi|text]
- Determines which module (core orange, bioinformatics add-on, or text mining add-on) to test (default: orange).
- -s, --single
- This option tells xtest.py to run each regression test only once, not the default three times.
Results of Regression Tests
Test of a particular script can have five different outcomes.
- crash
- The script crashed Orange with core dump (or general protection error or whatever)
- error
- The script raised an (unhandled) exception
- random
- The script ran OK, but gave different results in each iteration
- changed
- The script ran OK, but the result differs from the prescribed output
- OK
- Everything is OK. The
results subdirectory includes the files with expected results of the regression test. You've got all these from CVS. Their names equal the name of the script with a .txt extension appended. So, for tree2.py, the expected output would be in tree2.py.txt. If you also find a file tree2.py.linux2.txt (where instead linux2 you can see whatever is returned by the Python function sys.platform on a particular system), then this script's output is supposed to be different on this particular platform. The possible reasons are given below.
These are therefore the "official outputs". If you find no other files beginning with tree2.py, you're OK. Otherwise, you may find tree2.py.crash.txt, tree2.py.changed.txt, tree2.py.random1.txt and tree2.py.random2.txt or tree2.py.changed.txt. You can compare the output from this file (whichever you have) with the expected, or in case of a "random" output, compare the two results given by the same script.
Documentation scripts should, in principle, give the same output on all systems. There are, however, differences in floating point handling. A small one is in rounding: if the decimal begin cut off is 5, MS rounds the last decimal up, while gcc adds it to the closes odd number. xtest.py takes this into account when comparing files. The second difference - one for which we need the separate output files - occurs due to differences in code optimization; changing the order of floating-point operations can affect the result, and small changes can accumulate to larger differences.
Statistics on Module Coverage
Run the script called doc_statistics.py from the doc directory in core Orange distribution to obtain essentially statistics on how well do regression tests cover each of the Orange's modules. By default, the script will report statistics on core Orange distribution, but can be given an option obi or text to check the corresponding add-ons. Example: