Orange

Regression Tests Scripts

Scripts for regression test are all Python scripts that are located in the first level subdirectories of the doc directory within core Orange distribution and distribution of add-on modules. For core Orange, these subdirectories are reference, modules and ofb. For add-ons, the only such subdirectory is modules. Any exceptions for Python scritps in these directories that are not intended to be regression scripts should be listed in the file exclude-from-regression.txt that resides in this directory, one file name per line. The line should include only the name of the file with its extension.

When writing the test scripts, consider that:

  • test scripts should be short (typically, up to several tens of lines of code,
  • they should run fast (consider having hundreeds of scripts, each running for 10 minutes), runtime should typically be few seconds at most,
  • they should not rely on any external or any locally available internal resources that do not come up with installation (e.g., include all necessary data sets in the same directory, but make sure they are small).

Running Regression Test Scripts

The environment for script-based regression testing is on SVN in the module testing, directory regressionTests. You can install this environment anywhere: it will use orngEnviron to figure our where is your current installation of Orange and add-ons.

The module contains a script xtest.py which executes all regression scripts (see above) and checks whether they finish without the error, whether they give the same result if they are run for multiple times, and whether that result is the same as the expected output of the script. The files with the expect output for each script also come with regressionTests module within the subdirectory results.

Following are few examples that illustrate how to run regression tests:

> python xtest.py test > python xtest.py test tree > python xtest.py update > python xtest.py test --module=orange25 > python xtest.py test --module=orange25 tree > python xtest.py > python xtest.py report --module=obi

Running xtest.py

xtest.py takes several arguments: a command and options, added by an optional names of strings that should appear in the names of regression files to be tested.

update
Command update tells xtest.py to run the scripts which failed the last time. This is the default command. It assumes that you have run the test command already; it will not do anything if you run it immediately after a fresh installation from SVN.
test
Regression testing using all available scripts.
report
Prints out the report.
errors
Similar to report, except that it only prints out the failed tests.
help
Prints out the command line arguments.

Here are the options:

--module [orange|obi|text]
Determines which module (core orange, bioinformatics add-on, or text mining add-on) to test (default: orange).
-s, --single
This option tells xtest.py to run each regression test only once, not the default three times.

Results of Regression Tests

Test of a particular script can have five different outcomes.

crash
The script crashed Orange with core dump (or general protection error or whatever)
error
The script raised an (unhandled) exception
random
The script ran OK, but gave different results in each iteration
changed
The script ran OK, but the result differs from the prescribed output
OK
Everything is OK.
The results subdirectory includes the files with expected results of the regression test. You've got all these from CVS. Their names equal the name of the script with a .txt extension appended. So, for tree2.py, the expected output would be in tree2.py.txt. If you also find a file tree2.py.linux2.txt (where instead linux2 you can see whatever is returned by the Python function sys.platform on a particular system), then this script's output is supposed to be different on this particular platform. The possible reasons are given below.

These are therefore the "official outputs". If you find no other files beginning with tree2.py, you're OK. Otherwise, you may find tree2.py.crash.txt, tree2.py.changed.txt, tree2.py.random1.txt and tree2.py.random2.txt or tree2.py.changed.txt. You can compare the output from this file (whichever you have) with the expected, or in case of a "random" output, compare the two results given by the same script.

Documentation scripts should, in principle, give the same output on all systems. There are, however, differences in floating point handling. A small one is in rounding: if the decimal begin cut off is 5, MS rounds the last decimal up, while gcc adds it to the closes odd number. xtest.py takes this into account when comparing files. The second difference - one for which we need the separate output files - occurs due to differences in code optimization; changing the order of floating-point operations can affect the result, and small changes can accumulate to larger differences.

Statistics on Module Coverage

Run the script called doc_statistics.py from the doc directory in core Orange distribution to obtain essentially statistics on how well do regression tests cover each of the Orange's modules. By default, the script will report statistics on core Orange distribution, but can be given an option obi or text to check the corresponding add-ons. Example:

> python doc_statistics.py obi DOCUMENTATION OF MODULES There are 14 modules, 10 with documentation Of 14 modules, 4 are not documented: obiData, obiDicty, obiExpression, obiGenomicsUpdate REGRESSION TESTING SCRIPTS Regression scripts for documentation section: modules 11 Regression scripts that use a specific documented module: obiChem 0 obiProb 0 obiTaxonomy 0 obiGeneMatch 1 obiKEGG 1 obiMeSH 1 obiGsea 2 obiAssess 3 obiGO 3 obiGeneSets 4