Ignore:
Timestamp:
03/02/12 00:05:35 (2 years ago)
Author:
janezd <janez.demsar@…>
Branch:
default
Message:

Half-polished Orange.evaluation documentation

File:
1 edited

Legend:

Unmodified
Added
Removed
  • docs/reference/rst/Orange.evaluation.testing.rst

    r10339 r10414  
    66================================== 
    77 
    8 There are many ways to test prediction models on data. Orange includes 
    9 methods for cross-validation, leave-one out, random sampling and learning 
    10 curves. This methods handle learning of models and prediction of new 
    11 examples; they return :obj:`ExperimentResults` which can be passed to 
    12 :obj:`~Orange.evaluation.scoring` functions to evaluate model. 
     8Module :obj:`Orange.evaluation.testing` contains methods for 
     9cross-validation, leave-one out, random sampling and learning 
     10curves. These procedures split the data onto training and testing set 
     11and use the training data to induce models; models then make 
     12predictions for testing data. Predictions are collected in 
     13:obj:`ExperimentResults`, together with the actual classes and some 
     14other data. The latter can be given to functions 
     15:obj:`~Orange.evaluation.scoring` that compute the performance scores 
     16of models. 
    1317 
    1418.. literalinclude:: code/testing-example.py 
     19 
     20The following call makes 100 iterations of 70:30 test and stores all the 
     21induced classifiers. :: 
     22 
     23     res = Orange.evaluation.testing.proportion_test(learners, iris, 0.7, 100, store_classifiers=1) 
    1524 
    1625Different evaluation techniques are implemented as instance methods of 
    1726:obj:`Evaluation` class. For ease of use, an instance of this class is 
    1827created at module loading time and instance methods are exposed as functions 
    19 with the same name in Orange.evaluation.testing namespace. 
     28in :obj:`Orange.evaluation.testing`. 
     29 
     30Randomness in tests 
     31=================== 
     32 
     33If evaluation method uses random sampling, parameter 
     34``random_generator`` can be used to either provide either a random 
     35seed or an instance of :obj:`~Orange.misc.Random`. If omitted, a new 
     36instance of random generator is constructed for each call of the 
     37method with random seed 0. 
     38 
     39.. note:: 
     40 
     41    Running the same script twice will generally give the same 
     42    results. 
     43 
     44For conducting a repeatable set of experiments, construct an instance 
     45of :obj:`~Orange.misc.Random` and pass it to all of them. This way, 
     46all methods will use different random numbers, but they will be the 
     47same for each run of the script. 
     48 
     49For truly random number, set seed to a random number generated with 
     50python random generator. Since python's random generator is reset each 
     51time python is loaded with current system time as seed, results of the 
     52script will be different each time you run it. 
    2053 
    2154.. autoclass:: Evaluation 
    22    :members: 
    2355 
    24 .. autoclass:: ExperimentResults 
    25     :members: 
     56   .. automethod:: cross_validation 
    2657 
    27 .. autoclass:: TestedExample 
    28     :members: 
     58   .. automethod:: leave_one_out 
    2959 
    30 Generating random numbers 
    31 ========================= 
     60   .. automethod:: proportion_test 
    3261 
    33 Many evaluation 
     62   .. automethod:: test_with_indices 
    3463 
    35 *stratified* 
    36     Tells whether to stratify the random selections. Its default value is 
    37     :obj:`orange.StratifiedIfPossible` which stratifies selections 
    38     if the class variable is discrete and has no unknown values. 
     64   .. automethod:: one_fold_with_indices 
    3965 
    40 *random_generator* 
    41     If evaluation method relies on randomness, parameter ``random_generator`` 
    42     can be used to either provide a random seed or an instance of 
    43     :obj:`~Orange.misc.Random` which will be used to generate random numbers. 
     66   .. automethod:: learn_and_test_on_learn_data 
    4467 
    45     By default, a new instance of random generator is constructed for each 
    46     call of the method with random seed 0. 
     68   .. automethod:: learn_and_test_on_test_data 
    4769 
    48     If you use more than one method that is based on randomness, 
    49     you can construct an instance of :obj:`~Orange.misc.Random` and pass it 
    50     to all of them. This way, all methods will use different random numbers, 
    51     but this numbers will be the same for each run of the script. 
     70   .. automethod:: learning_curve(learners, examples, cv_indices=None, proportion_indices=None, proportions=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], preprocessors=(), random_generator=0, callback=None)¶ 
    5271 
    53     For truly random number, set seed to a random number generated with 
    54     python random generator. Since python's random generator is reset each 
    55     time python is loaded with current system time as seed, 
    56     results of the script will be different each time you run it. 
     72   .. automethod:: learning_curve_n(learners, examples, folds=10, proportions=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], stratification=StratifiedIfPossible, preprocessors=(), random_generator=0, callback=None) 
    5773 
    58 *preprocessors* 
    59     A list of preprocessors. It consists of tuples ``(c, preprocessor)``, 
    60     where ``c`` determines whether the preprocessor will be applied 
    61     to the learning set (``"L"``), test set (``"T"``) or to both 
    62     (``"B"``). The latter is applied first, when the example set is still 
    63     undivided. The ``"L"`` and ``"T"`` preprocessors are applied on the 
    64     separated subsets. Preprocessing testing examples is allowed only 
    65     on experimental procedures that do not report the TestedExample's 
    66     in the same order as examples in the original set. The second item 
    67     in the tuple, preprocessor can be either a pure Orange or a pure 
    68     Python preprocessor, that is, any function or callable class that 
    69     accepts a table of examples and weight, and returns a preprocessed 
    70     table and weight. 
     74   .. automethod:: learning_curve_with_test_data(learners, learn_set, test_set, times=10, proportions=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], stratification=StratifiedIfPossible, preprocessors=(), random_generator=0, store_classifiers=False, store_examples=False) 
    7175 
    72     This example will demonstrate the devastating effect of 100% class 
    73     noise on learning. :: 
     76   .. automethod:: test_on_data 
    7477 
    75         classnoise = orange.Preprocessor_addClassNoise(proportion=1.0) 
    76         res = Orange.evaluation.testing.proportion_test(learners, data, 0.7, 100, pps = [("L", classnoise)]) 
    77  
    78 *store_classifiers (keyword argument)* 
    79     If this flag is set, the testing procedure will store the constructed 
    80     classifiers. For each iteration of the test (eg for each fold in 
    81     cross validation, for each left out example in leave-one-out...), 
    82     the list of classifiers is appended to the ExperimentResults' 
    83     field classifiers. 
    84  
    85     The script below makes 100 repetitions of 70:30 test and store the 
    86     classifiers it induces. :: 
    87  
    88         res = Orange.evaluation.testing.proportion_test(learners, data, 0.7, 
    89         100, store_classifiers=1) 
    90  
    91  
    92 Knowing classes :obj:`TestedExample` that stores results of testing 
    93 for a single test example and :obj:`ExperimentResults` that stores a list of 
    94 TestedExamples along with some other data on experimental procedures 
    95 and classifiers used, is important if you would like to write your own 
    96 measures of quality of models, compatible the sampling infrastructure 
    97 provided by Orange. If not, you can skip the remainder of this page. 
    98  
    99  
    100  
    101 References 
    102 ========== 
    103  
    104 Salzberg, S. L. (1997). On comparing classifiers: Pitfalls to avoid 
    105 and a recommended approach. Data Mining and Knowledge Discovery 1, 
    106 pages 317-328. 
    107  
Note: See TracChangeset for help on using the changeset viewer.