source: orange/docs/reference/rst/Orange.evaluation.testing.rst @ 10192:2e2fcc44248e

Revision 10192:2e2fcc44248e, 4.3 KB checked in by anzeh <anze.staric@…>, 2 years ago (diff)

Updated documentation for evaluation.testing.

Line 
1.. index:: Testing, Sampling
2.. automodule:: Orange.evaluation.testing
3
4==================================
5Sampling and Testing (``testing``)
6==================================
7
8There are many ways to test prediction models on data. Orange includes
9methods for cross-validation, leave-one out, random sampling and learning
10curves. This methods handle learning of models and prediction of new
11examples; they return :obj:`ExperimentResults` which can be passed to
12:obj:`~Orange.evaluation.scoring` functions to evaluate model.
13
14.. literalinclude:: code/testing-example.py
15
16Different evaluation techniques are implemented as instance methods of
17:obj:`Evaluation` class. For ease of use, an instance of this class in
18created at module loading time and instance methods are exposed as functions
19with the same name in Orange.evaluation.testing namespace.
20
21.. autoclass:: Evaluation
22   :members:
23
24.. autoclass:: ExperimentResults
25    :members:
26
27.. autoclass:: TestedExample
28    :members:
29
30Generating random numbers
31=========================
32
33Many evaluation
34
35*stratified*
36    Tells whether to stratify the random selections. Its default value is
37    :obj:`orange.StratifiedIfPossible` which stratifies selections
38    if the class variable is discrete and has no unknown values.
39
40*random_generator*
41    If evaluation method relies on randomness, parameter ``random_generator``
42    can be used to either provide a random seed or an instance of
43    :obj:`~Orange.misc.Random` which will be used to generate random numbers.
44
45    By default, a new instance of random generator is constructed for each
46    call of the method with random seed 0.
47
48    If you use more than one method that is based on randomness,
49    you can construct an instance of :obj:`~Orange.misc.Random` and pass it
50    to all of them. This way, all methods will use different random numbers,
51    but this numbers will be the same for each run of the script.
52
53    For truly random number, set seed to a random number generated with
54    python random generator. Since python's random generator is reset each
55    time python is loaded with current system time as seed,
56    results of the script will be different each time you run it.
57
58*preprocessors*
59    A list of preprocessors. It consists of tuples ``(c, preprocessor)``,
60    where ``c`` determines whether the preprocessor will be applied
61    to the learning set (``"L"``), test set (``"T"``) or to both
62    (``"B"``). The latter is applied first, when the example set is still
63    undivided. The ``"L"`` and ``"T"`` preprocessors are applied on the
64    separated subsets. Preprocessing testing examples is allowed only
65    on experimental procedures that do not report the TestedExample's
66    in the same order as examples in the original set. The second item
67    in the tuple, preprocessor can be either a pure Orange or a pure
68    Python preprocessor, that is, any function or callable class that
69    accepts a table of examples and weight, and returns a preprocessed
70    table and weight.
71
72    This example will demonstrate the devastating effect of 100% class
73    noise on learning. ::
74
75        classnoise = orange.Preprocessor_addClassNoise(proportion=1.0)
76        res = Orange.evaluation.testing.proportion_test(learners, data, 0.7, 100, pps = [("L", classnoise)])
77
78*store_classifiers (keyword argument)*
79    If this flag is set, the testing procedure will store the constructed
80    classifiers. For each iteration of the test (eg for each fold in
81    cross validation, for each left out example in leave-one-out...),
82    the list of classifiers is appended to the ExperimentResults'
83    field classifiers.
84
85    The script below makes 100 repetitions of 70:30 test and store the
86    classifiers it induces. ::
87
88        res = Orange.evaluation.testing.proportion_test(learners, data, 0.7,
89        100, store_classifiers=1)
90
91
92Knowing classes :obj:`TestedExample` that stores results of testing
93for a single test example and :obj:`ExperimentResults` that stores a list of
94TestedExamples along with some other data on experimental procedures
95and classifiers used, is important if you would like to write your own
96measures of quality of models, compatible the sampling infrastructure
97provided by Orange. If not, you can skip the remainder of this page.
98
99
100
101References
102==========
103
104Salzberg, S. L. (1997). On comparing classifiers: Pitfalls to avoid
105and a recommended approach. Data Mining and Knowledge Discovery 1,
106pages 317-328.
107
Note: See TracBrowser for help on using the repository browser.