source: orange/docs/reference/rst/Orange.evaluation.testing.rst @ 9506:da6b728a44de

Revision 9506:da6b728a44de, 7.5 KB checked in by anze <anze.staric@…>, 2 years ago (diff)

Updated documentation for evaluation.testing.

Line 
1.. index:: Testing, Sampling
2.. automodule:: Orange.evaluation.testing
3
4==================================
5Sampling and Testing (``testing``)
6==================================
7
8This module includes functions for data sampling and splitting, and for
9testing learners. It implements cross-validation, leave-one out, random
10sampling and learning curves. All functions return their results in the same
11format - an instance of :obj:`ExperimentResults`, described at the end of the
12page, or, in case of learning curves, a list of :obj:`ExperimentResults`. This
13object(s) can be passed to statistical function for model evaluation
14(classification accuracy, Brier score, ROC analysis...) available in
15module :obj:`Orange.evaluation.scoring`.
16
17Your scripts will thus basically conduct experiments using methods of
18:obj:`Evaluation` class and functions in  :obj:`Orange.evaluation.testing`,
19covered on this page and then evaluate the results by functions in
20:obj:`Orange.evaluation.scoring`. For those interested in writing their own
21statistical measures of the quality of models,
22description of :obj:`TestedExample` and :obj:`ExperimentResults` are
23available at the end of this page.
24
25.. note:: Orange has been "de-randomized". Running the same script twice
26    will generally give the same results, unless special care is taken to
27    randomize it. This is opposed to the previous versions where special
28    care needed to be taken to make experiments repeatable.
29    See arguments :obj:`randseed` and :obj:`randomGenerator` for the
30    explanation.
31
32Example scripts in this section suppose that the data is loaded and a
33list of learning algorithms is prepared.
34
35part of :download:`testing-test.py <code/testing-test.py>` (uses :download:`voting.tab <code/voting.tab>`)
36
37.. literalinclude:: code/testing-test.py
38    :start-after: import random
39    :end-before: def printResults(res)
40
41After testing is done, classification accuracies can be computed and
42printed by the following function.
43
44.. literalinclude:: code/testing-test.py
45    :pyobject: printResults
46
47Common Arguments
48================
49
50Many function in this module use a set of common arguments, which we define here.
51
52*learners*
53    A list of learning algorithms. These can be either pure Orange objects
54    (such as :obj:`Orange.classification.bayes.NaiveLearner`) or Python
55    classes or functions written in pure Python (anything that can be
56    called with the same arguments and results as Orange's classifiers
57    and performs similar function).
58
59*examples, learnset, testset*
60    Examples, given as an :obj:`Orange.data.Table` (some functions need an undivided
61    set of examples while others need examples that are already split
62    into two sets). If examples are weighted, pass them as a tuple
63    ``(examples, weightID)``. Weights are respected by learning and testing,
64    but not by sampling. When selecting 10% of examples, this means 10%
65    by number, not by weights. There is also no guarantee that sums
66    of example weights will be (at least roughly) equal for folds in
67    cross validation.
68
69*strat*
70    Tells whether to stratify the random selections. Its default value is
71    :obj:`orange.StratifiedIfPossible` which stratifies selections
72    if the class variable is discrete and has no unknown values.
73
74*randseed (obsolete: indicesrandseed), randomGenerator*
75    Random seed (``randseed``) or random generator (``randomGenerator``) for
76    random selection of examples. If omitted, random seed of 0 is used and
77    the same test will always select the same examples from the example
78    set. There are various slightly different ways to randomize it.
79
80    *
81      Set ``randomGenerator`` to :obj:`orange.globalRandom`. The function's
82      selection will depend upon Orange's global random generator that
83      is reset (with random seed 0) when Orange is imported. The Script's
84      output will therefore depend upon what you did after Orange was
85      first imported in the current Python session. ::
86
87          res = Orange.evaluation.testing.proportion_test(learners, data, 0.7,
88              randomGenerator=orange.globalRandom)
89
90    *
91      Construct a new :obj:`orange.RandomGenerator`. The code below,
92      for instance, will produce different results in each iteration,
93      but overall the same results each time it's run.
94
95      .. literalinclude:: code/testing-test.py
96        :start-after: but the same each time the script is run
97        :end-before: # End
98
99    *
100      Set the random seed (argument ``randseed``) to a random
101      number. Python has a global random generator that is reset when
102      Python is loaded, using the current system time for a seed. With this,
103      results will be (in general) different each time the script is run.
104
105
106      .. literalinclude:: code/testing-test.py
107        :start-after: proportionsTest that will give different results each time it is run
108        :end-before: # End
109
110
111      The same module also provides random generators as object, so
112      that you can have independent local random generators in case you
113      need them.
114
115*preprocessors*
116    A list of preprocessors. It consists of tuples ``(c, preprocessor)``,
117    where ``c`` determines whether the preprocessor will be applied
118    to the learning set (``"L"``), test set (``"T"``) or to both
119    (``"B"``). The latter is applied first, when the example set is still
120    undivided. The ``"L"`` and ``"T"`` preprocessors are applied on the
121    separated subsets. Preprocessing testing examples is allowed only
122    on experimental procedures that do not report the TestedExample's
123    in the same order as examples in the original set. The second item
124    in the tuple, preprocessor can be either a pure Orange or a pure
125    Python preprocessor, that is, any function or callable class that
126    accepts a table of examples and weight, and returns a preprocessed
127    table and weight.
128
129    This example will demonstrate the devastating effect of 100% class
130    noise on learning. ::
131
132        classnoise = orange.Preprocessor_addClassNoise(proportion=1.0)
133        res = Orange.evaluation.testing.proportion_test(learners, data, 0.7, 100, pps = [("L", classnoise)])
134
135*proportions*
136    Gives the proportions of learning examples at which the tests are
137    to be made, where applicable. The default is ``[0.1, 0.2, ..., 1.0]``.
138
139*store_classifiers (keyword argument)*
140    If this flag is set, the testing procedure will store the constructed
141    classifiers. For each iteration of the test (eg for each fold in
142    cross validation, for each left out example in leave-one-out...),
143    the list of classifiers is appended to the ExperimentResults'
144    field classifiers.
145
146    The script below makes 100 repetitions of 70:30 test and store the
147    classifiers it induces. ::
148
149        res = Orange.evaluation.testing.proportion_test(learners, data, 0.7,
150        100, store_classifiers=1)
151
152Sampling and Testing Functions
153==============================
154
155.. autoclass:: Evaluation
156   :members:
157
158Classes
159=======
160
161Knowing classes :obj:`TestedExample` that stores results of testing
162for a single test example and :obj:`ExperimentResults` that stores a list of
163TestedExamples along with some other data on experimental procedures
164and classifiers used, is important if you would like to write your own
165measures of quality of models, compatible the sampling infrastructure
166provided by Orange. If not, you can skip the remainder of this page.
167
168.. autoclass:: TestedExample
169    :members:
170
171.. autoclass:: ExperimentResults
172    :members:
173
174References
175==========
176
177Salzberg, S. L. (1997). On comparing classifiers: Pitfalls to avoid
178and a recommended approach. Data Mining and Knowledge Discovery 1,
179pages 317-328.
180
Note: See TracBrowser for help on using the repository browser.