source: orange/docs/reference/rst/Orange.evaluation.testing.rst @ 9696:26fc745b27f6

Revision 9696:26fc745b27f6, 7.5 KB checked in by Jure Zbontar <jure.zbontar@…>, 2 years ago (diff)

Renamed RandomGenerator to Random in docs.

Line 
1.. index:: Testing, Sampling
2.. automodule:: Orange.evaluation.testing
3
4==================================
5Sampling and Testing (``testing``)
6==================================
7
8This module includes functions for data sampling and splitting, and for
9testing learners. It implements cross-validation, leave-one out, random
10sampling and learning curves. All functions return their results in the same
11format - an instance of :obj:`ExperimentResults`, described at the end of the
12page, or, in case of learning curves, a list of :obj:`ExperimentResults`. This
13object(s) can be passed to statistical function for model evaluation
14(classification accuracy, Brier score, ROC analysis...) available in
15module :obj:`Orange.evaluation.scoring`.
16
17Your scripts will thus basically conduct experiments using methods of
18:obj:`Evaluation` class and functions in  :obj:`Orange.evaluation.testing`,
19covered on this page and then evaluate the results by functions in
20:obj:`Orange.evaluation.scoring`. For those interested in writing their own
21statistical measures of the quality of models,
22description of :obj:`TestedExample` and :obj:`ExperimentResults` are
23available at the end of this page.
24
25.. note:: Orange has been "de-randomized". Running the same script twice
26    will generally give the same results, unless special care is taken to
27    randomize it. This is opposed to the previous versions where special
28    care needed to be taken to make experiments repeatable.
29    See argument obj:`random_generator` for the explanation.
30
31Example scripts in this section suppose that the data is loaded and a
32list of learning algorithms is prepared.
33
34part of :download:`testing-test.py <code/testing-test.py>` (uses :download:`voting.tab <code/voting.tab>`)
35
36.. literalinclude:: code/testing-test.py
37    :start-after: import random
38    :end-before: def printResults(res)
39
40After testing is done, classification accuracies can be computed and
41printed by the following function.
42
43.. literalinclude:: code/testing-test.py
44    :pyobject: printResults
45
46Common Arguments
47================
48
49Many function in this module use a set of common arguments, which we define here.
50
51*learners*
52    A list of learning algorithms. These can be either pure Orange objects
53    (such as :obj:`Orange.classification.bayes.NaiveLearner`) or Python
54    classes or functions written in pure Python (anything that can be
55    called with the same arguments and results as Orange's classifiers
56    and performs similar function).
57
58*examples, learnset, testset*
59    Examples, given as an :obj:`Orange.data.Table` (some functions need an undivided
60    set of examples while others need examples that are already split
61    into two sets). If examples are weighted, pass them as a tuple
62    ``(examples, weightID)``. Weights are respected by learning and testing,
63    but not by sampling. When selecting 10% of examples, this means 10%
64    by number, not by weights. There is also no guarantee that sums
65    of example weights will be (at least roughly) equal for folds in
66    cross validation.
67
68*strat*
69    Tells whether to stratify the random selections. Its default value is
70    :obj:`orange.StratifiedIfPossible` which stratifies selections
71    if the class variable is discrete and has no unknown values.
72
73*randseed (obsolete: indicesrandseed), random_generator*
74    Random seed (``randseed``) or random generator (``random_generator``) for
75    random selection of examples. If omitted, random seed of 0 is used and
76    the same test will always select the same examples from the example
77    set. There are various slightly different ways to randomize it.
78
79    *
80      Set ``random_generator`` to :obj:`orange.globalRandom`. The function's
81      selection will depend upon Orange's global random generator that
82      is reset (with random seed 0) when Orange is imported. The Script's
83      output will therefore depend upon what you did after Orange was
84      first imported in the current Python session. ::
85
86          res = Orange.evaluation.testing.proportion_test(learners, data, 0.7,
87              random_generator=orange.globalRandom)
88
89    *
90      Construct a new :obj:`Orange.misc.Random`. The code below,
91      for instance, will produce different results in each iteration,
92      but overall the same results each time it's run.
93
94      .. literalinclude:: code/testing-test.py
95        :start-after: but the same each time the script is run
96        :end-before: # End
97
98    *
99      Set the random seed (argument ``randseed``) to a random
100      number. Python has a global random generator that is reset when
101      Python is loaded, using the current system time for a seed. With this,
102      results will be (in general) different each time the script is run.
103
104
105      .. literalinclude:: code/testing-test.py
106        :start-after: proportionsTest that will give different results each time it is run
107        :end-before: # End
108
109
110      The same module also provides random generators as object, so
111      that you can have independent local random generators in case you
112      need them.
113
114*preprocessors*
115    A list of preprocessors. It consists of tuples ``(c, preprocessor)``,
116    where ``c`` determines whether the preprocessor will be applied
117    to the learning set (``"L"``), test set (``"T"``) or to both
118    (``"B"``). The latter is applied first, when the example set is still
119    undivided. The ``"L"`` and ``"T"`` preprocessors are applied on the
120    separated subsets. Preprocessing testing examples is allowed only
121    on experimental procedures that do not report the TestedExample's
122    in the same order as examples in the original set. The second item
123    in the tuple, preprocessor can be either a pure Orange or a pure
124    Python preprocessor, that is, any function or callable class that
125    accepts a table of examples and weight, and returns a preprocessed
126    table and weight.
127
128    This example will demonstrate the devastating effect of 100% class
129    noise on learning. ::
130
131        classnoise = orange.Preprocessor_addClassNoise(proportion=1.0)
132        res = Orange.evaluation.testing.proportion_test(learners, data, 0.7, 100, pps = [("L", classnoise)])
133
134*proportions*
135    Gives the proportions of learning examples at which the tests are
136    to be made, where applicable. The default is ``[0.1, 0.2, ..., 1.0]``.
137
138*store_classifiers (keyword argument)*
139    If this flag is set, the testing procedure will store the constructed
140    classifiers. For each iteration of the test (eg for each fold in
141    cross validation, for each left out example in leave-one-out...),
142    the list of classifiers is appended to the ExperimentResults'
143    field classifiers.
144
145    The script below makes 100 repetitions of 70:30 test and store the
146    classifiers it induces. ::
147
148        res = Orange.evaluation.testing.proportion_test(learners, data, 0.7,
149        100, store_classifiers=1)
150
151Sampling and Testing Functions
152==============================
153
154.. autoclass:: Evaluation
155   :members:
156
157Classes
158=======
159
160Knowing classes :obj:`TestedExample` that stores results of testing
161for a single test example and :obj:`ExperimentResults` that stores a list of
162TestedExamples along with some other data on experimental procedures
163and classifiers used, is important if you would like to write your own
164measures of quality of models, compatible the sampling infrastructure
165provided by Orange. If not, you can skip the remainder of this page.
166
167.. autoclass:: TestedExample
168    :members:
169
170.. autoclass:: ExperimentResults
171    :members:
172
173References
174==========
175
176Salzberg, S. L. (1997). On comparing classifiers: Pitfalls to avoid
177and a recommended approach. Data Mining and Knowledge Discovery 1,
178pages 317-328.
179
Note: See TracBrowser for help on using the repository browser.