source: orange/orange/Orange/evaluation/testing.py @ 7616:c392c6b940c3

Revision 7616:c392c6b940c3, 40.0 KB checked in by ales_erjavec <ales.erjavec@…>, 3 years ago (diff)
  • fixed some old style exceptions
Line 
1"""
2.. index:: Testing, Sampling
3
4====================
5Sampling and Testing
6====================
7
8This module includes functions for data sampling and splitting, and for
9testing learners. It implements cross-validation, leave-one out, random
10sampling and learning curves. All functions return their results in the same
11format - an instance of :obj:`ExperimentResults`, described at the end of the
12page, or, in case of learning curves, a list of :obj:`ExperimentResults`. This
13object(s) can be passed to statistical function for model evaluation
14(classification accuracy, Brier score, ROC analysis...) available in
15module :obj:`Orange.evaluation.scoring`.
16
17Your scripts will thus basically conduct experiments using functions in
18:obj:`Orange.evaluation.testing`, covered on this page and then evaluate
19the results by functions in :obj:`Orange.evaluation.scoring`. For those
20interested in writing their own statistical measures of the quality of
21models, description of :obj:`TestedExample` and :obj:`ExperimentResults`
22are available at the end of this page.
23
24.. note:: Orange has been "de-randomized". Running the same script twice
25    will generally give the same results, unless special care is taken to
26    randomize it. This is opposed to the previous versions where special
27    care needed to be taken to make experiments repeatable. See arguments
28    :obj:`randseed` and :obj:`randomGenerator` for the explanation.
29
30Example scripts in this section suppose that the data is loaded and a
31list of learning algorithms is prepared.
32
33part of `testing-test.py`_ (uses `voting.tab`_)
34
35.. literalinclude:: code/testing-test.py
36    :start-after: import random
37    :end-before: def printResults(res)
38
39After testing is done, classification accuracies can be computed and
40printed by the following function.
41
42.. literalinclude:: code/testing-test.py
43    :pyobject: printResults
44
45.. _voting.tab: code/voting.tab
46.. _testing-test.py: code/testing-test.py
47
48Common Arguments
49================
50
51Many function in this module use a set of common arguments, which we define here.
52
53*learners*
54    A list of learning algorithms. These can be either pure Orange objects
55    (such as :obj:`Orange.classification.bayes.NaiveLearner`) or Python
56    classes or functions written in pure Python (anything that can be
57    called with the same arguments and results as Orange's classifiers
58    and performs similar function).
59
60*examples, learnset, testset*
61    Examples, given as an :obj:`Orange.data.Table` (some functions need an undivided
62    set of examples while others need examples that are already split
63    into two sets). If examples are weighted, pass them as a tuple
64    ``(examples, weightID)``. Weights are respected by learning and testing,
65    but not by sampling. When selecting 10% of examples, this means 10%
66    by number, not by weights. There is also no guarantee that sums
67    of example weights will be (at least roughly) equal for folds in
68    cross validation.
69
70*strat*
71    Tells whether to stratify the random selections. Its default value is
72    :obj:`orange.StratifiedIfPossible` which stratifies selections
73    if the class variable is discrete and has no unknown values.
74
75*randseed (obsolete: indicesrandseed), randomGenerator*
76    Random seed (``randseed``) or random generator (``randomGenerator``) for
77    random selection of examples. If omitted, random seed of 0 is used and
78    the same test will always select the same examples from the example
79    set. There are various slightly different ways to randomize it.
80
81    *
82      Set ``randomGenerator`` to :obj:`orange.globalRandom`. The function's
83      selection will depend upon Orange's global random generator that
84      is reset (with random seed 0) when Orange is imported. The Script's
85      output will therefore depend upon what you did after Orange was
86      first imported in the current Python session. ::
87
88          res = orngTest.proportionTest(learners, data, 0.7,
89              randomGenerator=orange.globalRandom)
90
91    *
92      Construct a new :obj:`orange.RandomGenerator`. The code below,
93      for instance, will produce different results in each iteration,
94      but overall the same results each time it's run.
95
96      .. literalinclude:: code/testing-test.py
97        :start-after: but the same each time the script is run
98        :end-before: # End
99
100    *
101      Set the random seed (argument ``randseed``) to a random
102      number. Python has a global random generator that is reset when
103      Python is loaded, using the current system time for a seed. With this,
104      results will be (in general) different each time the script is run.
105
106
107      .. literalinclude:: code/testing-test.py
108        :start-after: proportionsTest that will give different results each time it is run
109        :end-before: # End
110
111
112      The same module also provides random generators as object, so
113      that you can have independent local random generators in case you
114      need them.
115
116*pps*
117    A list of preprocessors. It consists of tuples ``(c, preprocessor)``,
118    where ``c`` determines whether the preprocessor will be applied
119    to the learning set (``"L"``), test set (``"T"``) or to both
120    (``"B"``). The latter is applied first, when the example set is still
121    undivided. The ``"L"`` and ``"T"`` preprocessors are applied on the
122    separated subsets. Preprocessing testing examples is allowed only
123    on experimental procedures that do not report the TestedExample's
124    in the same order as examples in the original set. The second item
125    in the tuple, preprocessor can be either a pure Orange or a pure
126    Python preprocessor, that is, any function or callable class that
127    accepts a table of examples and weight, and returns a preprocessed
128    table and weight.
129
130    This example will demonstrate the devastating effect of 100% class
131    noise on learning. ::
132
133        classnoise = orange.Preprocessor_addClassNoise(proportion=1.0)
134        res = orngTest.proportionTest(learners, data, 0.7, 100, pps = [("L", classnoise)])
135
136*proportions*
137    Gives the proportions of learning examples at which the tests are
138    to be made, where applicable. The default is ``[0.1, 0.2, ..., 1.0]``.
139
140*storeClassifiers (keyword argument)*
141    If this flag is set, the testing procedure will store the constructed
142    classifiers. For each iteration of the test (eg for each fold in
143    cross validation, for each left out example in leave-one-out...),
144    the list of classifiers is appended to the ExperimentResults'
145    field classifiers.
146
147    The script below makes 100 repetitions of 70:30 test and store the
148    classifiers it induces. ::
149
150        res = orngTest.proportionTest(learners, data, 0.7, 100, storeClassifier=1)
151
152*verbose (keyword argument)*
153    Several functions can report their progress if you add a keyword
154    argument ``verbose=1``.
155
156Sampling and Testing Functions
157==============================
158
159.. autofunction:: proportionTest
160.. autofunction:: leaveOneOut
161.. autofunction:: crossValidation
162.. autofunction:: testWithIndices
163.. autofunction:: learningCurve
164.. autofunction:: learningCurveN
165.. autofunction:: learningCurveWithTestData
166.. autofunction:: learnAndTestOnTestData
167.. autofunction:: learnAndTestOnLearnData
168.. autofunction:: testOnData
169
170Classes
171=======
172
173Knowing classes :obj:`TestedExample` that stores results of testing
174for a single test example and :obj:`ExperimentResults` that stores a list of
175TestedExamples along with some other data on experimental procedures
176and classifiers used, is important if you would like to write your own
177measures of quality of models, compatible the sampling infrastructure
178provided by Orange. If not, you can skip the remainder of this page.
179
180.. autoclass:: TestedExample
181    :members:
182
183.. autoclass:: ExperimentResults
184    :members:
185
186References
187==========
188
189Salzberg, S. L. (1997). On comparing classifiers: Pitfalls to avoid
190and a recommended approach. Data Mining and Knowledge Discovery 1,
191pages 317-328.
192
193"""
194
195import Orange
196from Orange.misc import demangleExamples, getobjectname, printVerbose
197import exceptions, cPickle, os, os.path
198
199#### Some private stuff
200
201def encodePP(pps):
202    pps=""
203    for pp in pps:
204        objname = getobjectname(pp[1], "")
205        if len(objname):
206            pps+="_"+objname
207        else:
208            return "*"
209    return pps
210
211#### Data structures
212
213class TestedExample:
214    """
215    TestedExample stores predictions of different classifiers for a single testing example.
216
217    .. attribute:: classes
218
219        A list of predictions of type Value, one for each classifier.
220
221    .. attribute:: probabilities
222       
223        A list of probabilities of classes, one for each classifier.
224
225    .. attribute:: iterationNumber
226
227        Iteration number (e.g. fold) in which the TestedExample was created/tested.
228
229    .. attribute:: actualClass
230
231        The correct class of the example
232
233    .. attribute:: weight
234
235        Example's weight. Even if the example set was not weighted,
236        this attribute is present and equals 1.0.
237
238    :param iterationNumber:
239    :paramtype iterationNumber: type???
240    :param actualClass:
241    :paramtype actualClass: type???
242    :param n:
243    :paramtype n: int
244    :param weight:
245    :paramtype weight: float
246
247    """
248
249    def __init__(self, iterationNumber=None, actualClass=None, n=0, weight=1.0):
250        self.classes = [None]*n
251        self.probabilities = [None]*n
252        self.iterationNumber = iterationNumber
253        self.actualClass= actualClass
254        self.weight = weight
255   
256    def addResult(self, aclass, aprob):
257        """Appends a new result (class and probability prediction by a single classifier) to the classes and probabilities field."""
258   
259        if type(aclass.value)==float:
260            self.classes.append(float(aclass))
261            self.probabilities.append(aprob)
262        else:
263            self.classes.append(int(aclass))
264            self.probabilities.append(list(aprob))
265
266    def setResult(self, i, aclass, aprob):
267        """Sets the result of the i-th classifier to the given values."""
268        if type(aclass.value)==float:
269            self.classes[i] = float(aclass)
270            self.probabilities[i] = aprob
271        else:
272            self.classes[i] = int(aclass)
273            self.probabilities[i] = list(aprob)
274
275class ExperimentResults(object):
276    """
277    ``ExperimentResults`` stores results of one or more repetitions of
278    some test (cross validation, repeated sampling...) under the same
279    circumstances.
280
281    .. attribute:: results
282
283        A list of instances of TestedExample, one for each example in
284        the dataset.
285
286    .. attribute:: classifiers
287
288        A list of classifiers, one element for each repetition (eg
289        fold). Each element is a list of classifiers, one for each
290        learner. This field is used only if storing is enabled by
291        ``storeClassifiers=1``.
292
293    .. attribute:: numberOfIterations
294
295        Number of iterations. This can be the number of folds
296        (in cross validation) or the number of repetitions of some
297        test. ``TestedExample``'s attribute ``iterationNumber`` should
298        be in range ``[0, numberOfIterations-1]``.
299
300    .. attribute:: numberOfLearners
301
302        Number of learners. Lengths of lists classes and probabilities
303        in each :obj:`TestedExample` should equal ``numberOfLearners``.
304
305    .. attribute:: loaded
306
307        If the experimental method supports caching and there are no
308        obstacles for caching (such as unknown random seeds), this is a
309        list of boolean values. Each element corresponds to a classifier
310        and tells whether the experimental results for that classifier
311        were computed or loaded from the cache.
312
313    .. attribute:: weights
314
315        A flag telling whether the results are weighted. If ``False``,
316        weights are still present in ``TestedExamples``, but they are
317        all ``1.0``. Clear this flag, if your experimental procedure
318        ran on weighted testing examples but you would like to ignore
319        the weights in statistics.
320
321    """
322    def __init__(self, iterations, classifierNames, classValues, weights, baseClass=-1, **argkw):
323        self.classValues = classValues
324        self.classifierNames = classifierNames
325        self.numberOfIterations = iterations
326        self.numberOfLearners = len(classifierNames)
327        self.results = []
328        self.classifiers = []
329        self.loaded = None
330        self.baseClass = baseClass
331        self.weights = weights
332        self.__dict__.update(argkw)
333
334    def loadFromFiles(self, learners, filename):
335        self.loaded = []
336     
337        for i in range(len(learners)):
338            f = None
339            try:
340                f = open(".\\cache\\"+filename % getobjectname(learners[i], "*"), "rb")
341                d = cPickle.load(f)
342                for ex in range(len(self.results)):
343                    tre = self.results[ex]
344                    if (tre.actualClass, tre.iterationNumber) != d[ex][0]:
345                        raise SystemError, "mismatching example tables or sampling"
346                    self.results[ex].setResult(i, d[ex][1][0], d[ex][1][1])
347                self.loaded.append(1)
348            except exceptions.Exception:
349                self.loaded.append(0)
350            if f:
351                f.close()
352               
353        return not 0 in self.loaded               
354               
355    def saveToFiles(self, learners, filename):
356        """
357        Saves and load testing results. ``learners`` is a list of learners and
358        ``filename`` is a template for the filename. The attribute loaded is
359        initialized so that it contains 1's for the learners whose data
360        was loaded and 0's for learners which need to be tested. The
361        function returns 1 if all the files were found and loaded,
362        and 0 otherwise.
363
364        The data is saved in a separate file for each classifier. The
365        file is a binary pickle file containing a list of tuples
366        ``((x.actualClass, x.iterationNumber), (x.classes[i],
367        x.probabilities[i]))`` where ``x`` is a :obj:`TestedExample`
368        and ``i`` is the index of a learner.
369
370        The file resides in the directory ``./cache``. Its name consists
371        of a template, given by a caller. The filename should contain
372        a %s which is replaced by name, shortDescription, description,
373        func_doc or func_name (in that order) attribute of the learner
374        (this gets extracted by orngMisc.getobjectname). If a learner
375        has none of these attributes, its class name is used.
376
377        Filename should include enough data to make sure that it
378        indeed contains the right experimental results. The function
379        :obj:`learningCurve`, for example, forms the name of the file
380        from a string ``"{learningCurve}"``, the proportion of learning
381        examples, random seeds for cross-validation and learning set
382        selection, a list of preprocessors' names and a checksum for
383        examples. Of course you can outsmart this, but it should suffice
384        in most cases.
385
386        """
387
388        for i in range(len(learners)):
389            if self.loaded[i]:
390                continue
391           
392            fname=".\\cache\\"+filename % getobjectname(learners[i], "*")
393            if not "*" in fname:
394                if not os.path.isdir("cache"):
395                    os.mkdir("cache")
396                f=open(fname, "wb")
397                pickler=cPickle.Pickler(f, 1)
398                pickler.dump([(  (x.actualClass, x.iterationNumber), (x.classes[i], x.probabilities[i])  ) for x in self.results])
399                f.close()
400
401    def remove(self, index):
402        """remove one learner from evaluation results"""
403        for r in self.results:
404            del r.classes[index]
405            del r.probabilities[index]
406        del self.classifierNames[index]
407        self.numberOfLearners -= 1
408
409    def add(self, results, index, replace=-1):
410        """add evaluation results (for one learner)"""
411        if len(self.results)<>len(results.results):
412            raise SystemError, "mismatch in number of test cases"
413        if self.numberOfIterations<>results.numberOfIterations:
414            raise SystemError, "mismatch in number of iterations (%d<>%d)" % \
415                  (self.numberOfIterations, results.numberOfIterations)
416        if len(self.classifiers) and len(results.classifiers)==0:
417            raise SystemError, "no classifiers in results"
418
419        if replace < 0 or replace >= self.numberOfLearners: # results for new learner
420            self.classifierNames.append(results.classifierNames[index])
421            self.numberOfLearners += 1
422            for i,r in enumerate(self.results):
423                r.classes.append(results.results[i].classes[index])
424                r.probabilities.append(results.results[i].probabilities[index])
425            if len(self.classifiers):
426                for i in range(self.numberOfIterations):
427                    self.classifiers[i].append(results.classifiers[i][index])
428        else: # replace results of existing learner
429            self.classifierNames[replace] = results.classifierNames[index]
430            for i,r in enumerate(self.results):
431                r.classes[replace] = results.results[i].classes[index]
432                r.probabilities[replace] = results.results[i].probabilities[index]
433            if len(self.classifiers):
434                for i in range(self.numberOfIterations):
435                    self.classifiers[replace] = results.classifiers[i][index]
436
437#### Experimental procedures
438
439def leaveOneOut(learners, examples, pps=[], indicesrandseed="*", **argkw):
440
441    """leave-one-out evaluation of learners on a data set
442
443    Performs a leave-one-out experiment with the given list of learners
444    and examples. This is equivalent to performing len(examples)-fold
445    cross validation. Function accepts additional keyword arguments for
446    preprocessing, storing classifiers and verbose output.
447
448    """
449
450    (examples, weight) = demangleExamples(examples)
451    return testWithIndices(learners, examples, range(len(examples)), indicesrandseed, pps, **argkw)
452    # return testWithIndices(learners, examples, range(len(examples)), pps=pps, argkw)
453
454# apply(testWithIndices, (learners, (examples, weight), indices, indicesrandseed, pps), argkw)
455
456
457def proportionTest(learners, examples, learnProp, times=10,
458                   strat=Orange.core.MakeRandomIndices.StratifiedIfPossible,
459                   pps=[], callback=None, **argkw):
460    """train-and-test evaluation (train on a subset, test on remaing examples)
461
462    Splits the data with ``learnProp`` of examples in the learning
463    and the rest in the testing set. The test is repeated for a given
464    number of times (default 10). Division is stratified by default. The
465    Function also accepts keyword arguments for randomization and
466    storing classifiers.
467
468    100 repetitions of the so-called 70:30 test in which 70% of examples
469    are used for training and 30% for testing is done by::
470
471        res = orngTest.proportionTest(learners, data, 0.7, 100)
472
473    Note that Python allows naming the arguments; instead of "100" you
474    can use "times=100" to increase the clarity (not so with keyword
475    arguments, such as ``storeClassifiers``, ``randseed`` or ``verbose``
476    that must always be given with a name).
477
478    """
479   
480    # randomGenerator is set either to what users provided or to orange.RandomGenerator(0)
481    # If we left it None or if we set MakeRandomIndices2.randseed, it would give same indices each time it's called
482    randomGenerator = argkw.get("indicesrandseed", 0) or argkw.get("randseed", 0) or argkw.get("randomGenerator", 0)
483    pick = Orange.core.MakeRandomIndices2(stratified = strat, p0 = learnProp, randomGenerator = randomGenerator)
484   
485    examples, weight = demangleExamples(examples)
486    classVar = examples.domain.classVar
487    if classVar.varType == Orange.data.Type.Discrete:
488        values = list(classVar.values)
489        baseValue = classVar.baseValue
490    else:
491        baseValue = values = None
492    testResults = ExperimentResults(times, [l.name for l in learners], values, weight!=0, baseValue)
493
494    for time in range(times):
495        indices = pick(examples)
496        learnset = examples.selectref(indices, 0)
497        testset = examples.selectref(indices, 1)
498        learnAndTestOnTestData(learners, (learnset, weight), (testset, weight), testResults, time, pps, **argkw)
499        if callback: callback()
500    return testResults
501
502def crossValidation(learners, examples, folds=10,
503                    strat=Orange.core.MakeRandomIndices.StratifiedIfPossible,
504                    pps=[], indicesrandseed="*", **argkw):
505    """cross-validation evaluation of learners
506
507    Performs a cross validation with the given number of folds.
508
509    """
510    (examples, weight) = demangleExamples(examples)
511    if indicesrandseed!="*":
512        indices = Orange.core.MakeRandomIndicesCV(examples, folds, randseed=indicesrandseed, stratified = strat)
513    else:
514        randomGenerator = argkw.get("randseed", 0) or argkw.get("randomGenerator", 0)
515        indices = Orange.core.MakeRandomIndicesCV(examples, folds, stratified = strat, randomGenerator = randomGenerator)
516    return testWithIndices(learners, (examples, weight), indices, indicesrandseed, pps, **argkw)
517
518
519def learningCurveN(learners, examples, folds=10,
520                   strat=Orange.core.MakeRandomIndices.StratifiedIfPossible,
521                   proportions=Orange.core.frange(0.1), pps=[], **argkw):
522    """Construct a learning curve for learners.
523
524    A simpler interface for the function :obj:`learningCurve`. Instead
525    of methods for preparing indices, it simply takes the number of folds
526    and a flag telling whether we want a stratified cross-validation or
527    not. This function does not return a single :obj:`ExperimentResults` but
528    a list of them, one for each proportion. ::
529
530        prop = [0.2, 0.4, 0.6, 0.8, 1.0]
531        res = orngTest.learningCurveN(learners, data, folds = 5, proportions = prop)
532        for i, p in enumerate(prop):
533            print "%5.3f:" % p,
534            printResults(res[i])
535
536    This function basically prepares a random generator and example selectors
537    (``cv`` and ``pick``) and calls :obj:`learningCurve`.
538
539    """
540
541    seed = argkw.get("indicesrandseed", -1) or argkw.get("randseed", -1)
542    if seed:
543        randomGenerator = Orange.core.RandomGenerator(seed)
544    else:
545        randomGenerator = argkw.get("randomGenerator", Orange.core.RandomGenerator())
546       
547    if strat:
548        cv=Orange.core.MakeRandomIndicesCV(folds = folds, stratified = strat, randomGenerator = randomGenerator)
549        pick=Orange.core.MakeRandomIndices2(stratified = strat, randomGenerator = randomGenerator)
550    else:
551        cv=Orange.core.RandomIndicesCV(folds = folds, stratified = strat, randomGenerator = randomGenerator)
552        pick=Orange.core.RandomIndices2(stratified = strat, randomGenerator = randomGenerator)
553    return apply(learningCurve, (learners, examples, cv, pick, proportions, pps), argkw)
554
555
556def learningCurve(learners, examples, cv=None, pick=None, proportions=Orange.core.frange(0.1), pps=[], **argkw):
557    """
558    Computes learning curves using a procedure recommended by Salzberg
559    (1997). It first prepares data subsets (folds). For each proportion,
560    it performs the cross-validation, but taking only a proportion of
561    examples for learning.
562
563    Arguments ``cv`` and ``pick`` give the methods for preparing
564    indices for cross-validation and random selection of learning
565    examples. If they are not given, :obj:`orange.MakeRandomIndicesCV` and
566    :obj:`orange.MakeRandomIndices2` are used, both will be stratified and the
567    cross-validation will be 10-fold. Proportions is a list of proportions
568    of learning examples.
569
570    The function can save time by loading experimental existing data for
571    any test that were already conducted and saved. Also, the computed
572    results are stored for later use. You can enable this by adding
573    a keyword argument ``cache=1``. Another keyword deals with progress
574    report. If you add ``verbose=1``, the function will print the proportion
575    and the fold number.
576
577    """
578    verb = argkw.get("verbose", 0)
579    cache = argkw.get("cache", 0)
580    callback = argkw.get("callback", 0)
581
582    for pp in pps:
583        if pp[0]!="L":
584            raise SystemError, "cannot preprocess testing examples"
585
586    if not cv or not pick:   
587        seed = argkw.get("indicesrandseed", -1) or argkw.get("randseed", -1)
588        if seed:
589            randomGenerator = Orange.core.RandomGenerator(seed)
590        else:
591            randomGenerator = argkw.get("randomGenerator", Orange.core.RandomGenerator())
592        if not cv:
593            cv = Orange.core.MakeRandomIndicesCV(folds=10, stratified=Orange.core.MakeRandomIndices.StratifiedIfPossible, randomGenerator = randomGenerator)
594        if not pick:
595            pick = Orange.core.MakeRandomIndices2(stratified=Orange.core.MakeRandomIndices.StratifiedIfPossible, randomGenerator = randomGenerator)
596
597    examples, weight = demangleExamples(examples)
598    folds = cv(examples)
599    ccsum = hex(examples.checksum())[2:]
600    ppsp = encodePP(pps)
601    nLrn = len(learners)
602
603    allResults=[]
604    for p in proportions:
605        printVerbose("Proportion: %5.3f" % p, verb)
606
607        if (cv.randseed<0) or (pick.randseed<0):
608            cache = 0
609        else:
610            fnstr = "{learningCurve}_%s_%s_%s_%s%s-%s" % ("%s", p, cv.randseed, pick.randseed, ppsp, ccsum)
611            if "*" in fnstr:
612                cache = 0
613
614        conv = examples.domain.classVar.varType == Orange.data.Type.Discrete and int or float
615        testResults = ExperimentResults(cv.folds, [l.name for l in learners], examples.domain.classVar.values.native(), weight!=0, examples.domain.classVar.baseValue)
616        testResults.results = [TestedExample(folds[i], conv(examples[i].getclass()), nLrn, examples[i].getweight(weight))
617                               for i in range(len(examples))]
618
619        if cache and testResults.loadFromFiles(learners, fnstr):
620            printVerbose("  loaded from cache", verb)
621        else:
622            for fold in range(cv.folds):
623                printVerbose("  fold %d" % fold, verb)
624               
625                # learning
626                learnset = examples.selectref(folds, fold, negate=1)
627                learnset = learnset.selectref(pick(learnset, p0=p), 0)
628                if not len(learnset):
629                    continue
630               
631                for pp in pps:
632                    learnset = pp[1](learnset)
633
634                classifiers = [None]*nLrn
635                for i in range(nLrn):
636                    if not cache or not testResults.loaded[i]:
637                        classifiers[i] = learners[i](learnset, weight)
638
639                # testing
640                for i in range(len(examples)):
641                    if (folds[i]==fold):
642                        # This is to prevent cheating:
643                        ex = Orange.data.Instance(examples[i])
644                        ex.setclass("?")
645                        for cl in range(nLrn):
646                            if not cache or not testResults.loaded[cl]:
647                                cls, pro = classifiers[cl](ex, Orange.core.GetBoth)
648                                testResults.results[i].setResult(cl, cls, pro)
649                if callback: callback()
650            if cache:
651                testResults.saveToFiles(learners, fnstr)
652
653        allResults.append(testResults)
654       
655    return allResults
656
657
658def learningCurveWithTestData(learners, learnset, testset, times=10,
659                              proportions=Orange.core.frange(0.1),
660                              strat=Orange.core.MakeRandomIndices.StratifiedIfPossible, pps=[], **argkw):
661    """
662    This function is suitable for computing a learning curve on datasets,
663    where learning and testing examples are split in advance. For each
664    proportion of learning examples, it randomly select the requested
665    number of learning examples, builds the models and tests them on the
666    entire testset. The whole test is repeated for the given number of
667    times for each proportion. The result is a list of :obj:`ExperimentResults`,
668    one for each proportion.
669
670    In the following scripts, examples are pre-divided onto training
671    and testing set. Learning curves are computed in which 20, 40, 60,
672    80 and 100 percents of the examples in the former set are used for
673    learning and the latter set is used for testing. Random selection
674    of the given proportion of learning set is repeated for five times.
675
676    .. literalinclude:: code/testing-test.py
677        :start-after: Learning curve with pre-separated data
678        :end-before: # End
679
680
681    """
682    verb = argkw.get("verbose", 0)
683
684    learnset, learnweight = demangleExamples(learnset)
685    testweight = demangleExamples(testset)[1]
686   
687    randomGenerator = argkw.get("indicesrandseed", 0) or argkw.get("randseed", 0) or argkw.get("randomGenerator", 0)
688    pick = Orange.core.MakeRandomIndices2(stratified = strat, randomGenerator = randomGenerator)
689    allResults=[]
690    for p in proportions:
691        printVerbose("Proportion: %5.3f" % p, verb)
692        testResults = ExperimentResults(times, [l.name for l in learners],
693                                        testset.domain.classVar.values.native(),
694                                        testweight!=0, testset.domain.classVar.baseValue)
695        testResults.results = []
696       
697        for t in range(times):
698            printVerbose("  repetition %d" % t, verb)
699            learnAndTestOnTestData(learners, (learnset.selectref(pick(learnset, p), 0), learnweight),
700                                   testset, testResults, t)
701
702        allResults.append(testResults)
703       
704    return allResults
705
706   
707def testWithIndices(learners, examples, indices, indicesrandseed="*", pps=[], callback=None, **argkw):
708    """
709    Performs a cross-validation-like test. The difference is that the
710    caller provides indices (each index gives a fold of an example) which
711    do not necessarily divide the examples into folds of (approximately)
712    same sizes. In fact, the function :obj:`crossValidation` is actually written
713    as a single call to ``testWithIndices``.
714
715    ``testWithIndices`` takes care the ``TestedExamples`` are in the same order
716    as the corresponding examples in the original set. Preprocessing of
717    testing examples is thus not allowed. The computed results can be
718    saved in files or loaded therefrom if you add a keyword argument
719    ``cache=1``. In this case, you also have to specify the random seed
720    which was used to compute the indices (argument ``indicesrandseed``;
721    if you don't there will be no caching.
722
723    """
724
725    verb = argkw.get("verbose", 0)
726    cache = argkw.get("cache", 0)
727    storeclassifiers = argkw.get("storeclassifiers", 0) or argkw.get("storeClassifiers", 0)
728    cache = cache and not storeclassifiers
729
730    examples, weight = demangleExamples(examples)
731    nLrn = len(learners)
732
733    if not examples:
734        raise ValueError("Test data set with no examples")
735    if not examples.domain.classVar:
736        raise ValueError("Test data set without class attribute")
737   
738##    for pp in pps:
739##        if pp[0]!="L":
740##            raise SystemError, "cannot preprocess testing examples"
741
742    nIterations = max(indices)+1
743    if examples.domain.classVar.varType == Orange.data.Type.Discrete:
744        values = list(examples.domain.classVar.values)
745        basevalue = examples.domain.classVar.baseValue
746    else:
747        basevalue = values = None
748
749    conv = examples.domain.classVar.varType == Orange.data.Type.Discrete and int or float       
750    testResults = ExperimentResults(nIterations, [getobjectname(l) for l in learners], values, weight!=0, basevalue)
751    testResults.results = [TestedExample(indices[i], conv(examples[i].getclass()), nLrn, examples[i].getweight(weight))
752                           for i in range(len(examples))]
753
754    if argkw.get("storeExamples", 0):
755        testResults.examples = examples
756       
757    ccsum = hex(examples.checksum())[2:]
758    ppsp = encodePP(pps)
759    fnstr = "{TestWithIndices}_%s_%s%s-%s" % ("%s", indicesrandseed, ppsp, ccsum)
760    if "*" in fnstr:
761        cache = 0
762
763    if cache and testResults.loadFromFiles(learners, fnstr):
764        printVerbose("  loaded from cache", verb)
765    else:
766        for fold in range(nIterations):
767            # learning
768            learnset = examples.selectref(indices, fold, negate=1)
769            if not len(learnset):
770                continue
771            testset = examples.selectref(indices, fold, negate=0)
772            if not len(testset):
773                continue
774           
775            for pp in pps:
776                if pp[0]=="B":
777                    learnset = pp[1](learnset)
778                    testset = pp[1](testset)
779
780            for pp in pps:
781                if pp[0]=="L":
782                    learnset = pp[1](learnset)
783                elif pp[0]=="T":
784                    testset = pp[1](testset)
785                elif pp[0]=="LT":
786                    (learnset, testset) = pp[1](learnset, testset)
787
788            if not learnset:
789                raise SystemError, "no training examples after preprocessing"
790
791            if not testset:
792                raise SystemError, "no test examples after preprocessing"
793
794            classifiers = [None]*nLrn
795            for i in range(nLrn):
796                if not cache or not testResults.loaded[i]:
797                    classifiers[i] = learners[i](learnset, weight)
798            if storeclassifiers:   
799                testResults.classifiers.append(classifiers)
800
801            # testing
802            tcn = 0
803            for i in range(len(examples)):
804                if (indices[i]==fold):
805                    # This is to prevent cheating:
806                    ex = Orange.data.Instance(testset[tcn])
807                    ex.setclass("?")
808                    tcn += 1
809                    for cl in range(nLrn):
810                        if not cache or not testResults.loaded[cl]:
811                            cr = classifiers[cl](ex, Orange.core.GetBoth)                                     
812                            if cr[0].isSpecial():
813                                raise "Classifier %s returned unknown value" % (classifiers[cl].name or ("#%i" % cl))
814                            testResults.results[i].setResult(cl, cr[0], cr[1])
815            if callback:
816                callback()
817        if cache:
818            testResults.saveToFiles(learners, fnstr)
819       
820    return testResults
821
822
823def learnAndTestOnTestData(learners, learnset, testset, testResults=None, iterationNumber=0, pps=[], callback=None, **argkw):
824    """
825    This function performs no sampling on its own: two separate datasets
826    need to be passed, one for training and the other for testing. The
827    function preprocesses the data, induces the model and tests it. The
828    order of filters is peculiar, but it makes sense when compared to
829    other methods that support preprocessing of testing examples. The
830    function first applies preprocessors marked ``"B"`` (both sets), and only
831    then the preprocessors that need to processor only one of the sets.
832
833    You can pass an already initialized :obj:`ExperimentResults` (argument
834    ``results``) and an iteration number (``iterationNumber``). Results
835    of the test will be appended with the given iteration
836    number. This is because :obj:`learnAndTestWithTestData`
837    gets called by other functions, like :obj:`proportionTest` and
838    :obj:`learningCurveWithTestData`. If you omit the parameters, a new
839    :obj:`ExperimentResults` will be created.
840
841    """
842    storeclassifiers = argkw.get("storeclassifiers", 0) or argkw.get("storeClassifiers", 0)
843    storeExamples = argkw.get("storeExamples", 0)
844
845    learnset, learnweight = demangleExamples(learnset)
846    testset, testweight = demangleExamples(testset)
847    storeclassifiers = argkw.get("storeclassifiers", 0) or argkw.get("storeClassifiers", 0)
848   
849    for pp in pps:
850        if pp[0]=="B":
851            learnset = pp[1](learnset)
852            testset = pp[1](testset)
853
854    for pp in pps:
855        if pp[0]=="L":
856            learnset = pp[1](learnset)
857        elif pp[0]=="T":
858            testset = pp[1](testset)
859        elif pp[0]=="LT":
860            learnset, testset = pp[1](learnset, testset)
861           
862    classifiers = []
863    for learner in learners:
864        classifiers.append(learner(learnset, learnweight))
865        if callback:
866            callback()
867    classifiers = [learner(learnset, learnweight) for learner in learners]
868    for i in range(len(learners)): classifiers[i].name = getattr(learners[i], 'name', 'noname')
869    testResults = testOnData(classifiers, (testset, testweight), testResults, iterationNumber, storeExamples)
870    if storeclassifiers:
871        testResults.classifiers.append(classifiers)
872    return testResults
873
874
875def learnAndTestOnLearnData(learners, learnset, testResults=None, iterationNumber=0, pps=[], callback=None, **argkw):
876    """
877    This function is similar to the above, except that it learns and
878    tests on the same data. If first preprocesses the data with ``"B"``
879    preprocessors on the whole data, and afterwards any ``"L"`` or ``"T"``
880    preprocessors on separate datasets. Then it induces the model from
881    the learning set and tests it on the testing set.
882
883    As with :obj:`learnAndTestOnTestData`, you can pass an already initialized
884    :obj:`ExperimentResults` (argument ``results``) and an iteration number to the
885    function. In this case, results of the test will be appended with
886    the given iteration number.
887
888    """
889
890    storeclassifiers = argkw.get("storeclassifiers", 0) or argkw.get("storeClassifiers", 0)
891    storeExamples = argkw.get("storeExamples", 0)
892
893    learnset, learnweight = demangleExamples(learnset)
894
895    hasLorT = 0   
896    for pp in pps:
897        if pp[0]=="B":
898            learnset = pp[1](learnset)
899        else:
900            hasLorT = 1
901
902    if hasLorT:
903        testset = Orange.data.Table(learnset)
904        for pp in pps:
905            if pp[0]=="L":
906                learnset = pp[1](learnset)
907            elif pp[0]=="T":
908                testset = pp[1](testset)
909            elif pp[0]=="LT":
910                learnset, testset = pp[1](learnset, testset)
911    else:
912        testset = learnset   
913
914    classifiers = []
915    for learner in learners:
916        classifiers.append(learner(learnset, learnweight))
917        if callback:
918            callback()
919    for i in range(len(learners)): classifiers[i].name = getattr(learners[i], "name", "noname")
920    testResults = testOnData(classifiers, (testset, learnweight), testResults, iterationNumber, storeExamples)
921    if storeclassifiers:
922        testResults.classifiers.append(classifiers)
923    return testResults
924
925
926def testOnData(classifiers, testset, testResults=None, iterationNumber=0, storeExamples = False, **argkw):
927    """
928    This function gets a list of classifiers, not learners like the other
929    functions in this module. It classifies each testing example with
930    each classifier. You can pass an existing :obj:`ExperimentResults`
931    and iteration number, like in :obj:`learnAndTestWithTestData`
932    (which actually calls :obj:`testWithTestData`). If you don't, a new
933    :obj:`ExperimentResults` will be created.
934
935    """
936
937    testset, testweight = demangleExamples(testset)
938
939    if not testResults:
940        classVar = testset.domain.classVar
941        if testset.domain.classVar.varType == Orange.data.Type.Discrete:
942            values = classVar.values.native()
943            baseValue = classVar.baseValue
944        else:
945            values = None
946            baseValue = -1
947        testResults=ExperimentResults(1, [l.name for l in classifiers], values, testweight!=0, baseValue)
948
949    examples = getattr(testResults, "examples", False)
950    if examples and len(examples):
951        # We must not modify an example table we do not own, so we clone it the
952        # first time we have to add to it
953        if not getattr(testResults, "examplesCloned", False):
954            testResults.examples = Orange.data.Table(testResults.examples)
955            testResults.examplesCloned = True
956        testResults.examples.extend(testset)
957    else:
958        # We do not clone at the first iteration - cloning might never be needed at all...
959        testResults.examples = testset
960   
961    conv = testset.domain.classVar.varType == Orange.data.Type.Discrete and int or float
962    for ex in testset:
963        te = TestedExample(iterationNumber, conv(ex.getclass()), 0, ex.getweight(testweight))
964
965        for classifier in classifiers:
966            # This is to prevent cheating:
967            ex2 = Orange.data.Instance(ex)
968            ex2.setclass("?")
969            cr = classifier(ex2, Orange.core.GetBoth)
970            te.addResult(cr[0], cr[1])
971        testResults.results.append(te)
972       
973    return testResults
Note: See TracBrowser for help on using the repository browser.