Changeset 7514:0c03955ef45b in orange


Ignore:
Timestamp:
02/04/11 19:45:14 (3 years ago)
Author:
jzbontar <jure.zbontar@…>
Branch:
default
Convert:
c3e2015cf86d36e2fa6f5c782cd3f64e3324caa4
Message:

documenting module testing

Location:
orange
Files:
2 edited

Legend:

Unmodified
Added
Removed
  • orange/Orange/evaluation/testing.py

    r7421 r7514  
    4848Common Arguments 
    4949================ 
     50 
     51Many function in this module use a set of common arguments, which we define here. 
     52 
     53*learners* 
     54    A list of learning algorithms. These can be either pure Orange objects 
     55    (such as :obj:`Orange.classification.bayes.NaiveLearner`) or Python 
     56    classes or functions written in pure Python (anything that can be 
     57    called with the same arguments and results as Orange's classifiers 
     58    and performs similar function). 
     59 
     60*examples, learnset, testset* 
     61    Examples, given as an :obj:`Orange.data.Table` (some functions need an undivided 
     62    set of examples while others need examples that are already split 
     63    into two sets). If examples are weighted, pass them as a tuple 
     64    (examples, weightID). Weights are respected by learning and testing, 
     65    but not by sampling. When selecting 10% of examples, this means 10% 
     66    by number, not by weights. There is also no guarantee that sums 
     67    of example weights will be (at least roughly) equal for folds in 
     68    cross validation. 
     69 
     70*strat* 
     71    Tells whether to stratify the random selections. Its default value is 
     72    :obj:`Orange.core.StratifiedIfPossible` which stratifies selections 
     73    if the class variable is discrete and has no unknown values. 
     74 
     75*randseed (obsolete: indicesrandseed), randomGenerator* 
     76    Random seed (``randseed``) or random generator (``randomGenerator``) for 
     77    random selection of examples. If omitted, random seed of 0 is used and 
     78    the same test will always select the same examples from the example 
     79    set. There are various slightly different ways to randomize it. 
     80 
     81    *  
     82      Set ``randomGenerator`` to :obj:`Orange.core.globalRandom`. The function's 
     83      selection will depend upon Orange's global random generator that 
     84      is reset (with random seed 0) when Orange is imported. The Script's 
     85      output will therefore depend upon what you did after Orange was 
     86      first imported in the current Python session. :: 
     87 
     88          res = orngTest.proportionTest(learners, data, 0.7, 
     89              randomGenerator=orange.globalRandom)  
     90 
     91    *  
     92      Construct a new :obj:`Orange.core.RandomGenerator`. The code below, 
     93      for instance, will produce different results in each iteration, 
     94      but overall the same results each time it's run. 
     95 
     96      .. literalinclude:: code/testing-test.py 
     97        :start-after: but the same each time the script is run 
     98        :end-before: # End 
     99 
     100    * 
     101      Set the random seed (argument ``randseed``) to a random 
     102      number. Python has a global random generator that is reset when 
     103      Python is loaded, using the current system time for a seed. With this, 
     104      results will be (in general) different each time the script is run. 
     105 
     106 
     107      .. literalinclude:: code/testing-test.py 
     108        :start-after: proportionsTest that will give different results each time it is run 
     109        :end-before: # End 
     110 
     111 
     112      The same module also provides random generators as object, so 
     113      that you can have independent local random generators in case you 
     114      need them. 
     115 
     116*pps* 
     117    A list of preprocessors. It consists of tuples ``(c, preprocessor)``, 
     118    where ``c`` determines whether the preprocessor will be applied 
     119    to the learning set (``"L"``), test set (``"T"``) or to both 
     120    (``"B"``). The latter is applied first, when the example set is still 
     121    undivided. The ``"L"`` and ``"T"`` preprocessors are applied on the 
     122    separated subsets. Preprocessing testing examples is allowed only 
     123    on experimental procedures that do not report the TestedExample's 
     124    in the same order as examples in the original set. The second item 
     125    in the tuple, preprocessor can be either a pure Orange or a pure 
     126    Python preprocessor, that is, any function or callable class that 
     127    accepts a table of examples and weight, and returns a preprocessed 
     128    table and weight. 
     129 
     130    This example will demonstrate the devastating effect of 100% class 
     131    noise on learning. :: 
     132 
     133        classnoise = orange.Preprocessor_addClassNoise(proportion=1.0)  
     134        res = orngTest.proportionTest(learners, data, 0.7, 100, pps = [("L", classnoise)])  
     135 
     136*proportions* 
     137    Gives the proportions of learning examples at which the tests are 
     138    to be made, where applicable. The default is ``[0.1, 0.2, ..., 1.0]``. 
     139 
     140*storeClassifiers (keyword argument)* 
     141    If this flag is set, the testing procedure will store the constructed 
     142    classifiers. For each iteration of the test (eg for each fold in 
     143    cross validation, for each left out example in leave-one-out...), 
     144    the list of classifiers is appended to the ExperimentResults' 
     145    field classifiers. 
     146 
     147    The script below makes 100 repetitions of 70:30 test and store the 
     148    classifiers it induces. :: 
     149 
     150        res = orngTest.proportionTest(learners, data, 0.7, 100, storeClassifier=1) 
     151 
     152*verbose (keyword argument)* 
     153    Several functions can report their progress if you add a keyword 
     154    argument ``verbose=1``. 
     155 
     156Sampling and Testing Functions 
     157============================== 
     158 
     159.. autofunction:: proportionTest 
     160.. autofunction:: leaveOneOut 
     161.. autofunction:: crossValidation 
     162.. autofunction:: testWithIndices 
     163.. autofunction:: learningCurve 
     164.. autofunction:: learningCurveN 
     165.. autofunction:: learningCurveWithTestData 
     166.. autofunction:: learnAndTestOnTestData 
     167.. autofunction:: learnAndTestOnLearnData 
     168.. autofunction:: testOnData 
     169 
     170Classes 
     171======= 
     172 
     173Knowing classes :obj:`TestedExample` that stores results of testing 
     174for a single test example and ExperimentResults that stores a list of 
     175TestedExamples along with some other data on experimental procedures 
     176and classifiers used, is important if you would like to write your own 
     177measures of quality of models, compatible the sampling infrastructure 
     178provided by Orange. If not, you can skip the remainder of this page. 
     179 
     180.. autoclass:: TestedExample 
     181    :members: 
     182 
     183.. autoclass:: ExperimentResults 
     184    :members: 
    50185 
    51186""" 
     
    70205 
    71206class TestedExample: 
     207    """ 
     208    TestedExample stores predictions of different classifiers for a single testing example. 
     209 
     210    .. attribute:: classes 
     211 
     212        A list of predictions of type Value, one for each classifier. 
     213 
     214    .. attribute:: probabilities 
     215         
     216        A list of probabilities of classes, one for each classifier. 
     217 
     218    .. attribute:: iterationNumber 
     219 
     220        Iteration number (e.g. fold) in which the TestedExample was created/tested. 
     221 
     222    .. attribute:: actualClass 
     223 
     224        The correct class of the example 
     225 
     226    .. attribute:: weight 
     227 
     228        Example's weight. Even if the example set was not weighted, 
     229        this attribute is present and equals 1.0. 
     230 
     231    :param iterationNumber: 
     232    :paramtype iterationNumber: type??? 
     233    :param actualClass: 
     234    :paramtype actualClass: type??? 
     235    :param n: 
     236    :paramtype n: int 
     237    :param weight: 
     238    :paramtype weight: float 
     239 
     240    """ 
     241 
    72242    def __init__(self, iterationNumber=None, actualClass=None, n=0, weight=1.0): 
    73243        self.classes = [None]*n 
     
    78248     
    79249    def addResult(self, aclass, aprob): 
     250        """Appends a new result (class and probability prediction by a single classifier) to the classes and probabilities field.""" 
     251     
    80252        if type(aclass.value)==float: 
    81253            self.classes.append(float(aclass)) 
     
    86258 
    87259    def setResult(self, i, aclass, aprob): 
     260        """Sets the result of the i-th classifier to the given values.""" 
    88261        if type(aclass.value)==float: 
    89262            self.classes[i] = float(aclass) 
     
    94267 
    95268class ExperimentResults(object): 
     269    """ ExperimentResults stores results of one or more repetitions of 
     270    some test (cross validation, repeated sampling...) under the same 
     271    circumstances. 
     272 
     273    """ 
    96274    def __init__(self, iterations, classifierNames, classValues, weights, baseClass=-1, **argkw): 
    97275        self.classValues = classValues 
     
    180358 
    181359def leaveOneOut(learners, examples, pps=[], indicesrandseed="*", **argkw): 
    182     """leave-one-out evaluation of learners on a data set""" 
     360 
     361    """leave-one-out evaluation of learners on a data set 
     362 
     363    Performs a leave-one-out experiment with the given list of learners 
     364    and examples. This is equivalent to performing len(examples)-fold 
     365    cross validation. Function accepts additional keyword arguments for 
     366    preprocessing, storing classifiers and verbose output. 
     367 
     368    """ 
     369 
    183370    (examples, weight) = demangleExamples(examples) 
    184371    return testWithIndices(learners, examples, range(len(examples)), indicesrandseed, pps, **argkw) 
     
    191378                   strat=Orange.core.MakeRandomIndices.StratifiedIfPossible, 
    192379                   pps=[], callback=None, **argkw): 
    193     """train-and-test evaluation (train on a subset, test on remaing examples)""" 
     380    """train-and-test evaluation (train on a subset, test on remaing examples) 
     381 
     382    Splits the data with ``learnProp`` of examples in the learning 
     383    and the rest in the testing set. The test is repeated for a given 
     384    number of times (default 10). Division is stratified by default. The 
     385    Function also accepts keyword arguments for randomization and 
     386    storing classifiers. 
     387 
     388    100 repetitions of the so-called 70:30 test in which 70% of examples 
     389    are used for training and 30% for testing is done by:: 
     390 
     391        res = orngTest.proportionTest(learners, data, 0.7, 100)  
     392 
     393    Note that Python allows naming the arguments; instead of "100" you 
     394    can use "times=100" to increase the clarity (not so with keyword 
     395    arguments, such as ``storeClassifiers``, ``randseed`` or ``verbose`` 
     396    that must always be given with a name). 
     397 
     398    """ 
     399     
    194400    # randomGenerator is set either to what users provided or to Orange.core.RandomGenerator(0) 
    195401    # If we left it None or if we set MakeRandomIndices2.randseed, it would give same indices each time it's called 
     
    217423                    strat=Orange.core.MakeRandomIndices.StratifiedIfPossible, 
    218424                    pps=[], indicesrandseed="*", **argkw): 
    219     """cross-validation evaluation of learners""" 
     425    """cross-validation evaluation of learners 
     426 
     427    Performs a cross validation with the given number of folds. 
     428 
     429    """ 
    220430    (examples, weight) = demangleExamples(examples) 
    221431    if indicesrandseed!="*": 
     
    230440                   strat=Orange.core.MakeRandomIndices.StratifiedIfPossible, 
    231441                   proportions=Orange.core.frange(0.1), pps=[], **argkw): 
    232     """construct a learning curve for learners""" 
     442    """Construct a learning curve for learners. 
     443 
     444    A simpler interface for the function :obj:`learningCurve`. Instead 
     445    of methods for preparing indices, it simply takes the number of folds 
     446    and a flag telling whether we want a stratified cross-validation or 
     447    not. This function does not return a single ``ExperimentResults`` but 
     448    a list of them, one for each proportion. :: 
     449 
     450        prop = [0.2, 0.4, 0.6, 0.8, 1.0] 
     451        res = orngTest.learningCurveN(learners, data, folds = 5, proportions = prop) 
     452        for i, p in enumerate(prop): 
     453            print "%5.3f:" % p, 
     454            printResults(res[i]) 
     455 
     456    This function basically prepares a random generator and example selectors 
     457    (``cv`` and ``pick``) and calls :obj:`learningCurve`. 
     458 
     459    """ 
     460 
    233461    seed = argkw.get("indicesrandseed", -1) or argkw.get("randseed", -1) 
    234462    if seed: 
     
    247475 
    248476def learningCurve(learners, examples, cv=None, pick=None, proportions=Orange.core.frange(0.1), pps=[], **argkw): 
     477    """ 
     478    Computes learning curves using a procedure recommended by Salzberg 
     479    (1997). It first prepares data subsets (folds). For each proportion, 
     480    it performs the cross-validation, but taking only a proportion of 
     481    examples for learning. 
     482 
     483    Arguments ``cv`` and ``pick`` give the methods for preparing 
     484    indices for cross-validation and random selection of learning 
     485    examples. If they are not given, :obj:`Orange.core.MakeRandomIndicesCV` and 
     486    :obj:`Orange.core.MakeRandomIndices2` are used, both will be stratified and the 
     487    cross-validation will be 10-fold. Proportions is a list of proportions 
     488    of learning examples. 
     489 
     490    The function can save time by loading experimental existing data for 
     491    any test that were already conducted and saved. Also, the computed 
     492    results are stored for later use. You can enable this by adding 
     493    a keyword argument ``cache=1``. Another keyword deals with progress 
     494    report. If you add ``verbose=1``, the function will print the proportion 
     495    and the fold number. 
     496 
     497    """ 
    249498    verb = argkw.get("verbose", 0) 
    250499    cache = argkw.get("cache", 0) 
     
    330579                              proportions=Orange.core.frange(0.1), 
    331580                              strat=Orange.core.MakeRandomIndices.StratifiedIfPossible, pps=[], **argkw): 
     581    """ 
     582    This function is suitable for computing a learning curve on datasets, 
     583    where learning and testing examples are split in advance. For each 
     584    proportion of learning examples, it randomly select the requested 
     585    number of learning examples, builds the models and tests them on the 
     586    entire testset. The whole test is repeated for the given number of 
     587    times for each proportion. The result is a list of ExperimentResults, 
     588    one for each proportion. 
     589 
     590    In the following scripts, examples are pre-divided onto training 
     591    and testing set. Learning curves are computed in which 20, 40, 60, 
     592    80 and 100 percents of the examples in the former set are used for 
     593    learning and the latter set is used for testing. Random selection 
     594    of the given proportion of learning set is repeated for five times. 
     595 
     596    .. literalinclude:: code/testing-test.py 
     597        :start-after: Learning curve with pre-separated data 
     598        :end-before: # End 
     599 
     600 
     601    """ 
    332602    verb = argkw.get("verbose", 0) 
    333603 
     
    356626    
    357627def testWithIndices(learners, examples, indices, indicesrandseed="*", pps=[], callback=None, **argkw): 
     628    """ 
     629    Performs a cross-validation-like test. The difference is that the 
     630    caller provides indices (each index gives a fold of an example) which 
     631    do not necessarily divide the examples into folds of (approximately) 
     632    same sizes. In fact, the function :obj:`crossValidation` is actually written 
     633    as a single call to ``testWithIndices``. 
     634 
     635    ``testWithIndices`` takes care the ``TestedExamples`` are in the same order 
     636    as the corresponding examples in the original set. Preprocessing of 
     637    testing examples is thus not allowed. The computed results can be 
     638    saved in files or loaded therefrom if you add a keyword argument 
     639    ``cache=1``. In this case, you also have to specify the random seed 
     640    which was used to compute the indices (argument ``indicesrandseed``; 
     641    if you don't there will be no caching. 
     642 
     643    """ 
     644 
    358645    verb = argkw.get("verbose", 0) 
    359646    cache = argkw.get("cache", 0) 
     
    455742 
    456743def learnAndTestOnTestData(learners, learnset, testset, testResults=None, iterationNumber=0, pps=[], callback=None, **argkw): 
     744    """ 
     745    This function performs no sampling on its own: two separate datasets 
     746    need to be passed, one for training and the other for testing. The 
     747    function preprocesses the data, induces the model and tests it. The 
     748    order of filters is peculiar, but it makes sense when compared to 
     749    other methods that support preprocessing of testing examples. The 
     750    function first applies preprocessors marked ``"B"`` (both sets), and only 
     751    then the preprocessors that need to processor only one of the sets. 
     752 
     753    You can pass an already initialized :obj:`ExperimentResults` (argument 
     754    ``results``) and an iteration number (``iterationNumber``). Results 
     755    of the test will be appended with the given iteration 
     756    number. This is because :obj:`learnAndTestWithTestData` 
     757    gets called by other functions, like :obj:`proportionTest` and 
     758    :obj:`learningCurveWithTestData`. If you omit the parameters, a new 
     759    :obj:`ExperimentResults` will be created. 
     760 
     761    """ 
    457762    storeclassifiers = argkw.get("storeclassifiers", 0) or argkw.get("storeClassifiers", 0) 
    458763    storeExamples = argkw.get("storeExamples", 0) 
     
    489794 
    490795def learnAndTestOnLearnData(learners, learnset, testResults=None, iterationNumber=0, pps=[], callback=None, **argkw): 
     796    """ 
     797    This function is similar to the above, except that it learns and 
     798    tests on the same data. If first preprocesses the data with ``"B"`` 
     799    preprocessors on the whole data, and afterwards any ``"L"`` or ``"T"`` 
     800    preprocessors on separate datasets. Then it induces the model from 
     801    the learning set and tests it on the testing set. 
     802 
     803    As with :obj:`learnAndTestOnTestData`, you can pass an already initialized 
     804    :obj:`ExperimentResults` (argument ``results``) and an iteration number to the 
     805    function. In this case, results of the test will be appended with 
     806    the given iteration number. 
     807 
     808    """ 
     809 
    491810    storeclassifiers = argkw.get("storeclassifiers", 0) or argkw.get("storeClassifiers", 0) 
    492811    storeExamples = argkw.get("storeExamples", 0) 
     
    526845 
    527846def testOnData(classifiers, testset, testResults=None, iterationNumber=0, storeExamples = False, **argkw): 
     847    """ 
     848    This function gets a list of classifiers, not learners like the other 
     849    functions in this module. It classifies each testing example with 
     850    each classifier. You can pass an existing :obj:`ExperimentResults` 
     851    and iteration number, like in :obj:`learnAndTestWithTestData` 
     852    (which actually calls :obj:`testWithTestData`). If you don't, a new 
     853    :obj:`ExperimentResults` will be created. 
     854 
     855    """ 
     856 
    528857    testset, testweight = demangleExamples(testset) 
    529858 
  • orange/doc/Orange/rst/code/testing-test.py

    r7421 r7514  
    2121    printResults(res) 
    2222 
    23 print "\nproportionsTest that will give different results, but the same each \ 
    24 time the script is run" 
     23print "\nproportionsTest that will give different results, \ 
     24but the same each time the script is run" 
    2525myRandom = Orange.core.RandomGenerator() 
    2626for i in range(3): 
     
    2828        randomGenerator=myRandom) 
    2929    printResults(res) 
     30# End 
    3031 
    3132print "\nproportionsTest that will give different results each time it is run" 
     
    3435        randseed=random.randint(0, 100)) 
    3536    printResults(res) 
     37# End 
    3638 
    3739print "\nproportionsTest + storing classifiers" 
     
    5254    print "%5.3f:" % prop[i], 
    5355    printResults(res[i]) 
     56# End 
    5457 
    5558print "\nLearning curve with pre-separated data" 
     
    6265    print "%5.3f:" % prop[i], 
    6366    printResults(res[i]) 
     67# End 
    6468 
    6569print "\nLearning and testing on pre-separated data" 
Note: See TracChangeset for help on using the changeset viewer.