Changeset 7514:0c03955ef45b in orange

02/04/11 19:45:14 (3 years ago)
jzbontar <jure.zbontar@…>

documenting module testing

2 edited


  • orange/Orange/evaluation/

    r7421 r7514  
    4848Common Arguments 
     51Many function in this module use a set of common arguments, which we define here. 
     54    A list of learning algorithms. These can be either pure Orange objects 
     55    (such as :obj:`Orange.classification.bayes.NaiveLearner`) or Python 
     56    classes or functions written in pure Python (anything that can be 
     57    called with the same arguments and results as Orange's classifiers 
     58    and performs similar function). 
     60*examples, learnset, testset* 
     61    Examples, given as an :obj:`` (some functions need an undivided 
     62    set of examples while others need examples that are already split 
     63    into two sets). If examples are weighted, pass them as a tuple 
     64    (examples, weightID). Weights are respected by learning and testing, 
     65    but not by sampling. When selecting 10% of examples, this means 10% 
     66    by number, not by weights. There is also no guarantee that sums 
     67    of example weights will be (at least roughly) equal for folds in 
     68    cross validation. 
     71    Tells whether to stratify the random selections. Its default value is 
     72    :obj:`Orange.core.StratifiedIfPossible` which stratifies selections 
     73    if the class variable is discrete and has no unknown values. 
     75*randseed (obsolete: indicesrandseed), randomGenerator* 
     76    Random seed (``randseed``) or random generator (``randomGenerator``) for 
     77    random selection of examples. If omitted, random seed of 0 is used and 
     78    the same test will always select the same examples from the example 
     79    set. There are various slightly different ways to randomize it. 
     81    *  
     82      Set ``randomGenerator`` to :obj:`Orange.core.globalRandom`. The function's 
     83      selection will depend upon Orange's global random generator that 
     84      is reset (with random seed 0) when Orange is imported. The Script's 
     85      output will therefore depend upon what you did after Orange was 
     86      first imported in the current Python session. :: 
     88          res = orngTest.proportionTest(learners, data, 0.7, 
     89              randomGenerator=orange.globalRandom)  
     91    *  
     92      Construct a new :obj:`Orange.core.RandomGenerator`. The code below, 
     93      for instance, will produce different results in each iteration, 
     94      but overall the same results each time it's run. 
     96      .. literalinclude:: code/ 
     97        :start-after: but the same each time the script is run 
     98        :end-before: # End 
     100    * 
     101      Set the random seed (argument ``randseed``) to a random 
     102      number. Python has a global random generator that is reset when 
     103      Python is loaded, using the current system time for a seed. With this, 
     104      results will be (in general) different each time the script is run. 
     107      .. literalinclude:: code/ 
     108        :start-after: proportionsTest that will give different results each time it is run 
     109        :end-before: # End 
     112      The same module also provides random generators as object, so 
     113      that you can have independent local random generators in case you 
     114      need them. 
     117    A list of preprocessors. It consists of tuples ``(c, preprocessor)``, 
     118    where ``c`` determines whether the preprocessor will be applied 
     119    to the learning set (``"L"``), test set (``"T"``) or to both 
     120    (``"B"``). The latter is applied first, when the example set is still 
     121    undivided. The ``"L"`` and ``"T"`` preprocessors are applied on the 
     122    separated subsets. Preprocessing testing examples is allowed only 
     123    on experimental procedures that do not report the TestedExample's 
     124    in the same order as examples in the original set. The second item 
     125    in the tuple, preprocessor can be either a pure Orange or a pure 
     126    Python preprocessor, that is, any function or callable class that 
     127    accepts a table of examples and weight, and returns a preprocessed 
     128    table and weight. 
     130    This example will demonstrate the devastating effect of 100% class 
     131    noise on learning. :: 
     133        classnoise = orange.Preprocessor_addClassNoise(proportion=1.0)  
     134        res = orngTest.proportionTest(learners, data, 0.7, 100, pps = [("L", classnoise)])  
     137    Gives the proportions of learning examples at which the tests are 
     138    to be made, where applicable. The default is ``[0.1, 0.2, ..., 1.0]``. 
     140*storeClassifiers (keyword argument)* 
     141    If this flag is set, the testing procedure will store the constructed 
     142    classifiers. For each iteration of the test (eg for each fold in 
     143    cross validation, for each left out example in leave-one-out...), 
     144    the list of classifiers is appended to the ExperimentResults' 
     145    field classifiers. 
     147    The script below makes 100 repetitions of 70:30 test and store the 
     148    classifiers it induces. :: 
     150        res = orngTest.proportionTest(learners, data, 0.7, 100, storeClassifier=1) 
     152*verbose (keyword argument)* 
     153    Several functions can report their progress if you add a keyword 
     154    argument ``verbose=1``. 
     156Sampling and Testing Functions 
     159.. autofunction:: proportionTest 
     160.. autofunction:: leaveOneOut 
     161.. autofunction:: crossValidation 
     162.. autofunction:: testWithIndices 
     163.. autofunction:: learningCurve 
     164.. autofunction:: learningCurveN 
     165.. autofunction:: learningCurveWithTestData 
     166.. autofunction:: learnAndTestOnTestData 
     167.. autofunction:: learnAndTestOnLearnData 
     168.. autofunction:: testOnData 
     173Knowing classes :obj:`TestedExample` that stores results of testing 
     174for a single test example and ExperimentResults that stores a list of 
     175TestedExamples along with some other data on experimental procedures 
     176and classifiers used, is important if you would like to write your own 
     177measures of quality of models, compatible the sampling infrastructure 
     178provided by Orange. If not, you can skip the remainder of this page. 
     180.. autoclass:: TestedExample 
     181    :members: 
     183.. autoclass:: ExperimentResults 
     184    :members: 
    71206class TestedExample: 
     207    """ 
     208    TestedExample stores predictions of different classifiers for a single testing example. 
     210    .. attribute:: classes 
     212        A list of predictions of type Value, one for each classifier. 
     214    .. attribute:: probabilities 
     216        A list of probabilities of classes, one for each classifier. 
     218    .. attribute:: iterationNumber 
     220        Iteration number (e.g. fold) in which the TestedExample was created/tested. 
     222    .. attribute:: actualClass 
     224        The correct class of the example 
     226    .. attribute:: weight 
     228        Example's weight. Even if the example set was not weighted, 
     229        this attribute is present and equals 1.0. 
     231    :param iterationNumber: 
     232    :paramtype iterationNumber: type??? 
     233    :param actualClass: 
     234    :paramtype actualClass: type??? 
     235    :param n: 
     236    :paramtype n: int 
     237    :param weight: 
     238    :paramtype weight: float 
     240    """ 
    72242    def __init__(self, iterationNumber=None, actualClass=None, n=0, weight=1.0): 
    73243        self.classes = [None]*n 
    79249    def addResult(self, aclass, aprob): 
     250        """Appends a new result (class and probability prediction by a single classifier) to the classes and probabilities field.""" 
    80252        if type(aclass.value)==float: 
    81253            self.classes.append(float(aclass)) 
    87259    def setResult(self, i, aclass, aprob): 
     260        """Sets the result of the i-th classifier to the given values.""" 
    88261        if type(aclass.value)==float: 
    89262            self.classes[i] = float(aclass) 
    95268class ExperimentResults(object): 
     269    """ ExperimentResults stores results of one or more repetitions of 
     270    some test (cross validation, repeated sampling...) under the same 
     271    circumstances. 
     273    """ 
    96274    def __init__(self, iterations, classifierNames, classValues, weights, baseClass=-1, **argkw): 
    97275        self.classValues = classValues 
    181359def leaveOneOut(learners, examples, pps=[], indicesrandseed="*", **argkw): 
    182     """leave-one-out evaluation of learners on a data set""" 
     361    """leave-one-out evaluation of learners on a data set 
     363    Performs a leave-one-out experiment with the given list of learners 
     364    and examples. This is equivalent to performing len(examples)-fold 
     365    cross validation. Function accepts additional keyword arguments for 
     366    preprocessing, storing classifiers and verbose output. 
     368    """ 
    183370    (examples, weight) = demangleExamples(examples) 
    184371    return testWithIndices(learners, examples, range(len(examples)), indicesrandseed, pps, **argkw) 
    191378                   strat=Orange.core.MakeRandomIndices.StratifiedIfPossible, 
    192379                   pps=[], callback=None, **argkw): 
    193     """train-and-test evaluation (train on a subset, test on remaing examples)""" 
     380    """train-and-test evaluation (train on a subset, test on remaing examples) 
     382    Splits the data with ``learnProp`` of examples in the learning 
     383    and the rest in the testing set. The test is repeated for a given 
     384    number of times (default 10). Division is stratified by default. The 
     385    Function also accepts keyword arguments for randomization and 
     386    storing classifiers. 
     388    100 repetitions of the so-called 70:30 test in which 70% of examples 
     389    are used for training and 30% for testing is done by:: 
     391        res = orngTest.proportionTest(learners, data, 0.7, 100)  
     393    Note that Python allows naming the arguments; instead of "100" you 
     394    can use "times=100" to increase the clarity (not so with keyword 
     395    arguments, such as ``storeClassifiers``, ``randseed`` or ``verbose`` 
     396    that must always be given with a name). 
     398    """ 
    194400    # randomGenerator is set either to what users provided or to Orange.core.RandomGenerator(0) 
    195401    # If we left it None or if we set MakeRandomIndices2.randseed, it would give same indices each time it's called 
    217423                    strat=Orange.core.MakeRandomIndices.StratifiedIfPossible, 
    218424                    pps=[], indicesrandseed="*", **argkw): 
    219     """cross-validation evaluation of learners""" 
     425    """cross-validation evaluation of learners 
     427    Performs a cross validation with the given number of folds. 
     429    """ 
    220430    (examples, weight) = demangleExamples(examples) 
    221431    if indicesrandseed!="*": 
    230440                   strat=Orange.core.MakeRandomIndices.StratifiedIfPossible, 
    231441                   proportions=Orange.core.frange(0.1), pps=[], **argkw): 
    232     """construct a learning curve for learners""" 
     442    """Construct a learning curve for learners. 
     444    A simpler interface for the function :obj:`learningCurve`. Instead 
     445    of methods for preparing indices, it simply takes the number of folds 
     446    and a flag telling whether we want a stratified cross-validation or 
     447    not. This function does not return a single ``ExperimentResults`` but 
     448    a list of them, one for each proportion. :: 
     450        prop = [0.2, 0.4, 0.6, 0.8, 1.0] 
     451        res = orngTest.learningCurveN(learners, data, folds = 5, proportions = prop) 
     452        for i, p in enumerate(prop): 
     453            print "%5.3f:" % p, 
     454            printResults(res[i]) 
     456    This function basically prepares a random generator and example selectors 
     457    (``cv`` and ``pick``) and calls :obj:`learningCurve`. 
     459    """ 
    233461    seed = argkw.get("indicesrandseed", -1) or argkw.get("randseed", -1) 
    234462    if seed: 
    248476def learningCurve(learners, examples, cv=None, pick=None, proportions=Orange.core.frange(0.1), pps=[], **argkw): 
     477    """ 
     478    Computes learning curves using a procedure recommended by Salzberg 
     479    (1997). It first prepares data subsets (folds). For each proportion, 
     480    it performs the cross-validation, but taking only a proportion of 
     481    examples for learning. 
     483    Arguments ``cv`` and ``pick`` give the methods for preparing 
     484    indices for cross-validation and random selection of learning 
     485    examples. If they are not given, :obj:`Orange.core.MakeRandomIndicesCV` and 
     486    :obj:`Orange.core.MakeRandomIndices2` are used, both will be stratified and the 
     487    cross-validation will be 10-fold. Proportions is a list of proportions 
     488    of learning examples. 
     490    The function can save time by loading experimental existing data for 
     491    any test that were already conducted and saved. Also, the computed 
     492    results are stored for later use. You can enable this by adding 
     493    a keyword argument ``cache=1``. Another keyword deals with progress 
     494    report. If you add ``verbose=1``, the function will print the proportion 
     495    and the fold number. 
     497    """ 
    249498    verb = argkw.get("verbose", 0) 
    250499    cache = argkw.get("cache", 0) 
    330579                              proportions=Orange.core.frange(0.1), 
    331580                              strat=Orange.core.MakeRandomIndices.StratifiedIfPossible, pps=[], **argkw): 
     581    """ 
     582    This function is suitable for computing a learning curve on datasets, 
     583    where learning and testing examples are split in advance. For each 
     584    proportion of learning examples, it randomly select the requested 
     585    number of learning examples, builds the models and tests them on the 
     586    entire testset. The whole test is repeated for the given number of 
     587    times for each proportion. The result is a list of ExperimentResults, 
     588    one for each proportion. 
     590    In the following scripts, examples are pre-divided onto training 
     591    and testing set. Learning curves are computed in which 20, 40, 60, 
     592    80 and 100 percents of the examples in the former set are used for 
     593    learning and the latter set is used for testing. Random selection 
     594    of the given proportion of learning set is repeated for five times. 
     596    .. literalinclude:: code/ 
     597        :start-after: Learning curve with pre-separated data 
     598        :end-before: # End 
     601    """ 
    332602    verb = argkw.get("verbose", 0) 
    357627def testWithIndices(learners, examples, indices, indicesrandseed="*", pps=[], callback=None, **argkw): 
     628    """ 
     629    Performs a cross-validation-like test. The difference is that the 
     630    caller provides indices (each index gives a fold of an example) which 
     631    do not necessarily divide the examples into folds of (approximately) 
     632    same sizes. In fact, the function :obj:`crossValidation` is actually written 
     633    as a single call to ``testWithIndices``. 
     635    ``testWithIndices`` takes care the ``TestedExamples`` are in the same order 
     636    as the corresponding examples in the original set. Preprocessing of 
     637    testing examples is thus not allowed. The computed results can be 
     638    saved in files or loaded therefrom if you add a keyword argument 
     639    ``cache=1``. In this case, you also have to specify the random seed 
     640    which was used to compute the indices (argument ``indicesrandseed``; 
     641    if you don't there will be no caching. 
     643    """ 
    358645    verb = argkw.get("verbose", 0) 
    359646    cache = argkw.get("cache", 0) 
    456743def learnAndTestOnTestData(learners, learnset, testset, testResults=None, iterationNumber=0, pps=[], callback=None, **argkw): 
     744    """ 
     745    This function performs no sampling on its own: two separate datasets 
     746    need to be passed, one for training and the other for testing. The 
     747    function preprocesses the data, induces the model and tests it. The 
     748    order of filters is peculiar, but it makes sense when compared to 
     749    other methods that support preprocessing of testing examples. The 
     750    function first applies preprocessors marked ``"B"`` (both sets), and only 
     751    then the preprocessors that need to processor only one of the sets. 
     753    You can pass an already initialized :obj:`ExperimentResults` (argument 
     754    ``results``) and an iteration number (``iterationNumber``). Results 
     755    of the test will be appended with the given iteration 
     756    number. This is because :obj:`learnAndTestWithTestData` 
     757    gets called by other functions, like :obj:`proportionTest` and 
     758    :obj:`learningCurveWithTestData`. If you omit the parameters, a new 
     759    :obj:`ExperimentResults` will be created. 
     761    """ 
    457762    storeclassifiers = argkw.get("storeclassifiers", 0) or argkw.get("storeClassifiers", 0) 
    458763    storeExamples = argkw.get("storeExamples", 0) 
    490795def learnAndTestOnLearnData(learners, learnset, testResults=None, iterationNumber=0, pps=[], callback=None, **argkw): 
     796    """ 
     797    This function is similar to the above, except that it learns and 
     798    tests on the same data. If first preprocesses the data with ``"B"`` 
     799    preprocessors on the whole data, and afterwards any ``"L"`` or ``"T"`` 
     800    preprocessors on separate datasets. Then it induces the model from 
     801    the learning set and tests it on the testing set. 
     803    As with :obj:`learnAndTestOnTestData`, you can pass an already initialized 
     804    :obj:`ExperimentResults` (argument ``results``) and an iteration number to the 
     805    function. In this case, results of the test will be appended with 
     806    the given iteration number. 
     808    """ 
    491810    storeclassifiers = argkw.get("storeclassifiers", 0) or argkw.get("storeClassifiers", 0) 
    492811    storeExamples = argkw.get("storeExamples", 0) 
    527846def testOnData(classifiers, testset, testResults=None, iterationNumber=0, storeExamples = False, **argkw): 
     847    """ 
     848    This function gets a list of classifiers, not learners like the other 
     849    functions in this module. It classifies each testing example with 
     850    each classifier. You can pass an existing :obj:`ExperimentResults` 
     851    and iteration number, like in :obj:`learnAndTestWithTestData` 
     852    (which actually calls :obj:`testWithTestData`). If you don't, a new 
     853    :obj:`ExperimentResults` will be created. 
     855    """ 
    528857    testset, testweight = demangleExamples(testset) 
  • orange/doc/Orange/rst/code/

    r7421 r7514  
    2121    printResults(res) 
    23 print "\nproportionsTest that will give different results, but the same each \ 
    24 time the script is run" 
     23print "\nproportionsTest that will give different results, \ 
     24but the same each time the script is run" 
    2525myRandom = Orange.core.RandomGenerator() 
    2626for i in range(3): 
    2828        randomGenerator=myRandom) 
    2929    printResults(res) 
     30# End 
    3132print "\nproportionsTest that will give different results each time it is run" 
    3435        randseed=random.randint(0, 100)) 
    3536    printResults(res) 
     37# End 
    3739print "\nproportionsTest + storing classifiers" 
    5254    print "%5.3f:" % prop[i], 
    5355    printResults(res[i]) 
     56# End 
    5558print "\nLearning curve with pre-separated data" 
    6265    print "%5.3f:" % prop[i], 
    6366    printResults(res[i]) 
     67# End 
    6569print "\nLearning and testing on pre-separated data" 
Note: See TracChangeset for help on using the changeset viewer.