Changeset 10199:862759340b24 in orange


Ignore:
Timestamp:
02/13/12 16:51:54 (2 years ago)
Author:
anzeh <anze.staric@…>
Branch:
default
Message:

Fixed documentation warnings.

Files:
2 edited

Legend:

Unmodified
Added
Removed
  • Orange/misc/__init__.py

    r10197 r10199  
    342342 
    343343.. automodule:: Orange.misc.environ 
     344 
    344345""" 
    345346import environ 
  • docs/reference/rst/Orange.evaluation.scoring.rst

    r10176 r10199  
    5050.. autofunction:: Brier_score 
    5151 
    52 .. autofunction:: AUC 
    53  
    54     .. attribute:: AUC.ByWeightedPairs (or 0) 
    55  
    56         Computes AUC for each pair of classes (ignoring instances of all other 
    57         classes) and averages the results, weighting them by the number of 
    58         pairs of instances from these two classes (e.g. by the product of 
    59         probabilities of the two classes). AUC computed in this way still 
    60         behaves as concordance index, e.g., gives the probability that two 
    61         randomly chosen instances from different classes will be correctly 
    62         recognized (this is of course true only if the classifier knows 
    63         from which two classes the instances came). 
    64  
    65     .. attribute:: AUC.ByPairs (or 1) 
    66  
    67         Similar as above, except that the average over class pairs is not 
    68         weighted. This AUC is, like the binary, independent of class 
    69         distributions, but it is not related to concordance index any more. 
    70  
    71     .. attribute:: AUC.WeightedOneAgainstAll (or 2) 
    72  
    73         For each class, it computes AUC for this class against all others (that 
    74         is, treating other classes as one class). The AUCs are then averaged by 
    75         the class probabilities. This is related to concordance index in which 
    76         we test the classifier's (average) capability for distinguishing the 
    77         instances from a specified class from those that come from other classes. 
    78         Unlike the binary AUC, the measure is not independent of class 
    79         distributions. 
    80  
    81     .. attribute:: AUC.OneAgainstAll (or 3) 
    82  
    83         As above, except that the average is not weighted. 
    84  
    85    In case of multiple folds (for instance if the data comes from cross 
    86    validation), the computation goes like this. When computing the partial 
    87    AUCs for individual pairs of classes or singled-out classes, AUC is 
    88    computed for each fold separately and then averaged (ignoring the number 
    89    of instances in each fold, it's just a simple average). However, if a 
    90    certain fold doesn't contain any instances of a certain class (from the 
    91    pair), the partial AUC is computed treating the results as if they came 
    92    from a single-fold. This is not really correct since the class 
    93    probabilities from different folds are not necessarily comparable, 
    94    yet this will most often occur in a leave-one-out experiments, 
    95    comparability shouldn't be a problem. 
    96  
    97    Computing and printing out the AUC's looks just like printing out 
    98    classification accuracies (except that we call AUC instead of 
    99    CA, of course):: 
    100  
    101        AUCs = Orange.evaluation.scoring.AUC(res) 
    102        for l in range(len(learners)): 
    103            print "%10s: %5.3f" % (learners[l].name, AUCs[l]) 
    104  
    105    For vehicle, you can run exactly this same code; it will compute AUCs 
    106    for all pairs of classes and return the average weighted by probabilities 
    107    of pairs. Or, you can specify the averaging method yourself, like this:: 
    108  
    109        AUCs = Orange.evaluation.scoring.AUC(resVeh, Orange.evaluation.scoring.AUC.WeightedOneAgainstAll) 
    110  
    111    The following snippet tries out all four. (We don't claim that this is 
    112    how the function needs to be used; it's better to stay with the default.):: 
    113  
    114        methods = ["by pairs, weighted", "by pairs", "one vs. all, weighted", "one vs. all"] 
    115        print " " *25 + "  \tbayes\ttree\tmajority" 
    116        for i in range(4): 
    117            AUCs = Orange.evaluation.scoring.AUC(resVeh, i) 
    118            print "%25s: \t%5.3f\t%5.3f\t%5.3f" % ((methods[i], ) + tuple(AUCs)) 
    119  
    120    As you can see from the output:: 
    121  
    122                                    bayes   tree    majority 
    123               by pairs, weighted:  0.789   0.871   0.500 
    124                         by pairs:  0.791   0.872   0.500 
    125            one vs. all, weighted:  0.783   0.800   0.500 
    126                      one vs. all:  0.783   0.800   0.500 
     52.. autosingleton::AUC 
     53.. autoclass:: AucClass 
     54    :members: 
     55 
     56*AUC.ByWeightedPairs (or 0)* 
     57 
     58    Computes AUC for each pair of classes (ignoring instances of all other 
     59    classes) and averages the results, weighting them by the number of 
     60    pairs of instances from these two classes (e.g. by the product of 
     61    probabilities of the two classes). AUC computed in this way still 
     62    behaves as concordance index, e.g., gives the probability that two 
     63    randomly chosen instances from different classes will be correctly 
     64    recognized (this is of course true only if the classifier knows 
     65    from which two classes the instances came). 
     66 
     67*AUC.ByPairs (or 1)* 
     68 
     69    Similar as above, except that the average over class pairs is not 
     70    weighted. This AUC is, like the binary, independent of class 
     71    distributions, but it is not related to concordance index any more. 
     72 
     73*AUC.WeightedOneAgainstAll (or 2)* 
     74 
     75    For each class, it computes AUC for this class against all others (that 
     76    is, treating other classes as one class). The AUCs are then averaged by 
     77    the class probabilities. This is related to concordance index in which 
     78    we test the classifier's (average) capability for distinguishing the 
     79    instances from a specified class from those that come from other classes. 
     80    Unlike the binary AUC, the measure is not independent of class 
     81    distributions. 
     82 
     83*AUC.OneAgainstAll (or 3)* 
     84 
     85    As above, except that the average is not weighted. 
     86 
     87In case of multiple folds (for instance if the data comes from cross 
     88validation), the computation goes like this. When computing the partial 
     89AUCs for individual pairs of classes or singled-out classes, AUC is 
     90computed for each fold separately and then averaged (ignoring the number 
     91of instances in each fold, it's just a simple average). However, if a 
     92certain fold doesn't contain any instances of a certain class (from the 
     93pair), the partial AUC is computed treating the results as if they came 
     94from a single-fold. This is not really correct since the class 
     95probabilities from different folds are not necessarily comparable, 
     96yet this will most often occur in a leave-one-out experiments, 
     97comparability shouldn't be a problem. 
     98 
     99Computing and printing out the AUC's looks just like printing out 
     100classification accuracies (except that we call AUC instead of 
     101CA, of course):: 
     102 
     103   AUCs = Orange.evaluation.scoring.AUC(res) 
     104   for l in range(len(learners)): 
     105       print "%10s: %5.3f" % (learners[l].name, AUCs[l]) 
     106 
     107For vehicle, you can run exactly this same code; it will compute AUCs 
     108for all pairs of classes and return the average weighted by probabilities 
     109of pairs. Or, you can specify the averaging method yourself, like this:: 
     110 
     111   AUCs = Orange.evaluation.scoring.AUC(resVeh, Orange.evaluation.scoring.AUC.WeightedOneAgainstAll) 
     112 
     113The following snippet tries out all four. (We don't claim that this is 
     114how the function needs to be used; it's better to stay with the default.):: 
     115 
     116   methods = ["by pairs, weighted", "by pairs", "one vs. all, weighted", "one vs. all"] 
     117   print " " *25 + "  \tbayes\ttree\tmajority" 
     118   for i in range(4): 
     119       AUCs = Orange.evaluation.scoring.AUC(resVeh, i) 
     120       print "%25s: \t%5.3f\t%5.3f\t%5.3f" % ((methods[i], ) + tuple(AUCs)) 
     121 
     122As you can see from the output:: 
     123 
     124                               bayes   tree    majority 
     125          by pairs, weighted:  0.789   0.871   0.500 
     126                    by pairs:  0.791   0.872   0.500 
     127       one vs. all, weighted:  0.783   0.800   0.500 
     128                 one vs. all:  0.783   0.800   0.500 
    127129 
    128130.. autofunction:: AUC_single 
Note: See TracChangeset for help on using the changeset viewer.