Changeset 7840:69dc3fd3891a in orange


Ignore:
Timestamp:
04/14/11 09:14:06 (2 years ago)
Author:
mocnik <mocnik@…>
Branch:
default
Convert:
145119a52590fa17f6552c72dfb7c019285b9e0e
Message:

New underscore format for Orange.evaluation.scoring module.

File:
1 edited

Legend:

Unmodified
Added
Removed

  • orange/Orange/evaluation/scoring.py

    r7784 r7840  
    4343.. autofunction:: AP 
    4444 
    45 .. autofunction:: BrierScore 
     45.. autofunction:: Brier_score 
    4646 
    4747.. autofunction:: IS 
    … …  
    7272================ 
    7373 
    74 .. autofunction:: confusionMatrices 
     74.. autofunction:: confusion_matrices 
    7575 
    7676   **A positive-negative confusion matrix** is computed (a) if the class is 
    … …  
    8888    
    8989   We can also add the keyword argument :obj:`cutoff` 
    90    (e.g. confusionMatrices(results, cutoff=0.3); if we do, :obj:`confusionMatrices` 
     90   (e.g. confusion_matrices(results, cutoff=0.3); if we do, :obj:`confusion_matrices` 
    9191   will disregard the classifiers' class predictions and observe the predicted 
    9292   probabilities, and consider the prediction "positive" if the predicted 
    … …  
    9797   for naive Bayesian classifier:: 
    9898    
    99        cm = Orange.evaluation.scoring.confusionMatrices(res)[0] 
     99       cm = Orange.evaluation.scoring.confusion_matrices(res)[0] 
    100100       print "Confusion matrix for naive Bayes:" 
    101101       print "TP: %i, FP: %i, FN: %s, TN: %i" % (cm.TP, cm.FP, cm.FN, cm.TN) 
    102102        
    103        cm = Orange.evaluation.scoring.confusionMatrices(res, cutoff=0.2)[0] 
     103       cm = Orange.evaluation.scoring.confusion_matrices(res, cutoff=0.2)[0] 
    104104       print "Confusion matrix for naive Bayes:" 
    105105       print "TP: %i, FP: %i, FN: %s, TN: %i" % (cm.TP, cm.FP, cm.FN, cm.TN) 
    … …  
    122122   data set, we would compute the matrix like this:: 
    123123    
    124       cm = Orange.evaluation.scoring.confusionMatrices(resVeh, \ 
     124      cm = Orange.evaluation.scoring.confusion_matrices(resVeh, \ 
    125125vehicle.domain.classVar.values.index("van")) 
    126126    
    … …  
    152152   Here we see another example from `statExamples.py`_:: 
    153153    
    154        cm = Orange.evaluation.scoring.confusionMatrices(resVeh)[0] 
     154       cm = Orange.evaluation.scoring.confusion_matrices(resVeh)[0] 
    155155       classes = vehicle.domain.classVar.values 
    156156       print "\t"+"\t".join(classes) 
    … …  
    222222   part of `statExamples.py`_:: 
    223223    
    224        cm = Orange.evaluation.scoring.confusionMatrices(res) 
     224       cm = Orange.evaluation.scoring.confusion_matrices(res) 
    225225       print 
    226226       print "method\tsens\tspec" 
    … …  
    299299   of pairs. Or, you can specify the averaging method yourself, like this:: 
    300300    
    301        AUCs = Orange.evaluation.scoring.AUC(resVeh, orngStat.AUC.WeightedOneAgainstAll) 
     301       AUCs = Orange.evaluation.scoring.AUC(resVeh, Orange.evaluation.scoring.AUC.WeightedOneAgainstAll) 
    302302    
    303303   The following snippet tries out all four. (We don't claim that this is 
    … …  
    327327them, require that the results come from a test with a single iteration, 
    328328and they always compare one chosen class against all others. If you have 
    329 cross validation results, you can either use splitByIterations to split the 
     329cross validation results, you can either use split_by_iterations to split the 
    330330results by folds, call the function for each fold separately and then sum 
    331331the results up however you see fit, or you can set the ExperimentResults' 
    332332attribute numberOfIterations to 1, to cheat the function - at your own 
    333333responsibility for the statistical correctness. Regarding the multi-class 
    334 problems, if you don't chose a specific class, orngStat will use the class 
     334problems, if you don't chose a specific class, Orange.evaluation.scoring will use the class 
    335335attribute's baseValue at the time when results were computed. If baseValue 
    336336was not given at that time, 1 (that is, the second class) is used as default. 
    … …  
    346346.. autofunction:: AUCWilcoxon 
    347347 
    348 .. autofunction:: computeROC 
     348.. autofunction:: compute_ROC 
    349349 
    350350Comparison of Algorithms 
    … …  
    353353.. autofunction:: McNemar 
    354354 
    355 .. autofunction:: McNemarOfTwo 
     355.. autofunction:: McNemar_of_two 
    356356 
    357357========== 
    … …  
    420420================= 
    421421 
    422 .. autofunction:: splitByIterations 
     422.. autofunction:: split_by_iterations 
    423423 
    424424""" 
    … …  
    437437    return math.log(x)/math.log(2) 
    438438 
    439 def checkNonZero(x): 
     439def check_non_zero(x): 
    440440    """Throw Value Error when x = 0.0.""" 
    441441    if x==0.0: 
    … …  
    457457 
    458458 
    459 def splitByIterations(res): 
     459def split_by_iterations(res): 
    460460    """ Splits ExperimentResults of multiple iteratation test into a list 
    461461    of ExperimentResults, one for each iteration. 
    … …  
    471471 
    472472 
    473 def classProbabilitiesFromRes(res, **argkw): 
     473def class_probabilities_from_res(res, **argkw): 
    474474    """Calculate class probabilities""" 
    475475    probs = [0.0] * len(res.classValues) 
    … …  
    483483            probs[tex.actualClass] += tex.weight 
    484484            totweight += tex.weight 
    485         checkNonZero(totweight) 
     485        check_non_zero(totweight) 
    486486    return [prob/totweight for prob in probs] 
    487487 
    488488 
    489 def statisticsByFolds(stats, foldN, reportSE, iterationIsOuter): 
     489def statistics_by_folds(stats, foldN, reportSE, iterationIsOuter): 
    490490    # remove empty folds, turn the matrix so that learner is outer 
    491491    if iterationIsOuter: 
    … …  
    530530# Scores for evaluation of numeric predictions 
    531531 
    532 def checkArgkw(dct, lst): 
    533     """checkArgkw(dct, lst) -> returns true if any items have non-zero value in dct""" 
     532def check_argkw(dct, lst): 
     533    """check_argkw(dct, lst) -> returns true if any items have non-zero value in dct""" 
    534534    return reduce(lambda x,y: x or y, [dct.get(k, 0) for k in lst]) 
    535535 
    536 def regressionError(res, **argkw): 
    537     """regressionError(res) -> regression error (default: MSE)""" 
     536def regression_error(res, **argkw): 
     537    """regression_error(res) -> regression error (default: MSE)""" 
    538538    if argkw.get("SE", 0) and res.numberOfIterations > 1: 
    539539        # computes the scores for each iteration, then averages 
    … …  
    626626def MSE(res, **argkw): 
    627627    """ Computes mean-squared error. """ 
    628     return regressionError(res, **argkw) 
     628    return regression_error(res, **argkw) 
    629629     
    630630def RMSE(res, **argkw): 
    631631    """ Computes root mean-squared error. """ 
    632632    argkw.setdefault("sqrt", True) 
    633     return regressionError(res, **argkw) 
     633    return regression_error(res, **argkw) 
    634634 
    635635def MAE(res, **argkw): 
    636636    """ Computes mean absolute error. """ 
    637637    argkw.setdefault("abs", True) 
    638     return regressionError(res, **argkw) 
     638    return regression_error(res, **argkw) 
    639639 
    640640def RSE(res, **argkw): 
    641641    """ Computes relative squared error. """ 
    642642    argkw.setdefault("norm-sqr", True) 
    643     return regressionError(res, **argkw) 
     643    return regression_error(res, **argkw) 
    644644 
    645645def RRSE(res, **argkw): 
    … …  
    647647    argkw.setdefault("norm-sqr", True) 
    648648    argkw.setdefault("sqrt", True) 
    649     return regressionError(res, **argkw) 
     649    return regression_error(res, **argkw) 
    650650 
    651651def RAE(res, **argkw): 
    … …  
    653653    argkw.setdefault("abs", True) 
    654654    argkw.setdefault("norm-abs", True) 
    655     return regressionError(res, **argkw) 
     655    return regression_error(res, **argkw) 
    656656 
    657657def R2(res, **argkw): 
    … …  
    659659    argkw.setdefault("norm-sqr", True) 
    660660    argkw.setdefault("R2", True) 
    661     return regressionError(res, **argkw) 
     661    return regression_error(res, **argkw) 
    662662 
    663663def MSE_old(res, **argkw): 
    … …  
    731731        if type(res)==ConfusionMatrix: 
    732732            div = nm.TP+nm.FN+nm.FP+nm.TN 
    733             checkNonZero(div) 
     733            check_non_zero(div) 
    734734            ca = [(nm.TP+nm.TN)/div] 
    735735        else: 
    … …  
    744744                    CAs = map(lambda res, cls: res+(cls==tex.actualClass and tex.weight), CAs, tex.classes) 
    745745                    totweight += tex.weight 
    746             checkNonZero(totweight) 
     746            check_non_zero(totweight) 
    747747            ca = [x/totweight for x in CAs] 
    748748             
    … …  
    767767                foldN[tex.iterationNumber] += tex.weight 
    768768 
    769         return statisticsByFolds(CAsByFold, foldN, reportSE, False) 
     769        return statistics_by_folds(CAsByFold, foldN, reportSE, False) 
    770770 
    771771 
    … …  
    788788                APs = map(lambda res, probs: res + probs[tex.actualClass]*tex.weight, APs, tex.probabilities) 
    789789                totweight += tex.weight 
    790         checkNonZero(totweight) 
     790        check_non_zero(totweight) 
    791791        return [AP/totweight for AP in APs] 
    792792 
    … …  
    802802            foldN[tex.iterationNumber] += tex.weight 
    803803 
    804     return statisticsByFolds(APsByFold, foldN, reportSE, True) 
    805  
    806  
    807 def BrierScore(res, reportSE = False, **argkw): 
     804    return statistics_by_folds(APsByFold, foldN, reportSE, True) 
     805 
     806 
     807def Brier_score(res, reportSE = False, **argkw): 
    808808    """ Computes the Brier's score, defined as the average (over test examples) 
    809809    of sumx(t(x)-p(x))2, where x is a class, t(x) is 1 for the correct class 
    … …  
    834834                           res + tex.weight*reduce(lambda s, pi: s+pi**2, probs, 0) - 2*probs[tex.actualClass], MSEs, tex.probabilities) 
    835835            totweight = gettotweight(res) 
    836         checkNonZero(totweight) 
     836        check_non_zero(totweight) 
    837837        if reportSE: 
    838838            return [(max(x/totweight+1.0, 0), 0) for x in MSEs]  ## change this, not zero!!! 
    … …  
    854854            foldN[tex.iterationNumber] += tex.weight 
    855855 
    856     stats = statisticsByFolds(BSs, foldN, reportSE, True) 
     856    stats = statistics_by_folds(BSs, foldN, reportSE, True) 
    857857    if reportSE: 
    858858        return [(x+1.0, y) for x, y in stats] 
    … …  
    861861 
    862862def BSS(res, **argkw): 
    863     return [1-x/2 for x in apply(BrierScore, (res, ), argkw)] 
     863    return [1-x/2 for x in apply(Brier_score, (res, ), argkw)] 
    864864 
    865865def IS_ex(Pc, P): 
    … …  
    879879    """ 
    880880    if not apriori: 
    881         apriori = classProbabilitiesFromRes(res) 
     881        apriori = class_probabilities_from_res(res) 
    882882 
    883883    if res.numberOfIterations==1: 
    … …  
    918918            foldN[tex.iterationNumber] += tex.weight 
    919919 
    920     return statisticsByFolds(ISs, foldN, reportSE, False) 
     920    return statistics_by_folds(ISs, foldN, reportSE, False) 
    921921 
    922922 
    923923def Friedman(res, statistics, **argkw): 
    924924    sums = None 
    925     for ri in splitByIterations(res): 
     925    for ri in split_by_iterations(res): 
    926926        ranks = statc.rankdata(apply(statistics, (ri,), argkw)) 
    927927        if sums: 
    … …  
    939939def Wilcoxon(res, statistics, **argkw): 
    940940    res1, res2 = [], [] 
    941     for ri in splitByIterations(res): 
     941    for ri in split_by_iterations(res): 
    942942        stats = apply(statistics, (ri,), argkw) 
    943943        if (len(stats) != 2): 
    … …  
    947947    return statc.wilcoxont(res1, res2) 
    948948 
    949 def rankDifference(res, statistics, **argkw): 
     949def rank_difference(res, statistics, **argkw): 
    950950    if not res.results: 
    951951        raise TypeError, "no experiments" 
    … …  
    960960     
    961961class ConfusionMatrix: 
     962    """ Class ConfusionMatrix stores data about false and true 
     963    predictions compared to real class. It stores the number of 
     964    True Negatives, False Positive, False Negatives and True Positives. 
     965    """ 
    962966    def __init__(self): 
    963967        self.TP = self.FN = self.FP = self.TN = 0.0 
    … …  
    976980 
    977981 
    978 def confusionMatrices(res, classIndex=-1, **argkw): 
     982def confusion_matrices(res, classIndex=-1, **argkw): 
    979983    """ This function can compute two different forms of confusion matrix: 
    980984    one in which a certain class is marked as positive and the other(s) 
    … …  
    10371041 
    10381042# obsolete (renamed) 
    1039 computeConfusionMatrices = confusionMatrices 
    1040  
    1041  
    1042 def confusionChiSquare(confusionMatrix): 
     1043compute_confusion_matrices = confusion_matrices 
     1044 
     1045 
     1046def confusion_chi_square(confusionMatrix): 
    10431047    dim = len(confusionMatrix) 
    10441048    rowPriors = [sum(r) for r in confusionMatrix] 
    … …  
    11791183    return r 
    11801184 
    1181 def scottsPi(confm, bIsListOfMatrices=True): 
     1185def scotts_pi(confm, bIsListOfMatrices=True): 
    11821186   """Compute Scott's Pi for measuring inter-rater agreement for nominal data 
    11831187 
    … …  
    11881192   @param confm: confusion matrix, or list of confusion matrices. To obtain 
    11891193                           non-binary confusion matrix, call 
    1190                            orngStat.computeConfusionMatrices and set the 
     1194                           Orange.evaluation.scoring.compute_confusion_matrices and set the 
    11911195                           classIndex parameter to -2. 
    11921196   @param bIsListOfMatrices: specifies whether confm is list of matrices. 
    … …  
    11991203   if bIsListOfMatrices: 
    12001204       try: 
    1201            return [scottsPi(cm, bIsListOfMatrices=False) for cm in confm] 
     1205           return [scotts_pi(cm, bIsListOfMatrices=False) for cm in confm] 
    12021206       except TypeError: 
    12031207           # Nevermind the parameter, maybe this is a "conventional" binary 
    … …  
    12631267AROC = AUCWilcoxon # for backward compatibility, AROC is obsolote 
    12641268 
    1265 def compare2AUCs(res, lrn1, lrn2, classIndex=-1, **argkw): 
     1269def compare_2_AUCs(res, lrn1, lrn2, classIndex=-1, **argkw): 
    12661270    import corn 
    12671271    return corn.compare2ROCs(res, lrn1, lrn2, classIndex, res.weights and not argkw.get("unweighted")) 
    12681272 
    1269 compare2AROCs = compare2AUCs # for backward compatibility, compare2AROCs is obsolote 
    1270  
    1271      
    1272 def computeROC(res, classIndex=-1): 
     1273compare_2_AROCs = compare_2_AUCs # for backward compatibility, compare_2_AROCs is obsolote 
     1274 
     1275     
     1276def compute_ROC(res, classIndex=-1): 
    12731277    """ Computes a ROC curve as a list of (x, y) tuples, where x is  
    12741278    1-specificity and y is sensitivity. 
    … …  
    13021306## TC's implementation of algorithms, taken from: 
    13031307## T Fawcett: ROC Graphs: Notes and Practical Considerations for Data Mining Researchers, submitted to KDD Journal.  
    1304 def ROCslope((P1x, P1y, P1fscore), (P2x, P2y, P2fscore)): 
     1308def ROC_slope((P1x, P1y, P1fscore), (P2x, P2y, P2fscore)): 
    13051309    if (P1x == P2x): 
    13061310        return 1e300 
    13071311    return (P1y - P2y) / (P1x - P2x) 
    13081312 
    1309 def ROCaddPoint(P, R, keepConcavities=1): 
     1313def ROC_add_point(P, R, keepConcavities=1): 
    13101314    if keepConcavities: 
    13111315        R.append(P) 
    … …  
    13181322                T = R.pop() 
    13191323                T2 = R[-1] 
    1320                 if ROCslope(T2, T) > ROCslope(T, P): 
     1324                if ROC_slope(T2, T) > ROC_slope(T, P): 
    13211325                    R.append(T) 
    13221326                    R.append(P) 
    … …  
    13241328    return R 
    13251329 
    1326 def TCcomputeROC(res, classIndex=-1, keepConcavities=1): 
     1330def TC_compute_ROC(res, classIndex=-1, keepConcavities=1): 
    13271331    import corn 
    13281332    problists, tots = corn.computeROCCumulative(res, classIndex) 
    … …  
    13491353                else: 
    13501354                    fpr = 0.0 
    1351                 curve = ROCaddPoint((fpr, tpr, fPrev), curve, keepConcavities) 
     1355                curve = ROC_add_point((fpr, tpr, fPrev), curve, keepConcavities) 
    13521356                fPrev = f 
    13531357            thisPos, thisNeg = prob[1][1], prob[1][0] 
    … …  
    13621366        else: 
    13631367            fpr = 0.0 
    1364         curve = ROCaddPoint((fpr, tpr, f), curve, keepConcavities) ## ugly 
     1368        curve = ROC_add_point((fpr, tpr, f), curve, keepConcavities) ## ugly 
    13651369        results.append(curve) 
    13661370 
    … …  
    13691373## returns a list of points at the intersection of the tangential iso-performance line and the given ROC curve 
    13701374## for given values of FPcost, FNcost and pval 
    1371 def TCbestThresholdsOnROCcurve(FPcost, FNcost, pval, curve): 
     1375def TC_best_thresholds_on_ROC_curve(FPcost, FNcost, pval, curve): 
    13721376    m = (FPcost*(1.0 - pval)) / (FNcost*pval) 
    13731377 
    … …  
    14221426## for each (sub)set of input ROC curves 
    14231427## returns the average ROC curve and an array of (vertical) standard deviations 
    1424 def TCverticalAverageROC(ROCcurves, samples = 10): 
     1428def TC_vertical_average_ROC(ROCcurves, samples = 10): 
    14251429    def INTERPOLATE((P1x, P1y, P1fscore), (P2x, P2y, P2fscore), X): 
    14261430        if (P1x == P2x) or ((X > P1x) and (X > P2x)) or ((X < P1x) and (X < P2x)): 
    … …  
    14461450        elif fp < FPsample and i + 1 == len(ROC): # return the last 
    14471451            return ROC[i][1] 
    1448         raise ValueError, "cannot compute: TP_FOR_FP in TCverticalAverageROC" 
     1452        raise ValueError, "cannot compute: TP_FOR_FP in TC_vertical_average_ROC" 
    14491453        #return 0.0 
    14501454 
    … …  
    14811485## for each (sub)set of input ROC curves 
    14821486## returns the average ROC curve, an array of vertical standard deviations and an array of horizontal standard deviations 
    1483 def TCthresholdlAverageROC(ROCcurves, samples = 10): 
     1487def TC_threshold_average_ROC(ROCcurves, samples = 10): 
    14841488    def POINT_AT_THRESH(ROC, npts, thresh): 
    14851489        i = 0 
    … …  
    15461550##  - yesClassRugPoints is an array of (x, 1) points 
    15471551##  - noClassRugPoints is an array of (x, 0) points 
    1548 def computeCalibrationCurve(res, classIndex=-1): 
     1552def compute_calibration_curve(res, classIndex=-1): 
    15491553    import corn 
    15501554    ## merge multiple iterations into one 
    … …  
    16081612## returns an array of curve elements, where: 
    16091613##  - curve is an array of points ((TP+FP)/(P + N), TP/P, (th, FP/N)) on the Lift Curve 
    1610 def computeLiftCurve(res, classIndex=-1): 
     1614def compute_lift_curve(res, classIndex=-1): 
    16111615    import corn 
    16121616    ## merge multiple iterations into one 
    … …  
    16391643    self.C, self.D, self.T = C, D, T 
    16401644    
    1641 def isCDTEmpty(cdt): 
     1645def is_CDT_empty(cdt): 
    16421646    return cdt.C + cdt.D + cdt.T < 1e-20 
    16431647 
    16441648 
    1645 def computeCDT(res, classIndex=-1, **argkw): 
     1649def compute_CDT(res, classIndex=-1, **argkw): 
    16461650    """Obsolete, don't use""" 
    16471651    import corn 
    … …  
    16571661    if (res.numberOfIterations>1): 
    16581662        CDTs = [CDT() for i in range(res.numberOfLearners)] 
    1659         iterationExperiments = splitByIterations(res) 
     1663        iterationExperiments = split_by_iterations(res) 
    16601664        for exp in iterationExperiments: 
    16611665            expCDTs = corn.computeCDT(exp, classIndex, useweights) 
    … …  
    16651669                CDTs[i].T += expCDTs[i].T 
    16661670        for i in range(res.numberOfLearners): 
    1667             if isCDTEmpty(CDTs[0]): 
     1671            if is_CDT_empty(CDTs[0]): 
    16681672                return corn.computeCDT(res, classIndex, useweights) 
    16691673         
    … …  
    16741678## THIS FUNCTION IS OBSOLETE AND ITS AVERAGING OVER FOLDS IS QUESTIONABLE 
    16751679## DON'T USE IT 
    1676 def ROCsFromCDT(cdt, **argkw): 
     1680def ROCs_from_CDT(cdt, **argkw): 
    16771681    """Obsolete, don't use""" 
    16781682    if type(cdt) == list: 
    1679         return [ROCsFromCDT(c) for c in cdt] 
     1683        return [ROCs_from_CDT(c) for c in cdt] 
    16801684 
    16811685    C, D, T = cdt.C, cdt.D, cdt.T 
    … …  
    17051709    return res 
    17061710 
    1707 AROCFromCDT = ROCsFromCDT  # for backward compatibility, AROCFromCDT is obsolote 
     1711AROC_from_CDT = ROCs_from_CDT  # for backward compatibility, AROC_from_CDT is obsolote 
    17081712 
    17091713 
    … …  
    17161720def AUC_x(cdtComputer, ite, all_ite, divideByIfIte, computerArgs): 
    17171721    cdts = cdtComputer(*(ite, ) + computerArgs) 
    1718     if not isCDTEmpty(cdts[0]): 
     1722    if not is_CDT_empty(cdts[0]): 
    17191723        return [(cdt.C+cdt.T/2)/(cdt.C+cdt.D+cdt.T)/divideByIfIte for cdt in cdts], True 
    17201724         
    17211725    if all_ite: 
    17221726         cdts = cdtComputer(*(all_ite, ) + computerArgs) 
    1723          if not isCDTEmpty(cdts[0]): 
     1727         if not is_CDT_empty(cdts[0]): 
    17241728             return [(cdt.C+cdt.T/2)/(cdt.C+cdt.D+cdt.T) for cdt in cdts], False 
    17251729 
    … …  
    17581762def AUC_binary(res, useWeights = True): 
    17591763    if res.numberOfIterations > 1: 
    1760         return AUC_iterations(AUC_i, splitByIterations(res), (-1, useWeights, res, res.numberOfIterations)) 
     1764        return AUC_iterations(AUC_i, split_by_iterations(res), (-1, useWeights, res, res.numberOfIterations)) 
    17611765    else: 
    17621766        return AUC_i(res, -1, useWeights)[0] 
    … …  
    17671771     
    17681772    if res.numberOfIterations > 1: 
    1769         iterations = splitByIterations(res) 
     1773        iterations = split_by_iterations(res) 
    17701774        all_ite = res 
    17711775    else: 
    … …  
    17781782 
    17791783    if method in [0, 2]: 
    1780         prob = classProbabilitiesFromRes(res) 
     1784        prob = class_probabilities_from_res(res) 
    17811785         
    17821786    if method <= 1: 
    … …  
    18521856 
    18531857    if res.numberOfIterations > 1: 
    1854         return AUC_iterations(AUC_i, splitByIterations(res), (classIndex, useWeights, res, res.numberOfIterations)) 
     1858        return AUC_iterations(AUC_i, split_by_iterations(res), (classIndex, useWeights, res, res.numberOfIterations)) 
    18551859    else: 
    18561860        return AUC_i( res, classIndex, useWeights)[0] 
    … …  
    18631867    """ 
    18641868    if res.numberOfIterations > 1: 
    1865         return AUC_iterations(AUC_ij, splitByIterations(res), (classIndex1, classIndex2, useWeights, res, res.numberOfIterations)) 
     1869        return AUC_iterations(AUC_ij, split_by_iterations(res), (classIndex1, classIndex2, useWeights, res, res.numberOfIterations)) 
    18661870    else: 
    18671871        return AUC_ij(res, classIndex1, classIndex2, useWeights) 
    … …  
    18761880     
    18771881        classes = vehicle.domain.classVar.values 
    1878         AUCmatrix = orngStat.AUC_matrix(resVeh)[0] 
     1882        AUCmatrix = Orange.evaluation.scoring.AUC_matrix(resVeh)[0] 
    18791883        print "\t"+"\t".join(classes[:-1]) 
    18801884        for className, AUCrow in zip(classes[1:], AUCmatrix[1:]): 
    … …  
    18851889     
    18861890    if res.numberOfIterations > 1: 
    1887         iterations, all_ite = splitByIterations(res), res 
     1891        iterations, all_ite = split_by_iterations(res), res 
    18881892    else: 
    18891893        iterations, all_ite = [res], None 
    18901894     
    18911895    aucs = [[[] for i in range(numberOfClasses)] for i in range(numberOfLearners)] 
    1892     prob = classProbabilitiesFromRes(res) 
     1896    prob = class_probabilities_from_res(res) 
    18931897         
    18941898    for classIndex1 in range(numberOfClasses): 
    … …  
    19511955 
    19521956 
    1953 def McNemarOfTwo(res, lrn1, lrn2): 
    1954     """ McNemarOfTwo computes a McNemar statistics for a pair of classifier, 
     1957def McNemar_of_two(res, lrn1, lrn2): 
     1958    """ McNemar_of_two computes a McNemar statistics for a pair of classifier, 
    19551959    specified by indices learner1 and learner2. 
    19561960    """ 
    … …  
    19841988        Returns F, p and average ranks 
    19851989    """ 
    1986     res_split = splitByIterations(res) 
     1990    res_split = split_by_iterations(res) 
    19871991    res = [stat(r) for r in res_split] 
    19881992     
    … …  
    19921996    for r in res: 
    19931997        ranks = [k-x+1 for x in statc.rankdata(r)] 
    1994         if stat==BrierScore: # reverse ranks for BrierScore (lower better) 
     1998        if stat==Brier_score: # reverse ranks for Brier_score (lower better) 
    19951999            ranks = [k+1-x for x in ranks] 
    19962000        sums = [ranks[i]+sums[i] for i in range(k)] 
    … …  
    20042008 
    20052009 
    2006 def WilcoxonPairs(res, avgranks, stat=CA): 
     2010def Wilcoxon_pairs(res, avgranks, stat=CA): 
    20072011    """ Returns a triangular matrix, where element[i][j] stores significance of difference 
    20082012        between i-th and j-th classifier, as computed by Wilcoxon test. The element is positive 
    … …  
    20112015        and, optionally, a statistics; greater values should mean better results.append 
    20122016    """ 
    2013     res_split = splitByIterations(res) 
     2017    res_split = split_by_iterations(res) 
    20142018    res = [stat(r) for r in res_split] 
    20152019 
    … …  
    20302034 
    20312035 
    2032 def plotLearningCurveLearners(file, allResults, proportions, learners, noConfidence=0): 
    2033     plotLearningCurve(file, allResults, proportions, [Orange.misc.getobjectname(learners[i], "Learner %i" % i) for i in range(len(learners))], noConfidence) 
    2034      
    2035 def plotLearningCurve(file, allResults, proportions, legend, noConfidence=0): 
     2036def plot_learning_curve_learners(file, allResults, proportions, learners, noConfidence=0): 
     2037    plot_learning_curve(file, allResults, proportions, [Orange.misc.getobjectname(learners[i], "Learner %i" % i) for i in range(len(learners))], noConfidence) 
     2038     
     2039def plot_learning_curve(file, allResults, proportions, legend, noConfidence=0): 
    20362040    import types 
    20372041    fopened=0 
    … …  
    20682072 
    20692073 
    2070 def printSingleROCCurveCoordinates(file, curve): 
     2074def print_single_ROC_curve_coordinates(file, curve): 
    20712075    import types 
    20722076    fopened=0 
    … …  
    20822086 
    20832087 
    2084 def plotROCLearners(file, curves, learners): 
    2085     plotROC(file, curves, [Orange.misc.getobjectname(learners[i], "Learner %i" % i) for i in range(len(learners))]) 
    2086      
    2087 def plotROC(file, curves, legend): 
     2088def plot_ROC_learners(file, curves, learners): 
     2089    plot_ROC(file, curves, [Orange.misc.getobjectname(learners[i], "Learner %i" % i) for i in range(len(learners))]) 
     2090     
     2091def plot_ROC(file, curves, legend): 
    20882092    import types 
    20892093    fopened=0 
    … …  
    21132117 
    21142118 
    2115 def plotMcNemarCurveLearners(file, allResults, proportions, learners, reference=-1): 
    2116     plotMcNemarCurve(file, allResults, proportions, [Orange.misc.getobjectname(learners[i], "Learner %i" % i) for i in range(len(learners))], reference) 
    2117  
    2118 def plotMcNemarCurve(file, allResults, proportions, legend, reference=-1): 
     2119def plot_McNemar_curve_learners(file, allResults, proportions, learners, reference=-1): 
     2120    plot_McNemar_curve(file, allResults, proportions, [Orange.misc.getobjectname(learners[i], "Learner %i" % i) for i in range(len(learners))], reference) 
     2121 
     2122def plot_McNemar_curve(file, allResults, proportions, legend, reference=-1): 
    21192123    if reference<0: 
    21202124        reference=len(legend)-1 
    … …  
    21382142    for i in tmap: 
    21392143        for p in range(len(proportions)): 
    2140             file.write("%f\t%f\n" % (proportions[p], McNemarOfTwo(allResults[p], i, reference))) 
     2144            file.write("%f\t%f\n" % (proportions[p], McNemar_of_two(allResults[p], i, reference))) 
    21412145        file.write("e\n\n") 
    21422146 
    … …  
    21442148        file.close() 
    21452149 
    2146 defaultPointTypes=("{$\\circ$}", "{$\\diamond$}", "{$+$}", "{$\\times$}", "{$|$}")+tuple([chr(x) for x in range(97, 122)]) 
    2147 defaultLineTypes=("\\setsolid", "\\setdashpattern <4pt, 2pt>", "\\setdashpattern <8pt, 2pt>", "\\setdashes", "\\setdots") 
    2148  
    2149 def learningCurveLearners2PiCTeX(file, allResults, proportions, **options): 
    2150     return apply(learningCurve2PiCTeX, (file, allResults, proportions), options) 
    2151      
    2152 def learningCurve2PiCTeX(file, allResults, proportions, **options): 
     2150default_point_types=("{$\\circ$}", "{$\\diamond$}", "{$+$}", "{$\\times$}", "{$|$}")+tuple([chr(x) for x in range(97, 122)]) 
     2151default_line_types=("\\setsolid", "\\setdashpattern <4pt, 2pt>", "\\setdashpattern <8pt, 2pt>", "\\setdashes", "\\setdots") 
     2152 
     2153def learning_curve_learners_to_PiCTeX(file, allResults, proportions, **options): 
     2154    return apply(learning_curve_to_PiCTeX, (file, allResults, proportions), options) 
     2155     
     2156def learning_curve_to_PiCTeX(file, allResults, proportions, **options): 
    21532157    import types 
    21542158    fopened=0 
    … …  
    21672171    yshift=float(options.get("yshift", -ntestexamples/20.)) 
    21682172     
    2169     pointtypes=options.get("pointtypes", defaultPointTypes) 
    2170     linetypes=options.get("linetypes", defaultLineTypes) 
     2173    pointtypes=options.get("pointtypes", default_point_types) 
     2174    linetypes=options.get("linetypes", default_line_types) 
    21712175 
    21722176    if options.has_key("numberedx"): 
    … …  
    22272231    del file 
    22282232 
    2229 def legendLearners2PiCTeX(file, learners, **options): 
    2230   return apply(legend2PiCTeX, (file, [Orange.misc.getobjectname(learners[i], "Learner %i" % i) for i in range(len(learners))]), options) 
    2231      
    2232 def legend2PiCTeX(file, legend, **options): 
     2233def legend_learners_to_PiCTeX(file, learners, **options): 
     2234  return apply(legend_to_PiCTeX, (file, [Orange.misc.getobjectname(learners[i], "Learner %i" % i) for i in range(len(learners))]), options) 
     2235     
     2236def legend_to_PiCTeX(file, legend, **options): 
    22332237    import types 
    22342238    fopened=0 
    … …  
    22372241        fopened=1 
    22382242 
    2239     pointtypes=options.get("pointtypes", defaultPointTypes) 
    2240     linetypes=options.get("linetypes", defaultLineTypes) 
     2243    pointtypes=options.get("pointtypes", default_point_types) 
     2244    linetypes=options.get("linetypes", default_line_types) 
    22412245 
    22422246    file.write("\\mbox{\n") 
    … …  
    23852389        return 
    23862390 
    2387     def printFigure(fig, *args, **kwargs): 
     2391    def print_figure(fig, *args, **kwargs): 
    23882392        canvas = FigureCanvasAgg(fig) 
    23892393        canvas.print_figure(*args, **kwargs) 
    … …  
    24242428        #get pairs of non significant methods 
    24252429 
    2426         def getLines(sums, hsd): 
     2430        def get_lines(sums, hsd): 
    24272431 
    24282432            #get all pairs 
    … …  
    24332437            #keep only longest 
    24342438             
    2435             def noLonger((i,j), notSig): 
     2439            def no_longer((i,j), notSig): 
    24362440                for i1,j1 in notSig: 
    24372441                    if (i1 <= i and j1 > j) or (i1 < i and j1 >= j): 
    … …  
    24392443                return True 
    24402444 
    2441             longest = [ (i,j) for i,j in notSig if noLonger((i,j),notSig) ] 
     2445            longest = [ (i,j) for i,j in notSig if no_longer((i,j),notSig) ] 
    24422446             
    24432447            return longest 
    24442448 
    2445         lines = getLines(ssums, cd) 
     2449        lines = get_lines(ssums, cd) 
    24462450        linesblank = 0.2 + 0.2 + (len(lines)-1)*0.1 
    24472451 
    … …  
    25262530 
    25272531        #non significance lines     
    2528         def drawLines(lines, side=0.05, height=0.1): 
     2532        def draw_lines(lines, side=0.05, height=0.1): 
    25292533            start = cline + 0.2 
    25302534            for l,r in lines:   
    … …  
    25322536                start += height 
    25332537 
    2534         drawLines(lines) 
     2538        draw_lines(lines) 
    25352539 
    25362540    elif cd: 
    … …  
    25412545        line([(end, cline + bigtick/2), (end, cline - bigtick/2)], linewidth=2.5) 
    25422546  
    2543     printFigure(fig, filename, **kwargs) 
     2547    print_figure(fig, filename, **kwargs) 
    25442548 
    25452549if __name__ == "__main__": 
Note: See TracChangeset for help on using the changeset viewer.