Changeset 10343:528f672373f5 in orange

02/23/12 15:18:45 (2 years ago)
Lan Zagar <lan.zagar@…>

Improved multitarget scoring.

2 edited


  • Orange/evaluation/

    r10339 r10343  
    25412541def mt_average_score(res, score, weights=None): 
    25422542    """ 
    2543     Average the scores of individual targets. 
    2545     :param score: Single-target scoring method. 
     2543    Compute individual scores for each target and return the (weighted) average. 
     2545    One method can be used to compute scores for all targets or a list of 
     2546    scoring methods can be passed to use different methods for different 
     2547    targets. In the latter case, care has to be taken if the ranges of scoring 
     2548    methods differ. 
     2549    For example, when the first target is scored from -1 to 1 (1 best) and the 
     2550    second from 0 to 1 (0 best), using `weights=[0.5,-1]` would scale both 
     2551    to a span of 1, and invert the second so that higher scores are better. 
     2553    :param score: Single-target scoring method or a list of such methods 
     2554                  (one for each target). 
    25462555    :param weights: List of real weights, one for each target, 
    25472556                    for a weighted average. 
    25522561    if res.number_of_learners < 1: 
    25532562        return [] 
     2563    n_classes = len(res.results[0].actual_class) 
    25542564    if weights is None: 
    2555         weights = [1. for _ in res.results[0].actual_class] 
     2565        weights = [1.] * n_classes 
     2566    if not hasattr(score, '__len__'): 
     2567        score = [score] * n_classes 
     2568    elif len(score) != n_classes: 
     2569        raise ValueError, "Number of scoring methods and targets do not match." 
    25562570    # save original classes 
    25572571    clsss = [te.classes for te in res.results] 
    25592573    # compute single target scores 
    25602574    single_scores = [] 
    2561     for i in range(len(clsss[0][0])): 
     2575    for i in range(n_classes): 
    25622576        for te, clss, aclss in zip(res.results, clsss, aclsss): 
    25632577            te.classes = [cls[i] for cls in clss] 
    25642578            te.actual_class = aclss[i] 
    2565         single_scores.append(score(res)) 
     2579        single_scores.append(score[i](res)) 
    25662580    # restore original classes 
    25672581    for te, clss, aclss in zip(res.results, clsss, aclsss): 
  • docs/reference/rst/Orange.evaluation.scoring.rst

    r10340 r10343  
    137137multiple target classes. They can be used with standard 
    138138:obj:`~Orange.evaluation.testing` procedures (e.g. 
    139 :obj:`~Orange.evaluation.testing.Evaluation.cross_validation`), but require special 
    140 scoring functions to compute a single score from the obtained 
     139:obj:`~Orange.evaluation.testing.Evaluation.cross_validation`), but require 
     140special scoring functions to compute a single score from the obtained 
     142Since different targets can vary in importance depending on the experiment, 
     143some methods have options to indicate this e.g. through weights or customized 
     144distance functions. These can also be used for normalization in case target 
     145values do not have the same scales. 
    143147.. autofunction:: mt_flattened_score 
    144148.. autofunction:: mt_average_score 
    146 The whole procedure of evaluating multi-target methods and computing the scores 
    147 (RMSE errors) is shown in the following example (:download:` <code/>`): 
     150The whole procedure of evaluating multi-target methods and computing 
     151the scores (RMSE errors) is shown in the following example 
     152(:download:` <code/>`). Because we consider 
     153the first target to be more important and the last not so much we will 
     154indicate this using appropriate weights. 
    149156.. literalinclude:: code/ 
Note: See TracChangeset for help on using the changeset viewer.