Changeset 9680:4689e76f99be in orange


Ignore:
Timestamp:
02/06/12 10:08:15 (2 years ago)
Author:
Matija Polajnar <matija.polajnar@…>
Branch:
default
Message:

Updated documentation and regression tests for reliability estimation. Closes #1058.

Files:
2 deleted
5 edited

Legend:

Unmodified
Added
Removed
  • docs/reference/rst/Orange.evaluation.reliability.rst

    r9372 r9680  
    11.. automodule:: Orange.evaluation.reliability 
     2 
     3.. index:: Reliability Estimation 
     4 
     5.. index:: 
     6   single: reliability; Reliability Estimation for Regression 
     7 
     8######################################## 
     9Reliability estimation (``reliability``) 
     10######################################## 
     11 
     12************************************* 
     13Reliability Estimation for Regression 
     14************************************* 
     15 
     16Reliability assessment statistically predicts reliability of single 
     17predictions. Most of implemented algorithms are taken from Comparison of 
     18approaches for estimating reliability of individual regression predictions, 
     19Zoran Bosnić, 2008. 
     20 
     21The following example shows basic usage of reliability estimation methods: 
     22 
     23.. literalinclude:: code/reliability-basic.py 
     24 
     25The important points of this example are: 
     26 * construction of reliability estimators using classes, 
     27   implemented in this module, 
     28 * construction of a reliability learner that bonds a regular learner 
     29   (:class:`~Orange.classification.knn.kNNLearner` in this case) with 
     30   reliability estimators, 
     31 * calling the constructed classifier with 
     32   :obj:`Orange.classification.Classifier.GetBoth` option to obtain class 
     33   probabilities; :obj:`probability` is the object that gets appended the 
     34   :obj:`reliability_estimate` attribute, an instance of 
     35   :class:`Orange.evaluation.reliability.Estimate`, by the reliability learner. 
     36 
     37It is also possible to do reliability estimation on whole data 
     38table, not only on single instance. Next example demonstrates usage of a 
     39cross-validation technique for reliability estimation. Reliability estimations 
     40for first 10 instances get printed: 
     41 
     42.. literalinclude:: code/reliability-run.py 
     43 
     44Reliability Methods 
     45=================== 
     46 
     47Sensitivity Analysis (SAvar and SAbias) 
     48--------------------------------------- 
     49.. autoclass:: SensitivityAnalysis 
     50 
     51Variance of bagged models (BAGV) 
     52-------------------------------- 
     53.. autoclass:: BaggingVariance 
     54 
     55Local cross validation reliability estimate (LCV) 
     56------------------------------------------------- 
     57.. autoclass:: LocalCrossValidation 
     58 
     59Local modeling of prediction error (CNK) 
     60---------------------------------------- 
     61.. autoclass:: CNeighbours 
     62 
     63Bagging variance c-neighbours (BVCK) 
     64------------------------------------ 
     65 
     66.. autoclass:: BaggingVarianceCNeighbours 
     67 
     68Mahalanobis distance 
     69-------------------- 
     70 
     71.. autoclass:: Mahalanobis 
     72 
     73Mahalanobis to center 
     74--------------------- 
     75 
     76.. autoclass:: MahalanobisToCenter 
     77 
     78Reliability estimation wrappers 
     79=============================== 
     80 
     81.. autoclass:: Learner 
     82    :members: 
     83 
     84.. autoclass:: Classifier 
     85    :members: 
     86 
     87Reliability estimation results 
     88============================== 
     89 
     90.. autoclass:: Estimate 
     91    :members: 
     92    :show-inheritance: 
     93 
     94There is a dictionary named :obj:`METHOD_NAME` that maps reliability estimation 
     95method IDs (ints) to method names (strings). 
     96 
     97In this module, there are also two constants for distinguishing signed and 
     98absolute reliability estimation measures:: 
     99 
     100  SIGNED = 0 
     101  ABSOLUTE = 1 
     102 
     103Reliability estimation scoring methods 
     104====================================== 
     105 
     106.. autofunction:: get_pearson_r 
     107 
     108.. autofunction:: get_pearson_r_by_iterations 
     109 
     110.. autofunction:: get_spearman_r 
     111 
     112Example of usage 
     113================ 
     114 
     115.. literalinclude:: code/reliability-long.py 
     116    :lines: 1-16 
     117 
     118This script prints out Pearson's R coefficient between reliability estimates 
     119and actual prediction errors, and a corresponding p-value, for each of the 
     120reliability estimation measures used by default. :: 
     121 
     122  Estimate               r       p 
     123  SAvar absolute        -0.077   0.454 
     124  SAbias signed         -0.165   0.105 
     125  SAbias absolute       -0.099   0.333 
     126  BAGV absolute          0.104   0.309 
     127  CNK signed             0.233   0.021 
     128  CNK absolute           0.057   0.579 
     129  LCV absolute           0.069   0.504 
     130  BVCK_absolute          0.092   0.368 
     131  Mahalanobis absolute   0.091   0.375 
     132 
     133 
     134References 
     135========== 
     136 
     137Bosnić, Z., Kononenko, I. (2007) `Estimation of individual prediction 
     138reliability using local sensitivity analysis. <http://www.springerlink 
     139.com/content/e27p2584387532g8/>`_ *Applied Intelligence* 29(3), pp. 187-203. 
     140 
     141Bosnić, Z., Kononenko, I. (2008) `Comparison of approaches for estimating 
     142reliability of individual regression predictions. <http://www.sciencedirect 
     143.com/science/article/pii/S0169023X08001080>`_ *Data & Knowledge Engineering* 
     14467(3), pp. 504-516. 
     145 
     146Bosnić, Z., Kononenko, I. (2010) `Automatic selection of reliability estimates 
     147for individual regression predictions. <http://journals.cambridge 
     148.org/abstract_S0269888909990154>`_ *The Knowledge Engineering Review* 25(1), 
     149pp. 27-47. 
     150 
  • docs/reference/rst/code/reliability-long.py

    r9372 r9680  
    1313print "Estimate               r       p" 
    1414for estimate in reliability_res: 
    15     print "%-20s %7.3f %7.3f" % (Orange.evaluation.reliability.METHOD_NAME[estimate[3]], \ 
     15    print "%-20s %7.3f %7.3f" % (Orange.evaluation.reliability.METHOD_NAME[estimate[3]], 
    1616                                 estimate[0], estimate[1]) 
    1717 
     
    2525print "Estimate               r       p" 
    2626for estimate in reliability_res: 
    27     print "%-20s %7.3f %7.3f" % (Orange.evaluation.reliability.METHOD_NAME[estimate[3]], \ 
     27    print "%-20s %7.3f %7.3f" % (Orange.evaluation.reliability.METHOD_NAME[estimate[3]], 
    2828                                 estimate[0], estimate[1]) 
    2929 
  • docs/reference/rst/code/reliability-run.py

    r9372 r9680  
    77results = Orange.evaluation.testing.cross_validation([reliability], table) 
    88 
    9 for estimate in results.results[0].probabilities[0].reliability_estimate: 
    10     print estimate.method_name, estimate.estimate 
     9for i, instance in enumerate(results.results[:10]): 
     10    print "Instance", i 
     11    for estimate in instance.probabilities[0].reliability_estimate: 
     12        print "  ", estimate.method_name, estimate.estimate 
  • orange/Orange/evaluation/reliability.py

    r9663 r9680  
    1 """ 
    2 ######################################## 
    3 Reliability estimation (``reliability``) 
    4 ######################################## 
    5  
    6 .. index:: Reliability Estimation 
    7  
    8 .. index:: 
    9    single: reliability; Reliability Estimation for Regression 
    10  
    11 ************************************* 
    12 Reliability Estimation for Regression 
    13 ************************************* 
    14  
    15 This module includes different implementations of algorithm used for 
    16 predicting reliability of single predictions. Most of the algorithm are taken 
    17 from Comparison of approaches for estimating reliability of individual 
    18 regression predictions, Zoran Bosnic 2008. 
    19  
    20 Next example shows basic reliability estimation usage  
    21 (:download:`reliability-basic.py <code/reliability-basic.py>`, uses :download:`housing.tab <code/housing.tab>`): 
    22  
    23 .. literalinclude:: code/reliability_basic.py 
    24  
    25 First we load our desired data table and choose on learner we want to use  
    26 reliability estimation on. We also want to calculate only the Mahalanobis and  
    27 local cross validation estimates with desired parameters. We learn our  
    28 estimator on data, and estimate the reliability for first instance of data table. 
    29 We output the estimates used and the numbers. 
    30  
    31 We can also do reliability estimation on whole data table not only on single 
    32 instance. Example shows us doing cross validation on the desired data table, 
    33 using default reliability estimates, and at the ending output reliability 
    34 estimates for the first instance of data table. 
    35 (:download:`reliability-run.py <code/reliability-run.py>`, uses :download:`housing.tab <code/housing.tab>`): 
    36  
    37 .. literalinclude:: code/reliability-run.py 
    38  
    39 Reliability estimation methods are computationally quite hard so it may take 
    40 a bit of time for this script to produce a result. In the above example we 
    41 first create a learner that we're interested in, in this example 
    42 k-nearest-neighbors, and use it inside reliability learner and do cross 
    43 validation to get the results. Now we output for the first example in the 
    44 data table all the reliability estimates and their names. 
    45  
    46 Reliability Methods 
    47 =================== 
    48  
    49 Sensitivity Analysis (SAvar and SAbias) 
    50 --------------------------------------- 
    51 .. autoclass:: SensitivityAnalysis 
    52  
    53 Variance of bagged models (BAGV) 
    54 -------------------------------- 
    55 .. autoclass:: BaggingVariance 
    56  
    57 Local cross validation reliability estimate (LCV) 
    58 ------------------------------------------------- 
    59 .. autoclass:: LocalCrossValidation 
    60  
    61 Local modeling of prediction error (CNK) 
    62 ---------------------------------------- 
    63 .. autoclass:: CNeighbours 
    64  
    65 Bagging variance c-neighbours (BVCK) 
    66 ------------------------------------ 
    67  
    68 .. autoclass:: BaggingVarianceCNeighbours 
    69  
    70 Mahalanobis distance 
    71 -------------------- 
    72  
    73 .. autoclass:: Mahalanobis 
    74  
    75 Mahalanobis to center 
    76 --------------------- 
    77  
    78 .. autoclass:: MahalanobisToCenter 
    79  
    80 Reliability estimate learner 
    81 ============================ 
    82  
    83 .. autoclass:: Learner 
    84     :members: 
    85  
    86 Reliability estimation scoring methods 
    87 ====================================== 
    88  
    89 .. autofunction:: get_pearson_r 
    90  
    91 .. autofunction:: get_pearson_r_by_iterations 
    92  
    93 .. autofunction:: get_spearman_r 
    94  
    95 Referencing 
    96 =========== 
    97  
    98 There is a dictionary named :data:`METHOD_NAME` which has stored names of 
    99 all the reliability estimates:: 
    100  
    101   METHOD_NAME = {0: "SAvar absolute", 1: "SAbias signed", 2: "SAbias absolute", 
    102                  3: "BAGV absolute", 4: "CNK signed", 5: "CNK absolute", 
    103                  6: "LCV absolute", 7: "BVCK_absolute", 8: "Mahalanobis absolute", 
    104                  10: "ICV"} 
    105  
    106 and also two constants for saying whether the estimate is signed or it's an 
    107 absolute value:: 
    108  
    109   SIGNED = 0 
    110   ABSOLUTE = 1 
    111  
    112 Example of usage 
    113 ================ 
    114  
    115 Here we will walk through a bit longer example of how to use the reliability 
    116 estimate module (:download:`reliability-long.py <code/reliability-long.py>`, uses :download:`prostate.tab <code/prostate.tab>`): 
    117  
    118 .. literalinclude:: code/reliability-long.py 
    119     :lines: 1-16 
    120  
    121 After loading the Orange library we open out dataset. We chose to work with 
    122 the kNNLearner, that also works on regression problems. Create out reliability 
    123 estimate learner and test it with cross validation.  
    124 Estimates are then compared using Pearson's coefficient to the prediction error. 
    125 The p-values are also computed:: 
    126  
    127   Estimate               r       p 
    128   SAvar absolute        -0.077   0.454 
    129   SAbias signed         -0.165   0.105 
    130   SAbias absolute       -0.099   0.333 
    131   BAGV absolute          0.104   0.309 
    132   CNK signed             0.233   0.021 
    133   CNK absolute           0.057   0.579 
    134   LCV absolute           0.069   0.504 
    135   BVCK_absolute          0.092   0.368 
    136   Mahalanobis absolute   0.091   0.375 
    137  
    138 .. literalinclude:: code/reliability-long.py 
    139     :lines: 18-28 
    140  
    141 Outputs:: 
    142  
    143   Estimate               r       p 
    144   BAGV absolute          0.126   0.220 
    145   CNK signed             0.233   0.021 
    146   CNK absolute           0.057   0.579 
    147   LCV absolute           0.069   0.504 
    148   BVCK_absolute          0.105   0.305 
    149   Mahalanobis absolute   0.091   0.375 
    150  
    151  
    152 As you can see in the above code you can also chose with reliability estimation 
    153 method do you want to use. You might want to do this to reduce computation time  
    154 or because you think they don't perform good enough. 
    155  
    156  
    157 References 
    158 ========== 
    159  
    160 Bosnic Z, Kononenko I (2007) `Estimation of individual prediction reliability using local 
    161 sensitivity analysis. <http://www.springerlink.com/content/e27p2584387532g8/>`_ 
    162 *Applied Intelligence* 29(3), 187-203. 
    163  
    164 Bosnic Z, Kononenko I (2008) `Comparison of approaches for estimating reliability of  
    165 individual regression predictions. 
    166 <http://www.sciencedirect.com/science/article/pii/S0169023X08001080>`_ 
    167 *Data & Knowledge Engineering* 67(3), 504-516. 
    168  
    169 Bosnic Z, Kononenko I (2010) `Automatic selection of reliability estimates for individual  
    170 regression predictions. 
    171 <http://journals.cambridge.org/abstract_S0269888909990154>`_ 
    172 *The Knowledge Engineering Review* 25(1), 27-47. 
    173  
    174 """ 
    1751import Orange 
    1762 
     
    23662def get_pearson_r(res): 
    23763    """ 
    238     Returns Pearsons coefficient between the prediction error and each of the 
    239     used reliability estimates. Function also return the p-value of each of 
     64    :param res: results of evaluation, done using learners, 
     65        wrapped into :class:`Orange.evaluation.reliability.Classifier`. 
     66    :type res: :class:`Orange.evaluation.testing.ExperimentResults` 
     67 
     68    Return Pearson's coefficient between the prediction error and each of the 
     69    used reliability estimates. Also, return the p-value of each of 
    24070    the coefficients. 
    24171    """ 
     
    25686def get_spearman_r(res): 
    25787    """ 
    258     Returns Spearmans coefficient between the prediction error and each of the 
    259     used reliability estimates. Function also return the p-value of each of 
     88    :param res: results of evaluation, done using learners, 
     89        wrapped into :class:`Orange.evaluation.reliability.Classifier`. 
     90    :type res: :class:`Orange.evaluation.testing.ExperimentResults` 
     91 
     92    Return Spearman's coefficient between the prediction error and each of the 
     93    used reliability estimates. Also, return the p-value of each of 
    26094    the coefficients. 
    26195    """ 
     
    276110def get_pearson_r_by_iterations(res): 
    277111    """ 
    278     Returns average Pearsons coefficient over all folds between prediction error 
     112    :param res: results of evaluation, done using learners, 
     113        wrapped into :class:`Orange.evaluation.reliability.Classifier`. 
     114    :type res: :class:`Orange.evaluation.testing.ExperimentResults` 
     115 
     116    Return average Pearson's coefficient over all folds between prediction error 
    279117    and each of the used estimates. 
    280118    """ 
     
    317155 
    318156class Estimate: 
     157    """ 
     158    Reliability estimate. Contains attributes that describe the results of 
     159    reliability estimation. 
     160 
     161    .. attribute:: estimate 
     162 
     163        A numerical reliability estimate. 
     164 
     165    .. attribute:: signed_or_absolute 
     166 
     167        Determines whether the method used gives a signed or absolute result. 
     168        Has a value of either :obj:`SIGNED` or :obj:`ABSOLUTE`. 
     169 
     170    .. attribute:: method 
     171 
     172        An integer ID of reliability estimation method used. 
     173 
     174    .. attribute:: method_name 
     175 
     176        Name (string) of reliability estimation method used. 
     177 
     178    .. attribute:: icv_method 
     179 
     180        An integer ID of reliability estimation method that performed best, 
     181        as determined by ICV, and of which estimate is stored in the 
     182        :obj:`estimate` field. (:obj:`None` when ICV was not used.) 
     183 
     184    .. attribute:: icv_method_name 
     185 
     186        Name (string) of reliability estimation method that performed best, 
     187        as determined by ICV. (:obj:`None` when ICV was not used.) 
     188 
     189    """ 
    319190    def __init__(self, estimate, signed_or_absolute, method, icv_method = -1): 
    320191        self.estimate = estimate 
     
    374245    """ 
    375246     
    376     :param e: List of possible e values for SAvar and SAbias reliability estimates, the default value is [0.01, 0.1, 0.5, 1.0, 2.0]. 
     247    :param e: List of possible :math:`\epsilon` values for SAvar and SAbias 
     248        reliability estimates. 
    377249    :type e: list of floats 
    378250     
    379251    :rtype: :class:`Orange.evaluation.reliability.SensitivityAnalysisClassifier` 
    380252     
    381     To estimate the reliabilty for given example we extend the learning set  
    382     with given example and labeling it with :math:`K + \epsilon (l_{max} - l_{min})`, 
    383     where K denotes the initial prediction, :math:`\epsilon` is sensitivity parameter and 
    384     :math:`l_{min}` and :math:`l_{max}` denote lower and the upper bound of 
    385     the learning examples. After computing different sensitivity predictions 
    386     using different values of e, the prediction are combined into SAvar and SAbias. 
    387     SAbias can be used as signed estimate or as absolute value of SAbias.  
     253    To estimate the reliabilty for given instance, the learning set is extended 
     254    with this instance, labeled with :math:`K + \epsilon (l_{max} - l_{min})`, 
     255    where :math:`K` denotes the initial prediction, 
     256    :math:`\epsilon` is sensitivity parameter and :math:`l_{min}` and 
     257    :math:`l_{max}` denote lower and the upper bound of the learning examples 
     258    . After computing different sensitivity predictions using different 
     259    values of :math:`\epsilon`, the prediction are combined into SAvar and 
     260    SAbias. SAbias can be used as signed estimate or as absolute value of 
     261    SAbias. 
    388262 
    389263    :math:`SAvar = \\frac{\sum_{\epsilon \in E}(K_{\epsilon} - K_{-\epsilon})}{|E|}` 
     
    454328    """ 
    455329     
    456     :param m: Number of bagged models to be used with BAGV estimate 
     330    :param m: Number of bagging models to be used with BAGV estimate 
    457331    :type m: int 
    458332     
    459333    :rtype: :class:`Orange.evaluation.reliability.BaggingVarianceClassifier` 
    460334     
    461     We construct m different bagging models of the original chosen learner and use 
    462     those predictions (:math:`K_i, i = 1, ..., m`) of given example to calculate the variance, which we use as 
    463     reliability estimator. 
     335    :math:`m` different bagging models are constructed and used to estimate 
     336    the value of dependent variable for a given instance. The variance of 
     337    those predictions is used as a prediction reliability estimate. 
    464338 
    465339    :math:`BAGV = \\frac{1}{m} \sum_{i=1}^{m} (K_i - K)^2` 
    466340 
    467     where 
    468  
    469     :math:`K = \\frac{\sum_{i=1}^{m} K_i}{m}` 
     341    where :math:`K = \\frac{\sum_{i=1}^{m} K_i}{m}` and :math:`K_i` are 
     342    predictions of individual constructed models. 
    470343     
    471344    """ 
     
    507380    :rtype: :class:`Orange.evaluation.reliability.LocalCrossValidationClassifier` 
    508381     
    509     We find k nearest neighbours to the given example and put them in 
    510     seperate dataset. On this dataset we do leave one out 
    511     validation using given model. Reliability estimate is then distance 
    512     weighted absolute prediction error. 
    513      
    514     1. define the set of k nearest neighours :math:`N = { (x_1, x_1),..., (x_k, c_k)}` 
    515     2. FOR EACH :math:`(x_i, c_i) \in N` 
    516      
    517       2.1. generare model M on :math:`N \\backslash (x_i, c_i)` 
    518      
    519       2.2. for :math:`(x_i, c_i)` compute LOO prediction :math:`K_i` 
    520      
    521       2.3. for :math:`(x_i, c_i)` compute LOO error :math:`E_i = | C_i - K_i |` 
    522      
     382    :math:`k` nearest neighbours to the given instance are found and put in 
     383    a separate data set. On this data set, a leave-one-out validation is 
     384    performed. Reliability estimate is then the distance weighted absolute 
     385    prediction error. 
     386 
     387    If a special value 0 is passed as :math:`k` (as is by default), 
     388    it is set as 1/20 of data set size (or 5, whichever is greater). 
     389     
     390    1. Determine the set of k nearest neighours :math:`N = { (x_1, c_1),..., 
     391       (x_k, c_k)}`. 
     392    2. On this set, compute leave-one-out predictions :math:`K_i` and 
     393       prediction errors :math:`E_i = | C_i - K_i |`. 
    523394    3. :math:`LCV(x) = \\frac{ \sum_{(x_i, c_i) \in N} d(x_i, x) * E_i }{ \sum_{(x_i, c_i) \in N} d(x_i, x) }` 
    524395     
     
    527398        self.k = k 
    528399     
    529     def __call__(self, examples, learner): 
     400    def __call__(self, instances, learner): 
    530401        nearest_neighbours_constructor = Orange.classification.knn.FindNearestConstructor() 
    531402        nearest_neighbours_constructor.distanceConstructor = Orange.distance.EuclideanConstructor() 
    532403         
    533404        distance_id = Orange.data.new_meta_id() 
    534         nearest_neighbours = nearest_neighbours_constructor(examples, 0, distance_id) 
     405        nearest_neighbours = nearest_neighbours_constructor(instances, 0, distance_id) 
    535406         
    536407        if self.k == 0: 
    537             self.k = max(5, len(examples)/20) 
     408            self.k = max(5, len(instances)/20) 
    538409         
    539410        return LocalCrossValidationClassifier(distance_id, nearest_neighbours, self.k, learner) 
     
    581452    :rtype: :class:`Orange.evaluation.reliability.CNeighboursClassifier` 
    582453     
    583     Estimate CNK is defined for unlabeled example as difference between 
    584     average label of the nearest neighbours and the examples prediction. CNK can 
    585     be used as a signed estimate or only as absolute value.  
     454    CNK is defined for an unlabeled instance as a difference between average 
     455    label of its nearest neighbours and its prediction. CNK can be used as a 
     456    signed or absolute estimate. 
    586457     
    587458    :math:`CNK = \\frac{\sum_{i=1}^{k}C_i}{k} - K` 
    588459     
    589     Where k denotes number of neighbors, C :sub:`i` denotes neighbours' labels and 
    590     K denotes the example's prediction. 
     460    where :math:`k` denotes number of neighbors, C :sub:`i` denotes neighbours' 
     461    labels and :math:`K` denotes the instance's prediction. 
    591462     
    592463    """ 
     
    627498    """ 
    628499     
    629     :param k: Number of nearest neighbours used in Mahalanobis estimate 
     500    :param k: Number of nearest neighbours used in Mahalanobis estimate. 
    630501    :type k: int 
    631502     
    632503    :rtype: :class:`Orange.evaluation.reliability.MahalanobisClassifier` 
    633504     
    634     Mahalanobis distance estimate is defined as `mahalanobis distance <http://en.wikipedia.org/wiki/Mahalanobis_distance>`_ to the 
    635     k nearest neighbours of chosen example. 
     505    Mahalanobis distance reliability estimate is defined as 
     506    `mahalanobis distance <http://en.wikipedia.org/wiki/Mahalanobis_distance>`_ 
     507    to the evaluated instance's :math:`k` nearest neighbours. 
    636508 
    637509     
     
    665537    :rtype: :class:`Orange.evaluation.reliability.MahalanobisToCenterClassifier` 
    666538     
    667     Mahalanobis distance to center estimate is defined as `mahalanobis distance <http://en.wikipedia.org/wiki/Mahalanobis_distance>`_ to the 
    668     centroid of the data. 
     539    Mahalanobis distance to center reliability estimate is defined as a 
     540    `mahalanobis distance <http://en.wikipedia.org/wiki/Mahalanobis_distance>`_ 
     541    between the predicted instance and the centroid of the data. 
    669542 
    670543     
     
    718591    :rtype: :class:`Orange.evaluation.reliability.BaggingVarianceCNeighboursClassifier` 
    719592     
    720     BVCK is a combination of Bagging variance and local modeling of prediction 
    721     error, for this estimate we take the average of both. 
     593    BVCK is a combination (average) of Bagging variance and local modeling of 
     594    prediction error. 
    722595     
    723596    """ 
     
    780653    Reliability estimation wrapper around a learner we want to test. 
    781654    Different reliability estimation algorithms can be used on the 
    782     chosen learner. This learner works as any other and can be used as one. 
    783     The only difference is when the classifier is called with a given 
    784     example instead of only return the value and probabilities, it also 
    785     attaches a list of reliability estimates to  
    786     :data:`probabilities.reliability_estimate`. 
    787     Each reliability estimate consists of a tuple  
    788     (estimate, signed_or_absolute, method). 
    789      
    790     :param box_learner: Learner we want to wrap into reliability estimation 
     655    chosen learner. This learner works as any other and can be used as one, 
     656    but it returns the classifier, wrapped into an instance of 
     657    :class:`Orange.evaluation.reliability.Classifier`. 
     658     
     659    :param box_learner: Learner we want to wrap into a reliability estimation 
     660        classifier. 
    791661    :type box_learner: learner 
    792662     
     
    834704     
    835705    def internal_cross_validation(self, examples, folds=10): 
    836         """ Performs the ususal internal cross validation for getting the best 
    837         reliability estimate. It uses the reliability estimators defined in  
    838         estimators attribute. Returns the id of the method that scored the  
    839         best. """ 
     706        """ Perform the internal cross validation for getting the best 
     707        reliability estimate. It uses the reliability estimators defined in 
     708        estimators attribute. 
     709 
     710        Returns the id of the method that scored the best. 
     711 
     712        :param examples: Data instances to use for ICV. 
     713        :type examples: :class:`Orange.data.Table` 
     714        :param folds: number of folds for ICV. 
     715        :type folds: int 
     716        :rtype: int 
     717 
     718        """ 
    840719        res = Orange.evaluation.testing.cross_validation([self], examples, folds=folds) 
    841720        results = get_pearson_r(res) 
     
    844723     
    845724    def internal_cross_validation_testing(self, examples, folds=10): 
    846         """ Performs internal cross validation (as in Automatic selection of 
     725        """ Perform internal cross validation (as in Automatic selection of 
    847726        reliability estimates for individual regression predictions, 
    848         Zoran Bosnic 2010) and return id of the method 
    849         that scored best on this data. """ 
     727        Zoran Bosnic, 2010) and return id of the method 
     728        that scored best on this data. 
     729 
     730        :param examples: Data instances to use for ICV. 
     731        :type examples: :class:`Orange.data.Table` 
     732        :param folds: number of folds for ICV. 
     733        :type folds: int 
     734        :rtype: int 
     735 
     736        """ 
    850737        cv_indices = Orange.core.MakeRandomIndicesCV(examples, folds) 
    851738         
     
    869756 
    870757class Classifier: 
     758    """ 
     759    A reliability estimation wrapper for classifiers. 
     760 
     761    What distinguishes this classifier is that the returned probabilities (if 
     762    :obj:`Orange.classification.Classifier.GetProbabilities` or 
     763    :obj:`Orange.classification.Classifier.GetBoth` is passed) contain an 
     764    additional attribute :obj:`reliability_estimate`, which is an instance of 
     765     :class:`~Orange.evaluation.reliability.Estimate`. 
     766 
     767    """ 
     768 
    871769    def __init__(self, examples, box_learner, estimators, blending, blending_domain, rf_classifier, **kwds): 
    872770        self.__dict__.update(kwds) 
     
    886784    def __call__(self, example, result_type=Orange.core.GetValue): 
    887785        """ 
    888         Classify and estimate a new instance. When you chose  
    889         Orange.core.GetBoth or Orange.core.getProbabilities, you can access  
    890         the reliability estimates inside probabilities.reliability_estimate. 
     786        Classify and estimate reliability of estimation for a new instance. 
     787        When :obj:`result_type` is set to 
     788        :obj:`Orange.classification.Classifier.GetBoth` or 
     789        :obj:`Orange.classification.Classifier.GetProbabilities`, 
     790        an additional attribute :obj:`reliability_estimate`, 
     791        which is an instance of 
     792        :class:`~Orange.evaluation.reliability.Estimate`, 
     793        is added to the distribution object. 
    891794         
    892795        :param instance: instance to be classified. 
  • testing/regressionTests/results/orange/orange25/reliability-run.py.txt

    r9659 r9680  
    1 SAvar absolute 10.701260376 
    2 SAbias signed -0.0246158599854 
    3 SAbias absolute 0.0246158599854 
    4 LCV absolute 5.16450013273 
    5 BVCK_absolute 0.0697603345819 
    6 BAGV absolute 0.0221634151355 
    7 CNK signed -0.117357254028 
    8 CNK absolute 0.117357254028 
    9 Mahalanobis absolute 5.70795369148 
    10 Mahalanobis to center 2.74633686492 
     1Instance 0 
     2   SAvar absolute 10.701260376 
     3   SAbias signed -0.0246158599854 
     4   SAbias absolute 0.0246158599854 
     5   LCV absolute 5.16450013273 
     6   BVCK_absolute 0.0697603345819 
     7   BAGV absolute 0.0221634151355 
     8   CNK signed -0.117357254028 
     9   CNK absolute 0.117357254028 
     10   Mahalanobis absolute 5.70795369148 
     11   Mahalanobis to center 2.74633686492 
     12Instance 1 
     13   SAvar absolute 10.7012599945 
     14   SAbias signed -0.0270875930786 
     15   SAbias absolute 0.0270875930786 
     16   LCV absolute 2.99310729294 
     17   BVCK_absolute 0.531783116131 
     18   BAGV absolute 0.00305025630936 
     19   CNK signed 1.06051597595 
     20   CNK absolute 1.06051597595 
     21   Mahalanobis absolute 4.45522534847 
     22   Mahalanobis to center 2.11993694001 
     23Instance 2 
     24   SAvar absolute 10.7012619019 
     25   SAbias signed -0.0769199371338 
     26   SAbias absolute 0.0769199371338 
     27   LCV absolute 4.46030248588 
     28   BVCK_absolute 0.805066890698 
     29   BAGV absolute 0.0025318526845 
     30   CNK signed -1.60760192871 
     31   CNK absolute 1.60760192871 
     32   Mahalanobis absolute 4.45998907089 
     33   Mahalanobis to center 2.20910797014 
     34Instance 3 
     35   SAvar absolute 10.7012609482 
     36   SAbias signed 0.00884370803833 
     37   SAbias absolute 0.00884370803833 
     38   LCV absolute 3.63051784733 
     39   BVCK_absolute 0.0919738920925 
     40   BAGV absolute 3.02787884721e-08 
     41   CNK signed 0.183947753906 
     42   CNK absolute 0.183947753906 
     43   Mahalanobis absolute 4.43703401089 
     44   Mahalanobis to center 2.66496913406 
     45Instance 4 
     46   SAvar absolute 10.701257515 
     47   SAbias signed 0.0423228263855 
     48   SAbias absolute 0.0423228263855 
     49   LCV absolute 3.32640218766 
     50   BVCK_absolute 0.561947292901 
     51   BAGV absolute 4.66539472109e-07 
     52   CNK signed 1.12389411926 
     53   CNK absolute 1.12389411926 
     54   Mahalanobis absolute 4.91473317146 
     55   Mahalanobis to center 2.69116657497 
     56Instance 5 
     57   SAvar absolute 10.7012573242 
     58   SAbias signed 0.273178482056 
     59   SAbias absolute 0.273178482056 
     60   LCV absolute 3.18996776229 
     61   BVCK_absolute 0.0292111702193 
     62   BAGV absolute 0.000202812240257 
     63   CNK signed 0.0582195281982 
     64   CNK absolute 0.0582195281982 
     65   Mahalanobis absolute 4.46615076065 
     66   Mahalanobis to center 2.58802566358 
     67Instance 6 
     68   SAvar absolute 10.7012595177 
     69   SAbias signed -0.101607465744 
     70   SAbias absolute 0.101607465744 
     71   LCV absolute 2.95812406923 
     72   BVCK_absolute 0.364750828033 
     73   BAGV absolute 0.00759766436671 
     74   CNK signed -0.721903991699 
     75   CNK absolute 0.721903991699 
     76   Mahalanobis absolute 4.64355945587 
     77   Mahalanobis to center 2.50879836044 
     78Instance 7 
     79   SAvar absolute 10.701257515 
     80   SAbias signed -0.133856678009 
     81   SAbias absolute 0.133856678009 
     82   LCV absolute 2.64825094326 
     83   BVCK_absolute 0.244493027281 
     84   BAGV absolute 0.0150492106905 
     85   CNK signed -0.473936843872 
     86   CNK absolute 0.473936843872 
     87   Mahalanobis absolute 3.15618926287 
     88   Mahalanobis to center 3.76633615153 
     89Instance 8 
     90   SAvar absolute 10.687993145 
     91   SAbias signed -0.0223073482513 
     92   SAbias absolute 0.0223073482513 
     93   LCV absolute 3.23505105569 
     94   BVCK_absolute 0.163932395748 
     95   BAGV absolute 2.24266747828e-06 
     96   CNK signed -0.327862548828 
     97   CNK absolute 0.327862548828 
     98   Mahalanobis absolute 6.07192790508 
     99   Mahalanobis to center 5.21631597918 
     100Instance 9 
     101   SAvar absolute 10.7012593269 
     102   SAbias signed 0.0480660915375 
     103   SAbias absolute 0.0480660915375 
     104   LCV absolute 3.09854127044 
     105   BVCK_absolute 0.283781303455 
     106   BAGV absolute 0.0357186585455 
     107   CNK signed -0.531843948364 
     108   CNK absolute 0.531843948364 
     109   Mahalanobis absolute 4.02153724432 
     110   Mahalanobis to center 3.93183348061 
Note: See TracChangeset for help on using the changeset viewer.