Ignore:
Timestamp:
02/06/12 10:08:15 (2 years ago)
Author:
Matija Polajnar <matija.polajnar@…>
Branch:
default
Message:

Updated documentation and regression tests for reliability estimation. Closes #1058.

File:
1 edited

Legend:

Unmodified
Added
Removed
  • docs/reference/rst/Orange.evaluation.reliability.rst

    r9372 r9680  
    11.. automodule:: Orange.evaluation.reliability 
     2 
     3.. index:: Reliability Estimation 
     4 
     5.. index:: 
     6   single: reliability; Reliability Estimation for Regression 
     7 
     8######################################## 
     9Reliability estimation (``reliability``) 
     10######################################## 
     11 
     12************************************* 
     13Reliability Estimation for Regression 
     14************************************* 
     15 
     16Reliability assessment statistically predicts reliability of single 
     17predictions. Most of implemented algorithms are taken from Comparison of 
     18approaches for estimating reliability of individual regression predictions, 
     19Zoran Bosnić, 2008. 
     20 
     21The following example shows basic usage of reliability estimation methods: 
     22 
     23.. literalinclude:: code/reliability-basic.py 
     24 
     25The important points of this example are: 
     26 * construction of reliability estimators using classes, 
     27   implemented in this module, 
     28 * construction of a reliability learner that bonds a regular learner 
     29   (:class:`~Orange.classification.knn.kNNLearner` in this case) with 
     30   reliability estimators, 
     31 * calling the constructed classifier with 
     32   :obj:`Orange.classification.Classifier.GetBoth` option to obtain class 
     33   probabilities; :obj:`probability` is the object that gets appended the 
     34   :obj:`reliability_estimate` attribute, an instance of 
     35   :class:`Orange.evaluation.reliability.Estimate`, by the reliability learner. 
     36 
     37It is also possible to do reliability estimation on whole data 
     38table, not only on single instance. Next example demonstrates usage of a 
     39cross-validation technique for reliability estimation. Reliability estimations 
     40for first 10 instances get printed: 
     41 
     42.. literalinclude:: code/reliability-run.py 
     43 
     44Reliability Methods 
     45=================== 
     46 
     47Sensitivity Analysis (SAvar and SAbias) 
     48--------------------------------------- 
     49.. autoclass:: SensitivityAnalysis 
     50 
     51Variance of bagged models (BAGV) 
     52-------------------------------- 
     53.. autoclass:: BaggingVariance 
     54 
     55Local cross validation reliability estimate (LCV) 
     56------------------------------------------------- 
     57.. autoclass:: LocalCrossValidation 
     58 
     59Local modeling of prediction error (CNK) 
     60---------------------------------------- 
     61.. autoclass:: CNeighbours 
     62 
     63Bagging variance c-neighbours (BVCK) 
     64------------------------------------ 
     65 
     66.. autoclass:: BaggingVarianceCNeighbours 
     67 
     68Mahalanobis distance 
     69-------------------- 
     70 
     71.. autoclass:: Mahalanobis 
     72 
     73Mahalanobis to center 
     74--------------------- 
     75 
     76.. autoclass:: MahalanobisToCenter 
     77 
     78Reliability estimation wrappers 
     79=============================== 
     80 
     81.. autoclass:: Learner 
     82    :members: 
     83 
     84.. autoclass:: Classifier 
     85    :members: 
     86 
     87Reliability estimation results 
     88============================== 
     89 
     90.. autoclass:: Estimate 
     91    :members: 
     92    :show-inheritance: 
     93 
     94There is a dictionary named :obj:`METHOD_NAME` that maps reliability estimation 
     95method IDs (ints) to method names (strings). 
     96 
     97In this module, there are also two constants for distinguishing signed and 
     98absolute reliability estimation measures:: 
     99 
     100  SIGNED = 0 
     101  ABSOLUTE = 1 
     102 
     103Reliability estimation scoring methods 
     104====================================== 
     105 
     106.. autofunction:: get_pearson_r 
     107 
     108.. autofunction:: get_pearson_r_by_iterations 
     109 
     110.. autofunction:: get_spearman_r 
     111 
     112Example of usage 
     113================ 
     114 
     115.. literalinclude:: code/reliability-long.py 
     116    :lines: 1-16 
     117 
     118This script prints out Pearson's R coefficient between reliability estimates 
     119and actual prediction errors, and a corresponding p-value, for each of the 
     120reliability estimation measures used by default. :: 
     121 
     122  Estimate               r       p 
     123  SAvar absolute        -0.077   0.454 
     124  SAbias signed         -0.165   0.105 
     125  SAbias absolute       -0.099   0.333 
     126  BAGV absolute          0.104   0.309 
     127  CNK signed             0.233   0.021 
     128  CNK absolute           0.057   0.579 
     129  LCV absolute           0.069   0.504 
     130  BVCK_absolute          0.092   0.368 
     131  Mahalanobis absolute   0.091   0.375 
     132 
     133 
     134References 
     135========== 
     136 
     137Bosnić, Z., Kononenko, I. (2007) `Estimation of individual prediction 
     138reliability using local sensitivity analysis. <http://www.springerlink 
     139.com/content/e27p2584387532g8/>`_ *Applied Intelligence* 29(3), pp. 187-203. 
     140 
     141Bosnić, Z., Kononenko, I. (2008) `Comparison of approaches for estimating 
     142reliability of individual regression predictions. <http://www.sciencedirect 
     143.com/science/article/pii/S0169023X08001080>`_ *Data & Knowledge Engineering* 
     14467(3), pp. 504-516. 
     145 
     146Bosnić, Z., Kononenko, I. (2010) `Automatic selection of reliability estimates 
     147for individual regression predictions. <http://journals.cambridge 
     148.org/abstract_S0269888909990154>`_ *The Knowledge Engineering Review* 25(1), 
     149pp. 27-47. 
     150 
Note: See TracChangeset for help on using the changeset viewer.