source: orange-reliability/docs/rst/Orange.evaluation.reliability.rst @ 53:ba8bc7d59e7a

Revision 53:ba8bc7d59e7a, 5.6 KB checked in by markotoplak, 7 months ago (diff)

Updates to documentation.

RevLine 
[4]1.. automodule:: Orange.evaluation.reliability
2
3.. index:: Reliability Estimation
4
5.. index::
6   single: reliability; Reliability Estimation for Regression
7
8##########################################################
9Reliability estimation (``Orange.evaluation.reliability``)
10##########################################################
11
[13]12********************************************************
13Reliability Estimation for Regression and Classification
14********************************************************
[4]15
[41]16Reliability assessment aims to predict reliabilities of individual
[42]17predictions. Most of implemented algorithms for regression described in
18[Bosnic2008]_ and in [Pevec2011]_ for classification.
[41]19
20We can use reliability estimation with any Orange learners. The following example:
21
22 * Constructs reliability estimators (implemented in this module),
[42]23 * The :obj:`Learner` wrapper combines a regular learner, here a :obj:`~Orange.classification.knn.kNNLearner`, with reliability estimators.
[41]24 * Obtains prediction probabilities from the constructed classifier
25   (:obj:`Orange.classification.Classifier.GetBoth` option). The resulting
[42]26   probabilities have an additional attribute, :obj:`reliability_estimate`,
27   that contains a list of :class:`Orange.evaluation.reliability.Estimate`.
[4]28
29.. literalinclude:: code/reliability-basic.py
30    :lines: 7-
31
[41]32We could also evaluate more examples. The next example prints reliability estimates
33for first 10 instances (with cross-validation):
[4]34
35.. literalinclude:: code/reliability-run.py
[39]36    :lines: 7-
[4]37
[42]38Reliability estimation wrappers
39===============================
40
41.. autoclass:: Learner
42   :members: __call__
43
44.. autoclass:: Classifier
45   :members: __call__
46
47
[4]48Reliability Methods
49===================
50
[42]51All measures except :math:`O_{ref}` work with regression. Classification is
[41]52supported by BAGV, LCV, CNK and DENS, :math:`O_{ref}`.
[5]53
[4]54Sensitivity Analysis (SAvar and SAbias)
55---------------------------------------
56.. autoclass:: SensitivityAnalysis
57
58Variance of bagged models (BAGV)
59--------------------------------
60.. autoclass:: BaggingVariance
61
62Local cross validation reliability estimate (LCV)
63-------------------------------------------------
64.. autoclass:: LocalCrossValidation
65
66Local modeling of prediction error (CNK)
67----------------------------------------
68.. autoclass:: CNeighbours
69
70Bagging variance c-neighbours (BVCK)
71------------------------------------
72
73.. autoclass:: BaggingVarianceCNeighbours(bagv=BaggingVariance(), cnk=CNeighbours())
74
75Mahalanobis distance
76--------------------
77
78.. autoclass:: Mahalanobis
79
80Mahalanobis to center
81---------------------
82
83.. autoclass:: MahalanobisToCenter
84
[5]85Density estimation using Parzen window (DENS)
86---------------------------------------------
87
88.. autoclass:: ParzenWindowDensityBased
89
[40]90Internal cross validation (ICV)
91-------------------------------
92
93.. autoclass:: ICV
94
95
96Stacked generalization (Stacking)
[42]97---------------------------------
[40]98
99.. autoclass:: Stacking
100
[13]101Reference Estimate for Classification (:math:`O_{ref}`)
102-------------------------------------------------------
103
104.. autoclass:: ReferenceExpectedError
105
[4]106Reliability estimation results
107==============================
108
[41]109.. data:: SIGNED
110   
111.. data:: ABSOLUTE
112
113    These constants distinguish signed and
114    absolute reliability estimation measures.
115
116.. data:: METHOD_NAME
117
118    A dictionary that that maps reliability estimation
119    method IDs (integerss) to method names (strings).
120
[4]121.. autoclass:: Estimate
122    :members:
123    :show-inheritance:
124
125
126
[39]127Reliability estimation scoring
128==============================
[4]129
130.. autofunction:: get_pearson_r
131
132.. autofunction:: get_pearson_r_by_iterations
133
134.. autofunction:: get_spearman_r
135
[39]136Example
137=======
[4]138
[41]139The following script prints Pearson's correlation coefficient (r) between reliability
140estimates and actual prediction errors, and a corresponding p-value, for
141default reliability estimation measures.
142
[4]143.. literalinclude:: code/reliability-long.py
[39]144    :lines: 7-22
[4]145
[41]146Results::
[39]147 
[4]148  Estimate               r       p
149  SAvar absolute        -0.077   0.454
150  SAbias signed         -0.165   0.105
[39]151  SAbias absolute        0.095   0.352
152  LCV absolute           0.069   0.504
153  BVCK absolute          0.060   0.562
154  BAGV absolute          0.078   0.448
[4]155  CNK signed             0.233   0.021
[39]156  CNK absolute           0.058   0.574
[4]157  Mahalanobis absolute   0.091   0.375
[39]158  Mahalanobis to center  0.096   0.349
[4]159
160References
161==========
162
[42]163.. [Bosnic2007]  Bosnić, Z., Kononenko, I. (2007) `Estimation of individual prediction reliability using local sensitivity analysis. <http://www.springerlink.com/content/e27p2584387532g8/>`_ *Applied Intelligence* 29(3), pp. 187-203.
[4]164
[42]165.. [Bosnic2008] Bosnić, Z., Kononenko, I. (2008) `Comparison of approaches for estimating reliability of individual regression predictions. <http://www.sciencedirect .com/science/article/pii/S0169023X08001080>`_ *Data & Knowledge Engineering* 67(3), pp. 504-516.
[4]166
[42]167.. [Bosnic2010] Bosnić, Z., Kononenko, I. (2010) `Automatic selection of reliability estimates for individual regression predictions. <http://journals.cambridge .org/abstract_S0269888909990154>`_ *The Knowledge Engineering Review* 25(1), pp. 27-47.
[4]168
[42]169.. [Pevec2011] Pevec, D., Štrumbelj, E., Kononenko, I. (2011) `Evaluating Reliability of Single Classifications of Neural Networks. <http://www.springerlink.com /content/48u881761h127r33/export-citation/>`_ *Adaptive and Natural Computing Algorithms*, 2011, pp. 22-30.
[53]170
171.. [Wolpert1992] Wolpert, David H. (1992) `Stacked generalization.` *Neural Networks*, Vol. 5, 1992,  pp. 241-259.
172
173.. [Dzeroski2004] Dzeroski, S. and Zenko, B. (2004) `Is combining classifiers with stacking better than selecting the best one?` *Machine Learning*, Vol. 54, 2004,  pp. 255-273.
Note: See TracBrowser for help on using the repository browser.