# source:orange-reliability/docs/rst/Orange.evaluation.reliability.rst@41:d2bc01f57cf8

Revision 41:d2bc01f57cf8, 5.6 KB checked in by markotoplak, 5 months ago (diff)

Line
1.. automodule:: Orange.evaluation.reliability
2
3.. index:: Reliability Estimation
4
5.. index::
6   single: reliability; Reliability Estimation for Regression
7
8##########################################################
9Reliability estimation (``Orange.evaluation.reliability``)
10##########################################################
11
12********************************************************
13Reliability Estimation for Regression and Classification
14********************************************************
15
16Reliability assessment aims to predict reliabilities of individual
17predictions.
18
19Most of implemented algorithms for regression described in
20"Comparison of approaches for estimating reliability of individual
21regression predictions, Zoran Bosnić, 2008" for regression and in
22in "Evaluating Reliability of Single
23Classifications of Neural Networks, Darko Pevec, 2011" for classification.
24
25We can use reliability estimation with any Orange learners. The following example:
26
27 * Constructs reliability estimators (implemented in this module),
28 * Combines a regular learner.
29   (:class:`~Orange.classification.knn.kNNLearner` in this case) with
30   reliability estimators.
31 * Obtains prediction probabilities from the constructed classifier
32   (:obj:`Orange.classification.Classifier.GetBoth` option). The resulting
33   probabilities have and additional attribute, :obj:`reliability_estimate`
34   attribute, :class:`Orange.evaluation.reliability.Estimate`.
35
36.. literalinclude:: code/reliability-basic.py
37    :lines: 7-
38
39We could also evaluate more examples. The next example prints reliability estimates
40for first 10 instances (with cross-validation):
41
42.. literalinclude:: code/reliability-run.py
43    :lines: 7-
44
45Reliability Methods
46===================
47
48For regression, you can use all the described measures except :math:`O_{ref}`. Classification is
49supported by BAGV, LCV, CNK and DENS, :math:`O_{ref}`.
50
51Sensitivity Analysis (SAvar and SAbias)
52---------------------------------------
53.. autoclass:: SensitivityAnalysis
54
55Variance of bagged models (BAGV)
56--------------------------------
57.. autoclass:: BaggingVariance
58
59Local cross validation reliability estimate (LCV)
60-------------------------------------------------
61.. autoclass:: LocalCrossValidation
62
63Local modeling of prediction error (CNK)
64----------------------------------------
65.. autoclass:: CNeighbours
66
67Bagging variance c-neighbours (BVCK)
68------------------------------------
69
70.. autoclass:: BaggingVarianceCNeighbours(bagv=BaggingVariance(), cnk=CNeighbours())
71
72Mahalanobis distance
73--------------------
74
75.. autoclass:: Mahalanobis
76
77Mahalanobis to center
78---------------------
79
80.. autoclass:: MahalanobisToCenter
81
82Density estimation using Parzen window (DENS)
83---------------------------------------------
84
85.. autoclass:: ParzenWindowDensityBased
86
87Internal cross validation (ICV)
88-------------------------------
89
90.. autoclass:: ICV
91
92
93Stacked generalization (Stacking)
94-------------------------------
95
96.. autoclass:: Stacking
97
98Reference Estimate for Classification (:math:`O_{ref}`)
99-------------------------------------------------------
100
101.. autoclass:: ReferenceExpectedError
102
103Reliability estimation wrappers
104===============================
105
106.. autoclass:: Learner(box_learner, name="Reliability estimation", estimators=[SensitivityAnalysis(), LocalCrossValidation(), BaggingVarianceCNeighbours(), Mahalanobis(), MahalanobisToCenter()], **kwds)
107    :members:
108
109.. autoclass:: Classifier
110    :members:
111
112Reliability estimation results
113==============================
114
115.. data:: SIGNED
116
117.. data:: ABSOLUTE
118
119    These constants distinguish signed and
120    absolute reliability estimation measures.
121
122.. data:: METHOD_NAME
123
124    A dictionary that that maps reliability estimation
125    method IDs (integerss) to method names (strings).
126
127.. autoclass:: Estimate
128    :members:
129    :show-inheritance:
130
131
132
133Reliability estimation scoring
134==============================
135
136.. autofunction:: get_pearson_r
137
138.. autofunction:: get_pearson_r_by_iterations
139
140.. autofunction:: get_spearman_r
141
142Example
143=======
144
145The following script prints Pearson's correlation coefficient (r) between reliability
146estimates and actual prediction errors, and a corresponding p-value, for
147default reliability estimation measures.
148
149.. literalinclude:: code/reliability-long.py
150    :lines: 7-22
151
152Results::
153
154  Estimate               r       p
155  SAvar absolute        -0.077   0.454
156  SAbias signed         -0.165   0.105
157  SAbias absolute        0.095   0.352
158  LCV absolute           0.069   0.504
159  BVCK absolute          0.060   0.562
160  BAGV absolute          0.078   0.448
161  CNK signed             0.233   0.021
162  CNK absolute           0.058   0.574
163  Mahalanobis absolute   0.091   0.375
164  Mahalanobis to center  0.096   0.349
165
166References
167==========
168
169Bosnić, Z., Kononenko, I. (2007) `Estimation of individual prediction
170reliability using local sensitivity analysis. <http://www.springerlink
171.com/content/e27p2584387532g8/>`_ *Applied Intelligence* 29(3), pp. 187-203.
172
173Bosnić, Z., Kononenko, I. (2008) `Comparison of approaches for estimating
174reliability of individual regression predictions. <http://www.sciencedirect
175.com/science/article/pii/S0169023X08001080>`_ *Data & Knowledge Engineering*
17667(3), pp. 504-516.
177
178Bosnić, Z., Kononenko, I. (2010) `Automatic selection of reliability estimates
179for individual regression predictions. <http://journals.cambridge
180.org/abstract_S0269888909990154>`_ *The Knowledge Engineering Review* 25(1),
181pp. 27-47.
182
183Pevec, D., Štrumbelj, E., Kononenko, I. (2011) `Evaluating Reliability of
184Single Classifications of Neural Networks. <http://www.springerlink.com