source: orange-reliability/docs/rst/Orange.evaluation.reliability.rst @ 5:f50b356a019c

Revision 5:f50b356a019c, 5.5 KB checked in by Matija Polajnar <matija.polajnar@…>, 23 months ago (diff)

Merge in Lan Umek's implementations of reliability estimation for classification.

Line 
1.. automodule:: Orange.evaluation.reliability
2
3.. index:: Reliability Estimation
4
5.. index::
6   single: reliability; Reliability Estimation for Regression
7
8##########################################################
9Reliability estimation (``Orange.evaluation.reliability``)
10##########################################################
11
12*************************************
13Reliability Estimation for Regression
14*************************************
15
16Reliability assessment statistically predicts reliability of single
17predictions. Most of implemented algorithms for regression are taken from
18Comparison of approaches for estimating reliability of individual
19regression predictions, Zoran Bosnić, 2008. Implementations for
20classification follow descriptions in Evaluating Reliability of Single
21Classifications of Neural Networks, Darko Pevec, 2011.
22
23The following example shows basic usage of reliability estimation methods:
24
25.. literalinclude:: code/reliability-basic.py
26    :lines: 7-
27
28The important points of this example are:
29 * construction of reliability estimators using classes,
30   implemented in this module,
31 * construction of a reliability learner that bonds a regular learner
32   (:class:`~Orange.classification.knn.kNNLearner` in this case) with
33   reliability estimators,
34 * calling the constructed classifier with
35   :obj:`Orange.classification.Classifier.GetBoth` option to obtain class
36   probabilities; :obj:`probability` is the object that gets appended the
37   :obj:`reliability_estimate` attribute, an instance of
38   :class:`Orange.evaluation.reliability.Estimate`, by the reliability learner.
39
40It is also possible to do reliability estimation on whole data
41table, not only on single instance. Next example demonstrates usage of a
42cross-validation technique for reliability estimation. Reliability estimations
43for first 10 instances get printed:
44
45.. literalinclude:: code/reliability-run.py
46    :lines: 11-
47
48Reliability Methods
49===================
50
51For regression, all the described measures can be used. Classification domains
52are supported by the following methods: BAGV, LCV, CNK and DENS.
53
54Sensitivity Analysis (SAvar and SAbias)
55---------------------------------------
56.. autoclass:: SensitivityAnalysis
57
58Variance of bagged models (BAGV)
59--------------------------------
60.. autoclass:: BaggingVariance
61
62Local cross validation reliability estimate (LCV)
63-------------------------------------------------
64.. autoclass:: LocalCrossValidation
65
66Local modeling of prediction error (CNK)
67----------------------------------------
68.. autoclass:: CNeighbours
69
70Bagging variance c-neighbours (BVCK)
71------------------------------------
72
73.. autoclass:: BaggingVarianceCNeighbours(bagv=BaggingVariance(), cnk=CNeighbours())
74
75Mahalanobis distance
76--------------------
77
78.. autoclass:: Mahalanobis
79
80Mahalanobis to center
81---------------------
82
83.. autoclass:: MahalanobisToCenter
84
85Density estimation using Parzen window (DENS)
86---------------------------------------------
87
88.. autoclass:: ParzenWindowDensityBased
89
90Reliability estimation wrappers
91===============================
92
93.. autoclass:: Learner(box_learner, name="Reliability estimation", estimators=[SensitivityAnalysis(), LocalCrossValidation(), BaggingVarianceCNeighbours(), Mahalanobis(), MahalanobisToCenter()], **kwds)
94    :members:
95
96.. autoclass:: Classifier
97    :members:
98
99Reliability estimation results
100==============================
101
102.. autoclass:: Estimate
103    :members:
104    :show-inheritance:
105
106There is a dictionary named :obj:`METHOD_NAME` that maps reliability estimation
107method IDs (ints) to method names (strings).
108
109In this module, there are also two constants for distinguishing signed and
110absolute reliability estimation measures::
111
112  SIGNED = 0
113  ABSOLUTE = 1
114
115Reliability estimation scoring methods
116======================================
117
118.. autofunction:: get_pearson_r
119
120.. autofunction:: get_pearson_r_by_iterations
121
122.. autofunction:: get_spearman_r
123
124Example of usage
125================
126
127.. literalinclude:: code/reliability-long.py
128    :lines: 11-26
129
130This script prints out Pearson's R coefficient between reliability estimates
131and actual prediction errors, and a corresponding p-value, for each of the
132reliability estimation measures used by default. ::
133
134  Estimate               r       p
135  SAvar absolute        -0.077   0.454
136  SAbias signed         -0.165   0.105
137  SAbias absolute       -0.099   0.333
138  BAGV absolute          0.104   0.309
139  CNK signed             0.233   0.021
140  CNK absolute           0.057   0.579
141  LCV absolute           0.069   0.504
142  BVCK_absolute          0.092   0.368
143  Mahalanobis absolute   0.091   0.375
144
145
146References
147==========
148
149Bosnić, Z., Kononenko, I. (2007) `Estimation of individual prediction
150reliability using local sensitivity analysis. <http://www.springerlink
151.com/content/e27p2584387532g8/>`_ *Applied Intelligence* 29(3), pp. 187-203.
152
153Bosnić, Z., Kononenko, I. (2008) `Comparison of approaches for estimating
154reliability of individual regression predictions. <http://www.sciencedirect
155.com/science/article/pii/S0169023X08001080>`_ *Data & Knowledge Engineering*
15667(3), pp. 504-516.
157
158Bosnić, Z., Kononenko, I. (2010) `Automatic selection of reliability estimates
159for individual regression predictions. <http://journals.cambridge
160.org/abstract_S0269888909990154>`_ *The Knowledge Engineering Review* 25(1),
161pp. 27-47.
162
163Pevec, D., Štrumbelj, E., Kononenko, I. (2011) `Evaluating Reliability of
164Single Classifications of Neural Networks. <http://www.springerlink.com
165/content/48u881761h127r33/export-citation/>`_ *Adaptive and Natural Computing
166Algorithms*, 2011, pp. 22-30.
Note: See TracBrowser for help on using the repository browser.