source: orange-reliability/docs/rst/Orange.evaluation.reliability.rst @ 40:d1cc6a016f2c

Revision 40:d1cc6a016f2c, 5.9 KB checked in by markotoplak, 7 months ago (diff)

Added stubs of ICV and Stacking to the documentation

Line 
1.. automodule:: Orange.evaluation.reliability
2
3.. index:: Reliability Estimation
4
5.. index::
6   single: reliability; Reliability Estimation for Regression
7
8##########################################################
9Reliability estimation (``Orange.evaluation.reliability``)
10##########################################################
11
12********************************************************
13Reliability Estimation for Regression and Classification
14********************************************************
15
16Reliability assessment statistically predicts reliability of single
17predictions. Most of implemented algorithms for regression are taken from
18Comparison of approaches for estimating reliability of individual
19regression predictions, Zoran Bosnić, 2008. Implementations for
20classification follow descriptions in Evaluating Reliability of Single
21Classifications of Neural Networks, Darko Pevec, 2011.
22
23The following example shows basic usage of reliability estimation methods:
24
25.. literalinclude:: code/reliability-basic.py
26    :lines: 7-
27
28The important points of this example are:
29 * construction of reliability estimators using classes,
30   implemented in this module,
31 * construction of a reliability learner that bonds a regular learner
32   (:class:`~Orange.classification.knn.kNNLearner` in this case) with
33   reliability estimators,
34 * calling the constructed classifier with
35   :obj:`Orange.classification.Classifier.GetBoth` option to obtain class
36   probabilities; :obj:`probability` is the object that gets appended the
37   :obj:`reliability_estimate` attribute, an instance of
38   :class:`Orange.evaluation.reliability.Estimate`, by the reliability learner.
39
40It is also possible to do reliability estimation on whole data
41table, not only on single instance. Next example demonstrates usage of a
42cross-validation technique for reliability estimation. Reliability estimations
43for first 10 instances get printed:
44
45.. literalinclude:: code/reliability-run.py
46    :lines: 7-
47
48Reliability Methods
49===================
50
51For regression, all the described measures can be used, except for the :math:`O_{ref}`. Classification domains
52are supported by the following methods: BAGV, LCV, CNK and DENS, :math:`O_{ref}`.
53
54Sensitivity Analysis (SAvar and SAbias)
55---------------------------------------
56.. autoclass:: SensitivityAnalysis
57
58Variance of bagged models (BAGV)
59--------------------------------
60.. autoclass:: BaggingVariance
61
62Local cross validation reliability estimate (LCV)
63-------------------------------------------------
64.. autoclass:: LocalCrossValidation
65
66Local modeling of prediction error (CNK)
67----------------------------------------
68.. autoclass:: CNeighbours
69
70Bagging variance c-neighbours (BVCK)
71------------------------------------
72
73.. autoclass:: BaggingVarianceCNeighbours(bagv=BaggingVariance(), cnk=CNeighbours())
74
75Mahalanobis distance
76--------------------
77
78.. autoclass:: Mahalanobis
79
80Mahalanobis to center
81---------------------
82
83.. autoclass:: MahalanobisToCenter
84
85Density estimation using Parzen window (DENS)
86---------------------------------------------
87
88.. autoclass:: ParzenWindowDensityBased
89
90Internal cross validation (ICV)
91-------------------------------
92
93.. autoclass:: ICV
94
95
96Stacked generalization (Stacking)
97-------------------------------
98
99.. autoclass:: Stacking
100
101Reference Estimate for Classification (:math:`O_{ref}`)
102-------------------------------------------------------
103
104.. autoclass:: ReferenceExpectedError
105
106Reliability estimation wrappers
107===============================
108
109.. autoclass:: Learner(box_learner, name="Reliability estimation", estimators=[SensitivityAnalysis(), LocalCrossValidation(), BaggingVarianceCNeighbours(), Mahalanobis(), MahalanobisToCenter()], **kwds)
110    :members:
111
112.. autoclass:: Classifier
113    :members:
114
115Reliability estimation results
116==============================
117
118.. autoclass:: Estimate
119    :members:
120    :show-inheritance:
121
122There is a dictionary named :obj:`METHOD_NAME` that maps reliability estimation
123method IDs (ints) to method names (strings).
124
125In this module, there are also two constants for distinguishing signed and
126absolute reliability estimation measures::
127
128  SIGNED = 0
129  ABSOLUTE = 1
130
131Reliability estimation scoring
132==============================
133
134.. autofunction:: get_pearson_r
135
136.. autofunction:: get_pearson_r_by_iterations
137
138.. autofunction:: get_spearman_r
139
140Example
141=======
142
143.. literalinclude:: code/reliability-long.py
144    :lines: 7-22
145
146This script prints out Pearson's R coefficient between reliability estimates
147and actual prediction errors, and a corresponding p-value, for each of the
148reliability estimation measures used by default. ::
149 
150  Estimate               r       p
151  SAvar absolute        -0.077   0.454
152  SAbias signed         -0.165   0.105
153  SAbias absolute        0.095   0.352
154  LCV absolute           0.069   0.504
155  BVCK absolute          0.060   0.562
156  BAGV absolute          0.078   0.448
157  CNK signed             0.233   0.021
158  CNK absolute           0.058   0.574
159  Mahalanobis absolute   0.091   0.375
160  Mahalanobis to center  0.096   0.349
161
162References
163==========
164
165Bosnić, Z., Kononenko, I. (2007) `Estimation of individual prediction
166reliability using local sensitivity analysis. <http://www.springerlink
167.com/content/e27p2584387532g8/>`_ *Applied Intelligence* 29(3), pp. 187-203.
168
169Bosnić, Z., Kononenko, I. (2008) `Comparison of approaches for estimating
170reliability of individual regression predictions. <http://www.sciencedirect
171.com/science/article/pii/S0169023X08001080>`_ *Data & Knowledge Engineering*
17267(3), pp. 504-516.
173
174Bosnić, Z., Kononenko, I. (2010) `Automatic selection of reliability estimates
175for individual regression predictions. <http://journals.cambridge
176.org/abstract_S0269888909990154>`_ *The Knowledge Engineering Review* 25(1),
177pp. 27-47.
178
179Pevec, D., Štrumbelj, E., Kononenko, I. (2011) `Evaluating Reliability of
180Single Classifications of Neural Networks. <http://www.springerlink.com
181/content/48u881761h127r33/export-citation/>`_ *Adaptive and Natural Computing
182Algorithms*, 2011, pp. 22-30.
Note: See TracBrowser for help on using the repository browser.