source: orange-reliability/docs/rst/Orange.evaluation.reliability.rst @ 42:75bf74617e81

Revision 42:75bf74617e81, 5.3 KB checked in by markotoplak, 7 months ago (diff)

Updates to the documentation.

Line 
1.. automodule:: Orange.evaluation.reliability
2
3.. index:: Reliability Estimation
4
5.. index::
6   single: reliability; Reliability Estimation for Regression
7
8##########################################################
9Reliability estimation (``Orange.evaluation.reliability``)
10##########################################################
11
12********************************************************
13Reliability Estimation for Regression and Classification
14********************************************************
15
16Reliability assessment aims to predict reliabilities of individual
17predictions. Most of implemented algorithms for regression described in
18[Bosnic2008]_ and in [Pevec2011]_ for classification.
19
20We can use reliability estimation with any Orange learners. The following example:
21
22 * Constructs reliability estimators (implemented in this module),
23 * The :obj:`Learner` wrapper combines a regular learner, here a :obj:`~Orange.classification.knn.kNNLearner`, with reliability estimators.
24 * Obtains prediction probabilities from the constructed classifier
25   (:obj:`Orange.classification.Classifier.GetBoth` option). The resulting
26   probabilities have an additional attribute, :obj:`reliability_estimate`,
27   that contains a list of :class:`Orange.evaluation.reliability.Estimate`.
28
29.. literalinclude:: code/reliability-basic.py
30    :lines: 7-
31
32We could also evaluate more examples. The next example prints reliability estimates
33for first 10 instances (with cross-validation):
34
35.. literalinclude:: code/reliability-run.py
36    :lines: 7-
37
38Reliability estimation wrappers
39===============================
40
41.. autoclass:: Learner
42   :members: __call__
43
44.. autoclass:: Classifier
45   :members: __call__
46
47
48Reliability Methods
49===================
50
51All measures except :math:`O_{ref}` work with regression. Classification is
52supported by BAGV, LCV, CNK and DENS, :math:`O_{ref}`.
53
54Sensitivity Analysis (SAvar and SAbias)
55---------------------------------------
56.. autoclass:: SensitivityAnalysis
57
58Variance of bagged models (BAGV)
59--------------------------------
60.. autoclass:: BaggingVariance
61
62Local cross validation reliability estimate (LCV)
63-------------------------------------------------
64.. autoclass:: LocalCrossValidation
65
66Local modeling of prediction error (CNK)
67----------------------------------------
68.. autoclass:: CNeighbours
69
70Bagging variance c-neighbours (BVCK)
71------------------------------------
72
73.. autoclass:: BaggingVarianceCNeighbours(bagv=BaggingVariance(), cnk=CNeighbours())
74
75Mahalanobis distance
76--------------------
77
78.. autoclass:: Mahalanobis
79
80Mahalanobis to center
81---------------------
82
83.. autoclass:: MahalanobisToCenter
84
85Density estimation using Parzen window (DENS)
86---------------------------------------------
87
88.. autoclass:: ParzenWindowDensityBased
89
90Internal cross validation (ICV)
91-------------------------------
92
93.. autoclass:: ICV
94
95
96Stacked generalization (Stacking)
97---------------------------------
98
99.. autoclass:: Stacking
100
101Reference Estimate for Classification (:math:`O_{ref}`)
102-------------------------------------------------------
103
104.. autoclass:: ReferenceExpectedError
105
106Reliability estimation results
107==============================
108
109.. data:: SIGNED
110   
111.. data:: ABSOLUTE
112
113    These constants distinguish signed and
114    absolute reliability estimation measures.
115
116.. data:: METHOD_NAME
117
118    A dictionary that that maps reliability estimation
119    method IDs (integerss) to method names (strings).
120
121.. autoclass:: Estimate
122    :members:
123    :show-inheritance:
124
125
126
127Reliability estimation scoring
128==============================
129
130.. autofunction:: get_pearson_r
131
132.. autofunction:: get_pearson_r_by_iterations
133
134.. autofunction:: get_spearman_r
135
136Example
137=======
138
139The following script prints Pearson's correlation coefficient (r) between reliability
140estimates and actual prediction errors, and a corresponding p-value, for
141default reliability estimation measures.
142
143.. literalinclude:: code/reliability-long.py
144    :lines: 7-22
145
146Results::
147 
148  Estimate               r       p
149  SAvar absolute        -0.077   0.454
150  SAbias signed         -0.165   0.105
151  SAbias absolute        0.095   0.352
152  LCV absolute           0.069   0.504
153  BVCK absolute          0.060   0.562
154  BAGV absolute          0.078   0.448
155  CNK signed             0.233   0.021
156  CNK absolute           0.058   0.574
157  Mahalanobis absolute   0.091   0.375
158  Mahalanobis to center  0.096   0.349
159
160References
161==========
162
163.. [Bosnic2007]  Bosnić, Z., Kononenko, I. (2007) `Estimation of individual prediction reliability using local sensitivity analysis. <http://www.springerlink.com/content/e27p2584387532g8/>`_ *Applied Intelligence* 29(3), pp. 187-203.
164
165.. [Bosnic2008] Bosnić, Z., Kononenko, I. (2008) `Comparison of approaches for estimating reliability of individual regression predictions. <http://www.sciencedirect .com/science/article/pii/S0169023X08001080>`_ *Data & Knowledge Engineering* 67(3), pp. 504-516.
166
167.. [Bosnic2010] Bosnić, Z., Kononenko, I. (2010) `Automatic selection of reliability estimates for individual regression predictions. <http://journals.cambridge .org/abstract_S0269888909990154>`_ *The Knowledge Engineering Review* 25(1), pp. 27-47.
168
169.. [Pevec2011] Pevec, D., Štrumbelj, E., Kononenko, I. (2011) `Evaluating Reliability of Single Classifications of Neural Networks. <http://www.springerlink.com /content/48u881761h127r33/export-citation/>`_ *Adaptive and Natural Computing Algorithms*, 2011, pp. 22-30.
Note: See TracBrowser for help on using the repository browser.