source: orange-reliability/docs/rst/Orange.evaluation.reliability.rst @ 54:2a1c28cec845

Revision 54:2a1c28cec845, 5.5 KB checked in by markotoplak, 6 months ago (diff)

Documentation updates.

Line 
1.. automodule:: Orange.evaluation.reliability
2
3.. index:: Reliability Estimation
4
5.. index::
6   single: reliability; Reliability Estimation for Regression
7
8##########################################################
9Reliability estimation (``Orange.evaluation.reliability``)
10##########################################################
11
12********************************************************
13Reliability Estimation for Regression and Classification
14********************************************************
15
16Reliability assessment aims to predict reliabilities of individual
17predictions. Most of the implemented algorithms for regression are described in
18[Bosnic2008]_; the algorithms for classification are described in [Pevec2011]_.
19
20We can use reliability estimation with any prediction method. The following example:
21
22 * Constructs reliability estimators (implemented in this module),
23 * The :obj:`Learner` wrapper combines a a prediction method (learner), here a :obj:`~Orange.classification.knn.kNNLearner`, with reliability estimators.
24 * Obtains prediction probabilities, which have an additional attribute,
25   :obj:`reliability_estimate`,
26   that contains a list of :class:`Orange.evaluation.reliability.Estimate`.
27
28.. literalinclude:: code/reliability-basic.py
29    :lines: 7-
30
31The next example prints reliability estimates
32for first 10 instances (with cross-validation):
33
34.. literalinclude:: code/reliability-run.py
35    :lines: 7-
36
37Reliability estimation wrappers
38===============================
39
40.. autoclass:: Learner
41   :members: __call__
42
43.. autoclass:: Classifier
44   :members: __call__
45
46
47Reliability Methods
48===================
49
50All measures except :math:`O_{ref}` work with regression. Classification is
51supported by BAGV, LCV, CNK and DENS, :math:`O_{ref}`.
52
53Sensitivity Analysis (SAvar and SAbias)
54---------------------------------------
55.. autoclass:: SensitivityAnalysis
56
57Variance of bagged models (BAGV)
58--------------------------------
59.. autoclass:: BaggingVariance
60
61Local cross validation reliability estimate (LCV)
62-------------------------------------------------
63.. autoclass:: LocalCrossValidation
64
65Local modeling of prediction error (CNK)
66----------------------------------------
67.. autoclass:: CNeighbours
68
69Bagging variance c-neighbours (BVCK)
70------------------------------------
71
72.. autoclass:: BaggingVarianceCNeighbours(bagv=BaggingVariance(), cnk=CNeighbours())
73
74Mahalanobis distance
75--------------------
76
77.. autoclass:: Mahalanobis
78
79Mahalanobis to center
80---------------------
81
82.. autoclass:: MahalanobisToCenter
83
84Density estimation using Parzen window (DENS)
85---------------------------------------------
86
87.. autoclass:: ParzenWindowDensityBased
88
89Internal cross validation (ICV)
90-------------------------------
91
92.. autoclass:: ICV
93
94
95Stacked generalization (Stacking)
96---------------------------------
97
98.. autoclass:: Stacking
99
100Reference Estimate for Classification (:math:`O_{ref}`)
101-------------------------------------------------------
102
103.. autoclass:: ReferenceExpectedError
104
105Reliability estimation results
106==============================
107
108.. data:: SIGNED
109   
110.. data:: ABSOLUTE
111
112    These constants distinguish signed and
113    absolute reliability estimation measures.
114
115.. data:: METHOD_NAME
116
117    A dictionary that that maps reliability estimation
118    method IDs (integers) to method names (strings).
119
120.. autoclass:: Estimate
121    :members:
122    :show-inheritance:
123
124
125
126Reliability estimation scoring
127==============================
128
129.. autofunction:: get_pearson_r
130
131.. autofunction:: get_pearson_r_by_iterations
132
133.. autofunction:: get_spearman_r
134
135Example
136=======
137
138The following script prints Pearson's correlation coefficient (r) between reliability
139estimates and actual prediction errors, and a corresponding p-value, for
140default reliability estimation measures.
141
142.. literalinclude:: code/reliability-long.py
143    :lines: 7-22
144
145Results::
146 
147  Estimate               r       p
148  SAvar absolute        -0.077   0.454
149  SAbias signed         -0.165   0.105
150  SAbias absolute        0.095   0.352
151  LCV absolute           0.069   0.504
152  BVCK absolute          0.060   0.562
153  BAGV absolute          0.078   0.448
154  CNK signed             0.233   0.021
155  CNK absolute           0.058   0.574
156  Mahalanobis absolute   0.091   0.375
157  Mahalanobis to center  0.096   0.349
158
159References
160==========
161
162.. [Bosnic2007]  Bosnić, Z., Kononenko, I. (2007) `Estimation of individual prediction reliability using local sensitivity analysis. <http://www.springerlink.com/content/e27p2584387532g8/>`_ *Applied Intelligence* 29(3), pp. 187-203.
163
164.. [Bosnic2008] Bosnić, Z., Kononenko, I. (2008) `Comparison of approaches for estimating reliability of individual regression predictions. <http://www.sciencedirect .com/science/article/pii/S0169023X08001080>`_ *Data & Knowledge Engineering* 67(3), pp. 504-516.
165
166.. [Bosnic2010] Bosnić, Z., Kononenko, I. (2010) `Automatic selection of reliability estimates for individual regression predictions. <http://journals.cambridge .org/abstract_S0269888909990154>`_ *The Knowledge Engineering Review* 25(1), pp. 27-47.
167
168.. [Pevec2011] Pevec, D., Štrumbelj, E., Kononenko, I. (2011) `Evaluating Reliability of Single Classifications of Neural Networks. <http://www.springerlink.com /content/48u881761h127r33/export-citation/>`_ *Adaptive and Natural Computing Algorithms*, 2011, pp. 22-30.
169
170.. [Wolpert1992] Wolpert, David H. (1992) `Stacked generalization.` *Neural Networks*, Vol. 5, 1992,  pp. 241-259.
171
172.. [Dzeroski2004] Dzeroski, S. and Zenko, B. (2004) `Is combining classifiers with stacking better than selecting the best one?` *Machine Learning*, Vol. 54, 2004,  pp. 255-273.
Note: See TracBrowser for help on using the repository browser.