source: orange/docs/reference/rst/Orange.evaluation.reliability.rst @ 9681:b278e50a6071

Revision 9681:b278e50a6071, 4.5 KB checked in by Matija Polajnar <matija.polajnar@…>, 2 years ago (diff)

Reliability: add headers to regression test scripts.

Line 
1.. automodule:: Orange.evaluation.reliability
2
3.. index:: Reliability Estimation
4
5.. index::
6   single: reliability; Reliability Estimation for Regression
7
8########################################
9Reliability estimation (``reliability``)
10########################################
11
12*************************************
13Reliability Estimation for Regression
14*************************************
15
16Reliability assessment statistically predicts reliability of single
17predictions. Most of implemented algorithms are taken from Comparison of
18approaches for estimating reliability of individual regression predictions,
19Zoran Bosnić, 2008.
20
21The following example shows basic usage of reliability estimation methods:
22
23.. literalinclude:: code/reliability-basic.py
24    :lines: 7-
25
26The important points of this example are:
27 * construction of reliability estimators using classes,
28   implemented in this module,
29 * construction of a reliability learner that bonds a regular learner
30   (:class:`~Orange.classification.knn.kNNLearner` in this case) with
31   reliability estimators,
32 * calling the constructed classifier with
33   :obj:`Orange.classification.Classifier.GetBoth` option to obtain class
34   probabilities; :obj:`probability` is the object that gets appended the
35   :obj:`reliability_estimate` attribute, an instance of
36   :class:`Orange.evaluation.reliability.Estimate`, by the reliability learner.
37
38It is also possible to do reliability estimation on whole data
39table, not only on single instance. Next example demonstrates usage of a
40cross-validation technique for reliability estimation. Reliability estimations
41for first 10 instances get printed:
42
43.. literalinclude:: code/reliability-run.py
44    :lines: 7-
45
46Reliability Methods
47===================
48
49Sensitivity Analysis (SAvar and SAbias)
50---------------------------------------
51.. autoclass:: SensitivityAnalysis
52
53Variance of bagged models (BAGV)
54--------------------------------
55.. autoclass:: BaggingVariance
56
57Local cross validation reliability estimate (LCV)
58-------------------------------------------------
59.. autoclass:: LocalCrossValidation
60
61Local modeling of prediction error (CNK)
62----------------------------------------
63.. autoclass:: CNeighbours
64
65Bagging variance c-neighbours (BVCK)
66------------------------------------
67
68.. autoclass:: BaggingVarianceCNeighbours
69
70Mahalanobis distance
71--------------------
72
73.. autoclass:: Mahalanobis
74
75Mahalanobis to center
76---------------------
77
78.. autoclass:: MahalanobisToCenter
79
80Reliability estimation wrappers
81===============================
82
83.. autoclass:: Learner
84    :members:
85
86.. autoclass:: Classifier
87    :members:
88
89Reliability estimation results
90==============================
91
92.. autoclass:: Estimate
93    :members:
94    :show-inheritance:
95
96There is a dictionary named :obj:`METHOD_NAME` that maps reliability estimation
97method IDs (ints) to method names (strings).
98
99In this module, there are also two constants for distinguishing signed and
100absolute reliability estimation measures::
101
102  SIGNED = 0
103  ABSOLUTE = 1
104
105Reliability estimation scoring methods
106======================================
107
108.. autofunction:: get_pearson_r
109
110.. autofunction:: get_pearson_r_by_iterations
111
112.. autofunction:: get_spearman_r
113
114Example of usage
115================
116
117.. literalinclude:: code/reliability-long.py
118    :lines: 7-22
119
120This script prints out Pearson's R coefficient between reliability estimates
121and actual prediction errors, and a corresponding p-value, for each of the
122reliability estimation measures used by default. ::
123
124  Estimate               r       p
125  SAvar absolute        -0.077   0.454
126  SAbias signed         -0.165   0.105
127  SAbias absolute       -0.099   0.333
128  BAGV absolute          0.104   0.309
129  CNK signed             0.233   0.021
130  CNK absolute           0.057   0.579
131  LCV absolute           0.069   0.504
132  BVCK_absolute          0.092   0.368
133  Mahalanobis absolute   0.091   0.375
134
135
136References
137==========
138
139Bosnić, Z., Kononenko, I. (2007) `Estimation of individual prediction
140reliability using local sensitivity analysis. <http://www.springerlink
141.com/content/e27p2584387532g8/>`_ *Applied Intelligence* 29(3), pp. 187-203.
142
143Bosnić, Z., Kononenko, I. (2008) `Comparison of approaches for estimating
144reliability of individual regression predictions. <http://www.sciencedirect
145.com/science/article/pii/S0169023X08001080>`_ *Data & Knowledge Engineering*
14667(3), pp. 504-516.
147
148Bosnić, Z., Kononenko, I. (2010) `Automatic selection of reliability estimates
149for individual regression predictions. <http://journals.cambridge
150.org/abstract_S0269888909990154>`_ *The Knowledge Engineering Review* 25(1),
151pp. 27-47.
152
Note: See TracBrowser for help on using the repository browser.