source: orange/docs/reference/rst/Orange.evaluation.reliability.rst @ 9680:4689e76f99be

Revision 9680:4689e76f99be, 4.5 KB checked in by Matija Polajnar <matija.polajnar@…>, 2 years ago (diff)

Updated documentation and regression tests for reliability estimation. Closes #1058.

Line 
1.. automodule:: Orange.evaluation.reliability
2
3.. index:: Reliability Estimation
4
5.. index::
6   single: reliability; Reliability Estimation for Regression
7
8########################################
9Reliability estimation (``reliability``)
10########################################
11
12*************************************
13Reliability Estimation for Regression
14*************************************
15
16Reliability assessment statistically predicts reliability of single
17predictions. Most of implemented algorithms are taken from Comparison of
18approaches for estimating reliability of individual regression predictions,
19Zoran Bosnić, 2008.
20
21The following example shows basic usage of reliability estimation methods:
22
23.. literalinclude:: code/reliability-basic.py
24
25The important points of this example are:
26 * construction of reliability estimators using classes,
27   implemented in this module,
28 * construction of a reliability learner that bonds a regular learner
29   (:class:`~Orange.classification.knn.kNNLearner` in this case) with
30   reliability estimators,
31 * calling the constructed classifier with
32   :obj:`Orange.classification.Classifier.GetBoth` option to obtain class
33   probabilities; :obj:`probability` is the object that gets appended the
34   :obj:`reliability_estimate` attribute, an instance of
35   :class:`Orange.evaluation.reliability.Estimate`, by the reliability learner.
36
37It is also possible to do reliability estimation on whole data
38table, not only on single instance. Next example demonstrates usage of a
39cross-validation technique for reliability estimation. Reliability estimations
40for first 10 instances get printed:
41
42.. literalinclude:: code/reliability-run.py
43
44Reliability Methods
45===================
46
47Sensitivity Analysis (SAvar and SAbias)
48---------------------------------------
49.. autoclass:: SensitivityAnalysis
50
51Variance of bagged models (BAGV)
52--------------------------------
53.. autoclass:: BaggingVariance
54
55Local cross validation reliability estimate (LCV)
56-------------------------------------------------
57.. autoclass:: LocalCrossValidation
58
59Local modeling of prediction error (CNK)
60----------------------------------------
61.. autoclass:: CNeighbours
62
63Bagging variance c-neighbours (BVCK)
64------------------------------------
65
66.. autoclass:: BaggingVarianceCNeighbours
67
68Mahalanobis distance
69--------------------
70
71.. autoclass:: Mahalanobis
72
73Mahalanobis to center
74---------------------
75
76.. autoclass:: MahalanobisToCenter
77
78Reliability estimation wrappers
79===============================
80
81.. autoclass:: Learner
82    :members:
83
84.. autoclass:: Classifier
85    :members:
86
87Reliability estimation results
88==============================
89
90.. autoclass:: Estimate
91    :members:
92    :show-inheritance:
93
94There is a dictionary named :obj:`METHOD_NAME` that maps reliability estimation
95method IDs (ints) to method names (strings).
96
97In this module, there are also two constants for distinguishing signed and
98absolute reliability estimation measures::
99
100  SIGNED = 0
101  ABSOLUTE = 1
102
103Reliability estimation scoring methods
104======================================
105
106.. autofunction:: get_pearson_r
107
108.. autofunction:: get_pearson_r_by_iterations
109
110.. autofunction:: get_spearman_r
111
112Example of usage
113================
114
115.. literalinclude:: code/reliability-long.py
116    :lines: 1-16
117
118This script prints out Pearson's R coefficient between reliability estimates
119and actual prediction errors, and a corresponding p-value, for each of the
120reliability estimation measures used by default. ::
121
122  Estimate               r       p
123  SAvar absolute        -0.077   0.454
124  SAbias signed         -0.165   0.105
125  SAbias absolute       -0.099   0.333
126  BAGV absolute          0.104   0.309
127  CNK signed             0.233   0.021
128  CNK absolute           0.057   0.579
129  LCV absolute           0.069   0.504
130  BVCK_absolute          0.092   0.368
131  Mahalanobis absolute   0.091   0.375
132
133
134References
135==========
136
137Bosnić, Z., Kononenko, I. (2007) `Estimation of individual prediction
138reliability using local sensitivity analysis. <http://www.springerlink
139.com/content/e27p2584387532g8/>`_ *Applied Intelligence* 29(3), pp. 187-203.
140
141Bosnić, Z., Kononenko, I. (2008) `Comparison of approaches for estimating
142reliability of individual regression predictions. <http://www.sciencedirect
143.com/science/article/pii/S0169023X08001080>`_ *Data & Knowledge Engineering*
14467(3), pp. 504-516.
145
146Bosnić, Z., Kononenko, I. (2010) `Automatic selection of reliability estimates
147for individual regression predictions. <http://journals.cambridge
148.org/abstract_S0269888909990154>`_ *The Knowledge Engineering Review* 25(1),
149pp. 27-47.
150
Note: See TracBrowser for help on using the repository browser.