source: orange-reliability/docs/rst/Orange.evaluation.reliability.rst @ 13:f98f0417ceb6

Revision 13:f98f0417ceb6, 5.7 KB checked in by Matija Polajnar <matija.polajnar@…>, 22 months ago (diff)

Documentation of reference method for classification.

Line 
1.. automodule:: Orange.evaluation.reliability
2
3.. index:: Reliability Estimation
4
5.. index::
6   single: reliability; Reliability Estimation for Regression
7
8##########################################################
9Reliability estimation (``Orange.evaluation.reliability``)
10##########################################################
11
12********************************************************
13Reliability Estimation for Regression and Classification
14********************************************************
15
16Reliability assessment statistically predicts reliability of single
17predictions. Most of implemented algorithms for regression are taken from
18Comparison of approaches for estimating reliability of individual
19regression predictions, Zoran Bosnić, 2008. Implementations for
20classification follow descriptions in Evaluating Reliability of Single
21Classifications of Neural Networks, Darko Pevec, 2011.
22
23The following example shows basic usage of reliability estimation methods:
24
25.. literalinclude:: code/reliability-basic.py
26    :lines: 7-
27
28The important points of this example are:
29 * construction of reliability estimators using classes,
30   implemented in this module,
31 * construction of a reliability learner that bonds a regular learner
32   (:class:`~Orange.classification.knn.kNNLearner` in this case) with
33   reliability estimators,
34 * calling the constructed classifier with
35   :obj:`Orange.classification.Classifier.GetBoth` option to obtain class
36   probabilities; :obj:`probability` is the object that gets appended the
37   :obj:`reliability_estimate` attribute, an instance of
38   :class:`Orange.evaluation.reliability.Estimate`, by the reliability learner.
39
40It is also possible to do reliability estimation on whole data
41table, not only on single instance. Next example demonstrates usage of a
42cross-validation technique for reliability estimation. Reliability estimations
43for first 10 instances get printed:
44
45.. literalinclude:: code/reliability-run.py
46    :lines: 11-
47
48Reliability Methods
49===================
50
51For regression, all the described measures can be used, except for the :math:`O_{ref}`. Classification domains
52are supported by the following methods: BAGV, LCV, CNK and DENS, :math:`O_{ref}`.
53
54Sensitivity Analysis (SAvar and SAbias)
55---------------------------------------
56.. autoclass:: SensitivityAnalysis
57
58Variance of bagged models (BAGV)
59--------------------------------
60.. autoclass:: BaggingVariance
61
62Local cross validation reliability estimate (LCV)
63-------------------------------------------------
64.. autoclass:: LocalCrossValidation
65
66Local modeling of prediction error (CNK)
67----------------------------------------
68.. autoclass:: CNeighbours
69
70Bagging variance c-neighbours (BVCK)
71------------------------------------
72
73.. autoclass:: BaggingVarianceCNeighbours(bagv=BaggingVariance(), cnk=CNeighbours())
74
75Mahalanobis distance
76--------------------
77
78.. autoclass:: Mahalanobis
79
80Mahalanobis to center
81---------------------
82
83.. autoclass:: MahalanobisToCenter
84
85Density estimation using Parzen window (DENS)
86---------------------------------------------
87
88.. autoclass:: ParzenWindowDensityBased
89
90Reference Estimate for Classification (:math:`O_{ref}`)
91-------------------------------------------------------
92
93.. autoclass:: ReferenceExpectedError
94
95Reliability estimation wrappers
96===============================
97
98.. autoclass:: Learner(box_learner, name="Reliability estimation", estimators=[SensitivityAnalysis(), LocalCrossValidation(), BaggingVarianceCNeighbours(), Mahalanobis(), MahalanobisToCenter()], **kwds)
99    :members:
100
101.. autoclass:: Classifier
102    :members:
103
104Reliability estimation results
105==============================
106
107.. autoclass:: Estimate
108    :members:
109    :show-inheritance:
110
111There is a dictionary named :obj:`METHOD_NAME` that maps reliability estimation
112method IDs (ints) to method names (strings).
113
114In this module, there are also two constants for distinguishing signed and
115absolute reliability estimation measures::
116
117  SIGNED = 0
118  ABSOLUTE = 1
119
120Reliability estimation scoring methods
121======================================
122
123.. autofunction:: get_pearson_r
124
125.. autofunction:: get_pearson_r_by_iterations
126
127.. autofunction:: get_spearman_r
128
129Example of usage
130================
131
132.. literalinclude:: code/reliability-long.py
133    :lines: 11-26
134
135This script prints out Pearson's R coefficient between reliability estimates
136and actual prediction errors, and a corresponding p-value, for each of the
137reliability estimation measures used by default. ::
138
139  Estimate               r       p
140  SAvar absolute        -0.077   0.454
141  SAbias signed         -0.165   0.105
142  SAbias absolute       -0.099   0.333
143  BAGV absolute          0.104   0.309
144  CNK signed             0.233   0.021
145  CNK absolute           0.057   0.579
146  LCV absolute           0.069   0.504
147  BVCK_absolute          0.092   0.368
148  Mahalanobis absolute   0.091   0.375
149
150
151References
152==========
153
154Bosnić, Z., Kononenko, I. (2007) `Estimation of individual prediction
155reliability using local sensitivity analysis. <http://www.springerlink
156.com/content/e27p2584387532g8/>`_ *Applied Intelligence* 29(3), pp. 187-203.
157
158Bosnić, Z., Kononenko, I. (2008) `Comparison of approaches for estimating
159reliability of individual regression predictions. <http://www.sciencedirect
160.com/science/article/pii/S0169023X08001080>`_ *Data & Knowledge Engineering*
16167(3), pp. 504-516.
162
163Bosnić, Z., Kononenko, I. (2010) `Automatic selection of reliability estimates
164for individual regression predictions. <http://journals.cambridge
165.org/abstract_S0269888909990154>`_ *The Knowledge Engineering Review* 25(1),
166pp. 27-47.
167
168Pevec, D., Štrumbelj, E., Kononenko, I. (2011) `Evaluating Reliability of
169Single Classifications of Neural Networks. <http://www.springerlink.com
170/content/48u881761h127r33/export-citation/>`_ *Adaptive and Natural Computing
171Algorithms*, 2011, pp. 22-30.
Note: See TracBrowser for help on using the repository browser.