#
source:
orange-reliability/docs/rst/Orange.evaluation.reliability.rst
@
39:5e0e4eb49fd1

Revision 39:5e0e4eb49fd1, 5.7 KB checked in by markotoplak, 7 months ago (diff) |
---|

Line | |
---|---|

1 | .. automodule:: Orange.evaluation.reliability |

2 | |

3 | .. index:: Reliability Estimation |

4 | |

5 | .. index:: |

6 | single: reliability; Reliability Estimation for Regression |

7 | |

8 | ########################################################## |

9 | Reliability estimation (``Orange.evaluation.reliability``) |

10 | ########################################################## |

11 | |

12 | ******************************************************** |

13 | Reliability Estimation for Regression and Classification |

14 | ******************************************************** |

15 | |

16 | Reliability assessment statistically predicts reliability of single |

17 | predictions. Most of implemented algorithms for regression are taken from |

18 | Comparison of approaches for estimating reliability of individual |

19 | regression predictions, Zoran Bosnić, 2008. Implementations for |

20 | classification follow descriptions in Evaluating Reliability of Single |

21 | Classifications of Neural Networks, Darko Pevec, 2011. |

22 | |

23 | The following example shows basic usage of reliability estimation methods: |

24 | |

25 | .. literalinclude:: code/reliability-basic.py |

26 | :lines: 7- |

27 | |

28 | The important points of this example are: |

29 | * construction of reliability estimators using classes, |

30 | implemented in this module, |

31 | * construction of a reliability learner that bonds a regular learner |

32 | (:class:`~Orange.classification.knn.kNNLearner` in this case) with |

33 | reliability estimators, |

34 | * calling the constructed classifier with |

35 | :obj:`Orange.classification.Classifier.GetBoth` option to obtain class |

36 | probabilities; :obj:`probability` is the object that gets appended the |

37 | :obj:`reliability_estimate` attribute, an instance of |

38 | :class:`Orange.evaluation.reliability.Estimate`, by the reliability learner. |

39 | |

40 | It is also possible to do reliability estimation on whole data |

41 | table, not only on single instance. Next example demonstrates usage of a |

42 | cross-validation technique for reliability estimation. Reliability estimations |

43 | for first 10 instances get printed: |

44 | |

45 | .. literalinclude:: code/reliability-run.py |

46 | :lines: 7- |

47 | |

48 | Reliability Methods |

49 | =================== |

50 | |

51 | For regression, all the described measures can be used, except for the :math:`O_{ref}`. Classification domains |

52 | are supported by the following methods: BAGV, LCV, CNK and DENS, :math:`O_{ref}`. |

53 | |

54 | Sensitivity Analysis (SAvar and SAbias) |

55 | --------------------------------------- |

56 | .. autoclass:: SensitivityAnalysis |

57 | |

58 | Variance of bagged models (BAGV) |

59 | -------------------------------- |

60 | .. autoclass:: BaggingVariance |

61 | |

62 | Local cross validation reliability estimate (LCV) |

63 | ------------------------------------------------- |

64 | .. autoclass:: LocalCrossValidation |

65 | |

66 | Local modeling of prediction error (CNK) |

67 | ---------------------------------------- |

68 | .. autoclass:: CNeighbours |

69 | |

70 | Bagging variance c-neighbours (BVCK) |

71 | ------------------------------------ |

72 | |

73 | .. autoclass:: BaggingVarianceCNeighbours(bagv=BaggingVariance(), cnk=CNeighbours()) |

74 | |

75 | Mahalanobis distance |

76 | -------------------- |

77 | |

78 | .. autoclass:: Mahalanobis |

79 | |

80 | Mahalanobis to center |

81 | --------------------- |

82 | |

83 | .. autoclass:: MahalanobisToCenter |

84 | |

85 | Density estimation using Parzen window (DENS) |

86 | --------------------------------------------- |

87 | |

88 | .. autoclass:: ParzenWindowDensityBased |

89 | |

90 | Reference Estimate for Classification (:math:`O_{ref}`) |

91 | ------------------------------------------------------- |

92 | |

93 | .. autoclass:: ReferenceExpectedError |

94 | |

95 | Reliability estimation wrappers |

96 | =============================== |

97 | |

98 | .. autoclass:: Learner(box_learner, name="Reliability estimation", estimators=[SensitivityAnalysis(), LocalCrossValidation(), BaggingVarianceCNeighbours(), Mahalanobis(), MahalanobisToCenter()], **kwds) |

99 | :members: |

100 | |

101 | .. autoclass:: Classifier |

102 | :members: |

103 | |

104 | Reliability estimation results |

105 | ============================== |

106 | |

107 | .. autoclass:: Estimate |

108 | :members: |

109 | :show-inheritance: |

110 | |

111 | There is a dictionary named :obj:`METHOD_NAME` that maps reliability estimation |

112 | method IDs (ints) to method names (strings). |

113 | |

114 | In this module, there are also two constants for distinguishing signed and |

115 | absolute reliability estimation measures:: |

116 | |

117 | SIGNED = 0 |

118 | ABSOLUTE = 1 |

119 | |

120 | Reliability estimation scoring |

121 | ============================== |

122 | |

123 | .. autofunction:: get_pearson_r |

124 | |

125 | .. autofunction:: get_pearson_r_by_iterations |

126 | |

127 | .. autofunction:: get_spearman_r |

128 | |

129 | Example |

130 | ======= |

131 | |

132 | .. literalinclude:: code/reliability-long.py |

133 | :lines: 7-22 |

134 | |

135 | This script prints out Pearson's R coefficient between reliability estimates |

136 | and actual prediction errors, and a corresponding p-value, for each of the |

137 | reliability estimation measures used by default. :: |

138 | |

139 | Estimate r p |

140 | SAvar absolute -0.077 0.454 |

141 | SAbias signed -0.165 0.105 |

142 | SAbias absolute 0.095 0.352 |

143 | LCV absolute 0.069 0.504 |

144 | BVCK absolute 0.060 0.562 |

145 | BAGV absolute 0.078 0.448 |

146 | CNK signed 0.233 0.021 |

147 | CNK absolute 0.058 0.574 |

148 | Mahalanobis absolute 0.091 0.375 |

149 | Mahalanobis to center 0.096 0.349 |

150 | |

151 | References |

152 | ========== |

153 | |

154 | Bosnić, Z., Kononenko, I. (2007) `Estimation of individual prediction |

155 | reliability using local sensitivity analysis. <http://www.springerlink |

156 | .com/content/e27p2584387532g8/>`_ *Applied Intelligence* 29(3), pp. 187-203. |

157 | |

158 | Bosnić, Z., Kononenko, I. (2008) `Comparison of approaches for estimating |

159 | reliability of individual regression predictions. <http://www.sciencedirect |

160 | .com/science/article/pii/S0169023X08001080>`_ *Data & Knowledge Engineering* |

161 | 67(3), pp. 504-516. |

162 | |

163 | Bosnić, Z., Kononenko, I. (2010) `Automatic selection of reliability estimates |

164 | for individual regression predictions. <http://journals.cambridge |

165 | .org/abstract_S0269888909990154>`_ *The Knowledge Engineering Review* 25(1), |

166 | pp. 27-47. |

167 | |

168 | Pevec, D., Štrumbelj, E., Kononenko, I. (2011) `Evaluating Reliability of |

169 | Single Classifications of Neural Networks. <http://www.springerlink.com |

170 | /content/48u881761h127r33/export-citation/>`_ *Adaptive and Natural Computing |

171 | Algorithms*, 2011, pp. 22-30. |

**Note:**See TracBrowser for help on using the repository browser.