Changeset 54:2a1c28cec845 in orangereliability
 Timestamp:
 10/07/13 16:34:49 (7 months ago)
 Branch:
 default
 Files:

 2 edited
Legend:
 Unmodified
 Added
 Removed

docs/rst/Orange.evaluation.reliability.rst
r53 r54 15 15 16 16 Reliability assessment aims to predict reliabilities of individual 17 predictions. Most of implemented algorithms for regressiondescribed in18 [Bosnic2008]_ and in [Pevec2011]_ for classification.17 predictions. Most of the implemented algorithms for regression are described in 18 [Bosnic2008]_; the algorithms for classification are described in [Pevec2011]_. 19 19 20 We can use reliability estimation with any Orange learners. The following example:20 We can use reliability estimation with any prediction method. The following example: 21 21 22 22 * Constructs reliability estimators (implemented in this module), 23 * The :obj:`Learner` wrapper combines a regular learner, here a :obj:`~Orange.classification.knn.kNNLearner`, with reliability estimators. 24 * Obtains prediction probabilities from the constructed classifier 25 (:obj:`Orange.classification.Classifier.GetBoth` option). The resulting 26 probabilities have an additional attribute, :obj:`reliability_estimate`, 23 * The :obj:`Learner` wrapper combines a a prediction method (learner), here a :obj:`~Orange.classification.knn.kNNLearner`, with reliability estimators. 24 * Obtains prediction probabilities, which have an additional attribute, 25 :obj:`reliability_estimate`, 27 26 that contains a list of :class:`Orange.evaluation.reliability.Estimate`. 28 27 … … 30 29 :lines: 7 31 30 32 We could also evaluate more examples.The next example prints reliability estimates31 The next example prints reliability estimates 33 32 for first 10 instances (with crossvalidation): 34 33 … … 117 116 118 117 A dictionary that that maps reliability estimation 119 method IDs (integers s) to method names (strings).118 method IDs (integers) to method names (strings). 120 119 121 120 .. autoclass:: Estimate 
orangecontrib/reliability/__init__.py
r53 r54 52 52 def get_pearson_r(res): 53 53 """ 54 :param res: results of evaluation, done using learners, 55 wrapped into :class:`Orange.evaluation.reliability.Classifier`. 54 :param res: Evaluation results with :obj:`reliability_estimate`. 56 55 :type res: :class:`Orange.evaluation.testing.ExperimentResults` 57 56 58 Return Pearson's coefficient between the prediction error and each of the 59 used reliability estimates. Also, return the pvalue of each of 60 the coefficients. 57 Pearson's coefficients between the prediction error and 58 reliability estimates with pvalues. 61 59 """ 62 60 prediction_error = get_prediction_error_list(res) … … 76 74 def get_spearman_r(res): 77 75 """ 78 :param res: results of evaluation, done using learners, 79 wrapped into :class:`Orange.evaluation.reliability.Classifier`. 76 :param res: Evaluation results with :obj:`reliability_estimate`. 80 77 :type res: :class:`Orange.evaluation.testing.ExperimentResults` 81 78 82 Return Spearman's coefficient between the prediction error and each of the 83 used reliability estimates. Also, return the pvalue of each of 84 the coefficients. 79 Spearman's coefficients between the prediction error and 80 reliability estimates with pvalues. 85 81 """ 86 82 prediction_error = get_prediction_error_list(res) … … 100 96 def get_pearson_r_by_iterations(res): 101 97 """ 102 :param res: results of evaluation, done using learners, 103 wrapped into :class:`Orange.evaluation.reliability.Classifier`. 98 :param res: Evaluation results with :obj:`reliability_estimate`. 104 99 :type res: :class:`Orange.evaluation.testing.ExperimentResults` 105 100 106 Return average Pearson's coefficient over all folds between prediction error107 and each of the used estimates.101 Pearson's coefficients between prediction error 102 and reliability estimates averaged over all folds. 108 103 """ 109 104 results_by_fold = Orange.evaluation.scoring.split_by_iterations(res) … … 112 107 number_of_folds = len(results_by_fold) 113 108 results = [0 for _ in xrange(number_of_estimates)] 109 M 114 110 sig = [0 for _ in xrange(number_of_estimates)] 115 111 method_list = [0 for _ in xrange(number_of_estimates)] … … 199 195 class Estimate: 200 196 """ 201 Reliability estimate. Contains attributes that describe the results of 202 reliability estimation. 197 Describes a reliability estimate. 203 198 204 199 .. attribute:: estimate 205 200 206 A numerical reliability estimate.201 Value of reliability. 207 202 208 203 .. attribute:: signed_or_absolute 209 204 210 Determines whether the method used givesa signed or absolute result.205 Determines whether the method returned a signed or absolute result. 211 206 Has a value of either :obj:`SIGNED` or :obj:`ABSOLUTE`. 212 207 213 208 .. attribute:: method 214 209 215 An integer ID of reliability estimation method used.210 An integer ID of the reliability estimation method used. 216 211 217 212 .. attribute:: method_name 218 213 219 Name (string) of reliability estimation method used.214 Name (string) of the reliability estimation method used. 220 215 221 216 """ … … 281 276 :rtype: :class:`Orange.evaluation.reliability.SensitivityAnalysisClassifier` 282 277 283 To estimate the reliability of prediction for a given instance, 284 the learning set is extended with that instance with the label changes to 278 The learning set is extended with that instancem, where the label is changed to 285 279 :math:`K + \epsilon (l_{max}  l_{min})` (:math:`K` is the initial prediction, 286 280 :math:`\epsilon` a sensitivity parameter, and :math:`l_{min}` and 287 :math:`l_{max}` the lower and upper bounds of labels on training data) 281 :math:`l_{max}` the lower and upper bounds of labels on training data). 288 282 Results for multiple values of :math:`\epsilon` are combined 289 into SAvar and SAbias. SAbias can be used either ina signed or absolute form.283 into SAvar and SAbias. SAbias has a signed or absolute form. 290 284 291 285 :math:`SAvar = \\frac{\sum_{\epsilon \in E}(K_{\epsilon}  K_{\epsilon})}{E}` … … 386 380 """ 387 381 388 :param m: Number of bagg ing models to be used with BAGV estimate382 :param m: Number of bagged models. Default: 50. 389 383 :type m: int 390 384 391 :param for 385 :param for_instances: Optional. If test instances 392 386 are given as a parameter, this class can compute their reliabilities 393 387 on the fly, which saves memory. … … 397 391 :rtype: :class:`Orange.evaluation.reliability.BaggingVarianceClassifier` 398 392 399 400 :math:`m` different bagging models are used to estimate 401 the value of dependent variable for a given instance. For regression, 402 the variance of predictions is a reliability 403 estimate: 393 For regression, BAGV is the variance of predictions: 404 394 405 395 :math:`BAGV = \\frac{1}{m} \sum_{i=1}^{m} (K_i  K)^2`, where … … 407 397 predictions of individual models. 408 398 409 For classification, 1 minus the average Euclidean distance between class 410 probability distributions predicted by the model, and distributions 411 predicted by the individual bagged models, is the BAGV reliability 412 measure. For classification, a greater value implies a better 413 prediction. 414 399 For classification, BAGV is 1 minus the average Euclidean 400 distance between class probability distributions predicted by the 401 model, and distributions predicted by the individual bagged model; 402 a greater value implies a better prediction. 403 415 404 This reliability measure can run out of memory if individual classifiers themselves 416 405 use a lot of memory; it needs :math:`m` times memory … … 502 491 :type distance: function 503 492 504 :param distance_weighted: For classification,493 :param distance_weighted: Relevant only for classification; 505 494 use an average distance between distributions, weighted by :math:`e^{d}`, 506 495 where :math:`d` is the distance between predicted instance and the … … 594 583 :rtype: :class:`Orange.evaluation.reliability.CNeighboursClassifier` 595 584 596 For regression, CNK is defineda difference585 For regression, CNK is a difference 597 586 between average label of its nearest neighbours and the prediction. CNK 598 587 can be either signed or absolute. A greater value implies greater prediction error. … … 1070 1059 class Learner: 1071 1060 """ 1072 Adds reliability estimation to any learner: multiple reliability estimation 1073 algorithms can be used simultaneously. 1074 This learner can be used as any other learner, 1061 Adds reliability estimation to any prediction method. 1062 This class can be used as any other Orange learner, 1075 1063 but returns the classifier wrapped into an instance of 1076 1064 :class:`Orange.evaluation.reliability.Classifier`. 1077 1065 1078 1066 :param box_learner: Learner to wrap into a reliability estimation 1079 1067 classifier. … … 1106 1094 1107 1095 def __call__(self, instances, weight=None, **kwds): 1108 """ Learn from the given table of data instances.1096 """Construct a classifier. 1109 1097 1110 :param instances: Data to learn from.1098 :param instances: Learning data. 1111 1099 :type instances: Orange.data.Table 1112 1100 :param weight: Id of meta attribute with weights of instances … … 1149 1137 def __call__(self, instance, result_type=Orange.core.GetValue): 1150 1138 """ 1151 Classify and estimate reliability of estimationfor a new instance.1139 Classify and estimate reliability for a new instance. 1152 1140 When :obj:`result_type` is set to 1153 1141 :obj:`Orange.classification.Classifier.GetBoth` or
Note: See TracChangeset
for help on using the changeset viewer.