Changeset 7944:e4b6c5bda633 in orange
 Timestamp:
 05/26/11 12:13:16 (3 years ago)
 Branch:
 default
 Convert:
 8147848132e28d3fb4d4e36b633d1275521cfb38
 File:

 1 edited
Legend:
 Unmodified
 Added
 Removed

orange/Orange/evaluation/reliability.py
r7923 r7944 29 29 30 30 To estimate the reliabilty for given example we extend the learning set 31 with given example and labeling it with K + e(l\ :sub:`max` \  l\ :sub:`min` \),32 where K denotes the initial prediction, eis sensitivity parameter and33 l\ :sub:`min` \ and l\ :sub:`max` \denote lower and the upper bound of31 with given example and labeling it with :math:`K + \epsilon (l_{max}  l_{min})`, 32 where K denotes the initial prediction, :math:`\epsilon` is sensitivity parameter and 33 :math:`l_{min}` and :math:`l_{max}` denote lower and the upper bound of 34 34 the learning examples. After computing different sensitivity predictions 35 35 using different values of e, the prediction are combined into SAvar and SAbias. 36 36 SAbias can be used as signed estimate or as absolute value of SAbias. 37 37 38 :math:`SAvar = \\frac{\sum_{\epsilon \in E}(K_{\epsilon}  K_{\epsilon})}{E}` 39 40 :math:`SAbias = \\frac{\sum_{\epsilon \in E} (K_{\epsilon}  K ) + (K_{\epsilon}  K)}{2 E}` 41 38 42 Variance of bagged models (BAGV) 39 43  40 44 41 45 We construct m different bagging models of the original chosen learner and use 42 those predictions of given example to calculate the variance, which we use as46 those predictions (:math:`K_i, i = 1, ..., m`) of given example to calculate the variance, which we use as 43 47 reliability estimator. 48 49 :math:`BAGV = \\frac{1}{m} \sum_{i=1}^{m} (K_i  K)^2` 50 51 where 52 53 :math:`K = \\frac{\sum_{i=1}^{m} K_i}{m}` 44 54 45 55 Local cross validation reliability estimate (LCV) … … 51 61 weighted absolute prediction error. 52 62 63 1. define the set of k nearest neighours :math:`N = { (x_1, x_1),..., (x_k, c_k)}` 64 2. FOR EACH :math:`(x_i, c_i) \in N` 65 66 2.1. generare model M on :math:`N \\backslash (x_i, c_i)` 67 68 2.2. for :math:`(x_i, c_i)` compute LOO prediction :math:`K_i` 69 70 2.3. for :math:`(x_i, c_i)` compute LOO error :math:`E_i =  C_i  K_i ` 71 72 3. :math:`LCV(x) = \\frac{ \sum_{(x_i, c_i) \in N} d(x_i, x) * E_i }{ \sum_{(x_i, c_i) \in N} d(x_i, x) }` 73 53 74 Local modeling of prediction error (CNK) 54 75  … … 57 78 average label of the nearest neighbours and the examples prediction. CNK can 58 79 be used as a signed estimate or only as absolute value. 80 81 :math:`CNK = \\frac{\sum_{i=1}^{k}C_i}{k}  K` 82 83 Where k denotes number of neighbors, C :sub:`i` denotes neighbours' labels and 84 K denotes the example's prediction. 59 85 60 86 Bagging variance cneighbours (BVCK) … … 278 304 :param weight: Id of meta attribute with weights of instances 279 305 :type weight: integer 280 :rtype: :class:` estimator.Classifier`306 :rtype: :class:`Orange.evaluation.reliability.Classifier` 281 307 """ 282 308 return Classifier(examples, self.learner, self.e, self.m, self.cnk_k, … … 440 466 SAvar /= len(self.e) 441 467 SAbias /= 2*len(self.e) 442 probabilities.reliability_estimate.append( (SAvar, ABSOLUTE, SAVAR_ABSOLUTE) )443 probabilities.reliability_estimate.append( (SAbias, SIGNED, SABIAS_SIGNED) )444 probabilities.reliability_estimate.append( (SAbias, ABSOLUTE, SABIAS_ABSOLUTE) )468 probabilities.reliability_estimate.append( Estimate(SAvar, ABSOLUTE, SAVAR_ABSOLUTE) ) 469 probabilities.reliability_estimate.append( Estimate(SAbias, SIGNED, SABIAS_SIGNED) ) 470 probabilities.reliability_estimate.append( Estimate(SAbias, ABSOLUTE, SABIAS_ABSOLUTE) ) 445 471 446 472 … … 462 488 463 489 if self.use[DO_BAGV]: 464 probabilities.reliability_estimate.append( (BAGV, ABSOLUTE, BAGV_ABSOLUTE) )490 probabilities.reliability_estimate.append( Estimate(BAGV, ABSOLUTE, BAGV_ABSOLUTE) ) 465 491 466 492 # For each of the classifiers … … 491 517 492 518 if self.use[DO_CNK]: 493 probabilities.reliability_estimate.append( (CNK, SIGNED, CNK_SIGNED) )494 probabilities.reliability_estimate.append( (CNK, ABSOLUTE, CNK_ABSOLUTE) )519 probabilities.reliability_estimate.append( Estimate(CNK, SIGNED, CNK_SIGNED) ) 520 probabilities.reliability_estimate.append( Estimate(CNK, ABSOLUTE, CNK_ABSOLUTE) ) 495 521 496 522 # Calculate local crossvalidation reliability estimate … … 524 550 LCV = LCVer / LCVdi if LCVdi != 0 else 0 525 551 526 probabilities.reliability_estimate.append( (LCV, ABSOLUTE, LCV_ABSOLUTE) )552 probabilities.reliability_estimate.append( Estimate(LCV, ABSOLUTE, LCV_ABSOLUTE) ) 527 553 528 554 # BVCK … … 534 560 BVCK = (BAGV + abs(CNK)) / 2 535 561 536 probabilities.reliability_estimate.append( (BVCK, ABSOLUTE, BVCK_ABSOLUTE) )562 probabilities.reliability_estimate.append( Estimate(BVCK, ABSOLUTE, BVCK_ABSOLUTE) ) 537 563 538 564 # Mahalanobis distance to 3 closest neighbours … … 544 570 mahalanobis_distance = sum(ex[self.mid].value for ex in self.nnm(example, 3)) 545 571 546 probabilities.reliability_estimate.append( (mahalanobis_distance, ABSOLUTE, MAHAL_ABSOLUTE) )572 probabilities.reliability_estimate.append( Estimate(mahalanobis_distance, ABSOLUTE, MAHAL_ABSOLUTE) ) 547 573 548 574 #probabilities.reliability_estimate = [SAvar, SAbias, BAGV, CNK, LCV, BVCK, mahalanobis_distance] … … 551 577 method = [SAvar, SAbias, SAbias, BAGV, CNK, CNK, LCV, BVCK, mahalanobis_distance] 552 578 ICV = method[self.icv_method] 553 probabilities.reliability_estimate.append( (ICV, self.icv_signed_or_absolute, 10, self.icv_method))579 probabilities.reliability_estimate.append( Estimate(ICV, self.icv_signed_or_absolute, 10, self.icv_method)) 554 580 555 581 if result_type == Orange.core.GetValue: … … 561 587 562 588 def get_reliability_estimation_list(res, i): 563 return [result.probabilities[0].reliability_estimate[i] [0] for result in res.results], res.results[0].probabilities[0].reliability_estimate[i][1], res.results[0].probabilities[0].reliability_estimate[i][2]589 return [result.probabilities[0].reliability_estimate[i].estimate for result in res.results], res.results[0].probabilities[0].reliability_estimate[i].signed_or_absolute, res.results[0].probabilities[0].reliability_estimate[i].method 564 590 565 591 def get_prediction_error_list(res): … … 628 654 return statc.betai (df * 0.5, 0.5, df/(df + t*t)) 629 655 656 class Estimate: 657 def __init__(self, estimate, signed_or_absolute, method, icv_method = 1): 658 self.estimate = estimate 659 self.signed_or_absolute = signed_or_absolute 660 self.method = method 661 self.method_name = METHOD_NAME[method] 662 self.icv_method = icv_method 663 self.icv_method_name = METHOD_NAME[icv_method] if icv_method != 1 else ""
Note: See TracChangeset
for help on using the changeset viewer.