Changeset 10414:ac84efa944dc in orange for Orange/evaluation/testing.py
 Timestamp:
 03/02/12 00:05:35 (2 years ago)
 Branch:
 default
 File:

 1 edited
Legend:
 Unmodified
 Added
 Removed

Orange/evaluation/testing.py
r10234 r10414 12 12 class TestedExample: 13 13 """ 14 TestedExample stores predictions of different classifiers for a single testing example. 15 16 :var classes: A list of predictions of type Value, one for each classifier. 17 :var probabilities: A list of probabilities of classes, one for each classifier. 18 :var iteration_number: Iteration number (e.g. fold) in which the TestedExample was created/tested. 19 :var actual_class: The correct class of the example 20 :var weight: Example's weight. Even if the example set was not weighted, this attribute is present and equals 1.0. 14 TestedExample stores predictions of different classifiers for a 15 single testing data instance. 16 17 .. attribute:: classes 18 19 A list of predictions of type Value, one for each classifier. 20 21 .. attribute:: probabilities 22 23 A list of probabilities of classes, one for each classifier. 24 25 .. attribute:: iteration_number 26 27 Iteration number (e.g. fold) in which the TestedExample was 28 created/tested. 29 30 .. attribute actual_class 31 32 The correct class of the example 33 34 .. attribute weight 35 36 Instance's weight; 1.0 if data was not weighted 21 37 """ 22 38 … … 37 53 38 54 def add_result(self, aclass, aprob): 39 """Append sa new result (class and probability prediction by a single classifier) to the classes and probabilities field."""55 """Append a new result (class and probability prediction by a single classifier) to the classes and probabilities field.""" 40 56 41 57 if isinstance(aclass, (list, tuple)): … … 50 66 51 67 def set_result(self, i, aclass, aprob): 52 """Set sthe result of the ith classifier to the given values."""68 """Set the result of the ith classifier to the given values.""" 53 69 if isinstance(aclass, (list, tuple)): 54 70 self.classes[i] = aclass … … 72 88 ``ExperimentResults`` stores results of one or more repetitions of 73 89 some test (cross validation, repeated sampling...) under the same 74 circumstances. 75 76 :var results: A list of instances of :obj:`TestedExample`, one for each example in the dataset. 77 :var classifiers: A list of classifiers, one element for each repetition (eg. fold). Each element is a list 78 of classifiers, one for each learner. This field is used only if storing is enabled by ``storeClassifiers=1``. 79 :var number_of_iterations: Number of iterations. This can be the number of folds (in cross validation) 80 or the number of repetitions of some test. :obj:`TestedExample`'s attribute ``iteration_number`` should 81 be in range ``[0, number_of_iterations1]``. 82 :var number_of_learners: Number of learners. Lengths of lists classes and probabilities in each :obj:`TestedExample` 83 should equal ``number_of_learners``. 84 :var loaded: If the experimental method supports caching and there are no obstacles for caching (such as unknown 85 random seeds), this is a list of boolean values. Each element corresponds to a classifier and tells whether the 86 experimental results for that classifier were computed or loaded from the cache. 87 :var weights: A flag telling whether the results are weighted. If ``False``, weights are still present 88 in :obj:`TestedExample`, but they are all ``1.0``. Clear this flag, if your experimental procedure ran on weighted 89 testing examples but you would like to ignore the weights in statistics. 90 circumstances. Instances of this class are constructed by sampling 91 and testing functions from module :obj:`Orange.evaluation.testing` 92 and used by methods in module :obj:`Orange.evaluation.scoring`. 93 94 .. attribute:: results 95 96 A list of instances of :obj:`TestedExample`, one for each 97 example in the dataset. 98 99 .. attribute:: number_of_iterations 100 101 Number of iterations. This can be the number of folds (in 102 cross validation) or the number of repetitions of some 103 test. :obj:`TestedExample`'s attribute ``iteration_number`` 104 should be in range ``[0, number_of_iterations1]``. 105 106 .. attribute:: number_of_learners 107 108 Number of learners. Lengths of lists classes and probabilities 109 in each :obj:`TestedExample` should equal 110 ``number_of_learners``. 111 112 .. attribute:: classifier_names 113 114 Stores the names of the classifiers. 115 116 .. attribute:: classifiers 117 118 A list of classifiers, one element for each iteration of 119 sampling and learning (eg. fold). Each element is a list of 120 classifiers, one for each learner. For instance, 121 ``classifiers[2][4]`` refers to the 3rd repetition, 5th 122 learning algorithm. 123 124 Note that functions from :obj:`~Orange.evaluation.testing` 125 only store classifiers it enabled by setting 126 ``storeClassifiers`` to ``1``. 127 128 .. 129 .. attribute:: loaded 130 131 If the experimental method supports caching and there are no 132 obstacles for caching (such as unknown random seeds), this is a 133 list of boolean values. Each element corresponds to a classifier 134 and tells whether the experimental results for that classifier 135 were computed or loaded from the cache. 136 137 .. attribute:: base_class 138 139 The reference class for measures like AUC. 140 141 .. attribute:: class_values 142 143 The list of class values. 144 145 .. attribute:: weights 146 147 A flag telling whether the results are weighted. If ``False``, 148 weights are still present in :obj:`TestedExample`, but they 149 are all ``1.0``. Clear this flag, if your experimental 150 procedure ran on weighted testing examples but you would like 151 to ignore the weights in statistics. 90 152 """ 91 153 @deprecated_keywords({"classifierNames": "classifier_names", … … 198 260 preprocessors=(), random_generator=0, callback=None, 199 261 store_classifiers=False, store_examples=False): 200 """Perform cross validation with specified number of folds. 201 202 :param learners: list of learners to be tested 203 :param examples: data table on which the learners will be tested 204 :param folds: number of folds to perform 205 :param stratified: sets, whether indices should be stratified 206 :param preprocessors: a list of preprocessors to be used on data. 207 :param random_generator: :obj:`Orange.misc.RandomGenerator` object. 208 :param callback: a function that will be called after each fold is 209 computed. 210 :param store_classifiers: if True, classifiers will be accessible in 211 test_results. 212 :param store_examples: if True, examples will be accessible in 213 test_results. 262 """Cross validation test with specified number of folds. 263 264 :param learners: list of learning algorithms 265 :param examples: data instances used for training and testing 266 :param folds: number of folds 267 :param stratified: tells whether to stratify the sampling 268 :param preprocessors: a list of preprocessors to be used on data (obsolete) 269 :param random_generator: random seed or generator (see above) 270 :param callback: a function that is called after finishing each fold 271 :param store_classifiers: if ``True``, classifiers are stored in results 272 :param store_examples: if ``True``, examples are stored in results 214 273 :return: :obj:`ExperimentResults` 215 274 """ … … 234 293 def leave_one_out(self, learners, examples, preprocessors=(), 235 294 callback=None, store_classifiers=False, store_examples=False): 236 """Perform leaveoneout evaluation of learners on a data set. 237 238 :param learners: list of learners to be tested 239 :param examples: data table on which the learners will be tested 240 :param preprocessors: a list of preprocessors to be used on data. 241 :param callback: a function that will be called after each fold is 242 computed. 243 :param store_classifiers: if True, classifiers will be accessible in 244 test_results. 245 :param store_examples: if True, examples will be accessible in 246 test_results. 295 """Leaveoneout evaluation of learning algorithms. 296 297 :param learners: list of learning algorithms 298 :param examples: data instances used for training and testing 299 :param preprocessors: a list of preprocessors (obsolete) 300 :param callback: a function that is called after finishing each fold 301 :param store_classifiers: if ``True``, classifiers are stored in results 302 :param store_examples: if ``True``, examples are stored in results 247 303 :return: :obj:`ExperimentResults` 248 304 """ … … 257 313 "pps":"preprocessors"}) 258 314 def test_with_indices(self, learners, examples, indices, preprocessors=(), 259 callback=None, store_classifiers=False, store_examples=False, 260 **kwargs): 315 callback=None, store_classifiers=False, store_examples=False): 261 316 """ 262 317 Perform a crossvalidationlike test. Examples for each fold are 263 318 selected based on given indices. 264 319 265 :param learners: list of learners to be tested 266 :param examples: data table on which the learners will be tested 267 :param indices: a list of integers that defines, which examples will be 268 used for testing in each fold. The number of indices should be 269 equal to the number of examples. 270 :param preprocessors: a list of preprocessors to be used on data. 271 :param callback: a function that will be called after each fold is 272 computed. 273 :param store_classifiers: if True, classifiers will be accessible in test_results. 274 :param store_examples: if True, examples will be accessible in test_results. 320 :param learners: list of learning algorithms 321 :param examples: data instances used for training and testing 322 :param indices: a list of integer indices that sort examples into folds; each index corresponds to an example from ``examples`` 323 :param preprocessors: a list of preprocessors (obsolete) 324 :param callback: a function that is called after each fold 325 :param store_classifiers: if ``True``, classifiers are stored in results 326 :param store_examples: if ``True``, examples are stored in results 275 327 :return: :obj:`ExperimentResults` 276 328 """ … … 279 331 raise ValueError("Test data set with no examples") 280 332 test_type = self.check_test_type(examples, learners) 281 if "cache" in kwargs:282 raise ValueError("This feature is no longer supported.")283 333 284 334 niterations = max(indices)+1 … … 310 360 311 361 def one_fold_with_indices(self, learners, examples, fold, indices, preprocessors=(), weight=0): 312 """ Perform one fold of crossvalidation like procedure using provided indices."""362 """Similar to :obj:`test_with_indices` except that it performs single fold of crossvalidation, given by argument ``fold``.""" 313 363 learn_set = examples.selectref(indices, fold, negate=1) 314 364 test_set = examples.selectref(indices, fold, negate=0) … … 338 388 callback=None, store_classifiers=False, store_examples=False): 339 389 """ 340 Perform a test where learners are trained and testedon the same data.341 342 :param learners: list of learn ers to be tested343 :param examples: data table on which the learners will be tested344 :param preprocessors: a list of preprocessors to be used on data.345 :param callback: a function that will be called after each fold is computed.346 :param store_classifiers: if True, classifiers will be accessible in test_results.347 :param store_examples: if True, examples will be accessible in test_results.390 Train learning algorithms and test them on the same data. 391 392 :param learners: list of learning algorithms 393 :param examples: data instances used for training and testing 394 :param preprocessors: a list of preprocessors (obsolete) 395 :param callback: a function that is called after each learning 396 :param store_classifiers: if ``True``, classifiers are stored in results 397 :param store_examples: if ``True``, examples are stored in results 348 398 :return: :obj:`ExperimentResults` 349 399 """ … … 386 436 callback=None, store_classifiers=False, store_examples=False): 387 437 """ 388 Perform a test, where learners are trained on one dataset and tested 389 on another. 390 391 :param learners: list of learners to be tested 392 :param learn_set: a dataset used for training 393 :param test_set: a dataset used for testing 394 :param preprocessors: a list of preprocessors to be used on data. 395 :param callback: a function that is be called after each classifier is computed. 396 :param store_classifiers: if True, classifiers will be accessible in test_results. 397 :param store_examples: if True, examples will be accessible in test_results. 438 Train learning algorithms on one data sets and test them on another. 439 440 :param learners: list of learning algorithms 441 :param learn_set: training instances 442 :param test_set: testing instances 443 :param preprocessors: a list of preprocessors (obsolete) 444 :param callback: a function that is called after each learning 445 :param store_classifiers: if ``True``, classifiers are stored in results 446 :param store_examples: if ``True``, examples are stored in results 398 447 :return: :obj:`ExperimentResults` 399 448 """ … … 436 485 callback=None, store_classifiers=False, store_examples=False): 437 486 """ 438 Perform a test, where learners are trained and tested on different data sets. Training and test sets are439 generated by proportionally splitting examples. 440 441 :param learners: list of learners to be tested442 :param examples: a dataset used for evaluation443 :param learning_proportion: proportion of examples to be used for training444 :param times: number of test repetitions445 :param stratification: use stratification when constructing train and test sets.446 :param preprocessors: a list of preprocessors to be used on data.447 :param callback: a function that is be called after each classifier is computed.448 :param store_classifiers: if True, classifiers will be accessible in test_results.449 :param store_examples: if True, examples will be accessible in test_results.487 Iteratively split the data into training and testing set, and train and test the learnign algorithms. 488 489 :param learners: list of learning algorithms 490 :param examples: data instances used for training and testing 491 :param learning_proportion: proportion of data used for training 492 :param times: number of iterations 493 :param stratification: use stratified sampling 494 :param preprocessors: a list of preprocessors (obsolete) 495 :param random_generator: random seed or generator (see above) 496 :param callback: a function that is called after each fold 497 :param store_classifiers: if ``True``, classifiers are stored in results 498 :param store_examples: if ``True``, examples are stored in results 450 499 :return: :obj:`ExperimentResults` 451 500 """ … … 499 548 models are trained on different portions of the training data. 500 549 501 :param learners: list of learners to be tested 502 :param examples: a dataset used for evaluation 503 :param cv_indices: indices used for crossvalidation 504 :param proportion_indices: indices for proportion selection 505 :param proportions: proportions of train data to be used 506 :param preprocessors: a list of preprocessors to be used on data. 507 :param callback: a function that is be called after each classifier is computed. 550 :param learners: list of learning algorithms 551 :param examples: data instances used for training and testing 552 :param cv_indices: indices used for cross validation (leave ``None`` for 10fold CV) 553 :param proportion_indices: indices for proportion selection (leave ``None`` to let the function construct the folds) 554 :param proportions: list of proportions of data used for training 555 :param preprocessors: a list of preprocessors (obsolete) 556 :param random_generator: random seed or generator (see above) 557 :param callback: a function that is be called after each learning 508 558 :return: list of :obj:`ExperimentResults` 509 559 """ … … 540 590 random_generator=0, callback=None): 541 591 """ 542 Compute a learning curve where each crossvalidation has given number of folds 543 and models are trained on specified proportion of training data. 544 545 :param learners: list of learners to be tested 546 :param examples: a dataset used for evaluation 592 Compute a learning curve using multiple crossvalidations where 593 models are trained on different portions of the training data. 594 Similar to :obj:`learning_curve` except for simpler arguments. 595 596 :param learners: list of learning algorithms 597 :param examples: data instances used for training and testing 547 598 :param folds: number of folds for crossvalidation 548 :param proportions: proportions of train data to be used 549 :param preprocessors: a list of preprocessors to be used on data. 550 :param callback: a function that is called after each classifier is computed. 599 :param proportions: list of proportions of data used for training 600 :param stratification: use stratified sampling 601 :param preprocessors: a list of preprocessors (obsolete) 602 :param random_generator: random seed or generator (see above) 603 :param callback: a function that is be called after each learning 551 604 :return: list of :obj:`ExperimentResults` 552 605 """ … … 566 619 """ 567 620 Compute a learning curve given two datasets. Models are learned on 568 proportion of the first dataset and then used to make predictions for 569 the second dataset. 570 571 :param learners: list of learners to be tested 572 :param learn_set: a dataset used for evaluation 573 :param test_set: a dataset used for evaluation 574 :param proportions: proportions of train data to be used 575 :param preprocessors: a list of preprocessors to be used on data. 621 proportion of the first dataset and then tested on the second. 622 623 :param learners: list of learning algorithms 624 :param learn_set: training data 625 :param test_set: testing data 626 :param times: number of iterations 627 :param straitification: use stratified sampling 628 :param proportions: a list of proportions of training data to be used 629 :param preprocessors: a list of preprocessors (obsolete) 630 :param random_generator: random seed or generator (see above) 631 :param store_classifiers: if ``True``, classifiers are stored in results 632 :param store_examples: if ``True``, examples are stored in results 576 633 :return: list of :obj:`ExperimentResults` 577 634 """ … … 615 672 def test_on_data(self, classifiers, examples, store_classifiers=False, store_examples=False): 616 673 """ 617 Test classifiers on examples618 619 :param classifiers: classifiers to test620 :param examples: examples to test on621 :param store_classifiers: if True, classifiers will be accessible in test_results.622 :param store_examples: if True, examples will be accessible in test_results.674 Test classifiers on the given data 675 676 :param classifiers: a list of classifiers 677 :param examples: testing data 678 :param store_classifiers: if ``True``, classifiers are stored in results 679 :param store_examples: if ``True``, examples are stored in results 623 680 """ 624 681 … … 687 744 688 745 def _preprocess_data(self, learn_set, test_set, preprocessors): 689 """Apply preprocessors to learn and test dataset """746 """Apply preprocessors to learn and test dataset (obsolete)""" 690 747 for p_type, preprocessor in preprocessors: 691 748 if p_type == "B":
Note: See TracChangeset
for help on using the changeset viewer.