Changeset 9506:da6b728a44de in orange


Ignore:
Timestamp:
01/03/12 10:06:21 (2 years ago)
Author:
anze <anze.staric@…>
Branch:
default
Convert:
8cce121011e1d4a44039957c832cf36547f0d298
Message:

Updated documentation for evaluation.testing.

Files:
2 edited

Legend:

Unmodified
Added
Removed
  • docs/reference/rst/Orange.evaluation.testing.rst

    r9372 r9506  
    1515module :obj:`Orange.evaluation.scoring`. 
    1616 
    17 Your scripts will thus basically conduct experiments using functions in 
    18 :obj:`Orange.evaluation.testing`, covered on this page and then evaluate 
    19 the results by functions in :obj:`Orange.evaluation.scoring`. For those 
    20 interested in writing their own statistical measures of the quality of 
    21 models, description of :obj:`TestedExample` and :obj:`ExperimentResults` 
    22 are available at the end of this page. 
     17Your scripts will thus basically conduct experiments using methods of 
     18:obj:`Evaluation` class and functions in  :obj:`Orange.evaluation.testing`, 
     19covered on this page and then evaluate the results by functions in 
     20:obj:`Orange.evaluation.scoring`. For those interested in writing their own 
     21statistical measures of the quality of models, 
     22description of :obj:`TestedExample` and :obj:`ExperimentResults` are 
     23available at the end of this page. 
    2324 
    2425.. note:: Orange has been "de-randomized". Running the same script twice 
     
    112113      need them. 
    113114 
    114 *pps* 
     115*preprocessors* 
    115116    A list of preprocessors. It consists of tuples ``(c, preprocessor)``, 
    116117    where ``c`` determines whether the preprocessor will be applied 
     
    136137    to be made, where applicable. The default is ``[0.1, 0.2, ..., 1.0]``. 
    137138 
    138 *storeClassifiers (keyword argument)* 
     139*store_classifiers (keyword argument)* 
    139140    If this flag is set, the testing procedure will store the constructed 
    140141    classifiers. For each iteration of the test (eg for each fold in 
     
    146147    classifiers it induces. :: 
    147148 
    148         res = Orange.evaluation.testing.proportion_test(learners, data, 0.7, 100, storeClassifier=1) 
    149  
    150 *verbose (keyword argument)* 
    151     Several functions can report their progress if you add a keyword 
    152     argument ``verbose=1``. 
     149        res = Orange.evaluation.testing.proportion_test(learners, data, 0.7, 
     150        100, store_classifiers=1) 
    153151 
    154152Sampling and Testing Functions 
    155153============================== 
    156154 
    157 .. autofunction:: proportion_test 
    158 .. autofunction:: leave_one_out 
    159 .. autofunction:: cross_validation 
    160 .. autofunction:: test_with_indices 
    161 .. autofunction:: learning_curve 
    162 .. autofunction:: learning_curve_n 
    163 .. autofunction:: learning_curve_with_test_data 
    164 .. autofunction:: learn_and_test_on_test_data 
    165 .. autofunction:: learn_and_test_on_learn_data 
    166 .. autofunction:: test_on_data 
     155.. autoclass:: Evaluation 
     156   :members: 
    167157 
    168158Classes 
  • orange/Orange/evaluation/testing.py

    r9505 r9506  
    128128 
    129129    def create_tested_example(self, fold, example): 
    130         actual = [example.getclass, example.get_classes][self.test_type]() 
     130        actual = example.getclass() if self.test_type == TEST_TYPE_SINGLE \ 
     131                                  else example.get_classes() 
    131132        return TestedExample(fold, 
    132133                             self.converter(actual), 
     
    173174        return str(self.__dict__) 
    174175 
    175  
    176176ExperimentResults = deprecated_members({"classValues": "class_values", 
    177177                                        "classifierNames": "classifier_names", 
     
    179179                                        "numberOfIterations": "number_of_iterations", 
    180180                                        "numberOfLearners": "number_of_learners" 
    181                                         })(ExperimentResults) 
    182  
    183 #### Experimental procedures 
     181})(ExperimentResults) 
     182 
     183     
    184184class Evaluation(object): 
     185    """Common methods for learner evaluation.""" 
     186 
    185187    @deprecated_keywords({"pps": "preprocessors", 
    186188                          "strat": "stratified", 
     
    190192                          "storeClassifiers": "store_classifiers", 
    191193                          "storeExamples": "store_examples"}) 
    192     def cross_validation(self, learners, examples, folds=10, stratified=Orange.core.MakeRandomIndices.StratifiedIfPossible, 
    193                         preprocessors=(), random_generator=0, callback=None, store_classifiers=False, store_examples=False): 
     194    def cross_validation(self, learners, examples, folds=10, 
     195            stratified=Orange.core.MakeRandomIndices.StratifiedIfPossible, 
     196            preprocessors=(), random_generator=0, callback=None, 
     197            store_classifiers=False, store_examples=False): 
    194198        """Perform cross validation with specified number of folds. 
    195199 
     
    199203        :param stratified: sets, whether indices should be stratified 
    200204        :param preprocessors: a list of preprocessors to be used on data. 
    201         :param random_generator: random seed or random generator for selection of indices 
    202         :param callback: a function that will be called after each fold is computed. 
    203         :param store_classifiers: if True, classifiers will be accessible in test_results. 
    204         :param store_examples: if True, examples will be accessible in test_results. 
     205        :param random_generator: random seed or random generator for selection 
     206               of indices 
     207        :param callback: a function that will be called after each fold is 
     208               computed. 
     209        :param store_classifiers: if True, classifiers will be accessible in 
     210               test_results. 
     211        :param store_examples: if True, examples will be accessible in 
     212               test_results. 
     213        :return: :obj:`ExperimentResults` 
    205214        """ 
    206215        (examples, weight) = demangle_examples(examples) 
    207216 
    208         indices = Orange.core.MakeRandomIndicesCV(examples, folds, stratified=stratified, random_generator=random_generator) 
    209         return self.test_with_indices(learners=learners, examples=(examples, weight), indices=indices, 
    210                                  preprocessors=preprocessors, 
    211                                  callback=callback, store_classifiers=store_classifiers, store_examples=store_examples) 
     217        indices = Orange.core.MakeRandomIndicesCV(examples, folds, 
     218            stratified=stratified, random_generator=random_generator) 
     219 
     220        return self.test_with_indices( 
     221            learners=learners, 
     222            examples=(examples, weight), 
     223            indices=indices, 
     224            preprocessors=preprocessors, 
     225            callback=callback, 
     226            store_classifiers=store_classifiers, 
     227            store_examples=store_examples) 
    212228 
    213229 
     
    216232                          "storeExamples": "store_examples"}) 
    217233    def leave_one_out(self, learners, examples, preprocessors=(), 
    218                       callback=None, store_classifiers=False, store_examples=False): 
     234            callback=None, store_classifiers=False, store_examples=False): 
    219235        """Perform leave-one-out evaluation of learners on a data set. 
    220236 
     
    222238        :param examples: data table on which the learners will be tested 
    223239        :param preprocessors: a list of preprocessors to be used on data. 
    224         :param callback: a function that will be called after each fold is computed. 
    225         :param store_classifiers: if True, classifiers will be accessible in test_results. 
    226         :param store_examples: if True, examples will be accessible in test_results. 
     240        :param callback: a function that will be called after each fold is 
     241               computed. 
     242        :param store_classifiers: if True, classifiers will be accessible in 
     243               test_results. 
     244        :param store_examples: if True, examples will be accessible in 
     245               test_results. 
     246        :return: :obj:`ExperimentResults` 
    227247        """ 
    228248        return self.test_with_indices(learners, examples, indices=range(len(examples)), preprocessors=preprocessors, 
    229249                                 callback=callback, store_classifiers=store_classifiers, store_examples=store_examples) 
    230      
    231     def check_test_type(self, instances, learners): 
    232         learner_is_mlc = [isinstance(l, Orange.multilabel.MultiLabelLearner) 
    233                           for l in learners] 
    234         multi_label = any(learner_is_mlc) 
    235         if multi_label and not all(learner_is_mlc): 
    236             raise ValueError("Test on mixed types of learners (MLC and non-MLC) not possible") 
    237          
    238         if multi_label and not instances.domain.class_vars: 
    239             raise ValueError("Test data with multiple labels (class vars) expected") 
    240         if not multi_label and not instances.domain.class_var: 
    241             raise ValueError("Test data set without class attributes") 
    242          
    243         return [TEST_TYPE_SINGLE, TEST_TYPE_MLC][multi_label] 
    244  
    245      
     250 
    246251    @deprecated_keywords({"storeExamples": "store_examples", 
    247252                          "storeClassifiers": "store_classifiers=True", 
    248253                          "pps":"preprocessors"}) 
    249254    def test_with_indices(self, learners, examples, indices, preprocessors=(), 
    250                           callback=None, store_classifiers=False, store_examples=False, **kwargs): 
    251         """ 
    252         Perform a cross-validation-like test. Examples for each fold are selected 
    253         based on given indices. 
     255            callback=None, store_classifiers=False, store_examples=False, 
     256            **kwargs): 
     257        """ 
     258        Perform a cross-validation-like test. Examples for each fold are 
     259        selected based on given indices. 
    254260 
    255261        :param learners: list of learners to be tested 
    256262        :param examples: data table on which the learners will be tested 
    257263        :param indices: a list of integers that defines, which examples will be 
    258          used for testing in each fold. The number of indices should be equal to 
    259          the number of examples. 
    260         :param preprocessors: a list of preprocessors to be used on data. 
    261         :param callback: a function that will be called after each fold is computed. 
     264               used for testing in each fold. The number of indices should be 
     265                equal to the number of examples. 
     266        :param preprocessors: a list of preprocessors to be used on data. 
     267        :param callback: a function that will be called after each fold is 
     268               computed. 
    262269        :param store_classifiers: if True, classifiers will be accessible in test_results. 
    263270        :param store_examples: if True, examples will be accessible in test_results. 
     271        :return: :obj:`ExperimentResults` 
    264272        """ 
    265273        examples, weight = demangle_examples(examples) 
     
    334342        :param store_classifiers: if True, classifiers will be accessible in test_results. 
    335343        :param store_examples: if True, examples will be accessible in test_results. 
     344        :return: :obj:`ExperimentResults` 
    336345        """ 
    337346 
     
    383392        :param store_classifiers: if True, classifiers will be accessible in test_results. 
    384393        :param store_examples: if True, examples will be accessible in test_results. 
     394        :return: :obj:`ExperimentResults` 
    385395        """ 
    386396        learn_set, learn_weight = demangle_examples(learn_set) 
     
    433443        :param store_classifiers: if True, classifiers will be accessible in test_results. 
    434444        :param store_examples: if True, examples will be accessible in test_results. 
     445        :return: :obj:`ExperimentResults` 
    435446        """ 
    436447        pick = Orange.core.MakeRandomIndices2(stratified = stratification, p0 = learning_proportion, randomGenerator = random_generator) 
     
    465476        return test_results 
    466477 
    467     def learning_curve(self, learners, examples, cv=None, pick=None, proportions=Orange.core.frange(0.1), 
     478    @deprecated_keywords({"storeExamples": "store_examples", 
     479                          "storeClassifiers": "store_classifiers", 
     480                          "learnProp": "learning_proportion", 
     481                          "strat": "stratification", 
     482                          "pps": "preprocessors", 
     483                          "indicesrandseed": "random_generator", 
     484                          "randseed": "random_generator", 
     485                          "randomGenerator": "random_generator"}) 
     486    def learning_curve(self, learners, examples, cv_indices=None, proportion_indices=None, proportions=Orange.core.frange(0.1), 
    468487                       preprocessors=(), random_generator=0, callback=None): 
    469488        """ 
    470         Computes learning curves using a procedure recommended by Salzberg 
    471         (1997). It first prepares data subsets (folds). For each proportion, 
    472         it performs the cross-validation, but taking only a proportion of 
    473         examples for learning. 
    474  
    475         Arguments ``cv`` and ``pick`` give the methods for preparing 
    476         indices for cross-validation and random selection of learning 
    477         examples. If they are not given, :obj:`orange.MakeRandomIndicesCV` and 
    478         :obj:`orange.MakeRandomIndices2` are used, both will be stratified and the 
    479         cross-validation will be 10-fold. Proportions is a list of proportions 
    480         of learning examples. 
    481  
    482         The function can save time by loading experimental existing data for 
    483         any test that were already conducted and saved. Also, the computed 
    484         results are stored for later use. You can enable this by adding 
    485         a keyword argument ``cache=1``. Another keyword deals with progress 
    486         report. If you add ``verbose=1``, the function will print the proportion 
    487         and the fold number. 
    488  
    489         """ 
    490         if cv is None: 
    491             cv = Orange.core.MakeRandomIndicesCV(folds=10, stratified=Orange.core.MakeRandomIndices.StratifiedIfPossible, randomGenerator = random_generator) 
    492         if pick is None: 
    493             pick = Orange.core.MakeRandomIndices2(stratified=Orange.core.MakeRandomIndices.StratifiedIfPossible, randomGenerator = random_generator) 
     489        Compute a learning curve using multiple cross-validations where 
     490        models are trained on different portions of the training data. 
     491 
     492        :param learners: list of learners to be tested 
     493        :param examples: a dataset used for evaluation 
     494        :param cv_indices: indices used for crossvalidation 
     495        :param proportion_indices: indices for proportion selection 
     496        :param proportions: proportions of train data to be used 
     497        :param preprocessors: a list of preprocessors to be used on data. 
     498        :param callback: a function that is be called after each classifier is computed. 
     499        :return: list of :obj:`ExperimentResults` 
     500        """ 
     501        if cv_indices is None: 
     502            cv_indices = Orange.core.MakeRandomIndicesCV(folds=10, stratified=Orange.core.MakeRandomIndices.StratifiedIfPossible, randomGenerator = random_generator) 
     503        if proportion_indices is None: 
     504            proportion_indices = Orange.core.MakeRandomIndices2(stratified=Orange.core.MakeRandomIndices.StratifiedIfPossible, randomGenerator = random_generator) 
    494505 
    495506        examples, weight = demangle_examples(examples) 
    496         indices = cv(examples) 
     507        indices = cv_indices(examples) 
    497508         
    498509        all_results=[] 
    499510        for p in proportions: 
    500511            def select_proportion_preprocessor(examples): 
    501                 return examples.selectref(pick(examples, p0=p), 0) 
     512                return examples.selectref(proportion_indices(examples, p0=p), 0) 
    502513 
    503514            test_results = self.test_with_indices(learners, examples, indices, 
     
    509520 
    510521    def learning_curve_n(self, learners, examples, folds=10, 
    511                        strat=Orange.core.MakeRandomIndices.StratifiedIfPossible, 
    512                        proportions=Orange.core.frange(0.1), pps=[], **argkw): 
    513         """Construct a learning curve for learners. 
    514  
    515         A simpler interface for the function :obj:`learning_curve`. Instead 
    516         of methods for preparing indices, it simply takes the number of folds 
    517         and a flag telling whether we want a stratified cross-validation or 
    518         not. This function does not return a single :obj:`ExperimentResults` but 
    519         a list of them, one for each proportion. :: 
    520  
    521             prop = [0.2, 0.4, 0.6, 0.8, 1.0] 
    522             res = Orange.evaluation.testing.learning_curve_n(learners, data, folds = 5, proportions = prop) 
    523             for i, p in enumerate(prop): 
    524                 print "%5.3f:" % p, 
    525                 printResults(res[i]) 
    526  
    527         This function basically prepares a random generator and example selectors 
    528         (``cv`` and ``pick``) and calls :obj:`learning_curve`. 
    529  
     522                       proportions=Orange.core.frange(0.1), preprocessors=(), callback=None): 
     523        """ 
     524        Compute a learning curve where each cross-validation has given number of folds 
     525        and models are trained on specified proportion of training data. 
     526 
     527        :param learners: list of learners to be tested 
     528        :param examples: a dataset used for evaluation 
     529        :param folds: number of folds for cross-validation 
     530        :param proportions: proportions of train data to be used 
     531        :param preprocessors: a list of preprocessors to be used on data. 
     532        :param callback: a function that is called after each classifier is computed. 
     533        :return: list of :obj:`ExperimentResults` 
    530534        """ 
    531535 
     
    544548        return learning_curve(learners, examples, cv, pick, proportions, pps, **argkw) 
    545549     
    546     def learning_curve_with_test_data(self, learners, learn_set, test_set, times=10, 
    547                                   proportions=Orange.core.frange(0.1), 
    548                                   stratification=Orange.core.MakeRandomIndices.StratifiedIfPossible, pps=[], random_generator=0): 
    549         """ 
    550         This function is suitable for computing a learning curve on datasets, 
    551         where learning and testing examples are split in advance. For each 
    552         proportion of learning examples, it randomly select the requested 
    553         number of learning examples, builds the models and tests them on the 
    554         entire testset. The whole test is repeated for the given number of 
    555         times for each proportion. The result is a list of :obj:`ExperimentResults`, 
    556         one for each proportion. 
    557  
    558         In the following scripts, examples are pre-divided onto training 
    559         and testing set. Learning curves are computed in which 20, 40, 60, 
    560         80 and 100 percents of the examples in the former set are used for 
    561         learning and the latter set is used for testing. Random selection 
    562         of the given proportion of learning set is repeated for five times. 
    563  
    564         .. literalinclude:: code/testing-test.py 
    565             :start-after: Learning curve with pre-separated data 
    566             :end-before: # End 
    567  
    568  
     550    def learning_curve_with_test_data(self, learners, learn_set, test_set, 
     551            times=10, proportions=Orange.core.frange(0.1), 
     552            stratification=Orange.core.MakeRandomIndices.StratifiedIfPossible, 
     553            preprocessors=(), random_generator=0): 
     554        """ 
     555        Compute a learning curve given two datasets. Models are learned on 
     556        proportion of the first dataset and then used to make predictions for 
     557        the second dataset. 
     558 
     559        :param learners: list of learners to be tested 
     560        :param examples: a dataset used for evaluation 
     561        :param folds: number of folds for cross-validation 
     562        :param proportions: proportions of train data to be used 
     563        :param preprocessors: a list of preprocessors to be used on data. 
     564        :param callback: a function that is called after each classifier is computed. 
     565        :return: list of :obj:`ExperimentResults` 
    569566        """ 
    570567        learn_set, learn_weight = demangle_examples(learn_set) 
     
    611608 
    612609        examples, weight = demangle_examples(examples) 
    613         test_type = self.check_test_type(examples, learners) 
     610        test_type = self.check_test_type(examples, classifiers) 
    614611 
    615612        test_results = ExperimentResults(1, 
     
    697694                return "*" 
    698695        return pps 
     696 
     697    def check_test_type(self, instances, models): 
     698        model_is_mlc = [isinstance(m, Orange.multilabel.MultiLabelLearner) or 
     699                        isinstance(m, Orange.multilabel.MultiLabelClassifier) 
     700                          for m in models] 
     701        multi_label = any(model_is_mlc) 
     702        if multi_label and not all(model_is_mlc): 
     703            raise ValueError("Test on mixed types of learners (MLC and non-MLC) not possible") 
     704 
     705        if multi_label and not instances.domain.class_vars: 
     706            raise ValueError("Test data with multiple labels (class vars) expected") 
     707        if not multi_label and not instances.domain.class_var: 
     708            raise ValueError("Test data set without class attributes") 
     709 
     710        return TEST_TYPE_MLC if multi_label else TEST_TYPE_SINGLE 
    699711     
    700712default_evaluation = _default_evaluation = Evaluation() 
Note: See TracChangeset for help on using the changeset viewer.