Changeset 10414:ac84efa944dc in orange


Ignore:
Timestamp:
03/02/12 00:05:35 (2 years ago)
Author:
janezd <janez.demsar@…>
Branch:
default
Message:

Half-polished Orange.evaluation documentation

Files:
3 edited

Legend:

Unmodified
Added
Removed
  • Orange/evaluation/testing.py

    r10234 r10414  
    1212class TestedExample: 
    1313    """ 
    14     TestedExample stores predictions of different classifiers for a single testing example. 
    15  
    16     :var classes: A list of predictions of type Value, one for each classifier. 
    17     :var probabilities: A list of probabilities of classes, one for each classifier. 
    18     :var iteration_number: Iteration number (e.g. fold) in which the TestedExample was created/tested. 
    19     :var actual_class: The correct class of the example 
    20     :var weight: Example's weight. Even if the example set was not weighted, this attribute is present and equals 1.0. 
     14    TestedExample stores predictions of different classifiers for a 
     15    single testing data instance. 
     16 
     17    .. attribute:: classes 
     18 
     19        A list of predictions of type Value, one for each classifier. 
     20 
     21    .. attribute:: probabilities 
     22 
     23        A list of probabilities of classes, one for each classifier. 
     24 
     25    .. attribute:: iteration_number 
     26 
     27        Iteration number (e.g. fold) in which the TestedExample was 
     28        created/tested. 
     29 
     30    .. attribute actual_class 
     31 
     32        The correct class of the example 
     33 
     34    .. attribute weight 
     35 
     36        Instance's weight; 1.0 if data was not weighted 
    2137    """ 
    2238 
     
    3753 
    3854    def add_result(self, aclass, aprob): 
    39         """Appends a new result (class and probability prediction by a single classifier) to the classes and probabilities field.""" 
     55        """Append a new result (class and probability prediction by a single classifier) to the classes and probabilities field.""" 
    4056     
    4157        if isinstance(aclass, (list, tuple)): 
     
    5066 
    5167    def set_result(self, i, aclass, aprob): 
    52         """Sets the result of the i-th classifier to the given values.""" 
     68        """Set the result of the i-th classifier to the given values.""" 
    5369        if isinstance(aclass, (list, tuple)): 
    5470            self.classes[i] = aclass 
     
    7288    ``ExperimentResults`` stores results of one or more repetitions of 
    7389    some test (cross validation, repeated sampling...) under the same 
    74     circumstances. 
    75  
    76     :var results: A list of instances of :obj:`TestedExample`, one for each example in the dataset. 
    77     :var classifiers: A list of classifiers, one element for each repetition (eg. fold). Each element is a list 
    78       of classifiers, one for each learner. This field is used only if storing is enabled by ``storeClassifiers=1``. 
    79     :var number_of_iterations: Number of iterations. This can be the number of folds (in cross validation) 
    80       or the number of repetitions of some test. :obj:`TestedExample`'s attribute ``iteration_number`` should 
    81       be in range ``[0, number_of_iterations-1]``. 
    82     :var number_of_learners: Number of learners. Lengths of lists classes and probabilities in each :obj:`TestedExample` 
    83       should equal ``number_of_learners``. 
    84     :var loaded: If the experimental method supports caching and there are no obstacles for caching (such as unknown 
    85       random seeds), this is a list of boolean values. Each element corresponds to a classifier and tells whether the 
    86       experimental results for that classifier were computed or loaded from the cache. 
    87     :var weights: A flag telling whether the results are weighted. If ``False``, weights are still present 
    88       in :obj:`TestedExample`, but they are all ``1.0``. Clear this flag, if your experimental procedure ran on weighted 
    89       testing examples but you would like to ignore the weights in statistics. 
     90    circumstances. Instances of this class are constructed by sampling 
     91    and testing functions from module :obj:`Orange.evaluation.testing` 
     92    and used by methods in module :obj:`Orange.evaluation.scoring`. 
     93 
     94    .. attribute:: results 
     95 
     96        A list of instances of :obj:`TestedExample`, one for each 
     97        example in the dataset. 
     98 
     99    .. attribute:: number_of_iterations 
     100 
     101        Number of iterations. This can be the number of folds (in 
     102        cross validation) or the number of repetitions of some 
     103        test. :obj:`TestedExample`'s attribute ``iteration_number`` 
     104        should be in range ``[0, number_of_iterations-1]``. 
     105 
     106    .. attribute:: number_of_learners 
     107 
     108        Number of learners. Lengths of lists classes and probabilities 
     109        in each :obj:`TestedExample` should equal 
     110        ``number_of_learners``. 
     111 
     112    .. attribute:: classifier_names 
     113 
     114        Stores the names of the classifiers. 
     115 
     116    .. attribute:: classifiers 
     117 
     118        A list of classifiers, one element for each iteration of 
     119        sampling and learning (eg. fold). Each element is a list of 
     120        classifiers, one for each learner. For instance, 
     121        ``classifiers[2][4]`` refers to the 3rd repetition, 5th 
     122        learning algorithm. 
     123 
     124        Note that functions from :obj:`~Orange.evaluation.testing` 
     125        only store classifiers it enabled by setting 
     126        ``storeClassifiers`` to ``1``. 
     127 
     128    .. 
     129        .. attribute:: loaded 
     130 
     131            If the experimental method supports caching and there are no 
     132            obstacles for caching (such as unknown random seeds), this is a 
     133            list of boolean values. Each element corresponds to a classifier 
     134            and tells whether the experimental results for that classifier 
     135            were computed or loaded from the cache. 
     136 
     137    .. attribute:: base_class 
     138 
     139       The reference class for measures like AUC. 
     140 
     141    .. attribute:: class_values 
     142 
     143        The list of class values. 
     144 
     145    .. attribute:: weights 
     146 
     147        A flag telling whether the results are weighted. If ``False``, 
     148        weights are still present in :obj:`TestedExample`, but they 
     149        are all ``1.0``. Clear this flag, if your experimental 
     150        procedure ran on weighted testing examples but you would like 
     151        to ignore the weights in statistics. 
    90152    """ 
    91153    @deprecated_keywords({"classifierNames": "classifier_names", 
     
    198260            preprocessors=(), random_generator=0, callback=None, 
    199261            store_classifiers=False, store_examples=False): 
    200         """Perform cross validation with specified number of folds. 
    201  
    202         :param learners: list of learners to be tested 
    203         :param examples: data table on which the learners will be tested 
    204         :param folds: number of folds to perform 
    205         :param stratified: sets, whether indices should be stratified 
    206         :param preprocessors: a list of preprocessors to be used on data. 
    207         :param random_generator: :obj:`Orange.misc.RandomGenerator` object. 
    208         :param callback: a function that will be called after each fold is 
    209                computed. 
    210         :param store_classifiers: if True, classifiers will be accessible in 
    211                test_results. 
    212         :param store_examples: if True, examples will be accessible in 
    213                test_results. 
     262        """Cross validation test with specified number of folds. 
     263 
     264        :param learners: list of learning algorithms 
     265        :param examples: data instances used for training and testing 
     266        :param folds: number of folds 
     267        :param stratified: tells whether to stratify the sampling 
     268        :param preprocessors: a list of preprocessors to be used on data (obsolete) 
     269        :param random_generator: random seed or generator (see above) 
     270        :param callback: a function that is called after finishing each fold 
     271        :param store_classifiers: if ``True``, classifiers are stored in results 
     272        :param store_examples: if ``True``, examples are stored in results 
    214273        :return: :obj:`ExperimentResults` 
    215274        """ 
     
    234293    def leave_one_out(self, learners, examples, preprocessors=(), 
    235294            callback=None, store_classifiers=False, store_examples=False): 
    236         """Perform leave-one-out evaluation of learners on a data set. 
    237  
    238         :param learners: list of learners to be tested 
    239         :param examples: data table on which the learners will be tested 
    240         :param preprocessors: a list of preprocessors to be used on data. 
    241         :param callback: a function that will be called after each fold is 
    242                computed. 
    243         :param store_classifiers: if True, classifiers will be accessible in 
    244                test_results. 
    245         :param store_examples: if True, examples will be accessible in 
    246                test_results. 
     295        """Leave-one-out evaluation of learning algorithms. 
     296 
     297        :param learners: list of learning algorithms 
     298        :param examples: data instances used for training and testing 
     299        :param preprocessors: a list of preprocessors (obsolete) 
     300        :param callback: a function that is called after finishing each fold 
     301        :param store_classifiers: if ``True``, classifiers are stored in results 
     302        :param store_examples: if ``True``, examples are stored in results 
    247303        :return: :obj:`ExperimentResults` 
    248304        """ 
     
    257313                          "pps":"preprocessors"}) 
    258314    def test_with_indices(self, learners, examples, indices, preprocessors=(), 
    259             callback=None, store_classifiers=False, store_examples=False, 
    260             **kwargs): 
     315            callback=None, store_classifiers=False, store_examples=False): 
    261316        """ 
    262317        Perform a cross-validation-like test. Examples for each fold are 
    263318        selected based on given indices. 
    264319 
    265         :param learners: list of learners to be tested 
    266         :param examples: data table on which the learners will be tested 
    267         :param indices: a list of integers that defines, which examples will be 
    268                used for testing in each fold. The number of indices should be 
    269                equal to the number of examples. 
    270         :param preprocessors: a list of preprocessors to be used on data. 
    271         :param callback: a function that will be called after each fold is 
    272                computed. 
    273         :param store_classifiers: if True, classifiers will be accessible in test_results. 
    274         :param store_examples: if True, examples will be accessible in test_results. 
     320        :param learners: list of learning algorithms 
     321        :param examples: data instances used for training and testing 
     322        :param indices: a list of integer indices that sort examples into folds; each index corresponds to an example from ``examples`` 
     323        :param preprocessors: a list of preprocessors (obsolete) 
     324        :param callback: a function that is called after each fold 
     325        :param store_classifiers: if ``True``, classifiers are stored in results 
     326        :param store_examples: if ``True``, examples are stored in results 
    275327        :return: :obj:`ExperimentResults` 
    276328        """ 
     
    279331            raise ValueError("Test data set with no examples") 
    280332        test_type = self.check_test_type(examples, learners) 
    281         if "cache" in kwargs: 
    282             raise ValueError("This feature is no longer supported.") 
    283333 
    284334        niterations = max(indices)+1 
     
    310360 
    311361    def one_fold_with_indices(self, learners, examples, fold, indices, preprocessors=(), weight=0): 
    312         """Perform one fold of cross-validation like procedure using provided indices.""" 
     362        """Similar to :obj:`test_with_indices` except that it performs single fold of cross-validation, given by argument ``fold``.""" 
    313363        learn_set = examples.selectref(indices, fold, negate=1) 
    314364        test_set = examples.selectref(indices, fold, negate=0) 
     
    338388                                     callback=None, store_classifiers=False, store_examples=False): 
    339389        """ 
    340         Perform a test where learners are trained and tested on the same data. 
    341  
    342         :param learners: list of learners to be tested 
    343         :param examples: data table on which the learners will be tested 
    344         :param preprocessors: a list of preprocessors to be used on data. 
    345         :param callback: a function that will be called after each fold is computed. 
    346         :param store_classifiers: if True, classifiers will be accessible in test_results. 
    347         :param store_examples: if True, examples will be accessible in test_results. 
     390        Train learning algorithms and test them on the same data. 
     391 
     392        :param learners: list of learning algorithms 
     393        :param examples: data instances used for training and testing 
     394        :param preprocessors: a list of preprocessors (obsolete) 
     395        :param callback: a function that is called after each learning 
     396        :param store_classifiers: if ``True``, classifiers are stored in results 
     397        :param store_examples: if ``True``, examples are stored in results 
    348398        :return: :obj:`ExperimentResults` 
    349399        """ 
     
    386436                                    callback=None, store_classifiers=False, store_examples=False): 
    387437        """ 
    388         Perform a test, where learners are trained on one dataset and tested 
    389         on another. 
    390  
    391         :param learners: list of learners to be tested 
    392         :param learn_set: a dataset used for training 
    393         :param test_set: a dataset used for testing 
    394         :param preprocessors: a list of preprocessors to be used on data. 
    395         :param callback: a function that is be called after each classifier is computed. 
    396         :param store_classifiers: if True, classifiers will be accessible in test_results. 
    397         :param store_examples: if True, examples will be accessible in test_results. 
     438        Train learning algorithms on one data sets and test them on another. 
     439 
     440        :param learners: list of learning algorithms 
     441        :param learn_set: training instances 
     442        :param test_set: testing instances 
     443        :param preprocessors: a list of preprocessors (obsolete) 
     444        :param callback: a function that is called after each learning 
     445        :param store_classifiers: if ``True``, classifiers are stored in results 
     446        :param store_examples: if ``True``, examples are stored in results 
    398447        :return: :obj:`ExperimentResults` 
    399448        """ 
     
    436485                   callback=None, store_classifiers=False, store_examples=False): 
    437486        """ 
    438         Perform a test, where learners are trained and tested on different data sets. Training and test sets are 
    439         generated by proportionally splitting examples. 
    440  
    441         :param learners: list of learners to be tested 
    442         :param examples: a dataset used for evaluation 
    443         :param learning_proportion: proportion of examples to be used for training 
    444         :param times: number of test repetitions 
    445         :param stratification: use stratification when constructing train and test sets. 
    446         :param preprocessors: a list of preprocessors to be used on data. 
    447         :param callback: a function that is be called after each classifier is computed. 
    448         :param store_classifiers: if True, classifiers will be accessible in test_results. 
    449         :param store_examples: if True, examples will be accessible in test_results. 
     487        Iteratively split the data into training and testing set, and train and test the learnign algorithms. 
     488 
     489        :param learners: list of learning algorithms 
     490        :param examples: data instances used for training and testing 
     491        :param learning_proportion: proportion of data used for training 
     492        :param times: number of iterations 
     493        :param stratification: use stratified sampling 
     494        :param preprocessors: a list of preprocessors (obsolete) 
     495        :param random_generator: random seed or generator (see above) 
     496        :param callback: a function that is called after each fold 
     497        :param store_classifiers: if ``True``, classifiers are stored in results 
     498        :param store_examples: if ``True``, examples are stored in results 
    450499        :return: :obj:`ExperimentResults` 
    451500        """ 
     
    499548        models are trained on different portions of the training data. 
    500549 
    501         :param learners: list of learners to be tested 
    502         :param examples: a dataset used for evaluation 
    503         :param cv_indices: indices used for crossvalidation 
    504         :param proportion_indices: indices for proportion selection 
    505         :param proportions: proportions of train data to be used 
    506         :param preprocessors: a list of preprocessors to be used on data. 
    507         :param callback: a function that is be called after each classifier is computed. 
     550        :param learners: list of learning algorithms 
     551        :param examples: data instances used for training and testing 
     552        :param cv_indices: indices used for cross validation (leave ``None`` for 10-fold CV) 
     553        :param proportion_indices: indices for proportion selection (leave ``None`` to let the function construct the folds) 
     554        :param proportions: list of proportions of data used for training 
     555        :param preprocessors: a list of preprocessors (obsolete) 
     556        :param random_generator: random seed or generator (see above) 
     557        :param callback: a function that is be called after each learning 
    508558        :return: list of :obj:`ExperimentResults` 
    509559        """ 
     
    540590                       random_generator=0, callback=None): 
    541591        """ 
    542         Compute a learning curve where each cross-validation has given number of folds 
    543         and models are trained on specified proportion of training data. 
    544  
    545         :param learners: list of learners to be tested 
    546         :param examples: a dataset used for evaluation 
     592        Compute a learning curve using multiple cross-validations where 
     593        models are trained on different portions of the training data. 
     594        Similar to :obj:`learning_curve` except for simpler arguments. 
     595 
     596        :param learners: list of learning algorithms 
     597        :param examples: data instances used for training and testing 
    547598        :param folds: number of folds for cross-validation 
    548         :param proportions: proportions of train data to be used 
    549         :param preprocessors: a list of preprocessors to be used on data. 
    550         :param callback: a function that is called after each classifier is computed. 
     599        :param proportions: list of proportions of data used for training 
     600        :param stratification: use stratified sampling 
     601        :param preprocessors: a list of preprocessors (obsolete) 
     602        :param random_generator: random seed or generator (see above) 
     603        :param callback: a function that is be called after each learning 
    551604        :return: list of :obj:`ExperimentResults` 
    552605        """ 
     
    566619        """ 
    567620        Compute a learning curve given two datasets. Models are learned on 
    568         proportion of the first dataset and then used to make predictions for 
    569         the second dataset. 
    570  
    571         :param learners: list of learners to be tested 
    572         :param learn_set: a dataset used for evaluation 
    573         :param test_set: a dataset used for evaluation 
    574         :param proportions: proportions of train data to be used 
    575         :param preprocessors: a list of preprocessors to be used on data. 
     621        proportion of the first dataset and then tested on the second. 
     622 
     623        :param learners: list of learning algorithms 
     624        :param learn_set: training data 
     625        :param test_set: testing data 
     626        :param times: number of iterations 
     627        :param straitification: use stratified sampling 
     628        :param proportions: a list of proportions of training data to be used 
     629        :param preprocessors: a list of preprocessors (obsolete) 
     630        :param random_generator: random seed or generator (see above) 
     631        :param store_classifiers: if ``True``, classifiers are stored in results 
     632        :param store_examples: if ``True``, examples are stored in results 
    576633        :return: list of :obj:`ExperimentResults` 
    577634        """ 
     
    615672    def test_on_data(self, classifiers, examples, store_classifiers=False, store_examples=False): 
    616673        """ 
    617         Test classifiers on examples 
    618  
    619         :param classifiers: classifiers to test 
    620         :param examples: examples to test on 
    621         :param store_classifiers: if True, classifiers will be accessible in test_results. 
    622         :param store_examples: if True, examples will be accessible in test_results. 
     674        Test classifiers on the given data 
     675 
     676        :param classifiers: a list of classifiers 
     677        :param examples: testing data 
     678        :param store_classifiers: if ``True``, classifiers are stored in results 
     679        :param store_examples: if ``True``, examples are stored in results 
    623680        """ 
    624681 
     
    687744     
    688745    def _preprocess_data(self, learn_set, test_set, preprocessors): 
    689         """Apply preprocessors to learn and test dataset""" 
     746        """Apply preprocessors to learn and test dataset (obsolete)""" 
    690747        for p_type, preprocessor in preprocessors: 
    691748            if p_type == "B": 
  • docs/reference/rst/Orange.evaluation.rst

    r9372 r10414  
    33############################# 
    44 
     5Evaluation of prediction modules is split into two parts. Module 
     6:obj:`Orange.evaluation.testing` contains procedures that sample data, 
     7train learning algorithms and test models. All procedures return 
     8results as an instance of 
     9:obj:`~Orange.evaluation.testing.ExperimentResults` that is described 
     10below. Module :obj:`Orange.evaluation.scoring` uses such data to 
     11compute various performance scores like classification accuracy and 
     12AUC. 
     13 
     14There is a third module, which is unrelated to this 
     15scheme,:obj:`Orange.evaluation.reliability`, that assesses the reliability 
     16of individual predictions. 
     17 
    518.. toctree:: 
    619   :maxdepth: 1 
    720 
     21   Orange.evaluation.testing 
    822   Orange.evaluation.scoring 
    9    Orange.evaluation.testing 
    1023   Orange.evaluation.reliability 
    1124 
     25Classes for storing the experimental results 
     26-------------------------------------------- 
     27 
     28 
     29The following two classes are used for storing the results of experiments by :obj:`Orange.evaluation.testing` and computing of scores by :obj:`Orange.evaluation.scoring`. Instances of this class seldom need to be constructed and used outside of these two modules. 
     30 
     31.. py:currentmodule:: Orange.evaluation.testing 
     32 
     33.. autoclass:: ExperimentResults(iterations, classifier_names, class_values=None, weights=None, base_class=-1) 
     34    :members: 
     35 
     36.. autoclass:: TestedExample 
     37    :members: 
  • docs/reference/rst/Orange.evaluation.testing.rst

    r10339 r10414  
    66================================== 
    77 
    8 There are many ways to test prediction models on data. Orange includes 
    9 methods for cross-validation, leave-one out, random sampling and learning 
    10 curves. This methods handle learning of models and prediction of new 
    11 examples; they return :obj:`ExperimentResults` which can be passed to 
    12 :obj:`~Orange.evaluation.scoring` functions to evaluate model. 
     8Module :obj:`Orange.evaluation.testing` contains methods for 
     9cross-validation, leave-one out, random sampling and learning 
     10curves. These procedures split the data onto training and testing set 
     11and use the training data to induce models; models then make 
     12predictions for testing data. Predictions are collected in 
     13:obj:`ExperimentResults`, together with the actual classes and some 
     14other data. The latter can be given to functions 
     15:obj:`~Orange.evaluation.scoring` that compute the performance scores 
     16of models. 
    1317 
    1418.. literalinclude:: code/testing-example.py 
     19 
     20The following call makes 100 iterations of 70:30 test and stores all the 
     21induced classifiers. :: 
     22 
     23     res = Orange.evaluation.testing.proportion_test(learners, iris, 0.7, 100, store_classifiers=1) 
    1524 
    1625Different evaluation techniques are implemented as instance methods of 
    1726:obj:`Evaluation` class. For ease of use, an instance of this class is 
    1827created at module loading time and instance methods are exposed as functions 
    19 with the same name in Orange.evaluation.testing namespace. 
     28in :obj:`Orange.evaluation.testing`. 
     29 
     30Randomness in tests 
     31=================== 
     32 
     33If evaluation method uses random sampling, parameter 
     34``random_generator`` can be used to either provide either a random 
     35seed or an instance of :obj:`~Orange.misc.Random`. If omitted, a new 
     36instance of random generator is constructed for each call of the 
     37method with random seed 0. 
     38 
     39.. note:: 
     40 
     41    Running the same script twice will generally give the same 
     42    results. 
     43 
     44For conducting a repeatable set of experiments, construct an instance 
     45of :obj:`~Orange.misc.Random` and pass it to all of them. This way, 
     46all methods will use different random numbers, but they will be the 
     47same for each run of the script. 
     48 
     49For truly random number, set seed to a random number generated with 
     50python random generator. Since python's random generator is reset each 
     51time python is loaded with current system time as seed, results of the 
     52script will be different each time you run it. 
    2053 
    2154.. autoclass:: Evaluation 
    22    :members: 
    2355 
    24 .. autoclass:: ExperimentResults 
    25     :members: 
     56   .. automethod:: cross_validation 
    2657 
    27 .. autoclass:: TestedExample 
    28     :members: 
     58   .. automethod:: leave_one_out 
    2959 
    30 Generating random numbers 
    31 ========================= 
     60   .. automethod:: proportion_test 
    3261 
    33 Many evaluation 
     62   .. automethod:: test_with_indices 
    3463 
    35 *stratified* 
    36     Tells whether to stratify the random selections. Its default value is 
    37     :obj:`orange.StratifiedIfPossible` which stratifies selections 
    38     if the class variable is discrete and has no unknown values. 
     64   .. automethod:: one_fold_with_indices 
    3965 
    40 *random_generator* 
    41     If evaluation method relies on randomness, parameter ``random_generator`` 
    42     can be used to either provide a random seed or an instance of 
    43     :obj:`~Orange.misc.Random` which will be used to generate random numbers. 
     66   .. automethod:: learn_and_test_on_learn_data 
    4467 
    45     By default, a new instance of random generator is constructed for each 
    46     call of the method with random seed 0. 
     68   .. automethod:: learn_and_test_on_test_data 
    4769 
    48     If you use more than one method that is based on randomness, 
    49     you can construct an instance of :obj:`~Orange.misc.Random` and pass it 
    50     to all of them. This way, all methods will use different random numbers, 
    51     but this numbers will be the same for each run of the script. 
     70   .. automethod:: learning_curve(learners, examples, cv_indices=None, proportion_indices=None, proportions=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], preprocessors=(), random_generator=0, callback=None)¶ 
    5271 
    53     For truly random number, set seed to a random number generated with 
    54     python random generator. Since python's random generator is reset each 
    55     time python is loaded with current system time as seed, 
    56     results of the script will be different each time you run it. 
     72   .. automethod:: learning_curve_n(learners, examples, folds=10, proportions=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], stratification=StratifiedIfPossible, preprocessors=(), random_generator=0, callback=None) 
    5773 
    58 *preprocessors* 
    59     A list of preprocessors. It consists of tuples ``(c, preprocessor)``, 
    60     where ``c`` determines whether the preprocessor will be applied 
    61     to the learning set (``"L"``), test set (``"T"``) or to both 
    62     (``"B"``). The latter is applied first, when the example set is still 
    63     undivided. The ``"L"`` and ``"T"`` preprocessors are applied on the 
    64     separated subsets. Preprocessing testing examples is allowed only 
    65     on experimental procedures that do not report the TestedExample's 
    66     in the same order as examples in the original set. The second item 
    67     in the tuple, preprocessor can be either a pure Orange or a pure 
    68     Python preprocessor, that is, any function or callable class that 
    69     accepts a table of examples and weight, and returns a preprocessed 
    70     table and weight. 
     74   .. automethod:: learning_curve_with_test_data(learners, learn_set, test_set, times=10, proportions=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0], stratification=StratifiedIfPossible, preprocessors=(), random_generator=0, store_classifiers=False, store_examples=False) 
    7175 
    72     This example will demonstrate the devastating effect of 100% class 
    73     noise on learning. :: 
     76   .. automethod:: test_on_data 
    7477 
    75         classnoise = orange.Preprocessor_addClassNoise(proportion=1.0) 
    76         res = Orange.evaluation.testing.proportion_test(learners, data, 0.7, 100, pps = [("L", classnoise)]) 
    77  
    78 *store_classifiers (keyword argument)* 
    79     If this flag is set, the testing procedure will store the constructed 
    80     classifiers. For each iteration of the test (eg for each fold in 
    81     cross validation, for each left out example in leave-one-out...), 
    82     the list of classifiers is appended to the ExperimentResults' 
    83     field classifiers. 
    84  
    85     The script below makes 100 repetitions of 70:30 test and store the 
    86     classifiers it induces. :: 
    87  
    88         res = Orange.evaluation.testing.proportion_test(learners, data, 0.7, 
    89         100, store_classifiers=1) 
    90  
    91  
    92 Knowing classes :obj:`TestedExample` that stores results of testing 
    93 for a single test example and :obj:`ExperimentResults` that stores a list of 
    94 TestedExamples along with some other data on experimental procedures 
    95 and classifiers used, is important if you would like to write your own 
    96 measures of quality of models, compatible the sampling infrastructure 
    97 provided by Orange. If not, you can skip the remainder of this page. 
    98  
    99  
    100  
    101 References 
    102 ========== 
    103  
    104 Salzberg, S. L. (1997). On comparing classifiers: Pitfalls to avoid 
    105 and a recommended approach. Data Mining and Knowledge Discovery 1, 
    106 pages 317-328. 
    107  
Note: See TracChangeset for help on using the changeset viewer.