# Changeset 54:2a1c28cec845 in orange-reliability

Ignore:
Timestamp:
10/07/13 16:34:49 (6 months ago)
Branch:
default
Message:

Files:
2 edited

Unmodified
Removed
• ## docs/rst/Orange.evaluation.reliability.rst

 r53 Reliability assessment aims to predict reliabilities of individual predictions. Most of implemented algorithms for regression described in [Bosnic2008]_ and in [Pevec2011]_ for classification. predictions. Most of the implemented algorithms for regression are described in [Bosnic2008]_; the algorithms for classification are described in [Pevec2011]_. We can use reliability estimation with any Orange learners. The following example: We can use reliability estimation with any prediction method. The following example: * Constructs reliability estimators (implemented in this module), * The :obj:Learner wrapper combines a regular learner, here a :obj:~Orange.classification.knn.kNNLearner, with reliability estimators. * Obtains prediction probabilities from the constructed classifier (:obj:Orange.classification.Classifier.GetBoth option). The resulting probabilities have an additional attribute, :obj:reliability_estimate, * The :obj:Learner wrapper combines a a prediction method (learner), here a :obj:~Orange.classification.knn.kNNLearner, with reliability estimators. * Obtains prediction probabilities, which have an additional attribute, :obj:reliability_estimate, that contains a list of :class:Orange.evaluation.reliability.Estimate. :lines: 7- We could also evaluate more examples. The next example prints reliability estimates The next example prints reliability estimates for first 10 instances (with cross-validation): A dictionary that that maps reliability estimation method IDs (integerss) to method names (strings). method IDs (integers) to method names (strings). .. autoclass:: Estimate
• ## orangecontrib/reliability/__init__.py

 r53 def get_pearson_r(res): """ :param res: results of evaluation, done using learners, wrapped into :class:Orange.evaluation.reliability.Classifier. :param res: Evaluation results with :obj:reliability_estimate. :type res: :class:Orange.evaluation.testing.ExperimentResults Return Pearson's coefficient between the prediction error and each of the used reliability estimates. Also, return the p-value of each of the coefficients. Pearson's coefficients between the prediction error and reliability estimates with p-values. """ prediction_error = get_prediction_error_list(res) def get_spearman_r(res): """ :param res: results of evaluation, done using learners, wrapped into :class:Orange.evaluation.reliability.Classifier. :param res: Evaluation results with :obj:reliability_estimate. :type res: :class:Orange.evaluation.testing.ExperimentResults Return Spearman's coefficient between the prediction error and each of the used reliability estimates. Also, return the p-value of each of the coefficients. Spearman's coefficients between the prediction error and reliability estimates with p-values. """ prediction_error = get_prediction_error_list(res) def get_pearson_r_by_iterations(res): """ :param res: results of evaluation, done using learners, wrapped into :class:Orange.evaluation.reliability.Classifier. :param res: Evaluation results with :obj:reliability_estimate. :type res: :class:Orange.evaluation.testing.ExperimentResults Return average Pearson's coefficient over all folds between prediction error and each of the used estimates. Pearson's coefficients between prediction error and reliability estimates averaged over all folds. """ results_by_fold = Orange.evaluation.scoring.split_by_iterations(res) number_of_folds = len(results_by_fold) results = [0 for _ in xrange(number_of_estimates)] M sig = [0 for _ in xrange(number_of_estimates)] method_list = [0 for _ in xrange(number_of_estimates)] class Estimate: """ Reliability estimate. Contains attributes that describe the results of reliability estimation. Describes a reliability estimate. .. attribute:: estimate A numerical reliability estimate. Value of reliability. .. attribute:: signed_or_absolute Determines whether the method used gives a signed or absolute result. Determines whether the method returned a signed or absolute result. Has a value of either :obj:SIGNED or :obj:ABSOLUTE. .. attribute:: method An integer ID of reliability estimation method used. An integer ID of the reliability estimation method used. .. attribute:: method_name Name (string) of reliability estimation method used. Name (string) of the reliability estimation method used. """ :rtype: :class:Orange.evaluation.reliability.SensitivityAnalysisClassifier To estimate the reliability of prediction for a given instance, the learning set is extended with that instance with the label changes to The learning set is extended with that instancem, where the label is changed to :math:K + \epsilon (l_{max} - l_{min}) (:math:K is  the initial prediction, :math:\epsilon a sensitivity parameter, and :math:l_{min} and :math:l_{max} the lower and upper bounds of labels on training data) :math:l_{max} the lower and upper bounds of labels on training data). Results for multiple values of :math:\epsilon are combined into SAvar and SAbias. SAbias can be used either in a signed or absolute form. into SAvar and SAbias. SAbias has a signed or absolute form. :math:SAvar = \\frac{\sum_{\epsilon \in E}(K_{\epsilon} - K_{-\epsilon})}{|E|} """ :param m: Number of bagging models to be used with BAGV estimate :param m: Number of bagged models. Default: 50. :type m: int :param for instances:  Optional. If test instances :param for_instances:  Optional. If test instances are given as a parameter, this class can compute their reliabilities on the fly, which saves memory. :rtype: :class:Orange.evaluation.reliability.BaggingVarianceClassifier :math:m different bagging models are used to estimate the value of dependent variable for a given instance. For regression, the variance of predictions is a reliability estimate: For regression, BAGV is the variance of predictions: :math:BAGV = \\frac{1}{m} \sum_{i=1}^{m} (K_i - K)^2, where predictions of individual models. For classification, 1 minus the average Euclidean distance between class probability distributions predicted by the model, and distributions predicted by the individual bagged models, is the BAGV reliability measure. For classification, a greater value implies a better prediction. For classification, BAGV is 1 minus the average Euclidean distance between class probability distributions predicted by the model, and distributions predicted by the individual bagged model; a greater value implies a better prediction. This reliability measure can run out of memory if individual classifiers themselves use a lot of memory; it needs :math:m times memory :type distance: function :param distance_weighted: For classification, :param distance_weighted: Relevant only for classification; use an average distance between distributions, weighted by :math:e^{-d}, where :math:d is the distance between predicted instance and the :rtype: :class:Orange.evaluation.reliability.CNeighboursClassifier For regression, CNK is defined a difference For regression, CNK is a difference between average label of its nearest neighbours and the prediction. CNK can be either signed or absolute. A greater value implies greater prediction error. class Learner: """ Adds reliability estimation to any learner: multiple reliability estimation algorithms can be used simultaneously. This learner can be used as any other learner, Adds reliability estimation to any prediction method. This class can be used as any other Orange learner, but returns the classifier wrapped into an instance of :class:Orange.evaluation.reliability.Classifier. :param box_learner: Learner to wrap into a reliability estimation classifier. def __call__(self, instances, weight=None, **kwds): """Learn from the given table of data instances. """Construct a classifier. :param instances: Data to learn from. :param instances: Learning data. :type instances: Orange.data.Table :param weight: Id of meta attribute with weights of instances def __call__(self, instance, result_type=Orange.core.GetValue): """ Classify and estimate reliability of estimation for a new instance. Classify and estimate reliability for a new instance. When :obj:result_type is set to :obj:Orange.classification.Classifier.GetBoth or
Note: See TracChangeset for help on using the changeset viewer.