Changeset 41:d2bc01f57cf8 in orange-reliability for docs/rst/Orange.evaluation.reliability.rst


Ignore:
Timestamp:
10/03/13 14:50:28 (7 months ago)
Author:
markotoplak
Branch:
default
Message:

Documentation updates

File:
1 edited

Legend:

Unmodified
Added
Removed
  • docs/rst/Orange.evaluation.reliability.rst

    r40 r41  
    1414******************************************************** 
    1515 
    16 Reliability assessment statistically predicts reliability of single 
    17 predictions. Most of implemented algorithms for regression are taken from 
    18 Comparison of approaches for estimating reliability of individual 
    19 regression predictions, Zoran Bosnić, 2008. Implementations for 
    20 classification follow descriptions in Evaluating Reliability of Single 
    21 Classifications of Neural Networks, Darko Pevec, 2011. 
     16Reliability assessment aims to predict reliabilities of individual 
     17predictions.  
    2218 
    23 The following example shows basic usage of reliability estimation methods: 
     19Most of implemented algorithms for regression described in 
     20"Comparison of approaches for estimating reliability of individual 
     21regression predictions, Zoran Bosnić, 2008" for regression and in 
     22in "Evaluating Reliability of Single 
     23Classifications of Neural Networks, Darko Pevec, 2011" for classification. 
     24 
     25We can use reliability estimation with any Orange learners. The following example: 
     26 
     27 * Constructs reliability estimators (implemented in this module), 
     28 * Combines a regular learner. 
     29   (:class:`~Orange.classification.knn.kNNLearner` in this case) with 
     30   reliability estimators. 
     31 * Obtains prediction probabilities from the constructed classifier 
     32   (:obj:`Orange.classification.Classifier.GetBoth` option). The resulting 
     33   probabilities have and additional attribute, :obj:`reliability_estimate` 
     34   attribute, :class:`Orange.evaluation.reliability.Estimate`. 
    2435 
    2536.. literalinclude:: code/reliability-basic.py 
    2637    :lines: 7- 
    2738 
    28 The important points of this example are: 
    29  * construction of reliability estimators using classes, 
    30    implemented in this module, 
    31  * construction of a reliability learner that bonds a regular learner 
    32    (:class:`~Orange.classification.knn.kNNLearner` in this case) with 
    33    reliability estimators, 
    34  * calling the constructed classifier with 
    35    :obj:`Orange.classification.Classifier.GetBoth` option to obtain class 
    36    probabilities; :obj:`probability` is the object that gets appended the 
    37    :obj:`reliability_estimate` attribute, an instance of 
    38    :class:`Orange.evaluation.reliability.Estimate`, by the reliability learner. 
    39  
    40 It is also possible to do reliability estimation on whole data 
    41 table, not only on single instance. Next example demonstrates usage of a 
    42 cross-validation technique for reliability estimation. Reliability estimations 
    43 for first 10 instances get printed: 
     39We could also evaluate more examples. The next example prints reliability estimates 
     40for first 10 instances (with cross-validation): 
    4441 
    4542.. literalinclude:: code/reliability-run.py 
     
    4946=================== 
    5047 
    51 For regression, all the described measures can be used, except for the :math:`O_{ref}`. Classification domains 
    52 are supported by the following methods: BAGV, LCV, CNK and DENS, :math:`O_{ref}`. 
     48For regression, you can use all the described measures except :math:`O_{ref}`. Classification is 
     49supported by BAGV, LCV, CNK and DENS, :math:`O_{ref}`. 
    5350 
    5451Sensitivity Analysis (SAvar and SAbias) 
     
    116113============================== 
    117114 
     115.. data:: SIGNED 
     116     
     117.. data:: ABSOLUTE 
     118 
     119    These constants distinguish signed and 
     120    absolute reliability estimation measures. 
     121 
     122.. data:: METHOD_NAME 
     123 
     124    A dictionary that that maps reliability estimation 
     125    method IDs (integerss) to method names (strings). 
     126 
    118127.. autoclass:: Estimate 
    119128    :members: 
    120129    :show-inheritance: 
    121130 
    122 There is a dictionary named :obj:`METHOD_NAME` that maps reliability estimation 
    123 method IDs (ints) to method names (strings). 
    124131 
    125 In this module, there are also two constants for distinguishing signed and 
    126 absolute reliability estimation measures:: 
    127  
    128   SIGNED = 0 
    129   ABSOLUTE = 1 
    130132 
    131133Reliability estimation scoring 
     
    141143======= 
    142144 
     145The following script prints Pearson's correlation coefficient (r) between reliability  
     146estimates and actual prediction errors, and a corresponding p-value, for  
     147default reliability estimation measures.  
     148 
    143149.. literalinclude:: code/reliability-long.py 
    144150    :lines: 7-22 
    145151 
    146 This script prints out Pearson's R coefficient between reliability estimates 
    147 and actual prediction errors, and a corresponding p-value, for each of the 
    148 reliability estimation measures used by default. :: 
     152Results:: 
    149153   
    150154  Estimate               r       p 
Note: See TracChangeset for help on using the changeset viewer.