Index: docs/rst/Orange.evaluation.reliability.rst
===================================================================
 docs/rst/Orange.evaluation.reliability.rst (revision 40)
+++ docs/rst/Orange.evaluation.reliability.rst (revision 41)
@@ 14,32 +14,29 @@
********************************************************
Reliability assessment statistically predicts reliability of single
predictions. Most of implemented algorithms for regression are taken from
Comparison of approaches for estimating reliability of individual
regression predictions, Zoran BosniÄ‡, 2008. Implementations for
classification follow descriptions in Evaluating Reliability of Single
Classifications of Neural Networks, Darko Pevec, 2011.
+Reliability assessment aims to predict reliabilities of individual
+predictions.
The following example shows basic usage of reliability estimation methods:
+Most of implemented algorithms for regression described in
+"Comparison of approaches for estimating reliability of individual
+regression predictions, Zoran BosniÄ‡, 2008" for regression and in
+in "Evaluating Reliability of Single
+Classifications of Neural Networks, Darko Pevec, 2011" for classification.
+
+We can use reliability estimation with any Orange learners. The following example:
+
+ * Constructs reliability estimators (implemented in this module),
+ * Combines a regular learner.
+ (:class:`~Orange.classification.knn.kNNLearner` in this case) with
+ reliability estimators.
+ * Obtains prediction probabilities from the constructed classifier
+ (:obj:`Orange.classification.Classifier.GetBoth` option). The resulting
+ probabilities have and additional attribute, :obj:`reliability_estimate`
+ attribute, :class:`Orange.evaluation.reliability.Estimate`.
.. literalinclude:: code/reliabilitybasic.py
:lines: 7
The important points of this example are:
 * construction of reliability estimators using classes,
 implemented in this module,
 * construction of a reliability learner that bonds a regular learner
 (:class:`~Orange.classification.knn.kNNLearner` in this case) with
 reliability estimators,
 * calling the constructed classifier with
 :obj:`Orange.classification.Classifier.GetBoth` option to obtain class
 probabilities; :obj:`probability` is the object that gets appended the
 :obj:`reliability_estimate` attribute, an instance of
 :class:`Orange.evaluation.reliability.Estimate`, by the reliability learner.

It is also possible to do reliability estimation on whole data
table, not only on single instance. Next example demonstrates usage of a
crossvalidation technique for reliability estimation. Reliability estimations
for first 10 instances get printed:
+We could also evaluate more examples. The next example prints reliability estimates
+for first 10 instances (with crossvalidation):
.. literalinclude:: code/reliabilityrun.py
@@ 49,6 +46,6 @@
===================
For regression, all the described measures can be used, except for the :math:`O_{ref}`. Classification domains
are supported by the following methods: BAGV, LCV, CNK and DENS, :math:`O_{ref}`.
+For regression, you can use all the described measures except :math:`O_{ref}`. Classification is
+supported by BAGV, LCV, CNK and DENS, :math:`O_{ref}`.
Sensitivity Analysis (SAvar and SAbias)
@@ 116,16 +113,21 @@
==============================
+.. data:: SIGNED
+
+.. data:: ABSOLUTE
+
+ These constants distinguish signed and
+ absolute reliability estimation measures.
+
+.. data:: METHOD_NAME
+
+ A dictionary that that maps reliability estimation
+ method IDs (integerss) to method names (strings).
+
.. autoclass:: Estimate
:members:
:showinheritance:
There is a dictionary named :obj:`METHOD_NAME` that maps reliability estimation
method IDs (ints) to method names (strings).
In this module, there are also two constants for distinguishing signed and
absolute reliability estimation measures::

 SIGNED = 0
 ABSOLUTE = 1
Reliability estimation scoring
@@ 141,10 +143,12 @@
=======
+The following script prints Pearson's correlation coefficient (r) between reliability
+estimates and actual prediction errors, and a corresponding pvalue, for
+default reliability estimation measures.
+
.. literalinclude:: code/reliabilitylong.py
:lines: 722
This script prints out Pearson's R coefficient between reliability estimates
and actual prediction errors, and a corresponding pvalue, for each of the
reliability estimation measures used by default. ::
+Results::
Estimate r p