Ticket #997 (closed wish: fixed)

Opened 2 years ago

Last modified 2 years ago

Reliability widget

Reported by: blaz Owned by: ales
Milestone: 2.6 Component: canvas
Severity: immediate Keywords: reliability
Cc: Blocking:
Blocked By:

Description (last modified by blaz) (diff)

This is a specification of a new Reliability widget, implemented on the top of [orange.biolab.si/doc/orange25/Orange.evaluation.reliability.html Reliability estimation] module. For a given example and regressor, the code in the module contains different methods that can assess the reliability of the prediction of regressor. The plan is to build similar module for classification, but so far we only have it for regression.

Inputs: Learner (e.g., some regression algorithm, like regression trees) Training Data (ExampleTable, the data on which we report reliability) Test Data (ExampleTable)

Output: Reliability Scores (ExampleTable, for each reliability method this includes reliability score; optionally, it also includes prediction error and, also optionally, entire data set)

GUI:

Widget should implement all reliability methods from  Reliability module. User should choose which methods to use (some are extremely) slow and define their parameters. Following elements (boxes) constitute the GUI:

– Info – Includes a text box with two lines:

Regressor: <name of regressor> Training Data: Y instances, X features Test Data: Y1 instances, X1 features

Note: if test data is missing, all predictions of reliability are performed on Training Data. Test data is thus optional.

– Reliability methods – Includes a list of reliability methods, each preceeded with a check box and followed with a list of parameters. E.g. (CB=CheckBox):

CB Sensitivity analysis (variance)

Sensitivities: <list of e values, see documentation>

CB Sensitivity analysis (bias)

Sensitivities: <list of e values, see documentation>

CB Variance of bagged models

Models: <number>

CB Local cross validation

Nearest neighbors: <number>

CB Local modeling of prediction error

Nearest neighbors: <number>

CB Bagging variance c-neighbors

Models: <number> Nearest neighbors: <number>

CB Mahalanobis distance

Nearest neighbors: <number>

Default: Mahalanobis distance is selected by default, others are off.

– Output – CB Include prediction error CB Include original class and prediction CB Include input features

CB Commit on any change <Commit button> (disabled if no method is selected)

The output table should include as many rows as the input data. Columns should report on reliability, use the names that are abbreviations of each method (SAvar, SAbias, BAGV, LCV, CNK, BVCK, Mahalanobis). These are all added as meta values. Prediction error (if checked) is added as "Error" and "Abs Error" (error is original_class-prediction, the other is the absolute value), both added as meta. Original class is added as class, predicted class as "Prediction" meta attribute.

Jonna from AstraZeneca has also illustrated this widget through a figure: https://img.skitch.com/20111029-btfjxxby2dx88u32w96m6s782.jpg

Change History

comment:1 Changed 2 years ago by blaz

  • Description modified (diff)

comment:2 Changed 2 years ago by ales

  • Status changed from new to closed
  • Resolution set to fixed
Note: See TracTickets for help on using tickets.