#
source:
orange/docs/reference/rst/Orange.tuning.rst
@
10711:38a9de5ed525

Revision 10711:38a9de5ed525, 2.7 KB checked in by anze <anze.staric@…>, 2 years ago (diff) |
---|

Line | |
---|---|

1 | ############################### |

2 | Optimization (``optimization``) |

3 | ############################### |

4 | |

5 | .. automodule:: Orange.optimization |

6 | |

7 | .. index:: tuning |

8 | |

9 | Wrappers for Tuning Parameters and Thresholds |

10 | |

11 | Classes for two very useful purposes: tuning learning algorithm's parameters |

12 | using internal validation and tuning the threshold for classification into |

13 | positive class. |

14 | |

15 | ***************** |

16 | Tuning parameters |

17 | ***************** |

18 | |

19 | Two classes support tuning parameters. |

20 | :obj:`Orange.optimization.Tune1Parameter` for fitting a single parameter and |

21 | :obj:`Orange.optimization.TuneMParameters` fitting multiple parameters at once, |

22 | trying all possible combinations. When called with data and, optionally, id |

23 | of meta attribute with weights, they find the optimal setting of arguments |

24 | using cross validation. The classes can also be used as ordinary learning |

25 | algorithms - they are in fact derived from |

26 | :obj:`Orange.classification.Learner`. |

27 | |

28 | Both classes have a common parent, :obj:`Orange.optimization.TuneParameters`, |

29 | and a few common attributes. |

30 | |

31 | .. autoclass:: TuneParameters |

32 | :members: |

33 | |

34 | .. autoclass:: Tune1Parameter |

35 | :members: |

36 | |

37 | .. autoclass:: TuneMParameters |

38 | :members: |

39 | |

40 | ************************** |

41 | Setting Optimal Thresholds |

42 | ************************** |

43 | |

44 | Some models may perform well in terms of AUC which measures the ability to |

45 | distinguish between instances of two classes, but have low classifications |

46 | accuracies. The reason may be in the threshold: in binary problems, classifiers |

47 | usually classify into the more probable class, while sometimes, when class |

48 | distributions are highly skewed, a modified threshold would give better |

49 | accuracies. Here are two classes that can help. |

50 | |

51 | .. autoclass:: ThresholdLearner |

52 | :members: |

53 | |

54 | .. autoclass:: ThresholdClassifier |

55 | :members: |

56 | |

57 | Examples |

58 | ======== |

59 | |

60 | This is how you use the learner. |

61 | |

62 | part of :download:`optimization-thresholding1.py <code/optimization-thresholding1.py>` |

63 | |

64 | .. literalinclude:: code/optimization-thresholding1.py |

65 | |

66 | The output:: |

67 | |

68 | W/out threshold adjustement: 0.633 |

69 | With adjusted thredhold: 0.659 |

70 | With threshold at 0.80: 0.449 |

71 | |

72 | part of :download:`optimization-thresholding2.py <code/optimization-thresholding2.py>` |

73 | |

74 | .. literalinclude:: code/optimization-thresholding2.py |

75 | |

76 | The script first divides the data into training and testing subsets. It trains |

77 | a naive Bayesian classifier and than wraps it into |

78 | :obj:`Orange.optimization.ThresholdClassifiers` with thresholds of .2, .5 and |

79 | .8. The three models are tested on the left-out data, and we compute the |

80 | confusion matrices from the results. The printout:: |

81 | |

82 | 0.20: TP 60.000, TN 1.000 |

83 | 0.50: TP 42.000, TN 24.000 |

84 | 0.80: TP 2.000, TN 43.000 |

85 | |

86 | shows how the varying threshold changes the balance between the number of true |

87 | positives and negatives. |

88 | |

89 | .. autoclass:: PreprocessedLearner |

90 | :members: |

**Note:**See TracBrowser for help on using the repository browser.