## AUC error bars and pvalues

2 posts
• Page

**1**of**1**### AUC error bars and pvalues

Hello,

I'm interested in computing (1) error bars, and (2) significance levels, of the AUC from a cross-validated experiment.

For (1), the orngStat documentation says that the function AUCWilcoxon(results) will return the AUC and its standard error, but only for a single-iteration experiment. Indeed, when I give it the results from a call to orngTest.crossValidation, an error is generated. Is there any way to get the standard errors of the AUC of a cross-validated experiment?

For (2), I think what I want is to wrap up the entire experiment inside a permutation test, which would compute a p-value representing the probability of finding an AUC at least as extreme as the one computed without permuting the class labels. Is there a permutation test capability already built into Orange, or would I have to program this myself?

Thanks,

John

I'm interested in computing (1) error bars, and (2) significance levels, of the AUC from a cross-validated experiment.

For (1), the orngStat documentation says that the function AUCWilcoxon(results) will return the AUC and its standard error, but only for a single-iteration experiment. Indeed, when I give it the results from a call to orngTest.crossValidation, an error is generated. Is there any way to get the standard errors of the AUC of a cross-validated experiment?

For (2), I think what I want is to wrap up the entire experiment inside a permutation test, which would compute a p-value representing the probability of finding an AUC at least as extreme as the one computed without permuting the class labels. Is there a permutation test capability already built into Orange, or would I have to program this myself?

Thanks,

John

- Code: Select all
`res = orngTest.crossValidation([orange.BayesLearner()], data)`

res.numberOfIterations = 1

for ex in res.results:

res.iterationNumber = 0

But I'm not sure that this is statistically correct.

I like you idea for a permutation test very much - it seems more plausible than the other tests for AUC I've heard about. This is not yet in Orange, but it shouldn't take more that a few lines of Python.

2 posts
• Page

**1**of**1**