Orange Forum • View topic - Inconsistent SVM results

Inconsistent SVM results

Report bugs (or imagined bugs).
(Archived/read-only, please use our ticketing system for reporting bugs and their discussion.)
Forum rules
Archived/read-only, please use our ticketing system for reporting bugs and their discussion.

Inconsistent SVM results

Postby Guest » Wed Mar 11, 2009 14:23

There seems to be a bug in orngSVM which can be illustrated with the following example:

Code: Select all
# Load example dataset
data = orange.ExampleTable("iris.tab")

learners = []

# Create learner:
svm_1 = orngSVM.SVMLearner(kernel_type=0, C = 512, normalization=True, name="SVM-1" )
learners.append(svm_1)

svm_2 = orngSVM.SVMLearner(kernel_type=0, C = 512, normalization=True, name="SVM-2" )
learners.append(svm_2)


# Now, do cross-validation on data:
results = orngTest.crossValidation(learners, data, folds=10)

##Uncomment to reverse order of learners:
#learners.reverse()

#output the results
print "Learner CA IS Brier AUC"
for i in range(len(learners)):
    print "%-8s %5.3f %5.3f %5.3f %5.3f" % (learners[i].name, \
     orngStat.CA(results)[i], orngStat.IS(results)[i],
     orngStat.BrierScore(results)[i], orngStat.AUC(results)[i])


Simply: try two SVM-learners with the exact same parameters on the same data, and report the results.

This code outputs (first one without learners.reverse(), second on with learners.reverse()):

Code: Select all
Learner CA IS Brier AUC
SVM-1    0.960 1.429 0.049 0.997
SVM-2    0.967 1.431 0.047 0.997

Learner CA IS Brier AUC
SVM-2    0.960 1.429 0.049 0.997
SVM-1    0.967 1.431 0.047 0.997


This is quite strange, since this tells me that the performance of the SVMs is dependent on the order in which they are evaluated. As far as I understand SVMs there is no stochastic element in the learning/classification process (but please correct me if I am wrong). Note that I am assuming that the folds on which both learners are tested are the same, which is stated in the Orange documentation (http://www.ailab.si/orange/doc/modules/orngTest.htm).

Is this a bug, or am I missing something?

Postby Ales » Tue Mar 17, 2009 12:31

The SVM by itself is not stochastic, but the implementation used in orange uses random permutations for class probability estimation. You can turn this behavior off by setting probability=False in learner constructor. Example:

Code: Select all
svm_1 = orngSVM.SVMLearner(kernel_type=0, C = 512, probability=False, normalization=True, name="SVM-1" )
learners.append(svm_1)

svm_2 = orngSVM.SVMLearner(kernel_type=0, C = 512, probability=False, normalization=True, name="SVM-2" )
learners.append(svm_2)

Thanks!

Postby Guest » Wed Mar 18, 2009 10:32

Thank you, that solves my problems.

Note that the documentation states that the default value for probability is False, but this is not the case, so I had no idea that I was using it :)


Return to Bugs



cron