Orange Forum • View topic - EarthLearner gets memory corruption above a few terms

EarthLearner gets memory corruption above a few terms

A place to ask questions about methods in Orange and how they are used and other general support.

EarthLearner gets memory corruption above a few terms

Postby gegaron » Tue Jul 03, 2012 20:02

I am current using Orange.regression.earth.EarthLearner on synthetic data as a testbed for potential use at JPL/NASA for helping to predict CO2 values from atmospheric spectra.

EarthLearner has been working admirably on simple datasets, even up to 200 features x 100,000 samples. But when I wanted to create a more complex relationship in a small dataset (13 features x 10,000 samples) I noticed that many function parameter settings would cause EarthLearner to blow with a memory corruption warning.

If I request < 35 terms with this dataset, it will complete normally. If I request >= 35 terms, even with degree = 1, it will suffer a memory corruption error and terminate.

I would appreciate any advice on how to get around this error, as up until now EarthLearner was looking like an excellent candidate for our work and would definitely be part of our published result citations. I'm hoping you'll tell me I somehow am setting up the data wrong for EarthLearner originally, or some other easily repaired problem. But I doubt that given it's ability to handle my simple data with large numbers of terms, but not this smaller case with more complex relationships.

Thanks to anyone who can help me very much.

The specific reported error is:

*** glibc detected *** python: malloc(): memory corruption: 0x0000000010a77490 ***
======= Backtrace: =========
/lib64/libc.so.6[0x366ca71a1e]
/lib64/libc.so.6(__libc_malloc+0x6e)[0x366ca7374e]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0[0x2b95c8bd4f02]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0[0x2b95c8c5a45f]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0[0x2b95c8c5aaec]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0(Py_BuildValue+0x8f)[0x2b95c8c5ae6f]
/opt/local/depot/python-64/2.7.1/lib/python2.7/site-packages/numpy/linalg/lapack_lite.so[0x2b95cfcec6e4]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x66da)[0x2b95c8c3b7aa]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8c9)[0x2b95c8c3cd99]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5c12)[0x2b95c8c3ace2]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8c9)[0x2b95c8c3cd99]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5c12)[0x2b95c8c3ace2]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8c9)[0x2b95c8c3cd99]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0[0x2b95c8bc2fe2]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b95c8b95908]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0[0x2b95c8bf1965]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0[0x2b95c8bf1553]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0(PyObject_Call+0x68)[0x2b95c8b95908]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x1097)[0x2b95c8c36167]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5fd7)[0x2b95c8c3b0a7]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x8c9)[0x2b95c8c3cd99]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0(PyEval_EvalCode+0x32)[0x2b95c8c3ce12]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0(PyRun_FileExFlags+0xc1)[0x2b95c8c60251]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0(PyRun_SimpleFileExFlags+0x1f7)[0x2b95c8c60527]
/opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0(Py_Main+0xaf3)[0x2b95c8c726c3]
/lib64/libc.so.6(__libc_start_main+0xf4)[0x366ca1d994]
python[0x400639]
======= Memory map: ========
00400000-00401000 r-xp 00000000 00:1a 9824886 /opt/local/depot/python-64/2.7.1/bin/python
00600000-00601000 rw-p 00000000 00:1a 9824886 /opt/local/depot/python-64/2.7.1/bin/python
0dfee000-114c7000 rw-p 0dfee000 00:00 0 [heap]
366c600000-366c61c000 r-xp 00000000 fd:00 4456452 /lib64/ld-2.5.so
366c81c000-366c81d000 r--p 0001c000 fd:00 4456452 /lib64/ld-2.5.so
366c81d000-366c81e000 rw-p 0001d000 fd:00 4456452 /lib64/ld-2.5.so
366ca00000-366cb4d000 r-xp 00000000 fd:00 4456456 /lib64/libc-2.5.so
366cb4d000-366cd4d000 ---p 0014d000 fd:00 4456456 /lib64/libc-2.5.so
366cd4d000-366cd51000 r--p 0014d000 fd:00 4456456 /lib64/libc-2.5.so
366cd51000-366cd52000 rw-p 00151000 fd:00 4456456 /lib64/libc-2.5.so
366cd52000-366cd57000 rw-p 366cd52000 00:00 0
366ce00000-366ce82000 r-xp 00000000 fd:00 4456472 /lib64/libm-2.5.so
366ce82000-366d081000 ---p 00082000 fd:00 4456472 /lib64/libm-2.5.so
366d081000-366d082000 r--p 00081000 fd:00 4456472 /lib64/libm-2.5.so
366d082000-366d083000 rw-p 00082000 fd:00 4456472 /lib64/libm-2.5.so
366d200000-366d202000 r-xp 00000000 fd:00 4456478 /lib64/libdl-2.5.so
366d202000-366d402000 ---p 00002000 fd:00 4456478 /lib64/libdl-2.5.so
366d402000-366d403000 r--p 00002000 fd:00 4456478 /lib64/libdl-2.5.so
366d403000-366d404000 rw-p 00003000 fd:00 4456478 /lib64/libdl-2.5.so
366d600000-366d614000 r-xp 00000000 fd:00 4456480 /lib64/libz.so.1.2.3
366d614000-366d813000 ---p 00014000 fd:00 4456480 /lib64/libz.so.1.2.3
366d813000-366d814000 rw-p 00013000 fd:00 4456480 /lib64/libz.so.1.2.3
366da00000-366da16000 r-xp 00000000 fd:00 4456462 /lib64/libpthread-2.5.so
366da16000-366dc15000 ---p 00016000 fd:00 4456462 /lib64/libpthread-2.5.so
366dc15000-366dc16000 r--p 00015000 fd:00 4456462 /lib64/libpthread-2.5.so
366dc16000-366dc17000 rw-p 00016000 fd:00 4456462 /lib64/libpthread-2.5.so
366dc17000-366dc1b000 rw-p 366dc17000 00:00 0
366de00000-366de0d000 r-xp 00000000 fd:00 4456474 /lib64/libgcc_s-4.1.2-20080825.so.1
366de0d000-366e00d000 ---p 0000d000 fd:00 4456474 /lib64/libgcc_s-4.1.2-20080825.so.1
366e00d000-366e00e000 rw-p 0000d000 fd:00 4456474 /lib64/libgcc_s-4.1.2-20080825.so.1
366e200000-366e2e6000 r-xp 00000000 fd:00 5475260 /usr/lib64/libstdc++.so.6.0.8
366e2e6000-366e4e5000 ---p 000e6000 fd:00 5475260 /usr/lib64/libstdc++.so.6.0.8
366e4e5000-366e4eb000 r--p 000e5000 fd:00 5475260 /usr/lib64/libstdc++.so.6.0.8
366e4eb000-366e4ee000 rw-p 000eb000 fd:00 5475260 /usr/lib64/libstdc++.so.6.0.8
366e4ee000-366e500000 rw-p 366e4ee000 00:00 0
366f200000-366f23b000 r-xp 00000000 fd:00 4456501 /lib64/libsepol.so.1
366f23b000-366f43b000 ---p 0003b000 fd:00 4456501 /lib64/libsepol.so.1
366f43b000-366f43c000 rw-p 0003b000 fd:00 4456501 /lib64/libsepol.so.1
366f43c000-366f446000 rw-p 366f43c000 00:00 0
366f600000-366f621000 r-xp 00000000 fd:00 5480497 /usr/lib64/libjpeg.so.62.0.0
366f621000-366f820000 ---p 00021000 fd:00 5480497 /usr/lib64/libjpeg.so.62.0.0
366f820000-366f821000 rw-p 00020000 fd:00 5480497 /usr/lib64/libjpeg.so.62.0.0
366fe00000-366fe15000 r-xp 00000000 fd:00 4456502 /lib64/libselinux.so.1
366fe15000-3670015000 ---p 00015000 fd:00 4456502 /lib64/libselinux.so.1
3670015000-3670017000 rw-p 00015000 fd:00 4456502 /lib64/libselinux.so.1
3670017000-3670018000 rw-p 3670017000 00:00 0
3670200000-367027f000 r-xp 00000000 fd:00 5481072 /usr/lib64/libfreetype.so.6.3.10
367027f000-367047f000 ---p 0007f000 fd:00 5481072 /usr/lib64/libfreetype.so.6.3.10
367047f000-3670484000 rw-p 0007f000 fd:00 5481072 /usr/lib64/libfreetype.so.6.3.10
3671200000-3671223000 r-xp 00000000 fd:00 5480739 /usr/lib64/libpng12.so.0.10.0
3671223000-3671423000 ---p 00023000 fd:00 5480739 /usr/lib64/libpng12.so.0.10.0
3671423000-3671424000 rw-p 00023000 fd:00 5480739 /usr/lib64/libpng12.so.0.10.0
3673a00000-3673a11000 r-xp 00000000 fd:00 4456512 /lib64/libresolv-2.5.so
3673a11000-3673c11000 ---p 00011000 fd:00 4456512 /lib64/libresolv-2.5.so
3673c11000-3673c12000 r--p 00011000 fd:00 4456512 /lib64/libresolv-2.5.so
3673c12000-3673c13000 rw-p 00012000 fd:00 4456512 /lib64/libresolv-2.5.so
3673c13000-3673c15000 rw-p 3673c13000 00:00 0
3674200000-367424e000 r-xp 00000000 fd:00 5482235 /usr/lib64/libncurses.so.5.5
367424e000-367444e000 ---p 0004e000 fd:00 5482235 /usr/lib64/libncurses.so.5.5
367444e000-367445c000 rw-p 0004e000 fd:00 5482235 /usr/lib64/libncurses.so.5.5
367445c000-367445d000 rw-p 367445c000 00:00 0
3674600000-367472d000 r-xp 00000000 fd:00 4456593 /lib64/libcrypto.so.0.9.8e
367472d000-367492c000 ---p 0012d000 fd:00 4456593 /lib64/libcrypto.so.0.9.8e
367492c000-367494d000 rw-p 0012c000 fd:00 4456593 /lib64/libcrypto.so.0.9.8e
367494d000-3674951000 rw-p 367494d000 00:00 0
3675600000-3675602000 r-xp 00000000 fd:00 4456635 /lib64/libutil-2.5.so
3675602000-3675801000 ---p 00002000 fd:00 4456635 /lib64/libutil-2.5.so
3675801000-3675802000 r--p 00001000 fd:00 4456635 /lib64/libutil-2.5.so
3675802000-3675803000 rw-p 00002000 fd:00 4456635 /lib64/libutil-2.5.so
3676800000-3676802000 r-xp 00000000 fd:00 4456511 /lib64/libkeyutils-1.2.so
3676802000-3676a01000 ---p 00002000 fd:00 4456511 /lib64/libkeyutils-1.2.so
3676a01000-3676a02000 rw-p 00001000 fd:00 4456511 /lib64/libkeyutils-1.2.so
3677400000-3677408000 r-xp 00000000 fd:00 6000143 /usr/lib64/libkrb5support.so.0.1
3677408000-3677607000 ---p 00008000 fd:00 6000143 /usr/lib64/libkrb5support.so.0.1
3677607000-3677608000 rw-p 00007000 fd:00 6000143 /usr/lib64/libkrb5support.so.0.1
3677c00000-3677c24000 r-xp 00000000 fd:00 6000150 /usr/lib64/libk5crypto.so.3.1
3677c24000-3677e23000 ---p 00024000 fd:00 6000150 /usr/lib64/libk5crypto.so.3.1
3677e23000-3677e25000 rw-p 00023000 fd:00 6000150 /usr/lib64/libk5crypto.so.3.1
3a99800000-3a99802000 r-xp 00000000 fd:00 4456645 /lib64/libcom_err.so.2.1
3a99802000-3a99a01000 ---p 00002000 fd:00 4456645 /lib64/libcom_err.so.2.1
3a99a01000-3a99a02000 rw-p 00001000 fd:00 4456645 /lib64/libcom_err.so.2.1
3a99c00000-3a99c2c000 r-xp 00000000 fd:00 6000410 /usr/lib64/libgssapi_krb5.so.2.2
3a99c2c000-3a99e2c000 ---p 0002c000 fd:00 6000410 /usr/lib64/libgssapi_krb5.so.2.2
3a99e2c000-3a99e2e000 rw-p 0002c000 fd:00 6000410 /usr/lib64/libgssapi_krb5.so.2.2
3a9a000000-3a9a091000 r-xp 00000000 fd:00 6000027 /usr/lib64/libkrb5.so.3.3
3a9a091000-3a9a291000 ---p 00091000 fd:00 6000027 /usr/lib64/libkrb5.so.3.3
3a9a291000-3a9a295000 rw-p 00091000 fd:00 6000027 /usr/lib64/libkrb5.so.3.3
3a9a400000-3a9a446000 r-xp 00000000 fd:00 4456742 /lib64/libssl.so.0.9.8e
3a9a446000-3a9a646000 ---p 00046000 fd:00 4456742 /lib64/libssl.so.0.9.8e
3a9a646000-3a9a64c000 rw-p 00046000 fd:00 4456742 /lib64/libssl.so.0.9.8e
2b95c8b2f000-2b95c8b31000 rw-p 2b95c8b2f000 00:00 0
2b95c8b50000-2b95c8ccd000 r-xp 00000000 00:1a 289178551 /opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0
2b95c8ccd000-2b95c8ecd000 ---p 0017d000 00:1a 289178551 /opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0
2b95c8ecd000-2b95c8f0b000 rw-p 0017d000 00:1a 289178551 /opt/local/depot/python-64/2.7.1/lib/libpython2.7.so.1.0
2b95c8f0b000-2b95c8f9f000 rw-p 2b95c8f0b000 00:00 0
2b95c90d0000-2b95c9111000 rw-p 2b95c90d0000 00:00 0
2b95c9111000-2b95cc6e3000 r--p 00000000 fd:00 5966234 /usr/lib/locale/locale-archive
2b95cc6e3000-2b95cc6e9000 r-xp 00000000 00:1a 410654736 /opt/local/depot/python-64/2.7.1/lib/python2.7/lib-dynload/_collections.so
2b95cc6e9000-2b95cc8e9000 ---p 00006000 00:1a 410654736 /opt/local/depot/python-64/2.7.1/lib/python2.7/lib-dynload/_collections.so
2b95cc8e9000-2b95cc8eb000 rw-p 00006000 00:1a 410654736 /opt/local/depot/python-64/2.7.1/lib/python2.7/lib-dynload/_collections.so
2b95cc8eb000-2b95cc8f3000 r-xp 00000000 00:1a 410654779 /opt/local/depot/python-64/2.7.1/lib/python2.7/lib-dynload/operator.so
2b95cc8f3000-2b95ccaf2000 ---p 00008000 00:1a 410654779 /opt/local/depot/python-64/2.7.1/lib/python2.7/lib-dynload/operator.so
2b95ccaf2000-2b95ccaf4000 rw-p 00007000 00:1a 410654779 /opt/local/depot/python-64/2.7.1/lib/python2.7/lib-dynload/operator.so
2b95ccaf4000-2b95ccafe000 r-xp 00000000 00:1a 410654774 /opt/local/depot/python-64/2.7.1/lib/python2.7/lib-dynload/itertools.so
2b95ccafe000-2b95cccfd000 ---p 0000a000 00:1a 410654774 /opt/local/depot/python-64/2.7.1/lib/python2.7/lib-dynload/itertools.so
2b95cccfd000-2b95ccd02000 rw-p 00009000 00:1a 410654774 /opt/local/depot/python-64/2.7.1/lib/python2.7/lib-dynload/itertools.so
2b95ccd02000-2b95ccd04000 r-xp 00000000 00:1a 410654727 /opt/local/depot/python-64/2.7.1/lib/python2.7/lib-dynload/_bisect.so
2b95ccd04000-2b95ccf03000 ---p 00002000 00:1a 410654727 /opt/local/depot/python-64/2.7.1/lib/python2.7/lib-dynload/_bisect.so
2b95ccf03000-2b95ccf04000 rw-p 00001000 00:1a 410654727 /opt/local/depot/python-64/2.7.1/lib/python2.7/lib-dynload/_bisect.so
2b95ccf04000-2b95ccf07000 r-xp 00000000 00:1a 410654745 /opt/local/depot/python-64/2.7.1/lib/python2.7/lib-dynload/_heapq.so
2b95ccf07000-2b95cd107000 ---p 00003000 00:1a 410654745 /opt/local/depot/python-64/2.7.1/lib/python2.7/lib-dynload/_heapq.so
2b95cd107000-2b95cd109000 rw-p 00003000 00:1a 410654745 /opt/local/depot/python-64/2.7.1/lib/python2.7/lib-dynload/_heapq.so
2b95cd109000-2b95cd10b000 r-xp 00000000 00:1a 410654773 /opt/local/depot/python-64/2.7.1/lib/python2.7/lib-dynload/grp.so
2b95cd10b000-2b95cd30a000 ---p 00002000 00:1a 410654773 /opt/local/depot/python-64/2.7.1/lib/python2.7/lib-dynload/grp.so
2b95cd30a000-2b95cd30b000 rw-p 00001000 00:1a 410654773 /opt/local/depot/python-64/2.7.1/lib/python2.7/lib-dynload/grp.so
2b95cd30b000-2b95cd30f000 r-xp 00000000 00:1a 410654790 /opt/local/depot/python-64/2.7.1/lib/python2.7/lib-dynload/time.so
2b95cd30f000-2b95cd50e000 ---p 00004000 00:1a 410654790 /opt/local/depot/python-64/2.7.1/lib/python2.7/lib-dynload/time.so
2b95cd50e000-2b95cd510000 rw-p 00003000 00:1a 410654790 /opt/local/depot/python-64/2.7.1/lib/python2.7/lib-dynload/time.so
2b95cd510000-2b95cd514000 r-xp 00000000 00:1a 410654785 /opt/local/depot/python-64/2.7.1/lib/python2.Abort

Re: EarthLearner gets memory corruption above a few terms

Postby gegaron » Tue Jul 03, 2012 21:30

To be clearer, my versions:

python 2.7.1 (r271:86832, Sep 2 2011, 15:36:55)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-50)] on linux2

help(Orange.version)
full_version = '2.6a1.dev-Unknown'
hg_revision = 'Unknown'
release = False
short_version = '2.6a1'
version = '2.6a1.dev-Unknown'

uname -r
2.6.18-308.8.1.el5

Re: EarthLearner gets memory corruption above a few terms

Postby Ales » Wed Jul 04, 2012 11:53

I don't know what the problem could be.
Can you provide more details on the nature of the relations in the data (a script to generate it would be most helpful), if it is not confidential?

Can you also run the script producing the error under valgrid and post the output.

Re: EarthLearner gets memory corruption above a few terms

Postby gegaron » Thu Jul 05, 2012 19:22

The data are simply several features ranging between 0 and 1 randomly. The regression value to predict is the sum of f1 + sin(f2*pi) + f3^4 + f5^5 + f6*f7 etc. Just a bunch of highly nonlinear terms based on each feature, plus some feature interactions via products. If I use very simple synthetic data, like f1 + f2 + f3, I will still experience an occasional memory corruption event, but it is very rare.

The valgrind output appears to be too large for this editor (reports too few characters then dumps my message when I try to submit), so I will include the beginning and end in the hopes it is still useful:

(these all happen before my code runs, just importing Orange)

==12779== Memcheck, a memory error detector
==12779== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==12779== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==12779== Command: python -u /home/mandrake/acos_scripts/regression_parameter_sweep2.py ../interacting.txt 0
==12779==
==12779== Invalid read of size 4
==12779== at 0x4CB7E34: PyObject_Free (obmalloc.c:958)
==12779== by 0x4C86D9A: code_dealloc (codeobject.c:307)
==12779== by 0x4D2C7C9: load_source_module (import.c:1022)
==12779== by 0x4D2CF4F: import_submodule (import.c:2596)
==12779== by 0x4D2D421: load_next (import.c:2416)
==12779== by 0x4D2D6B7: PyImport_ImportModuleLevel (import.c:2137)
==12779== by 0x4D0DD1C: builtin___import__ (bltinmodule.c:49)
==12779== by 0x4C6F907: PyObject_Call (abstract.c:2529)
==12779== by 0x4D0E365: PyEval_CallObjectWithKeywords (ceval.c:3881)
==12779== by 0x4D128EB: PyEval_EvalFrameEx (ceval.c:2332)
==12779== by 0x4D16D98: PyEval_EvalCodeEx (ceval.c:3252)
==12779== by 0x4D16E11: PyEval_EvalCode (ceval.c:666)
==12779== Address 0x5116020 is 1,824 bytes inside a block of size 2,963 free'd
==12779== at 0x4A05D21: free (vg_replace_malloc.c:325)
==12779== by 0x4D30A14: PyMarshal_ReadLastObjectFromFile (marshal.c:1143)
==12779== by 0x4D2CA08: load_source_module (import.c:773)
==12779== by 0x4D2CF4F: import_submodule (import.c:2596)
==12779== by 0x4D2D421: load_next (import.c:2416)
==12779== by 0x4D2D6B7: PyImport_ImportModuleLevel (import.c:2137)
==12779== by 0x4D0DD1C: builtin___import__ (bltinmodule.c:49)
==12779== by 0x4C6F907: PyObject_Call (abstract.c:2529)
==12779== by 0x4D0E365: PyEval_CallObjectWithKeywords (ceval.c:3881)
==12779== by 0x4D128EB: PyEval_EvalFrameEx (ceval.c:2332)
==12779== by 0x4D16D98: PyEval_EvalCodeEx (ceval.c:3252)
==12779== by 0x4D16E11: PyEval_EvalCode (ceval.c:666)
==12779==
==12779== Invalid read of size 4
==12779== at 0x4CB7E34: PyObject_Free (obmalloc.c:958)
==12779== by 0x4CC96CB: tupledealloc (tupleobject.c:233)
==12779== by 0x4C86E22: code_dealloc (codeobject.c:308)
==12779== by 0x4D2C7C9: load_source_module (import.c:1022)
==12779== by 0x4D2CF4F: import_submodule (import.c:2596)
==12779== by 0x4D2D421: load_next (import.c:2416)
==12779== by 0x4D2D6B7: PyImport_ImportModuleLevel (import.c:2137)
==12779== by 0x4D0DD1C: builtin___import__ (bltinmodule.c:49)
==12779== by 0x4C6F907: PyObject_Call (abstract.c:2529)
==12779== by 0x4D0E365: PyEval_CallObjectWithKeywords (ceval.c:3881)
==12779== by 0x4D128EB: PyEval_EvalFrameEx (ceval.c:2332)
==12779== by 0x4D16D98: PyEval_EvalCodeEx (ceval.c:3252)
==12779== Address 0x5116020 is 1,824 bytes inside a block of size 2,963 free'd
==12779== at 0x4A05D21: free (vg_replace_malloc.c:325)
==12779== by 0x4D30A14: PyMarshal_ReadLastObjectFromFile (marshal.c:1143)
==12779== by 0x4D2CA08: load_source_module (import.c:773)
==12779== by 0x4D2CF4F: import_submodule (import.c:2596)
==12779== by 0x4D2D421: load_next (import.c:2416)
==12779== by 0x4D2D6B7: PyImport_ImportModuleLevel (import.c:2137)
==12779== by 0x4D0DD1C: builtin___import__ (bltinmodule.c:49)
==12779== by 0x4C6F907: PyObject_Call (abstract.c:2529)
==12779== by 0x4D0E365: PyEval_CallObjectWithKeywords (ceval.c:3881)
==12779== by 0x4D128EB: PyEval_EvalFrameEx (ceval.c:2332)
==12779== by 0x4D16D98: PyEval_EvalCodeEx (ceval.c:3252)
==12779== by 0x4D16E11: PyEval_EvalCode (ceval.c:666)
==12779==
==12779== Invalid read of size 4
==12779== at 0x4CB7E34: PyObject_Free (obmalloc.c:958)
==12779== by 0x4CC96CB: tupledealloc (tupleobject.c:233)
==12779== by 0x4C86E12: code_dealloc (codeobject.c:309)
==12779== by 0x4D2C7C9: load_source_module (import.c:1022)
==12779== by 0x4D2CF4F: import_submodule (import.c:2596)
==12779== by 0x4D2D421: load_next (import.c:2416)
==12779== by 0x4D2D6B7: PyImport_ImportModuleLevel (import.c:2137)
==12779== by 0x4D0DD1C: builtin___import__ (bltinmodule.c:49)
==12779== by 0x4C6F907: PyObject_Call (abstract.c:2529)
==12779== by 0x4D0E365: PyEval_CallObjectWithKeywords (ceval.c:3881)
==12779== by 0x4D128EB: PyEval_EvalFrameEx (ceval.c:2332)
==12779== by 0x4D16D98: PyEval_EvalCodeEx (ceval.c:3252)
==12779== Address 0x5116020 is 1,824 bytes inside a block of size 2,963 free'd
==12779== at 0x4A05D21: free (vg_replace_malloc.c:325)
==12779== by 0x4D30A14: PyMarshal_ReadLastObjectFromFile (marshal.c:1143)
==12779== by 0x4D2CA08: load_source_module (import.c:773)
==12779== by 0x4D2CF4F: import_submodule (import.c:2596)
==12779== by 0x4D2D421: load_next (import.c:2416)
==12779== by 0x4D2D6B7: PyImport_ImportModuleLevel (import.c:2137)
==12779== by 0x4D0DD1C: builtin___import__ (bltinmodule.c:49)
==12779== by 0x4C6F907: PyObject_Call (abstract.c:2529)
==12779== by 0x4D0E365: PyEval_CallObjectWithKeywords (ceval.c:3881)
==12779== by 0x4D128EB: PyEval_EvalFrameEx (ceval.c:2332)
==12779== by 0x4D16D98: PyEval_EvalCodeEx (ceval.c:3252)
==12779== by 0x4D16E11: PyEval_EvalCode (ceval.c:666)
==12779==

...


==12779==
==12779== Conditional jump or move depends on uninitialised value(s)
==12779== at 0x4CB7E3D: PyObject_Free (obmalloc.c:958)
==12779== by 0x4C68BE9: PyGrammar_AddAccelerators (acceler.c:124)
==12779== by 0x4C69B76: PyParser_New (parser.c:77)
==12779== by 0x4C69BD4: parsetok (parsetok.c:136)
==12779== by 0x4D37EEC: PyParser_ASTFromString (pythonrun.c:1442)
==12779== by 0x4D380C7: PyRun_StringFlags (pythonrun.c:1307)
==12779== by 0x4D14B67: PyEval_EvalFrameEx (ceval.c:4745)
==12779== by 0x4D16D98: PyEval_EvalCodeEx (ceval.c:3252)
==12779== by 0x4D14CE1: PyEval_EvalFrameEx (ceval.c:4108)
==12779== by 0x4D16D98: PyEval_EvalCodeEx (ceval.c:3252)
==12779== by 0x4D14CE1: PyEval_EvalFrameEx (ceval.c:4108)
==12779== by 0x4D150A6: PyEval_EvalFrameEx (ceval.c:4098)

...

==12779==
==12779== More than 100 errors detected. Subsequent errors
==12779== will still be recorded, but in less detail than before.

...

(now the errors after my code begins to run)

==12779== Conditional jump or move depends on uninitialised value(s)
==12779== at 0x11D8E6C0: parseMatrixContents(GCPtr<TExampleGenerator>, int const&, char const*, int const&, bool&, bool&, bool&, bool&, bool&, int&, std::vector<bool, std::allocator<bool> >&) (gslconversions.cpp:128)
==12779== by 0x11CCCC24: ExampleTable_toNumericOrMA(_object*, _object*, _object*, _object**, _object**) (lib_kernel.cpp:3468)
==12779== by 0x4D157A9: PyEval_EvalFrameEx (ceval.c:4012)
==12779== by 0x4D16D98: PyEval_EvalCodeEx (ceval.c:3252)
==12779== by 0x4D14CE1: PyEval_EvalFrameEx (ceval.c:4108)
==12779== by 0x4D16D98: PyEval_EvalCodeEx (ceval.c:3252)
==12779== by 0x4C9CFE1: function_call (funcobject.c:526)
==12779== by 0x4C6F907: PyObject_Call (abstract.c:2529)
==12779== by 0x4CCB964: slot_tp_new (typeobject.c:5694)
==12779== by 0x4CCB552: type_call (typeobject.c:712)
==12779== by 0x4C6F907: PyObject_Call (abstract.c:2529)
==12779== by 0x4D10166: PyEval_EvalFrameEx (ceval.c:4230)
==12779==
==12779== Conditional jump or move depends on uninitialised value(s)
==12779== at 0x11CCCD84: ExampleTable_toNumericOrMA(_object*, _object*, _object*, _object**, _object**) (lib_kernel.cpp:3502)
==12779== by 0x4D157A9: PyEval_EvalFrameEx (ceval.c:4012)
==12779== by 0x4D16D98: PyEval_EvalCodeEx (ceval.c:3252)
==12779== by 0x4D14CE1: PyEval_EvalFrameEx (ceval.c:4108)
==12779== by 0x4D16D98: PyEval_EvalCodeEx (ceval.c:3252)
==12779== by 0x4C9CFE1: function_call (funcobject.c:526)
==12779== by 0x4C6F907: PyObject_Call (abstract.c:2529)
==12779== by 0x4CCB964: slot_tp_new (typeobject.c:5694)
==12779== by 0x4CCB552: type_call (typeobject.c:712)
==12779== by 0x4C6F907: PyObject_Call (abstract.c:2529)
==12779== by 0x4D10166: PyEval_EvalFrameEx (ceval.c:4230)
==12779== by 0x4D150A6: PyEval_EvalFrameEx (ceval.c:4098)

valgrind: m_mallocfree.c:248 (get_bszB_as_is): Assertion 'bszB_lo == bszB_hi' failed.
valgrind: Heap block lo/hi size mismatch: lo = 18544, hi = 13813188272276672056.
This is probably caused by your program erroneously writing past the
end of a heap block and corrupting heap metadata. If you fix any
invalid writes reported by Memcheck, this assertion failure will
probably go away. Please try that before reporting this as a bug.

Re: EarthLearner gets memory corruption above a few terms

Postby Ales » Fri Jul 06, 2012 12:56

I fixed the use of uninitialized variable in parseMatrixContents and pushed the change to the main repo.

Can you update, rebuild orange and test again.

However I am not sure if this was the cause for the memory corruption. Are there any other 'invalid write' errors in valgrind's output?

Re: EarthLearner gets memory corruption above a few terms

Postby gegaron » Fri Jul 06, 2012 20:53

You are so responsive! Thanks so much!
I could not attach the entire valgrind.out before to this message, so I have hosted it instead on my own webpage. The link is:

http://www.myrandus.com/valgrind.out

That is the sum of information provided by valgrind. If you could do any other fixes that appear in that log, I would be glad to re-install Orange and do some more runs to see if the issue resolves itself.

Again, thanks deeply. This is helping my research quite a bit. I very much want to use Orange for this project.

-Lukas Mandrake

Re: EarthLearner gets memory corruption above a few terms

Postby Ales » Mon Jul 09, 2012 11:31

The errors seem to happen in lapack library when calling numpy.linalg.lstsq. This is called in the pruning step if the builtin earth subset evaluation fails.
My guess is there is something wrong with the numpy/lapack bridge.

Can you try replacing numpy.linalg.lstsq with scipy.linalg.lstsq? I.e.
Code: Select all
import numpy
import scipy.linalg
numpy.linalg.lstsq = scipy.linalg.lstsq
at the start of your script, before importing Orange?

Re: EarthLearner gets memory corruption above a few terms

Postby gegaron » Mon Jul 09, 2012 20:15

You, my friend, are a bonified miracle-worker. The scripts all completed in all parameter settings with results. No memory corruption was found.

I am deeply thankful for the assistance you've provided me. I can now proceed with the study itself.

If there is any way for me to post how helpful you've been to anyone above you or a funder / manager role, I would be glad to do so.

Have a fantastic day.


Return to Questions & Support