Orange Forum • View topic - toNumpy triggers buffer overflow detection [solved]

toNumpy triggers buffer overflow detection [solved]

Report bugs (or imagined bugs).
(Archived/read-only, please use our ticketing system for reporting bugs and their discussion.)
Forum rules
Archived/read-only, please use our ticketing system for reporting bugs and their discussion.

toNumpy triggers buffer overflow detection [solved]

Postby Feanor76 » Fri Mar 12, 2010 18:03

Hi folks,

So, I recently rebuilt my system for amd64. I recompiled and installed orange and ran a script that runs fine under x86 32-bit. This error occurs with both a mid November 2009 build and Mar-11 nightly build.

A few thoughts and then the python code and the error messages. I'm calling ExampleTable_toNumpy (defined in lib_kernel.cpp) and the crash is occuring in ExampleTable_toNumericOrMA (also in lib_kernel.cpp). It seems like the memory management is failing. It also appears that the overflow itself is in parseMatrixContents. I am unfamiliar with "fortify fail", I'm looking into it.

Code versions:
numpy-1.3.0
gcc-4.3.4
gsl-1.12
R-2.9.2
glibc-2.10.1
(I noticed parseMatrixContents was defined for gsl and R ... not sure if those are needed).

The code pulls some data and then converts it to a NumPy array:

def exampleTableToNameMapAndArray(table):
nToC = {}
for indx, a in enumerate(table.domain.attributes):
nToC[a.name] = indx
nt = table.toNumpy("a", 0, range.ExampleTable.Multinomial_Ignore)[0]
return nToC, nt

The result is the following:

*** buffer overflow detected ***: /usr/bin/python2.6 terminated
======= Backtrace: =========
/lib/libc.so.6(__fortify_fail+0x37)[0x7f9bdb6e2867]
/lib/libc.so.6[0x7f9bdb6e0680]
/lib/libc.so.6[0x7f9bdb6dfc1b]
/usr/lib/python2.6/site-packages/orange/orange.so(_Z12raiseWarningP7_objectPKcz+0xaa)[0x7f9bdb143f6a]
/usr/lib/python2.6/site-packages/orange/orange.so(_Z19parseMatrixContents5GCPtrI17TExampleGeneratorERKiPKcS3_RbS6_S6_S6_RiRSt6vectorIbSaIbEE+0x4f5)[0x7f9bdaf1eb65]
/usr/lib/python2.6/site-packages/orange/orange.so(_Z26ExampleTable_toNumericOrMAP7_objectS0_S0_PS0_S1_+0x1db)[0x7f9bdafd985b]
/usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x4d3d)[0x7f9bdc2d1e2d]
/usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5c05)[0x7f9bdc2d2cf5]
/usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x861)[0x7f9bdc2d3901]
/usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x4e48)[0x7f9bdc2d1f38]
/usr/lib/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5c05)[0x7f9bdc2d2cf5]
/usr/lib/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x861)[0x7f9bdc2d3901]
/usr/lib/libpython2.6.so.1.0(PyEval_EvalCode+0x32)[0x7f9bdc2d3a12]
/usr/lib/libpython2.6.so.1.0[0x7f9bdc2ed991]
/usr/lib/libpython2.6.so.1.0(PyRun_FileExFlags+0x96)[0x7f9bdc2eda66]
/usr/lib/libpython2.6.so.1.0(PyRun_SimpleFileExFlags+0x205)[0x7f9bdc2eeee5]
/usr/lib/libpython2.6.so.1.0(Py_Main+0xb2b)[0x7f9bdc2fad0b]
/lib/libc.so.6(__libc_start_main+0xe6)[0x7f9bdb61aa26]
/usr/bin/python2.6[0x400789]

Postby Feanor76 » Fri Mar 12, 2010 23:47

So, the "fortify" function call provided a clue. Here's the deal:

gcc-4.3.4 (possibly earlier versions as well?) have a compiler option that looks like this "-D_FORTIFY_SOURCE " turned on in gentoo distributions. The purpose of this flag is to catch buffer overflow errors and malicious stack smashing attempts (basically different ways to playing with memory for fun and profit). Don't know where else this might be enabled. If you get suspicous of this, you can run:

omega source # gcc -E -dM - </dev/null | grep FORTIFY
#define _FORTIFY_SOURCE 2

If it returns something, then fortify is enabled by default (i.e., it's "defined"). You can turn if off by undefining it with a command line compile option:

-U_FORTIFY_SOURCE

To disable it for the entire orange build, edit your COMPILEOPTIONS in source/makefile.defs to include -U_FORTIFY_SOURCE.

For example:

COMPILEOPTIONS = -fPIC -fpermissive -w -DLINUX -D$(MODULENAME)_EXPORTS -O2 $(CXXFLAGS) -U_FORTIFY_SOURCE

Best,
Mark

PS I'm not sure what behavior in parseMatrix... triggered this. Presumably there is some fast and loose memory work going on.

Postby Janez » Mon Mar 29, 2010 19:10

Presumably there is some fast and loose memory work going on.


It all happens before converting any data, and parseMatrix itself does not do anything suspicious with memory.

The source of the error is probably raiseWarning itself. I found an extern definition in which the buffer is too small (although the actual definition makes it large enough). While this is wrong, it shouldn't be a problem. I fixed it - could you update your sources and try again?

If it still doesn't work: raiseWarning is called from parseMatrix if there is an attribute of a type which cannot be converted to numPy, for instance a string. I tried with a string but couldn't reproduce the error (I don't have a 64-bit Linux at hand, so I tried it on Mac). Does your data include an attribute which is neither discrete nor continuous? Does it have a funky name -- say more than 200 characters?


Return to Bugs