Here's my somewhat complicated setup that has just begun to cause a StackOverflow exception a couple of days ago:
On my windows-based continuous integration platform I have got a Jenkins job that starts a Python script.
This Python script runs a cmake command, an msbuild call and then executes the newly compiled gtest-based test framwork.
The msbuild produces a dll and the gtest executable. The executable itself then loads the dll in order to test it.
A couple of days ago I made some changes in the source code of the dll that alter the memory footprint of some of my structures (basically just array lengths). It's plain C code. Now some of the tests exit with a stack-overflow exception.
I admit I'm putting some data structures on the stack that don't necessarily have to be there but it's the best I've got for information hiding in C (better than using static global variables).
if(myCondition)
{
int hugeBuffer[20000];
...
}
Apart from that there is no recursion or anything fancy going on that could be a legit source of trouble. Large chunks of data are always passed by reference (pointer).
Anyway, the stack overflow exception doesn't occur on my local machine running the gtest executable directly from Visual Studio unless I significantly reduce the reserved stack memory in the linker settings.
Then in debug mode I clearly run into a point where the stack just overflows at the beginning of a function.
Unfortunately I couldn't find any way of debugging how full the stack is. In VS I've only got the call stack window which doesn't show the current "fill level" of the stack.
So although you guys might kill mir for this I'm guessing I really just don't have enough stack memory available when running the Jenkins job.
So I'm wondering what step actually defines the amount of stack memory available for my DLL code. It's clearly less than the default 10MB I have in VisualStudio on my local machine.
In the msbuild step there is no STACK parameter used for the linker so I'm guessing the exe header should contain the same value as in Visual Studio (10MB?).
The Python script runs a subprocess.call which could ignore the value set by the linker and overwrite it. I could neither find any information on that nor on how to change the stack memory allocated. I don't even know whether it spawns a thread or a process which may also affect the stack size.
The DLL loading mechanism in windows is also somewhat mysterious to me but I'm guessing the dll uses the same stack as the executable using it. I'm using the LoadLibrary() macro from WinBase.h.
by sheer luck I found out that although the same CMakeLists.txt is used for creating the projects (locally and on Jenkins) the resulting projects generated from them differ.
My local projects (and the GUI-VisualStudio solution I manually created on the Server for error finding) had 10MB of stack reserved whereas the python-based call to CMake was only using the default value of 1MB which is not enough for the project.
This strange behavior of Cmake may be compensated for by adding this line to CMakeLists.txt:
# Set linker to use 10MB of stack memory for all gtest executables, the default of 1MB is not enough!
SET( CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} /STACK:\"10000000\"")
sorry to bother you :)
I'm glad it's over
Related
I built an EXE file from a Python script using PyInstaller, using
pyinstaller --onefile myscript.py
Packages I used:
pandas, numpy, imutils, opencv, logging, os, random, json, string, csv, datetime, uuid
The EXE runs fine on my PC. However, when I try it on another PC I get the error shown in this screenshot: https://www.screencast.com/t/msZrURL4v
Any idea what the problem is?
The error you post just says "I was looking for one specific DLL and did not find it".
Rather than installing other packages and extensions that might, or might not, be or somehow contain the right DLL, you now need to determine exactly what it is that isn't to be found.
I can suggest three complementary methods, none absolutely certain to pinpoint the exact problem (of course the voodoo method of "install some package at random and see whether it fixes it" might also work, and often does -- but that's magic, not computer science):
the quickest: check the pyimod03_importers.py file at line 714, see what it was doing when the exception was thrown. Due to Windows' library loading strategies, you might be handed a red herring, with a file reported not to be there when it actually is, because it relies on a second missing file whose name you won't be told.
the easiest: use a tool like SysInternals' DEPENDS.EXE to inspect the OMR.EXE file. This is almost guaranteed not to work in this case, because the needed imports might be specified in Python format, not in any form that DEPENDS.EXE will recognize.
the most comprehensive, but least easy: use a tool like SysInternals' PROCMON, set up the filters to exclude the background noise of Windows' idle state - there will be an awful lot of that - and then fake running OMR.EXE; exclude the additional noise generated by that. You'll need about fortyish filters to be set up. Finally run OMR.EXE. Near the end, you will see a series of attempt to load SOMETHING.DLL, all failed; the first is where the DLL is supposed to be (by either Python or OMR), the others are all suitable alternatives.
Then:
if the DLL is one of yours, find out how to pack them with the EXE bundle.
if it is not, you need to reliably assess where it can be found.
It might well be that the suggestion you were given - install MSVC redistributable that-version-or-other - was absolutely correct. Libraries with names like MSVCnn... belong to that package. MSO... files usually belong to Microsoft Office redistributables. MSJET... files are found in several Microsoft package, for example the .NET redistributable.
otherwise, Google and possibly MSDN Search Engine are your friends.
From past experience, I suggest setting up a virtual machine for testing, then seeing what packages are needed. This is because the first DLL crash will hide any subsequent ones, and you might need to repeat the above steps several times. The fact that the first library you need is supplied by the NETFX64 package and the second by the Microsoft Office runtime might be true, but when you find out that the second library is needed, you might also find out that the MSO runtime would have supplied the first also; so at that point, and not before, you discover that the NETFX64 package wasn't really needed, and can simplify your installation requirements to the MSO runtime alone.
Boiling down the requirements to a short list might be a lengthy task and you will want to restart the machine from scratch more than once. With a VM, that is easy to do.
(I've kept referring to the MSO runtime because I figure that your program will process a checkbox answers module, and will likey need or believe it needs some scanner recognition features, which the MSO runtime supplies. If that is so, they'll probably come last).
I have a python module that uses an external C++ library using a C++ extension build with distutils. When I compile the C++ library with the address sanitizer, -fsanitize option of GCC, I get a segfault when running unit tests. Now, initially I thought that it was because me using different compiler options for the two binaries, the C++ python extension and the library but now I am more and more convinced that this is because the address sanitizer found an error in the library and triggered a seg fault, as explained here.
This is also supported by the fact that if I compile the C++ library without the address sanitizer, everything works fine.
When I run unit tests, the program outputs very little information:
./run_unit_tests
Segmentation fault (core dumped)
Even looking at the core dump I was able to find only a stack trace pointing to the C++ library but no mention of address sanitizer.
I have tried to use ASAN_OPTIONS to redirect the sanitizer output to a file but the sanitizer apparently does not pick up the options:
ASAN_OPTIONS=help=1 ./run_unit_tests
Segmentation fault (core dumped)
What strategy should I take here to confirm that the seg fault coms from the sanitizer and possibly discover what kind of error it is?
First a few clarifications:
this is because the address sanitizer found an error in the library and triggered a seg fault
When Asan detects an error, it will always emit a friendly error message. Segfault means that
either instrumentation went wrong at some point
or (much less likely) instrumented code inadverently triggered some already existing critical bug
This is also supported by the fact that if I compile the C++ library with undefined behavior sanitizer, everything is working fine
UBSan is much simpler than ASan so in general you can not really share conclusions about them.
I have tried to use ASAN_OPTIONS to redirect the sanitizer output to a file but the sanitizer apparently does not pick up the options:
The fact that help=1 fails tells us that sanitized app segfaulted at early startup, before Asan was able to parse ASAN_OPTIONS and react appropriately. This usually happens when there is some basic issue in how Asan was enabled.
My guess is that you miss LD_PRELOAD=path/to/libasan.so environment setting which is required when when applying Asan to a single DSO instead of main application (as is the case with sanitized C/C++ plugins for interpreters, check Asan FAQ).
If this does not help, I suggest to provide more information (e.g. GCC version, symbolized stack at the point of segv).
I'm looking at statically link Python with my application. The reason for this is because in some test cases I've seen a 10% speed increase. My application uses the Python C-API heavily, and it seems that Whole Program Optimization is able to do some good optimizations. I expect Profile Guided Optimizations will gain a little more too. This is all being done in MSVC2015
So far I've recompiled the pythoncore project (python35.dll) into a static library and linked that with my application (let's call it myapp.exe). FYI other than changing the project type to static, the only other thing that needs doing is setting the define Py_NO_ENABLE_SHARED during the static lib compile, and when compiling myapp.exe. This works fine and it's how I was able to obtain the 10% speed improvement test result.
So the next step is continuing to support external python modules that have .pyd files (which are .dll files renamed to .pyd). These modules will have been compiled expecting to dynamically link with python35.dll, so I need to provide a workaround for that requirement, since all of the python functions are now embedded into myapp.exe.
First I use a .def file to export all of the public Python functions from myapp.exe. This works fine.
The missing piece is how do I create a python35.dll which redirects all the calls to the functions exported from myapp.exe.
My first attempt is using DLL forwarding. I made a custom python35.dll which has a .def file with lines such as:
PyArg_Parse=myapp.PyArg_Parse
In theory, this works. If I use Dependency Walker on socket.pyd, it correctly opens my python35.dll and shows that all the calls are being forwarded to myapp.exe.
However when actually running myapp.exe and trying to import socket, it fails to load the required entry points from myapp.exe. 'import socket' in Python will cause a LoadLibrary("socket.pyd") to occur. This will load my custom python35.dll implicitly. The failure occurs while trying to load python35.dll, it's unable to find the entry points for it's forwards. It seems like the reason for this is because myapp.exe won't be part of the library search path. I seem to be able to verify this by copying myapp.exe to myapp.dll. If I do that then the python35.dll load works, however this isn't a solution since that will result in 2 copies of the Python enviroment (one in myapp.exe, one in myapp.dll)
Possible other avenues I've also looked into but haven't found the right solution for:
Somehow getting .exe files to be part of the library search path
Using Windows manifest/configuration to redirect the library somehow
Manually using declspec(naked) and jmp statements to more explicitly wrap the .dll. I'm working in x64, so I don't think that's possible anymore?
I could manually do the whole Python API and wrap each function manually. This is doable if I can find a way to create the function definitions of all the exports so it's not an insane amount of manual work.
In summary, is there a way to redirect/forward calls to a .dll to functions/data exported from an .exe. Thanks!
I ended up going with the solution that #martineau suggested in the comments, which was to put all of my application, including Python, into a single .dll instead of an .exe. Then the .exe is just a simple file that calls into the .dll and does nothing else.
I am targetting an embedded platform with linux_rt, and would like to compile cpython. I am not asking whether python is appropriate for realtime, or its latency. I AM asking about compiling under platform constraints.
I would like an interpretter embedded in a C shared library, but will also accept an exectuable binary if needs be.
Any C compiling ive done is for mainstream OS deployment, and i usually just hit make install. Im not afraid to get a little dirty, but am afraid of longterm maintenance and repeatability.
To avoid as much memory overhead as possible, are there any compiler configurations that can be changed from defaults? Can I easily strip sections of the standard library I know will not be needed?
Target platform has a 600 MHz Celeron, and 256mb RAM. The required firmware is built for a v2.6 kernel (might be 2.4). The default OS image uses Busybox, and most standard system libraries are minimally available. The root filesystem is around 100mB (flash), although I will have an external memory card mounted and can extended root onto there.
Python should have 70% Cpu and 128mB ram at most times, although I could imagine sloppy execution of the interpretter at times, and on RT linux, that could start to add up. Just trying to take precautions before I dive in.
Looking for simple Do's or Don'ts. Reference to similar projects would be great, but I really want to stick with CPython where possible.
I do not have the target platform in the shop yet, so I cannot post any tests. Will have the unit in 2 weeks and will update this post, at that time, if needed.
make a VM with the target configuration to help you get started. VirtualBox or QEmu. If you don't have a root FS one place to start is TinyCore, which is very small, configurable, but also can run on your laptop -- http://www.linuxjournal.com/article/11023
According to the Python documentation, when compiling a Python extension on Windows, "you should use the same version of VC++ that was used to build Python itself". The explanation usually given is that the mismatch in VC runtime version will cause problems. However, it is possible to compile extensions using newer Visual Studio versions that appear to work just fine.
What are the cases where the different runtimes would cause problems? The most information I've seen on this topics was this thread on the python-dev mailing list. Is there a (hopefully small) set of use cases that lead to problematic behavior, or is it just a matter of luck that I haven't run into any trouble yet?
That mailing thread is the most comprehensive list that I've seen of cases where mismatched C runtimes causes problems. The general problem is each runtime doesn't share anything with the other runtime, each has its own separate state, and anything they expose externally can't be shared between the runtimes by your own code. The former problem means that each runtime has its own errno and the second the means that you can't use FILE *' objects opened with one runtime with the file I/O functions of the other.
Enumerating all the possible problems would mean enumerating the entire visible state (including indirectly visible state) of the of the runtimes, and then enumerating every value they can generate and receive that might be incompatible.
Somewhat offsetting this though is Microsoft's promise that object files (.OBJ) compiled with one version of the Microsoft C/C++ compiler should be compatible with subsequent versions of the compiler. This means for example, two different runtimes won't use a completely different set of values for errno (eg. ENOENT is always 2) because those values will appear as constants in object files.