I looked for a way to retrieve the path to a dll that is installed in the user's computer, but where the path could change depending on where they decided to install it. Couldn't find anything, so I wrote this with my own findings (feel free to add your own).
Some background:
I'm writing a module that loads Python into C++ (my users' machines have installed Python in their path, but the python version and path may vary between users)
However, I've found 2 issues:
The linker creates dll version dependencies even if I'm using functions available to any python 3 release (e.g. python 3.6 requires python36.dll).
The PYTHONPATH must be set in order to find the installed modules.
For the first issue, I used LoadLibrary to load at run time the appropriate dll, but that still left the burden of configuration on the user (he had to configure which dll was in his system, and where it was installed). Works fine if your user knows about his configuration, which is not the case for many of my users.
So that brings me to the guessing:
I was able to load python3.dll (which is located right next to the python36.dll or python38.dll) and I needed the path to the dll to calculate PTYHONPATH (and potentially, use python3 version to get the right dll to use, like python37.dll, python38.dll, etc.)
I don't think there is a nice way to allow for different Python minor versions (eg 3.6 or 3.8) unless you define a stable ABI for Python. However from my experience is that this makes for a poor subset of Python that doesn't support things like PyMemoryView. I would also suggest using delay load linker flags if you can isolate the Python part of your code to its own DLL. That way you can have a config file that reads where the Python library is located at runtime and load it from the appropriate path.
First, load the library using LoadLibraryA (or LoadLibraryW) and then use GetModuleFileNameA (or GetModuleFileNameW) to get the fullpath
//#include <stdio.h>
//#include <iostream>
HMODULE pythonLib = nullptr;
pythonLib = LoadLibraryA("python3.dll");
if (pythonLib != nullptr) {
char path[MAX_PATH];
GetModuleFileNameA(pythonLib, path, MAX_PATH);
std::cout << path << std::endl;
}
Related
I have a large program written in C++ that I wish to make usable via Python. I've written a python extension to expose an interface through which python code can call the C++ functions. The issue I'm having with this is that installing seems to be nontrivial.
All documentation I can find seems to indicate that I should create a setup.py which creates a distutils.core.Extension. In every example I've found, the Extension object being created is given a list of source files, which it compiles. If my code was one or two files, this would be fine. Unfortunately, it's dozens of files, and I use a number of relatively complicated visual studio build settings. As a result, building by listing .c files seems to be challenging to say the least.
I've currently configured my Python extension to build as a .dll and link against python39.lib. I tried changing the extension to .pyd and including the file in a manifest.in. After I created a setup.py and ran it, it created a .egg file that I verified did include the .pyd I created. However, after installing it, when I imported the module into python, the module was completely empty (and I verified that the PyInit_[module] function was not called). Python dll Extension Import says that I can import the dll if I change the extension to .pyd and place the file in the Dlls directory of python's installation. I've encountered two problems with this.
The first is that it seems to me that it's not very distributable like this. I'd like to package this into a python wheel, and I'm not sure how a wheel could do this. The second is even more problematic - it doesn't exactly work. It calls the initialization function of my extension, and I've verified in WinDbg that it's returning a python module. However, this is what I always get from the console.
>>> import bluespawn
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
SystemError: initialization of bluespawn did not return an extension module
The Python documentation has a section on publishing binary extensions, but for the past four years, it has been left as a placeholder. The github issue linked here isn't that helpful either; it boils down to either use distutils to build or use enscons to build. But since my build is a fairly complicated procedure, completely rewriting it to use enscons is less than desirable, to say the least.
It seems to me like placing the file in the DLLs directory is the wrong way of going about this. Given that I have a DLL and making setuptools compile everything itself seems infeasible, how should I go about installing my extension?
For reference, here's my initialization function, in case that's incorrect.
PyModuleDef bsModule{ PyModuleDef_HEAD_INIT, "bluespawn", "Bluespawn python bindings", -1, methods };
PyMODINIT_FUNC PyInit_bluespawn() {
PyObject* m;
Py_Initialize();
PyEval_InitThreads();
PyGILState_STATE state = PyGILState_Ensure(); // Crashes without this. Call to PyEval_InitThreads() required for this.
m = PyModule_Create(&bsModule);
PyGILState_Release(state);
Py_Finalize();
return m;
}
The python interface is available here: https://github.com/ION28/BLUESPAWN/blob/client-add-pylib/BLUESPAWN-win-client/src/user/python/PythonInterface.cpp
EDIT: I have a working solution that I am sure is not best practice. I created a very small C file that simply passes all calls it receives onto the large DLL I've already created. The C file is responsible for initializing the module, but everything else is handled inside the DLL. It works, but it seems like a very bad way of doing things. What I'm looking for is a better way of doing this.
Let me try and divide your post into two separate questions:
How to package a C++ library with a non-trivial compilation process using setuptools
Is it possible to distribute a python package with a precompiled library
1. How to package a C++ library with a non-trivial compilation process using setuptools
It is possible. I was quite surprised to see that setuptools offers many ways to override the compilation process, see the documentation here. For example, you can use the keyword argument extra_compile_args to pass extra arguments to the compiler.
In addition, as setup.py is a python file, you could relatively easily write some code to automatically collect all files needed for compilation. I'd done this myself in a project (github), and it worked quite well for me.
Here's some code from the setup.py:
libinjector = Extension('pyinjector.libinjector',
sources=[str(c.relative_to(PROJECT_ROOT))
for c in [LIBINJECTOR_WRAPPER, *LIBINJECTOR_SRC.iterdir()]
if c.suffix == '.c'],
include_dirs=[str(LIBINJECTOR_DIR.relative_to(PROJECT_ROOT) / 'include')],
export_symbols=['injector_attach', 'injector_inject', 'injector_detach'],
define_macros=[('EM_AARCH64', '183')])
2. Is it possible to distribute a python package with a precompiled library
I understand from your edit that you've managed to get it to work, but I'll say a few words anyway. Releasing precompiled binaries with your source distribution is possible, and it is possible to release your manually-compiled binaries in a wheel file as well, but it is not recommended.
The main reason is compatibility with the target architecture. First, you'll have to include two DLLs in your distribution, one for x64 and one for x86. Second, you might lose some nice optimizations, because you'll have to instruct the compiler to ignore optimizations available for the specific CPU type (note that this applies to normal wheel distributions as well). If you're compiling against windows SDK, you'll probably want to use the user's version too. In addition, including two DLLs in your release might grow it to an awkward size for a source distribution.
I have an application in C and at some point I need to solve a non-linear optimization problem. Unfortunately AFAIK there are very limited resources to do that in C (please let me know otherwise). However it is quite simple to do it in Python, e.g. scipy.optimize.minimize.
While I was trying to do that I encountered some of what it seems to be very frequent pitfalls, e.g. Python.h not found, module not loading, segmentation fault on function call, etc.
What is a quick and easy first-timer’s way to link the two programs?
There are some things that you have to make sure are in place in order to make this work:
Make sure you have Python installed (you may need the python-dev package).
Locate your Python.h file, e.g. by locate Python.h. One of the occurrences should be in a sub(sub)folder in the include folder, e.g. the path should be something like ../include/python2.7/Python.h.
Insert #include “<path_to_Python.h>" in your C code in order to be able to use the Python API.
Use any tutorial to call your Python function. I used this one and it did the trick. However there were a couple of small points missing:
Whenever you use any Py<Name> function, e.g. PyImport_Import(), always check the result to make sure there was no error, e.g.
// Load the module object
pModule = PyImport_Import(pName);
if (!pModule)
{
PyErr_Print();
printf("ERROR in pModule\n");
exit(1);
}
Immediately after initializing the Python interpreter, i.e. after Py_Initialize();, you have to append the current path to sys.path in order to be able to load your module (assuming it is located in your current directory):
PyObject *sys = PyImport_ImportModule("sys");
PyObject *path = PyObject_GetAttrString(sys, "path");
PyList_Append(path, PyString_FromString("."));
Keep in mind that when you give the name of your Python file, it has to be without the extension .py.
Lastly, you have to do the following during compiling/linking:
Remember the ../include/python2.7/Python.h file you used before? Include the include folder in the list of the header files directories with the -I option in the gcc options during compilation, e.g. -I /System/Library/Frameworks/Python.framework/Versions/2.7/include.
Also pass to the linker the folder with the required libraries. It should be inside the same folder where the include folder is located, e.g. -L /System/Library/Frameworks/Python.framework/Versions/2.7/lib, along with the -lpython2.7 option (of course adjusting it accordingly to your Python version).
Now you must be able to successfully compile and execute your C program that calls in it your Python program.
I hope this was helpful and good luck!
Sources:
How do you call Python code from C code?
http://www.linuxjournal.com/article/8497?page=0,1
http://www.codeproject.com/Articles/11805/Embedding-Python-in-C-C-Part-I
http://www.codeproject.com/Articles/11843/Embedding-Python-in-C-C-Part-II
Python C API doesn't load module
What sets up sys.path with Python, and when?
http://linux.die.net/man/1/gcc
PyObject segfault on function call
I have Python on my Ubuntu system, but gcc can't find Python.h
How do you call Python code from C code?
How can I get my own dll to use a dll in a specific location?
I have build an 32bit Excel dll (a .xll). I am using the Python C-Api in the dll. Problem is in general there is only a 64bit Python installation. The 64 python26.dll is located \System32 .
If I put the 32bit python26.dll in the Excel exe Folder or in \SysWOW64 everything is fine, but usually I have no access to those folders. Placing the 32 bit python dll in the xll folders does not help, it seems \System32 comes first.
I there any solution without linking explicitly using "LoadLibrary" and many times "GetProcAddress".?
Any option in VS2010?
Many thanks for your help.
You are right, the search order has the system directories first (and for a reason). This is a traditional situation with plugins. The applications themselves can store their dependent libraries in their own program folder and be fine, because that has the highest priority. Plugins commonly don't have that luxury.
The solution might be to use delayed loading for the DLL. If you have a program like this:
int main()
{
Py_Initialize();
}
...and you link it with python26.lib, it will create a static entry in the import table, pointing to python26.dll!Py_Initialize. Because the import table is processed before your code runs, you can't say which python26.dll you want to use, where to find it etc.
But if you set linker flag /DELAYLOAD:python26.dll and add MSVC's delayimp.lib to the build, it will postpone loading the Python DLL until some function from it is called. So now you can make sure you load the correct one: the import table has just the base name of the dependency (python26.dll) and if some library with the same name is already loaded, it will just reuse it. So you can preload the one you want from any location of your choice, then let the delayed loader do its job, like so:
int main()
{
LoadLibrary(TEXT("C:\\Program Files (x86)\\...\\...\\python26.dll"));
Py_Initialize();
}
But even with that, you might face another problem. The fact that the first loaded module with the correct base name is used by default means that even if you managed to do everything else right, there could still be some other Excel plugin or some silly Shell extension loaded (e.g. when the user goes to the "Open file" dialog), which would load their own python26.dll with possibly incompatible API, built with another compiler or anything else, which could ruin your day.
So I would also recommend you bundle your own custom version renamed to something like productname-python26.dll and link to that instead. There surely would never be a similarly named DLL by any other product, so you would be safe. In order to create a .lib file from a renamed DLL, try e.g. this guide.
I've embedded python on a mobile device successfully, but now how do I include a python library such as urllib?
Additionally, how can I include my own python scripts without a PYTHONPATH?
(please note: python is not installed on this system)
The easiest way is to create a .zip file containing all the python code you need and add this to your process's PYTHONPATH environment variable (via setenv()) prior to initializing the embedded Python interpreter. Usage of .pyd libraries can be done similarly by adding them to the same directory as the .zip and including the directory in the PYTHONPATH as well.
Usage of the setenv() call can cause trouble on Windows if you're mixing c-runtime versions. I spent many aggrivating hours learing that setenv() only sets the environment variables for the version of the c-runtime your compiler ships with. So if, for example, Python was built with VC++ 2005 and your compiler is VC++ 2008, you'll need to use an alternative mechanism. Browsing the sources for py2exe and/or PyInstaller may provide you with a better solution (since you're doing essentially the same thing as these tools) but a simple alternative is to "cheat" by using PyRun_SimpleString() to set the module search path from within Python itself.
snprintf(buff, "import sys\nsys.path.append("%s")\n", py_zip_filename)
PyRun_SimpleString(buff)
I'm developing on Windows, and I've searched everywhere without finding anyone talking about this kind of thing.
I made a C++ app on my desktop that embedded Python 3.1 using MSVC. I linked python31.lib and included python31.dll in the app's run folder alongside the executable. It works great. My extension and embedding code definitely works and there are no crashes.
I sent the run folder to my friend who doesn't have Python installed, and the app crashes for him during the scripting setup phase.
A few hours ago, I tried the app on my laptop that has Python 2.6 installed. I got the same crash behavior as my friend, and through debugging found that it was the Py_Initialize() call that fails.
I installed Python 3.1 on my laptop without changing the app code. I ran it and it runs perfectly. I uninstalled Python 3.1 and the app crashes again. I put in code in my app to dynamically link from the local python31.dll, to ensure that it was using it, but I still get the crash.
I don't know if the interpreter needs more than the DLL to start up or what. I haven't been able to find any resources on this. The Python documentation and other guides do not seem to ever address how to distribute your C/C++ applications that use Python embedding without having the users install Python locally. I know it's more of an issue on Windows than on Unix, but I've seen a number of Windows C/C++ applications that embed Python locally and I'm not sure how they do it.
What else do I need other than the DLL? Why does it work when I install Python and then stop working when I uninstall it? It sounds like it should be so trivial; maybe that's why nobody really talks about it. Nevertheless, I can't really explain how to deal with this crash issue.
Thank you very much in advance.
In addition to pythonxy.dll, you also need the entire Python library, i.e. the contents of the lib folder, plus the extension modules, i.e. the contents of the DLLs folder. Without the standard library, Python won't even start, since it tries to find os.py (in 3.x; string.py in 2.x). On startup, it imports a number of modules, in particular site.py.
There are various locations where it searches for the standard library; in your cases, it eventually finds it in the registry. Before, uses the executable name (as set through Py_SetProgramName) trying to find the landmark; it also checks for a file python31.zip which should be a zipped copy of the standard library. It also checks for a environment variable PYTHONHOME.
You are free to strip the library from stuff that you don't need; there are various tools that compute dependencies statically (modulefinder in particular).
If you want to minimize the number of files, you can
link all extension modules statically into your pythonxy.dll, or even link pythonxy.dll statically into your application
use the freeze tool; this will allow linking the byte code of the standard library into your pythonxy.dll.
(alternatively to 2.) use pythonxy.zip for the standard library.
Nice. And if you do not want to zip, copy Python26\DLLs & Python26\lib to your exe directory as:
.\myexe.exe
.\python26.dll
.\Python26\DLLs
.\Python26\lib
And then set PYTHONHOME with Py_SetPythonHome() API. Apparently, this API is not in the list of "allowed" calls before Py_Initialize();
Below worked for me on Windows (Python not installed):
#include "stdafx.h"
#include <iostream>
#include "Python.h"
using namespace std;
int _tmain(int argc, _TCHAR* argv[])
{
char pySearchPath[] = "Python26";
Py_SetPythonHome(pySearchPath);
Py_Initialize();
PyRun_SimpleString("from time import time,ctime\n"
"print 'Today is',ctime(time())\n");
//cerr << Py_GetPath() << endl;
Py_Finalize();
return 0;
}
Good that the search path is relative w.r.t the exe. Py_GetPath can show you where all it is looking for the modules.
A zip of the Python standard library worked for me with Python27.
I zipped the contents of Lib and dll, and made sure there was no additional python27-subfolder or Lib or dll subfolder. i.e. just a zip named python27.zip containing all the files.
I copied that zip and the python27.dll alongside the executable.
I wanted to add some additional info for others who might still be having troubles with this, as I was. I was eventually able to get my application working using the method proposed by user sambha, that is:
Program Files (x86)\
MyApplicationFolder\
MyApplication.exe
python27.dll
Python27\
DLLs\ (contents of DLLs folder)
Lib\ (contents of Lib folder)
...with one important addition: I also needed to install the MSVCR90.DLL. I'm using Python 2.7 and apparently python27.dll requires the MSVCR90.DLL (and maybe other MSVC*90.DLLs).
I solved this by downloading, installing, and running the 'vcredist_x86.exe' package from http://www.microsoft.com/en-us/download/details.aspx?id=29 . I think, though I am not certain, that you need to do it this way at least on Win7, as opposed to simply placing the MSVC*90.DLLs alongside your .exe as you may have done in the past. The Microsoft installer places the files and registers them in a special way under Win7.
I also tried the .zip file method but that did not work, even with the MSVCR90.DLL installed.