I am writing C extensions for python. I am just experimenting for the time being and I have written a hello world extension that looks like this :
#include <Python2.7/Python.h>
static PyObject* helloworld(PyObject* self)
{
return Py_BuildValue("s", "Hello, Python extensions!!");
}
static char helloworld_docs[] = "helloworld( ): Any message you want to put here!!\n";
static PyMethodDef helloworld_funcs[] = {
{"helloworld", (PyCFunction)helloworld, METH_NOARGS, helloworld_docs},
{NULL,NULL,0,NULL}
};
void inithelloworld(void)
{
Py_InitModule3("helloworld", helloworld_funcs,"Extension module example!");
}
the code works perfectly fine, after installing it from a setup.py file I wrote, and installing it from command line
python setup.py install
What I want is the following :
I want to use the C file as a python extension module, without installing it, that is I want to use it as just another python file in my project, and not a file that I need to install before my python modules get to use its functionality. Is there some way of doing this ?
You can simply compile the extension without installing (usually something like python setup.py build). Then you have to make sure the interpreter can find the compiled module (for example by copying it next to a script that imports it, or setting PYTHONPATH).
You can create your "own interpreter" by not extending python, but embedding it into your application. In that way, your objects will be always available for the users who are running your program. This is a pretty common thing to do in certain cases, for example look at the Blender project where all the bpy, bmesh and bge modules are already included.
The downside is, your users can't use the python command directly, they have to use your hello_world_python instead. (But of course you can provide your extension as a module as well.) And that also means, you have to compile and distribute your application for all platforms you want to support -- in case you want to distribute it as a binary, to make your users lives a bit easier.
For further information on embedding python into your program, read the propriate sections of the documentation:
Embedding Python in Another Application
Personal suggestion: Use Python 3.5 whenever you can, and stop supporting the old 2.x versions. For more information, read this article: Should I use Python 2 or Python 3 for my development activity?
Related
I have a large program written in C++ that I wish to make usable via Python. I've written a python extension to expose an interface through which python code can call the C++ functions. The issue I'm having with this is that installing seems to be nontrivial.
All documentation I can find seems to indicate that I should create a setup.py which creates a distutils.core.Extension. In every example I've found, the Extension object being created is given a list of source files, which it compiles. If my code was one or two files, this would be fine. Unfortunately, it's dozens of files, and I use a number of relatively complicated visual studio build settings. As a result, building by listing .c files seems to be challenging to say the least.
I've currently configured my Python extension to build as a .dll and link against python39.lib. I tried changing the extension to .pyd and including the file in a manifest.in. After I created a setup.py and ran it, it created a .egg file that I verified did include the .pyd I created. However, after installing it, when I imported the module into python, the module was completely empty (and I verified that the PyInit_[module] function was not called). Python dll Extension Import says that I can import the dll if I change the extension to .pyd and place the file in the Dlls directory of python's installation. I've encountered two problems with this.
The first is that it seems to me that it's not very distributable like this. I'd like to package this into a python wheel, and I'm not sure how a wheel could do this. The second is even more problematic - it doesn't exactly work. It calls the initialization function of my extension, and I've verified in WinDbg that it's returning a python module. However, this is what I always get from the console.
>>> import bluespawn
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
SystemError: initialization of bluespawn did not return an extension module
The Python documentation has a section on publishing binary extensions, but for the past four years, it has been left as a placeholder. The github issue linked here isn't that helpful either; it boils down to either use distutils to build or use enscons to build. But since my build is a fairly complicated procedure, completely rewriting it to use enscons is less than desirable, to say the least.
It seems to me like placing the file in the DLLs directory is the wrong way of going about this. Given that I have a DLL and making setuptools compile everything itself seems infeasible, how should I go about installing my extension?
For reference, here's my initialization function, in case that's incorrect.
PyModuleDef bsModule{ PyModuleDef_HEAD_INIT, "bluespawn", "Bluespawn python bindings", -1, methods };
PyMODINIT_FUNC PyInit_bluespawn() {
PyObject* m;
Py_Initialize();
PyEval_InitThreads();
PyGILState_STATE state = PyGILState_Ensure(); // Crashes without this. Call to PyEval_InitThreads() required for this.
m = PyModule_Create(&bsModule);
PyGILState_Release(state);
Py_Finalize();
return m;
}
The python interface is available here: https://github.com/ION28/BLUESPAWN/blob/client-add-pylib/BLUESPAWN-win-client/src/user/python/PythonInterface.cpp
EDIT: I have a working solution that I am sure is not best practice. I created a very small C file that simply passes all calls it receives onto the large DLL I've already created. The C file is responsible for initializing the module, but everything else is handled inside the DLL. It works, but it seems like a very bad way of doing things. What I'm looking for is a better way of doing this.
Let me try and divide your post into two separate questions:
How to package a C++ library with a non-trivial compilation process using setuptools
Is it possible to distribute a python package with a precompiled library
1. How to package a C++ library with a non-trivial compilation process using setuptools
It is possible. I was quite surprised to see that setuptools offers many ways to override the compilation process, see the documentation here. For example, you can use the keyword argument extra_compile_args to pass extra arguments to the compiler.
In addition, as setup.py is a python file, you could relatively easily write some code to automatically collect all files needed for compilation. I'd done this myself in a project (github), and it worked quite well for me.
Here's some code from the setup.py:
libinjector = Extension('pyinjector.libinjector',
sources=[str(c.relative_to(PROJECT_ROOT))
for c in [LIBINJECTOR_WRAPPER, *LIBINJECTOR_SRC.iterdir()]
if c.suffix == '.c'],
include_dirs=[str(LIBINJECTOR_DIR.relative_to(PROJECT_ROOT) / 'include')],
export_symbols=['injector_attach', 'injector_inject', 'injector_detach'],
define_macros=[('EM_AARCH64', '183')])
2. Is it possible to distribute a python package with a precompiled library
I understand from your edit that you've managed to get it to work, but I'll say a few words anyway. Releasing precompiled binaries with your source distribution is possible, and it is possible to release your manually-compiled binaries in a wheel file as well, but it is not recommended.
The main reason is compatibility with the target architecture. First, you'll have to include two DLLs in your distribution, one for x64 and one for x86. Second, you might lose some nice optimizations, because you'll have to instruct the compiler to ignore optimizations available for the specific CPU type (note that this applies to normal wheel distributions as well). If you're compiling against windows SDK, you'll probably want to use the user's version too. In addition, including two DLLs in your release might grow it to an awkward size for a source distribution.
I'm using an IronPython script engine in my C# application. It generally works ok, but for some reason it cannot find the "math" library (i.e. I can't "import math"). I checked my DLL's (IronPython.dll, Microsoft.Scripting, Microsoft.Dynamic) and they all seem to be ok and recent version (I copied them out of an IronPython 2.7.7.0 installation). However, when I try to execute an "import math" command, it says "No module named math". I can import "sys" and other modules ok, why not "math"?
Here's a simplified version of my code:
pyEngine = Python.CreateEngine();
outputStream = new NotifyingMemoryStream();
outputStream.TextWritten += OutputStream_TextWritten;
outputStreamWriter = new StreamWriter(outputStream) { AutoFlush = true };
pyEngine.Runtime.IO.SetOutput(outputStream, outputStreamWriter);
pyEngine.Runtime.IO.SetErrorOutput(outputStream, outputStreamWriter);
ScriptSource source = pyEngine.CreateScriptSourceFromString("import math" + Environment.NewLine + "math.log(10)", Microsoft.Scripting.SourceCodeKind.AutoDetect);
double b = source.Execute<double>()
The error occurs at the "double b = source..." line. Any help would be appreciated.
Just copying the IronPython DLLs out of an IronPython installation into your project does not get you the standard library.
The reason sys works is that it's one of a handful of special modules that are not just "builtins", but literally linked or frozen into the main interpreter. Most of the stdlib will not work.
The short version is: You have to either:
Also copy the standard library, or
Reference it in-place like this:
ICollection searchPaths = pythonEngine.GetSearchPaths();
searchPaths.add("D:\Absolute\Path\To\IronPython");
pythonEngine.SetSearchPaths(searchPaths);
Obviously, the latter solution won't work if you want to deploy or distribute your app.
According to this blog post and a few others, it looks like the way most people handle this is to NuGet the stdlib into your project instead of copying stuff around manually. And, while you're at it, to NuGet IronPython instead of copying and pasting the DLLs.
This still doesn't completely solve deploy/distribute, but from there, it's just a matter of configuring your build to copy whichever parts of that Lib you want into your target, basically the same way you're presumably already doing with the DLLs. If you copy some or all of the stdlib libs into your build target, they'll automatically be on your search path; if you copy them into some custom subdirectory instead, you'll of course need to add that as shown above.
If you don't plan to deploy/distribute, you still may want to configure the copy. But, if not, you can just add ..\\.. to your search path.
(By the way, all of this is definitely not what I vaguely remember doing… but then what I remember is probably horribly out of date.)
I have a Qt C++ project in which, for some functionality,I have added Python. The C++ function calls python script and returns the values.Like the below Example.
PyObject *pName,*pModule,*pFunc;
PyObject *pArgs,*pValue;
const char *module="getBeamData";
pName=PyUnicode_FromString(module);
pModule=PyImport_Import(pName);
if(pModule!=NULL){
pFunc=PyObject_GetAttrString(pModule,"getBeamDose");
if(pFunc&&PyCallable_Check(pFunc)){
pArgs=PyTuple_New(2);
PyTuple_SetItem(pArgs,0,PyBytes_FromString(path.toStdString().c_str()));
PyTuple_SetItem(pArgs,1,PyLong_FromLong(fraction));
pValue=PyObject_CallObject(pFunc,pArgs);
It calls the script getBeamdata.py. Functionality works fine. Do I have to install python and libraries(like numpy) for all customer and ship script ? Or is there any other way without sending script.
Without statically compiling your Python script you will need to distribute:
requirements.txt (for all libraries/modules required by your script).
details on the correct version of Python required by the script (2.7/3 etc).
Alternatively you could look into creating an executable using py2exe and distribute that instead, although you'll need to modify the way your program interacts with the Python script.
Some further details on distributing Python can be found here.
If I were you, I'd do it with some sort of installer.
Now in that installer, you can of course embed the wheel version of your script, which is in addition going to be installed using setuptools.
In that setuptools script you can mention dependencies, but that means that your client should have either Python installed on his/her computer, or you should ship your own version of Python embedded with the software.
Such functionality can be also be achieved using Boost.Python
I am starting to convert a lot of C stuff in python 3.
In C I defined a directory called "Toolbox", where i put all my functions i needed in different programs, so called libraries.
To use a specific library i had just to add the line
#include "/home/User/Toolbox/VectorFunctions.h"
into my source. So i was able to use the same library in different sources.
In python i tried to write some Toolbox functions and implement them into the source with import VectorFunctions, which works, as long as the file VectorFunctions.py is in the same directory as the source.
I there a way (I think there must be one...) telling python that VectorFunctions.py is located in a different directory, e.g. /home/User/Python_Toolbox?
Thanks for any comment!
What I would do is to organize these toolbox functions into one installable Python package bruno_toolbox, with its setup.py, and then install it into development mode to system site packages, using python setup.py develop, and then use the bruno_toolbox like any other package on the system, everywhere. Then if that package feels useful, I'd publish it to PyPI for the benefit of everyone.
You can use python path. Writing this code beginning of your program :
import sys
sys.path.append('/home/User/Python_Toolbox')
If you have VectorFunctions.py in this folder you can import it :
import VectorFunctions
I'm developing on Windows, and I've searched everywhere without finding anyone talking about this kind of thing.
I made a C++ app on my desktop that embedded Python 3.1 using MSVC. I linked python31.lib and included python31.dll in the app's run folder alongside the executable. It works great. My extension and embedding code definitely works and there are no crashes.
I sent the run folder to my friend who doesn't have Python installed, and the app crashes for him during the scripting setup phase.
A few hours ago, I tried the app on my laptop that has Python 2.6 installed. I got the same crash behavior as my friend, and through debugging found that it was the Py_Initialize() call that fails.
I installed Python 3.1 on my laptop without changing the app code. I ran it and it runs perfectly. I uninstalled Python 3.1 and the app crashes again. I put in code in my app to dynamically link from the local python31.dll, to ensure that it was using it, but I still get the crash.
I don't know if the interpreter needs more than the DLL to start up or what. I haven't been able to find any resources on this. The Python documentation and other guides do not seem to ever address how to distribute your C/C++ applications that use Python embedding without having the users install Python locally. I know it's more of an issue on Windows than on Unix, but I've seen a number of Windows C/C++ applications that embed Python locally and I'm not sure how they do it.
What else do I need other than the DLL? Why does it work when I install Python and then stop working when I uninstall it? It sounds like it should be so trivial; maybe that's why nobody really talks about it. Nevertheless, I can't really explain how to deal with this crash issue.
Thank you very much in advance.
In addition to pythonxy.dll, you also need the entire Python library, i.e. the contents of the lib folder, plus the extension modules, i.e. the contents of the DLLs folder. Without the standard library, Python won't even start, since it tries to find os.py (in 3.x; string.py in 2.x). On startup, it imports a number of modules, in particular site.py.
There are various locations where it searches for the standard library; in your cases, it eventually finds it in the registry. Before, uses the executable name (as set through Py_SetProgramName) trying to find the landmark; it also checks for a file python31.zip which should be a zipped copy of the standard library. It also checks for a environment variable PYTHONHOME.
You are free to strip the library from stuff that you don't need; there are various tools that compute dependencies statically (modulefinder in particular).
If you want to minimize the number of files, you can
link all extension modules statically into your pythonxy.dll, or even link pythonxy.dll statically into your application
use the freeze tool; this will allow linking the byte code of the standard library into your pythonxy.dll.
(alternatively to 2.) use pythonxy.zip for the standard library.
Nice. And if you do not want to zip, copy Python26\DLLs & Python26\lib to your exe directory as:
.\myexe.exe
.\python26.dll
.\Python26\DLLs
.\Python26\lib
And then set PYTHONHOME with Py_SetPythonHome() API. Apparently, this API is not in the list of "allowed" calls before Py_Initialize();
Below worked for me on Windows (Python not installed):
#include "stdafx.h"
#include <iostream>
#include "Python.h"
using namespace std;
int _tmain(int argc, _TCHAR* argv[])
{
char pySearchPath[] = "Python26";
Py_SetPythonHome(pySearchPath);
Py_Initialize();
PyRun_SimpleString("from time import time,ctime\n"
"print 'Today is',ctime(time())\n");
//cerr << Py_GetPath() << endl;
Py_Finalize();
return 0;
}
Good that the search path is relative w.r.t the exe. Py_GetPath can show you where all it is looking for the modules.
A zip of the Python standard library worked for me with Python27.
I zipped the contents of Lib and dll, and made sure there was no additional python27-subfolder or Lib or dll subfolder. i.e. just a zip named python27.zip containing all the files.
I copied that zip and the python27.dll alongside the executable.
I wanted to add some additional info for others who might still be having troubles with this, as I was. I was eventually able to get my application working using the method proposed by user sambha, that is:
Program Files (x86)\
MyApplicationFolder\
MyApplication.exe
python27.dll
Python27\
DLLs\ (contents of DLLs folder)
Lib\ (contents of Lib folder)
...with one important addition: I also needed to install the MSVCR90.DLL. I'm using Python 2.7 and apparently python27.dll requires the MSVCR90.DLL (and maybe other MSVC*90.DLLs).
I solved this by downloading, installing, and running the 'vcredist_x86.exe' package from http://www.microsoft.com/en-us/download/details.aspx?id=29 . I think, though I am not certain, that you need to do it this way at least on Win7, as opposed to simply placing the MSVC*90.DLLs alongside your .exe as you may have done in the past. The Microsoft installer places the files and registers them in a special way under Win7.
I also tried the .zip file method but that did not work, even with the MSVCR90.DLL installed.