I am trying to create an exe on Windows from a Python 3 package with a C extension module. In distutils, you can create an extension like this:
from distutils.core import setup, Extension
module1 = Extension('demo',
sources = ['demo.c'])
setup (name = 'PackageName',
version = '1.0',
description = 'This is a demo package',
ext_modules = [module1])
Than, the extension will be compiled with the appropriate compiler and placed alongside your other modules with the command:
python setup.py build_ext --inplace
cx_Freeze is a module that can package your code into an exe file along with a Python interpreter and the relevant packages. Then, an end user can use your program without having a Python installation. Unfortunately, cx_Freeze doesn't have an Extension class, and I cannot find a way to handle compilation with cx_Freeze.
One solution I am unsure about is to first build the extensions in place with distutils/setuptools, and then use cx_Freeze to create the executable. I don't want to reinvent the wheel though, so I wonder if someone else with more experience in this area has a solution.
I found a working solution. I can import Extension from distutils, and pass it into the setup from cx_Freeze:
from cx_Freeze import setup
from distutils.core import Extension
...
setup=(...
ext_modules=Extension(...))
This makes sense, since cx_Freeze is built on top of distutils. Originally, I was trying to use setuptools.setup, but that doesn't work.
Related
I have read How to enable `--embed` with cythonize? and Cython --embed flag in setup.py but this method does not work on Windows. Let's say we have:
# app.py
print("hello")
# build.py
import setuptools # allows us to avoid calling vcvarsall.bat, see https://stackoverflow.com/a/53172602/
from distutils.core import setup
from Cython.Build import cythonize
from Cython.Compiler import Options
Options.embed = "main"
setup(ext_modules=cythonize(r'app.py', build_dir="release"), script_args=['build'], options={'build': {'build_lib': 'release'}})
Running this build.py script on Python 3.8 for Windows does not produce an app.exe file like it would with the command line command:
cython app.py --embed
Instead, it produces a .pyd file.
How to use cythonize + embed from a Python script, producing a .exe, on Windows?
Solved: in fact the problem did not come from cythonize itself, but from the fact distutils.core.setup(...) is configured to compile+link into a .pyd instead of a .exe.
Here is the solution:
from distutils._msvccompiler import MSVCCompiler # "from distutils.msvccompiler" did not work for me!
from Cython.Compiler import Options
Options.embed = "main"
cythonize(r'src\app.py', build_dir="build")
compiler = MSVCCompiler()
compiler.compile([r"build\src\app.c"], include_dirs=["C:/Python38/include"])
compiler.link_executable([r"build\src\app.obj"], "app", libraries=["python38"], library_dirs=["C:/Python38/libs"], output_dir="release", extra_preargs=["/NOIMPLIB", "/NOEXP"])
The .exe will be in the release folder.
(Note: I also upgraded Cython to the latest version 0.29.30, it might have helped as well)
I am trying to link to my own C library from Cython, following the directions I've found on the web, including this answer:
Using Cython To Link Python To A Shared Library
I am running IPython through Spyder.
My setup.py looks like this:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
import numpy as np
setup(
ext_modules = cythonize(
[Extension("*",["*.pyx"],
libraries =["MyLib"],
extra_compile_args = ["-fopenmp","-O3"],
extra_link_args=["-L/path/to/lib"])
]),
include_dirs = [np.get_include()],
)
The file libMyLib.so is in /path/to/lib and it compiles fine.
I have a Python script in my IPython profile startup folder that does this
try:
os.environ["LD_LIBRARY_PATH"] += ":/path/to/lib"
except KeyError:
os.environ["LD_LIBRARY_PATH"] = "/path/to/lib"
I can confirm that this is running, because if I type os.environ["LD_LIBRARY_PATH"] into the IPython interpreter, it returns /path/to/lib
But when I try to load the Cython module (i.e. import mycythonmodule) I get:
ImportError: libMyLib.so: cannot open shared object file: No such file or directory
I've also tried putting libMyLib.so in other places to see if cython would find it:
In the directory where Python is running
On the Python path
In the same folder as the cython module
But it still doesn't find the shared library. The only way I can get it to find the library is by dropping it in /usr/lib, but I don't want it there, I want to be able to set the library path.
Am I missing something?
I'm self-answering, in case anyone else runs into the same problem. Looks like the answers are here:
Set LD_LIBRARY_PATH before importing in python
Changing LD_LIBRARY_PATH at runtime for ctypes
According to these answers (and my experience), the linker reads LD_LIBRARY_PATH when python is launched, so changing it from within python doesn't have any useful effect, at least not the effect I was hoping for. The only solution is to either wrap python in a shell script that sets LD_LIBRARY_PATH, or else drop the shared object somewhere on the linker search path.
Kind of a pain, but it is what it is.
I have fixed it by change setup.py.
I have a C++ dynamic shared library called "libtmsmdreader.so". and a header file named "TmsMdReader.hpp"
I wrapper C++ shared library to cython library called "tmsmdreader-pythonxxxxxx.so"
from setuptools import setup
from distutils.extension import Extension
from Cython.Build import cythonize
setup(
name="tmsmdreader",
ext_modules=cythonize([
Extension(
name="tmsmdreader",
language="c++",
sources=["TmsMdReaderApi.pyx"],
libraries=["tmsmdreader"],
library_dirs=["."],
include_dirs=["."],
extra_compile_args=["-std=c++14"],
compiler_directives={'language_level': 3},
runtime_library_dirs=["."])
]))
library_dirs=["."] and runtime_library_dirs=["."] can fixed LD_LIBRARY_PATH if libtmsmdreader.so in python scripy directory
I would like to create a setup.py script for a python package with several submodules that depend on both cython and f2py. I have attempted to use setuptools and numpy.distutils, but have so far failed:
Using setuptools
I am able to compile my cython extensions (and create an installation for the rest of the package) using setuptools. I have, however, been unable to figure out how to use setuptools to generate the f2py extension. After extensive searching, I only found rather old messages like this one that state that f2py modules must be compiled using numpy.distutils.
Using numpy.distutils
I am able to compile my f2py extensions (and create an installation for the rest of the package) using numpy.distutils. I have, however, been unable to figure out how to get numpy.distutils to compile my cython extensions as it always attempts to use pyrex to compile it (and I am using extensions specific to cython) recent. I have done a search to figure out how to get numpy.distutils for cython files and - at least as of a year ago - they recommend applying a monkey patch to numpy.distutils. It seems applying such a monkey patch also restricts the options that can be passed to Cython.
My question is: what is the recommended way to write a setup.py script for packages that depend on both f2py and cython? Is applying a patch to numpy.distutils really the way to go still?
You can just call each separately in your setup.py as in
http://answerpot.com/showthread.php?601643-cython%20and%20f2py
# Cython extension
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
setup(
ext_modules = [Extension( 'cext', ['cext.pyx'] )],
cmdclass = {'build_ext': build_ext},
script_args = ['build_ext', '--inplace'],
)
# Fortran extension
from numpy.distutils.core import setup, Extension
setup(
ext_modules = [Extension( 'fext', ['fext.f90'] )],
)
Your calling context (I think they call this namespace, not sure)
has to change as to what the current object Extension and function
setup() is.
The first setup() call, it's the distutils.extension.Extension
and distutils.core.setup()
The second setup() call, it's the numpy.distutils.core.Extension
and numpy.distutils.core.setup()
Turns out this is no longer true. With both setuptools and distutils (at least the numpy version) it is possible to have extensions with C, Cython and f2py. The only caveat is that to compile f2py modules one must always use numpy.distutils for both the setup and Extension functions. But setuptools can still be used for the installation (for example, allowing the installation of a developer version with python setup.py develop).
To use distutils exclusively you use the following:
from numpy.distutils.core import setup
from numpy.distutils.extension import Extension
To use setuptools, you need to import it before the distutils imports:
import setuptools
And then the rest of the code is identical:
from numpy import get_include
from Cython.Build import cythonize
NAME = 'my_package'
NUMPY_INC = get_include()
extensions = [
Extension(name=NAME + ".my_cython_ext",
include_dirs=[NUMPY_INC, "my_c_dir"]
sources=["my_cython_ext.pyx", "my_c_dir/my_ext_c_file.c"]),
Extension(name=NAME + ".my_f2py_ext",
sources=["my_f2py_ext.f"]),
]
extensions = cythonize(extensions)
setup(..., ext_modules=extensions)
Obviously you need to put all your other stuff in the setup() call. In the above I assume that you'll use numpy with Cython, along with an external C file (my_ext_c_file.c) that will be at my_c_dir/, and that the f2py module is only one Fortran file. Adjust as needed.
I use the following script to distribute a module containing pure python code.
from distutils.core import setup, Extension
import os
setup (name = 'mtester',
version = '0.1',
description = 'Python wrapper for libmtester',
packages=['mtester'],
package_dir={'mtester':'module'},
)
The problem I have is, I modified one of the files that uses an external library (a .so file), which I need to ship along with the existing module. I was suggested to use package_data to include the library. I modified the script to the following.
from distutils.core import setup, Extension
import os
data_dir = os.path.abspath('../lib64/')
setup (name = 'mtester',
version = '0.1',
description = 'Python wrapper for libmtester',
packages=['mtester'],
package_dir={'mtester':'module'},
package_data={'mtester':[data_dir+'mhelper.so']},
)
The problem is, adding package_data did not make any difference. This is not installing the mhelper.so in any location (neither in site-packages nor in site-packages/mtester).
System info: Fedora 10, 64 bit, python 2.5 (Yes it is ancient. But it is our build machine, and it needs to stay that way to maintain backward compatibility)
Any suggestions that would help me resolve this would be well appreciated!
Unfortunately package_data looks for files relative to the top of the package. One fix is to move the helper library under the module dir with the rest of the code:
% mv lib64/mhelper.so module/
Then modify the package_data argument accordingly:
package_data = {'mtester': ['mhelper.so']}
...
Then test:
% python setup.py bdist
% tar tf dist/mtester-0.1.linux-x86_64.tar.gz | grep mhelper
./usr/local/lib/python2.5/dist-packages/mtester/mhelper.so
I'd like to make a Python package containing some Cython code. I've got the the Cython code working nicely. However, now I want to know how best to package it.
For most people who just want to install the package, I'd like to include the .c file that Cython creates, and arrange for setup.py to compile that to produce the module. Then the user doesn't need Cython installed in order to install the package.
But for people who may want to modify the package, I'd also like to provide the Cython .pyx files, and somehow also allow for setup.py to build them using Cython (so those users would need Cython installed).
How should I structure the files in the package to cater for both these scenarios?
The Cython documentation gives a little guidance. But it doesn't say how to make a single setup.py that handles both the with/without Cython cases.
I've done this myself now, in a Python package simplerandom (BitBucket repo - EDIT: now github) (I don't expect this to be a popular package, but it was a good chance to learn Cython).
This method relies on the fact that building a .pyx file with Cython.Distutils.build_ext (at least with Cython version 0.14) always seems to create a .c file in the same directory as the source .pyx file.
Here is a cut-down version of setup.py which I hope shows the essentials:
from distutils.core import setup
from distutils.extension import Extension
try:
from Cython.Distutils import build_ext
except ImportError:
use_cython = False
else:
use_cython = True
cmdclass = {}
ext_modules = []
if use_cython:
ext_modules += [
Extension("mypackage.mycythonmodule", ["cython/mycythonmodule.pyx"]),
]
cmdclass.update({'build_ext': build_ext})
else:
ext_modules += [
Extension("mypackage.mycythonmodule", ["cython/mycythonmodule.c"]),
]
setup(
name='mypackage',
...
cmdclass=cmdclass,
ext_modules=ext_modules,
...
)
I also edited MANIFEST.in to ensure that mycythonmodule.c is included in a source distribution (a source distribution that is created with python setup.py sdist):
...
recursive-include cython *
...
I don't commit mycythonmodule.c to version control 'trunk' (or 'default' for Mercurial). When I make a release, I need to remember to do a python setup.py build_ext first, to ensure that mycythonmodule.c is present and up-to-date for the source code distribution. I also make a release branch, and commit the C file into the branch. That way I have a historical record of the C file that was distributed with that release.
Adding to Craig McQueen's answer: see below for how to override the sdist command to have Cython automatically compile your source files before creating a source distribution.
That way your run no risk of accidentally distributing outdated C sources. It also helps in the case where you have limited control over the distribution process e.g. when automatically creating distributions from continuous integration etc.
from distutils.command.sdist import sdist as _sdist
...
class sdist(_sdist):
def run(self):
# Make sure the compiled Cython files in the distribution are up-to-date
from Cython.Build import cythonize
cythonize(['cython/mycythonmodule.pyx'])
_sdist.run(self)
cmdclass['sdist'] = sdist
http://docs.cython.org/en/latest/src/userguide/source_files_and_compilation.html#distributing-cython-modules
It is strongly recommended that you distribute the generated .c files as well as your Cython sources, so that users can install your module without needing to have Cython available.
It is also recommended that Cython compilation not be enabled by default in the version you distribute. Even if the user has Cython installed, he probably doesn’t want to use it just to install your module. Also, the version he has may not be the same one you used, and may not compile your sources correctly.
This simply means that the setup.py file that you ship with will just be a normal distutils file on the generated .c files, for the basic example we would have instead:
from distutils.core import setup
from distutils.extension import Extension
setup(
ext_modules = [Extension("example", ["example.c"])]
)
The easiest is to include both but just use the c-file? Including the .pyx file is nice, but it's not needed once you have the .c file anyway. People who want to recompile the .pyx can install Pyrex and do it manually.
Otherwise you need to have a custom build_ext command for distutils that builds the C file first. Cython already includes one. http://docs.cython.org/src/userguide/source_files_and_compilation.html
What that documentation doesn't do is say how to make this conditional, but
try:
from Cython.distutils import build_ext
except ImportError:
from distutils.command import build_ext
Should handle it.
Including (Cython) generated .c files are pretty weird. Especially when we include that in git. I'd prefer to use setuptools_cython. When Cython is not available, it will build an egg which has built-in Cython environment, and then build your code using the egg.
A possible example: https://github.com/douban/greenify/blob/master/setup.py
Update(2017-01-05):
Since setuptools 18.0, there's no need to use setuptools_cython. Here is an example to build Cython project from scratch without setuptools_cython.
All other answers either rely on
distutils
importing from Cython.Build, which creates a chicken-and-egg problem between requiring cython via setup_requires and importing it.
A modern solution is to use setuptools instead, see this answer (automatic handling of Cython extensions requires setuptools 18.0, i.e., it's available for many years already). A modern standard setup.py with requirements handling, an entry point, and a cython module could look like this:
from setuptools import setup, Extension
with open('requirements.txt') as f:
requirements = f.read().splitlines()
setup(
name='MyPackage',
install_requires=requirements,
setup_requires=[
'setuptools>=18.0', # automatically handles Cython extensions
'cython>=0.28.4',
],
entry_points={
'console_scripts': [
'mymain = mypackage.main:main',
],
},
ext_modules=[
Extension(
'mypackage.my_cython_module',
sources=['mypackage/my_cython_module.pyx'],
),
],
)
The simple hack I came up with:
from distutils.core import setup
try:
from Cython.Build import cythonize
except ImportError:
from pip import pip
pip.main(['install', 'cython'])
from Cython.Build import cythonize
setup(…)
Just install Cython if it could not be imported. One should probably not share this code, but for my own dependencies it's good enough.
This is a setup script I wrote which makes it easier to include nested directories inside the build. One needs to run it from folder within a package.
Givig structure like this:
__init__.py
setup.py
test.py
subdir/
__init__.py
anothertest.py
setup.py
from setuptools import setup, Extension
from Cython.Distutils import build_ext
# from os import path
ext_names = (
'test',
'subdir.anothertest',
)
cmdclass = {'build_ext': build_ext}
# for modules in main dir
ext_modules = [
Extension(
ext,
[ext + ".py"],
)
for ext in ext_names if ext.find('.') < 0]
# for modules in subdir ONLY ONE LEVEL DOWN!!
# modify it if you need more !!!
ext_modules += [
Extension(
ext,
["/".join(ext.split('.')) + ".py"],
)
for ext in ext_names if ext.find('.') > 0]
setup(
name='name',
ext_modules=ext_modules,
cmdclass=cmdclass,
packages=["base", "base.subdir"],
)
# Build --------------------------
# python setup.py build_ext --inplace
Happy compiling ;)
The easiest way I found using only setuptools instead of the feature limited distutils is
from setuptools import setup
from setuptools.extension import Extension
try:
from Cython.Build import cythonize
except ImportError:
use_cython = False
else:
use_cython = True
ext_modules = []
if use_cython:
ext_modules += cythonize('package/cython_module.pyx')
else:
ext_modules += [Extension('package.cython_module',
['package/cython_modules.c'])]
setup(name='package_name', ext_modules=ext_modules)
I think I found a pretty good way of doing this by providing a custom build_ext command. The idea is the following:
I add the numpy headers by overriding finalize_options() and doing import numpy in the body of the function, which nicely avoids the problem of numpy not being available before setup() installs it.
If cython is available on the system, it hooks into the command's check_extensions_list() method and by cythonizes all out-of-date cython modules, replacing them with C extensions that can later handled by the build_extension() method. We just provide the latter part of the functionality in our module too: this means that if cython is not available but we have a C extension present, it still works, which allows you to do source distributions.
Here's the code:
import re, sys, os.path
from distutils import dep_util, log
from setuptools.command.build_ext import build_ext
try:
import Cython.Build
HAVE_CYTHON = True
except ImportError:
HAVE_CYTHON = False
class BuildExtWithNumpy(build_ext):
def check_cython(self, ext):
c_sources = []
for fname in ext.sources:
cname, matches = re.subn(r"(?i)\.pyx$", ".c", fname, 1)
c_sources.append(cname)
if matches and dep_util.newer(fname, cname):
if HAVE_CYTHON:
return ext
raise RuntimeError("Cython and C module unavailable")
ext.sources = c_sources
return ext
def check_extensions_list(self, extensions):
extensions = [self.check_cython(ext) for ext in extensions]
return build_ext.check_extensions_list(self, extensions)
def finalize_options(self):
import numpy as np
build_ext.finalize_options(self)
self.include_dirs.append(np.get_include())
This allows one to just write the setup() arguments without worrying about imports and whether one has cython available:
setup(
# ...
ext_modules=[Extension("_my_fast_thing", ["src/_my_fast_thing.pyx"])],
setup_requires=['numpy'],
cmdclass={'build_ext': BuildExtWithNumpy}
)