Develop Reference

How to use Cython to compile Python 3 into C

I'm trying to convert a Python 3 script into C and then compile that C file into an executable.
I have this simple python script:
def greet(name = ""):
print("Hello {0}".format(name if len(name) > 0 else "World"))
greet("Mango")
I've converted this script into C using:
cython greet.py -o greet.c
Then I've compiled the C file using:
cc greet.c -o greet
After I entered the last command I got the error:
fatal error: Python.h: No such file or directory compilation terminated.
After I got the error I went back and realised that I was using Python3 and that I had forgot the "3" after "cython".
So re-compiled the python script using:
cython3 greet.py -o greet.c
Then attempted to re-compile the C file using:
cc greet.c -o greet
Again this failed and threw the same error so I went searching on SO and Google and found these questions:
fatal error: Python.h: No such file or directory
I have Python on my Ubuntu system, but gcc can't find Python.h
https://askubuntu.com/questions/526708/fatal-error-python-h-no-file-or-directory
None of these answers in these questions work.
I've made sure that I have installed cython all of the correct dependencies using apt-get install and pip install sadly thought it still does not seem to work.

Check the documentation. It's not enough to do gcc x.c -o x.
This page explains compilation: http://docs.cython.org/src/reference/compilation.html
There's a lot more to it, but a direct answer is:
Compiling your .c files will vary depending on your operating system. Python documentation for writing extension modules should have some details for your system. Here we give an example on a Linux system:
$ gcc -shared -pthread -fPIC -fwrapv -O2 -Wall -fno-strict-aliasing
-I/usr/include/python2.7 -o yourmod.so yourmod.c
Of course in your situation it's going to be something closer to -I/usr/include/python3.4, or even $(pkg-config --libs --cflags python3). And you're not building with -shared, because you want an executable.
Shortest "this has to work" set of commands is:
cython3 --embed greet.py -o greet.c
gcc $(pkg-config --libs --cflags python3) greet.c -o greet
You need to install pkg-config if it's missing.

As #viraptor's answer shows you and as per my comment, your main problem is that you need to tell your C compiler (e.g. gcc) where to find the python headers required (pyconfig.h and Python.h). To do this, you need to pass a -I option to gcc.
The other answer suggests using pkg-config to add this to your command line. However, like you, with Ubuntu 14.04, cython3 and python3-dev installs, using this method leads the compiled program to exit with a segmentation fault for me.
So, I suggest you go back to basics. After
cython greet.py -o greet.c
Run the following command. It assumes that Python.h and friends are in the standard place (i.e. you've done a standard install of python3-dev)
gcc -I/usr/include/python3.4m -o greet greet.c -lpython3.4m
If that doesn't work - use find / -iname Python.h to find the location of the necessary files and alter the -I path accordingly.
In time, when you want to use cython on more complex programs, such as those that link to other C libraries, you'll need to learn about the other options you need to pass to gcc to get it to compile and link correctly. To get you going, though, the above should work (tested on Ubuntu 14.04 as per your spec)
P.S. I'm not sure why the pkg-config suggestion doesn't work - but for me it seems to add in an extra path to -I which breaks things.

Python 2:
python -m pip install --upgrade cython
Python 3:
python3 -m pip install --upgrade cython

Gensim with MinGW

I seem to be one of the many people struggling to install gensim on windows. I have trawled through countless forums but the errors poster there never appear to match my errors. So hopefully someone can point me in the right direction!
I am running Windows Server 2012 R2 Standard 64-bit. I have installed MinGW & Anaconda 2.2.0 (64-bit), which comes with Python 2.7.9.
I have added a file distutils.cfg into C:\Users\Sam\Anaconda\Lib\distutils with the contents:
[build]
compiler=mingw32
I have added C:\MinGW\bin to my Environment variables.
If I install gensim using pip I do not get any errors, until I try to run Word2Vec when I get the error:
C:\Users\sam.passmore\AppData\Local\Continuum\Anaconda\lib\site-packages\gensim\models\word2vec.py:459: UserWarning: C extension com
pilation failed, training will be slow. Install a C compiler and reinstall gensim for fast training.
So I have uninstalled gensim and tried to re-install using the mingw32 compiler, but this gives me this error:
python setup.py build --compiler=mingw32
c:\users\sam.passmore\appdata\local\continuum\anaconda\lib\site-packages\setuptools-14.3-py2.7.egg\setuptools\dist.py:282: UserWarni
ng: Normalizing '0.11.1-1' to '0.11.1.post1'
running build
running build_ext
building 'gensim.models.word2vec_inner' extension
C:\MinGW\bin\gcc.exe -DMS_WIN64 -mdll -O -Wall -Igensim\models -IC:\Users\sam.passmore\AppData\Local\Continuum\Anaconda\include -IC:
\Users\sam.passmore\AppData\Local\Continuum\Anaconda\PC -IC:\Users\sam.passmore\AppData\Local\Continuum\Anaconda\lib\site-packages\n
umpy\core\include -c ./gensim/models/word2vec_inner.c -o build\temp.win-amd64-2.7\Release\.\gensim\models\word2vec_inner.o
gcc: error: ./gensim/models/word2vec_inner.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.
command 'C:\\MinGW\\bin\\gcc.exe' failed with exit status 1
setup.py:82: UserWarning:
********************************************************************
WARNING: %s could not
be compiled. No C extensions are essential for gensim to run,
although they do result in significant speed improvements for some modules.
%s
Here are some hints for popular operating systems:
If you are seeing this message on Linux you probably need to
install GCC and/or the Python development package for your
version of Python.
Debian and Ubuntu users should issue the following command:
$ sudo apt-get install build-essential python-dev
RedHat, CentOS, and Fedora users should issue the following command:
$ sudo yum install gcc python-devel
If you are seeing this message on OSX please read the documentation
here:
http://api.mongodb.org/python/current/installation.html#osx
********************************************************************
The gensim.models.word2vec_inner extension moduleThe output above this warning shows how the compilation failed.
"The output above this warning shows how the compilation failed.")
building 'gensim.models.doc2vec_inner' extension
C:\MinGW\bin\gcc.exe -DMS_WIN64 -mdll -O -Wall -Igensim\models -IC:\Users\sam.passmore\AppData\Local\Continuum\Anaconda\include -IC:
\Users\sam.passmore\AppData\Local\Continuum\Anaconda\PC -IC:\Users\sam.passmore\AppData\Local\Continuum\Anaconda\lib\site-packages\n
umpy\core\include -c ./gensim/models/doc2vec_inner.c -o build\temp.win-amd64-2.7\Release\.\gensim\models\doc2vec_inner.o
gcc: error: ./gensim/models/doc2vec_inner.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.
command 'C:\\MinGW\\bin\\gcc.exe' failed with exit status 1
setup.py:82: UserWarning:
********************************************************************
WARNING: %s could not
be compiled. No C extensions are essential for gensim to run,
although they do result in significant speed improvements for some modules.
%s
Here are some hints for popular operating systems:
If you are seeing this message on Linux you probably need to
install GCC and/or the Python development package for your
version of Python.
Debian and Ubuntu users should issue the following command:
$ sudo apt-get install build-essential python-dev
RedHat, CentOS, and Fedora users should issue the following command:
$ sudo yum install gcc python-devel
If you are seeing this message on OSX please read the documentation
here:
http://api.mongodb.org/python/current/installation.html#osx
********************************************************************
The gensim.models.doc2vec_inner extension moduleThe output above this warning shows how the compilation failed.
"The output above this warning shows how the compilation failed."
I have exhausted all options I can think of or find, so if anyone could give some advice it would be much appreciated.

I managed to solve this after using conda install for gensim, rather than pip.
conda install gensim
I am not sure what other steps I have included above have contributed to the answer, but this was the last thing I did before I no longer was getting the 'Install a C compiler and reinstall gensim for fast training.' message.
During my research in trying to solve this problem I saw that the most common methods were adding the lines
[build]
compiler=mingw32
to the distutils.cfg file as well as ensuring MinGW is in your path. Also ensuring that the MinGW bit version is the same as your python version.

Python pip library search paths

I'm trying to install the python package cvxopt via pip into a virtual environment on a Centos 6 system.
Since lapack is a prerequisite, and since I don't have superuser access to the system, I've installed lapack locally using the following:
$ cmake -DCMAKE_INSTALL_PREFIX=/home/tom .
$ make install
I notice that after this step liblapack.a is located in my local lib directory, /home/tom/lib, as expected.
However, when I activate my virtual environment and run pip install cvxopt I get the following error:
<snip>
gcc -pthread -shared
build/temp.linux-x86_64-2.7/src/C/base.o
build/temp.linux-x86_64-2.7/src/C/dense.o
build/temp.linux-x86_64-2.7/src/C/sparse.o
-L/usr/lib -L/public/apps/python/2.7.4/lib
-lm -llapack -lblas -lpython2.7 -o build/lib.linux-x86_64-2.7/cvxopt/base.so
/usr/bin/ld: cannot find -llapack
collect2: ld returned 1 exit status
error: command 'gcc' failed with exit status 1
I've tried adding /home/tom/lib to both LD_LIBRARY_PATH and LD_RUN_PATH but neither of these seem to affect the gcc command line options that pip is using to build the module.
Can anyone tell me how to give library search paths to pip? Is there a better way?

After some experimentation, I was able to solve the problem. The first thing I had to do was enable the shared libraries for LAPACK:
~/src/lapack-3.5.0 $ cmake -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX=/home/tom .
~/src/lapack-3.5.0 $ make install
Then I exported LD_FLAGS to include the proper library search path:
export LD_FLAGS="-L/home/tom/lib"
Then I re-attempted the install of cvxopt from source into the virtual environment:
~/src/cvxopt-1.1.6 $ . venv/bin/activate
~/src/cvxopt-1.1.6 (venv)$ python setup.py build install
I didn't experiment with LD_LIBRARY_PATH or LD_RUN_PATH to determine whether they were still required with LD_FLAGS. However, I did determine that a shared LAPACK library was definitely required; cvxopt wouldn't link against the static library.

Compile Pygame 1.9.1 from source on Mac OSX 10.9 for Python 3.3

I'd like to start by saying that asking a question here isn't something I do lightly. I've now been attempting to install Pygame 1.9.1 from source for four hours, and I've run into several problems but was able to overcome each one. This is the first one I'm completely stumped on.
After I unpack pygame-1.9.1release.tar.gz and cd to the folder, I run 'python3 setup.py install'
/Library/Frameworks/SDL.framework/Versions/Current/Headers/SDL_syswm.h:58:10: fatal error: 'X11/Xlib.h' file not found
#include <X11/Xlib.h>
1 error generated.
error: command '/usr/bin/clang' failed with exit status 1
After some online research, I read that X11 is no longer included with OSX 10.9, so I installed XQuartz in the hope that it could serve as a replacement.
Xquartz does include 'Xlib.h', but it's filepath is /opt/X11/include/X11/Xlib.h, and the setup script doesn't find it here. I've tried several ways to fix this.
The error process specifically comes up in the build part of the install. I've tried using
python3 setup.py config --include-dirs /opt/X11/include
as well as
python3 setup.py config --include-dirs /opt/X11/include/X11
but get the same error.
I read online that installing the Xcode command line tools could fix this problem. I'm ~95% sure I had the command line tools before, but I tried to get them again anyway with 'xcode-select --install' only to be told that 'Can't install the software because it is currently unavailable from the Software Update server'. "No problem," I tell myself and just download the package from the Mac developers site. It doesn't help anything.
I've also tried symlinking with the command
sudo ln -s /usr/include/X11 /opt/X11/include/X11
Of everything I've tried, I understand this the least, do I could very well be doing the symlink command wrong.
Two other commands I tried are
export C_INCLUDE_PATH=/opt/X11/include
export CPLUS_INCLUDE_PATH=/opt/X11/include
Just looking at the documentation, I came across a debug option for the startup script.
The output of 'python3 setup.py build -g' is
running build
running build_py
running build_ext
building 'pygame.display' extension
/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -arch i386 -arch x86_64 -g -g -Ddarwin -I/Library/Frameworks/SDL.framework/Versions/Current/Headers -I/Library/Frameworks/Python.framework/Versions/3.3/include/python3.3m -c src/display.c -o build/temp.macosx-10.6-intel-3.3/src/display.o
In file included from src/display.c:30:
/Library/Frameworks/SDL.framework/Versions/Current/Headers/SDL_syswm.h:58:10: fatal error: 'X11/Xlib.h' file not found
#include <X11/Xlib.h>
^
1 error generated.
error: command '/usr/bin/clang' failed with exit status 1
I'm not sure what else I can do, but I'd really appreciate some help with this. Thanks in advance!

setup.py build|install commands accepts CFLAGS and LDFLAGS compiler options, as far as I know.
Thus here, you can do
CFLAGS=-I/opt/X11/include python3 setup.py install
You probably need the LDFLAGS as well; I'm not sure about the exact location, but something like
CFLAGS=-I/opt/X11/include LDFLAGS=-L/opt/X11/lib python3 setup.py install
seems logical.
After that, you may have to set your DYLD_LIBRARY_PATH to point to your X11 libraries. In case your installation proceeds but your PyGame script won't run and complains about not finding X11 libraries. Thus
export DYLD_LIBRARY_PATH=/opt/X11/lib
As for the symbolic link you tried to create: it's the wrong way 'round: the two paths should have been switched, so I guess you got an error because /opt/X11/include/X11 already exists. But with the above, no symlink is necessary.

I ran into a similar error when trying to install pygame into my PyCharm project's virtualenv. I solved it by cding to the SDL Framework directory where the error originated and copying the X11 header files into it:
$ cd /Library/Frameworks/SDL.framework/Versions/Current/Headers
$ cp -R /opt/X11/include/X11 ./
then $ pip install pygame ran without errors.

mysqldb build error

i remember installing Python + Django + MySQL + MySQLdb on my 32-bit Mac with Leopard 10.5.7.
I tried the same procedure with Mac Snow Leopard. But have unfortunately ran into a lot of errors...
i dont know but something weird is happening. Please look at the error log:
Amit-Vermas-MacBook:mysql-python-1.2.2 amitverma$ python setup.py build
running build
running build_py
copying MySQLdb/release.py -> build/lib.macosx-10.3-i386-2.5/MySQLdb
running build_ext
building '_mysql' extension
creating build/temp.macosx-10.3-i386-2.5
gcc-4.0 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 -Dversion_info=(1,2,2,'final',0) -D__version__=1.2.2 -I/usr/local/mysql/include -I/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5 -c _mysql.c -o build/temp.macosx-10.3-i386-2.5/_mysql.o -g -Os -arch x86_64 -fno-common -D_P1003_1B_VISIBLE -DSIGNAL_WITH_VIO_CLOSE -DSIGNALS_DONT_BREAK_READ -DIGNORE_SIGHUP_SIGQUIT -DDONT_DECLARE_CXA_PURE_VIRTUAL
In file included from /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/Python.h:57,
from pymemcompat.h:10,
from _mysql.c:29:
/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/pyport.h:761:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
In file included from _mysql.c:35:
/usr/local/mysql/include/my_config.h:1050:1: warning: "HAVE_WCSCOLL" redefined
In file included from /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/Python.h:8,
from pymemcompat.h:10,
from _mysql.c:29:
/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/pyconfig.h:721:1: warning: this is the location of the previous definition
In file included from _mysql.c:35:
/usr/local/mysql/include/my_config.h:1168:1: warning: "SIZEOF_LONG" redefined
In file included from /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/Python.h:8,
from pymemcompat.h:10,
from _mysql.c:29:
/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/pyconfig.h:811:1: warning: this is the location of the previous definition
In file included from _mysql.c:35:
/usr/local/mysql/include/my_config.h:1177:1: warning: "SIZEOF_PTHREAD_T" redefined
In file included from /Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/Python.h:8,
from pymemcompat.h:10,
from _mysql.c:29:
/Library/Frameworks/Python.framework/Versions/2.5/include/python2.5/pyconfig.h:820:1: warning: this is the location of the previous definition
error: command 'gcc-4.0' failed with exit status 1
Amit-Vermas-MacBook:mysql-python-1.2.2 amitverma$

This is my personal makefile rule for that
MYSQLDB_VERSION=1.2.3c1
MYSQLDB_TARGET=$(BUILD_FLAGS_DIR)/mysqldb
MYSQLDB_PACKAGE=MySQL-python-$(MYSQLDB_VERSION).tar.gz
MYSQLDB_PACKAGE_URL=http://downloads.sourceforge.net/project/mysql-python/mysql-python-test/$(MYSQLDB_VERSION)/$(MYSQLDB_PACKAGE)
.PHONY: mysqldb mysqldb-download
mysqldb: $(MYSQLDB_TARGET)
mysqldb-download: $(DOWNLOAD_DIR)/$(MYSQLDB_PACKAGE)
$(MYSQLDB_TARGET): $(INIT_TARGET) $(MYSQLDB_DEPS) $(DOWNLOAD_DIR)/$(MYSQLDB_PACKAGE)
-rm -rf $(UNPACK_DIR)/MySQL-python-$(MYSQLDB_VERSION)
tar -m -C $(UNPACK_DIR) -xzvf $(DOWNLOAD_DIR)/$(MYSQLDB_PACKAGE)
-cd $(UNPACK_DIR)/MySQL-python-$(MYSQLDB_VERSION); \
for patch in $(PATCH_DIR)/mysqldb-$(MYSQLDB_VERSION)_$(ARCH)_*; \
do patch -p1 < $$patch; \
done
cd $(UNPACK_DIR)/MySQL-python-$(MYSQLDB_VERSION); export CC="gcc -m64" FC="g95 -m64" CPPFLAGS="-I$(RUNTIME_DIR)/include" CFLAGS="-m64 -I$(RUNTIME_DIR)/include" LD_LIBRARY_PATH=$(RUNTIME_DIR)/lib64:$(RUNTIME_DIR)/lib:$$LD_LIBRARY_PATH PATH=$(RUNTIME_DIR)/bin:$$PATH PYTHONPATH=$(RUNTIME_DIR)/lib/python2.5/site-packages/; $(RUNTIME_DIR)/bin/python2.5 setup.py install --prefix=$(RUNTIME_DIR)
touch $(MYSQLDB_TARGET)
$(DOWNLOAD_DIR)/$(MYSQLDB_PACKAGE):
for package in $(MYSQLDB_PACKAGE_URL); \
do \
echo -n "Downloading $$package... "; \
cd $(DOWNLOAD_DIR); curl -L -O $$package; \
echo "done"; \
done
touch $#
ALL_RUNTIME_TARGETS+=$(MYSQLDB_TARGET)
ALL_DOWNLOAD_TARGETS+=$(DOWNLOAD_DIR)/$(MYSQLDB_PACKAGE)
And a patch
$ more mysqldb-1.2.3c1_x86_64-apple-darwin10_patch-000
diff -Naur MySQL-python-1.2.3c1/setup.py MySQL-python-1.2.3c1.new/setup.py
--- MySQL-python-1.2.3c1/setup.py 2008-10-18 02:12:31.000000000 +0200
+++ MySQL-python-1.2.3c1.new/setup.py 2009-10-08 22:59:05.000000000 +0200
## -13,6 +13,8 ##
from setup_windows import get_config
metadata, options = get_config()
+options["extra_compile_args"].remove("-arch")
+options["extra_compile_args"].remove("x86_64")
metadata['ext_modules'] = [Extension(sources=['_mysql.c'], **options)]
metadata['long_description'] = metadata['long_description'].replace(r'\n', '')
setup(**metadata)
And it works for me. I cannot guarantee, but... maybe you will find some interesting hint inside.
Please note that I am using a custom built compiler (for outdated reasons too ugly to delve in)

The most likely explanation is that you are trying to link a 64-bit version of the MySQL libraries with a 32-bit-only version of Python (currently, all of the python.org installers for OS X are 32-bit only). (You can verify that by using the file command on the library files in /usr/local/mysql/).
Some solutions:
use the Apple-supplied python2.6 on
Snow Leopard which is 64-bit
install a 32-bit version of the MySQL libraries
install a complete solution using MacPorts: install the base MacPorts
infrastructure and then install the MySQLdb adapter for python 2.6 (or 2.5) which will also install all necessary dependencies including a new
python and MySQL client libraries that should all work together correctly (and be
able to be updated by MacPorts):
sudo port install py26-mysql # or py25-mysql
For using MySQL with python on OS X, I recommend the last solution, that is, unless you really enjoy and have the time to do package management and installation. It will likely save you a lot of trouble over the long run.
P.S. MacPorts includes ports of django and PIL as well:
sudo port install py26-django py26-pil
EDIT:
To go the MacPorts route, follow the instructions I gave here to remove the effects of a python.org installer python. DO NOT attempt to delete or modify the Apple-installed Python files in /usr/bin or /System/Library; they are part of OS X. Then follow the instructions cited above to install MacPorts. In order to avoid interference with Apple- or third-party installs, MacPorts installs all of its files into a completely separate directory structure rooted at /opt/local. Thus, you will need to modify your .bash_profile to add /opt/local/bin to your $PATH. If you want the MacPorts versions to be found first, add something like:
export PATH="/opt/local/bin:${PATH}"
When you start a new terminal session, you should find the MacPorts python2.6 at python2.6. If you also want to make the command python point there:
$ sudo port install python_select
$ sudo python_select -l
Available versions:
current none python26 python26-apple
$ sudo python_select python26

The following blog post helped me compile MySQLdb 1.2.2 on the Mac:
http://www.mangoorange.com/2008/08/01/installing-python-mysqldb-122-on-mac-os-x/
However, later on I tried MySQLDB 1.2.3c1 and didn't have any problems compiling out of the box. 1.2.2 is several years old and causes deprecation warnings on Python 2.6. I would just make the switch to 1.2.3.c1 and see if that works for you.
1.2.3c1 is the latest version on PyPi.

It looks like you need to reinstall/update XCode (build tools)