Develop Reference

Cython/Python comparison invalid syntax error

I would like to compare Python with Cython in term of time execution, so I wrote two files:
fac.py
def factorial(n):
if n >= 1:
return n*factorial(n - 1)
return 1
fastfac.pyx
cpdef long fastfactorial(long n):
if n>= 1:
return n*fastfactorial(n - 1)
return 1
Then I wrote a setup file:
setup.py
from distutils.core import setup
from Cython.Build import cythonize
setup(ext_modules = cythonize('fastfac.pyx'))
From the Powershell I executed the two commands:
pip install Cython
python setup.py build_ext --inplace
From the second command I get the following message:
Compiling fastfac.pyx because it changed.
[1/1] Cythonizing fastfac.pyx
C:\Users\.....\venv\lib\site-packages\Cython\Compiler\Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: C:\Users\.....\fastfac.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
running build_ext
building 'fastfac' extension
error: Unable to find vcvarsall.bat
However I tried to make a comparison, so I wrote the file:
comparison.py
from fastfac import fastfactorial
from fac import factorial
from timeit import timeit
print(timeit('fastfactorial(20)', globals = globals(), number = 10000))
print(timeit('factorial(20)', globals = globals(), number = 10000))
When I run it, I get this error message:
Traceback (most recent call last):
File "C:/Users/...../comparison.py", line 1, in <module>
from fastfac import fastfactorial
ModuleNotFoundError: No module named 'fastfac'
It seems that in the file python.pyx the definition cpdef long fastfactorial(long n) is not recognized as a regular function definition but as a syntax error; in fact, if I try to run that file I get the error message:
File "C:/Users/...../fastfac.pyx", line 1
cpdef long fastfactorial(long n):
^
SyntaxError: invalid syntax
How can I solve? How can I correctly define a cpdef inside a .pyx file?
what am I missing?

The problem is not your definition of fastfactorial, it is the fact that your setup.py exited with an error and, presumably, without compiling fastfac into a c library. In general, you should always fix such errors.
Your error appears to be happening because you don't have a Microsoft Visual C++ compiler installed. You can follow the instructions in this answer to choose a version of Visual C++ to install.
You also have a warning about the language_level not being set. You shouldn't ignore warnings either so it is worth explicitly stating the level in your setup.py.
setup(ext_modules=cythonize('fastfac.pyx'), language_level=3)

Cython compiler error

I haven't been able to find much about this, but when attempting to compile one of my python scripts with cython, it gives me this error:
Error compiling Cython file:
------------------------------------------------------------
...
import traceback #line:24
import bcrypt #line:25
Y =str (os .path .dirname (os .path .abspath (__file__ )))#line:28
IH =open #line:29
IA =str #line:30
IK =print #line:31
^
------------------------------------------------------------
headlessobfu.pyx:29:4: Expected an identifier or literal
Traceback (most recent call last):
File "setup.py", line 5, in <module>
ext_modules = cythonize("headlessobfu.pyx")
File "C:\Users\justi\AppData\Local\Programs\Python\Python36-32\lib\site-packages\Cython\Build\Dependencies.py", line 1026, in cythonize
cythonize_one(*args)
File "C:\Users\justi\AppData\Local\Programs\Python\Python36-32\lib\site-packages\Cython\Build\Dependencies.py", line 1146, in cythonize_one
raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: headlessobfu.pyx
I haven't been able to find the exact cause of this error. I was able to get simple scripts to compile just fine. The python runs just fine on it's own. Is it a problem with my python formatting?
Here is the command line argument I am running:
py setup.py build_ext --inplace
If anyone has a solution please let me know. Thanks.

By default Cython assumes Python 2 syntax, even when you're using Python 3 (edit 2021: Cython 3 will change this and largely assume Python 3 syntax by default). Here's the minimum, complete example you should have created:
cy_print_test.pyx
x = print
If I compile it with Cython
cython cy_print_test.pyx
I get the error message that you do. print is not an identifier or literal because under Python 2 syntax it is a special statement.
However, if I compile it with Cython set to use Python 3 syntax:
cython -3 cy_print_test.pyx
it works fine - under Python 3 syntax print is a function and so this makes perfect sense.
Alternative ways of getting Python 3 syntax would be to add the following line to the start of your .pyx file
#cython: language_level=3
or to specify it as a compiler directive in setup.py:
from distutils.core import setup
from Cython.Build import cythonize
setup(
ext_modules = cythonize('cy_print_test.pyx', compiler_directives={'language_level': 3}),
)
(note that your setup.py should also form part of the minimum, complete example in your question)

Ok so this has to do with cython and python formatting. The code has to be totally obfuscated and with no errors or the compiler won't work. So work through the compiler and fix each error as it arises. Hope this helps.

we need to install the required dependencies, and I guess you are missing out the python3-dev.
Try these commands on terminal (cmd)
$ pip3 install cython
$ sudo apt install update
$ sudo apt install build-essential
$ sudo apt install python3-dev
then run the $ python3 setup.py build_ext --inplace again. it should work.

sphinx python with boost-python library not found

I'm trying to use sphinx to generate documentation for a package that I wrote. Everything was working great until I added some code that used another package I wrote that wraps some boost-python modules.
Now when I run make html I get this (edited for clarity):
$ make html
sphinx-build -b html -d build/doctrees source build/html
Running Sphinx v1.3.1
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 0 source files that are out of date
updating environment: 0 added, 1 changed, 0 removed
reading sources... [100%] my_project
/path/to/my_project/my_project/docs/source/my_project.rst:19: WARNING: autodoc: failed to import module u'my_project.my_module'; the following exception was raised:
Traceback (most recent call last):
File "/Users/harding/anaconda/lib/python2.7/site-packages/Sphinx-1.3.1-py2.7.egg/sphinx/ext/autodoc.py", line 385, in import_object
__import__(self.modname)
File "/path/to/my_project/__init__.py", line 12, in <module>
from . import module_that_uses_boost_python_classes
File "/path/to/module_that_uses_boost_python_classes.py", line 10, in <module>
import package_that_wraps_boost_python_classes
File "/path/to/package_that_wraps_boost_python_classes/__init__.py", line 21, in <module>
from native_boost_python_module import *
ImportError: dlopen(/path/to/native_boost_python_module.so, 2): Library not loaded: cpp_library.dylib
Referenced from: /path/to/native_boost_python_module.so
Reason: image not found
The problem is that when sphinx trys to import my module, it fails to load the linked c++ library from my boost-python related package. That package is installed via pip and it loads fine in all other situations.
I manually specified the full path to all the c++ libraries with install_name_tool -change and then sphinx works fine.
My question is this: how can I get sphinx to import my boost-python package without manually editing the c++ libraries manually with install_name_tool?
I have DYLD_LIBRARY_PATH set to include the directory that contains all the c++ libraries.
I'm on MacOS Sierra

Yocto setup.py requires pyserial

I am trying to package https://github.com/kliment/Printrun in a Yocto recipe but cannot get it working. My recipe currently looks as follows:
LICENSE = "AGPLv3"
LIC_FILES_CHKSUM = "file://COPYING;md5=d32239bcb673463ab874e80d47fae504"
PV = "2.7"
SRCREV = "0193a9dbe31458c45059bf2dcb0a9905b7bb06fc"
SRC_URI = "git://github.com/kliment/Printrun.git;protocol=git;branch=master"
RDEPENDS_${PN} = "python-cython \
python-pyserial \
"
S = "${WORKDIR}/git"
inherit distutils
I am assuming this is what I need to do because it has a setup.py that inherits from distutils? If so, this does not work and I get an error complaining about a lack of the serial module:
DEBUG: Executing shell function do_compile
WARNING: Failed to cythonize: No module named Cython.Build
Traceback (most recent call last):
File "setup.py", line 36, in <module>
from printrun.printcore import __version__ as printcore_version
File "/home/gerhard/Jethro/yocto/build/out-glibc/work/armv7at2hf-vfp-neon-angstrom-linux-gnueabi/printrun/2.7-r0/git/printrun/printcore.py", line 20, in <module>
from serial import Serial, SerialException, PARITY_ODD, PARITY_NONE
ImportError: No module named serial
ERROR: python setup.py build execution failed.
ERROR: Function failed: do_compile (log file is located at /home/gerhard/Jethro/yocto/build/out-glibc/work/armv7at2hf-vfp-neon-angstrom-linux-gnueabi/printrun/2.7-r0/temp/log.do_compile.15542)
I would also like to be able to compile the small cythonable module with cython. For some reason both cython and psyerial are not availible even though I added them as rdepends, what am I doing wrong?

You added dependency on python cython only on runtime. I guess you need to add also for compilation.
DEPENDS_${PN} = "python-cython \
python-pyserial"

Pandas installation on Mac OS X: ImportError (cannot import name hashtable)

I would like to build pandas from source rather than use a package manager because I am interested in contributing. The first time I tried to build pandas, these were the steps I took:
1) created the virtualenv
mkvirtualenv --no-site-packages pandas
2) activated the virtualenv
3) installed Anaconda CE. However, this was installed in ~/anaconda.
4) cloned pandas
5) built C extensions in place
(pandas)ems ~/.virtualenvs/pandas/localrepo/pandas> ~/anaconda/bin/python setup.py build_ext --inplace
6) built pandas
(pandas)ems ~/.virtualenvs/pandas/localrepo/pandas> ~/anaconda/bin/python setup.py build
7) ran nosetests on master branch
Tests failed:
(pandas)ems ~/.virtualenvs/pandas/localrepo/pandas> nosetests pandas
E
======================================================================
ERROR: Failure: ValueError (numpy.dtype has the wrong size, try recompiling)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/EmilyChen/.virtualenvs/pandas/lib/python2.7/site-packages/nose/loader.py", line 390, in loadTestsFromName
addr.filename, addr.module)
File "/Users/EmilyChen/.virtualenvs/pandas/lib/python2.7/site-packages/nose/importer.py", line 39, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/Users/EmilyChen/.virtualenvs/pandas/lib/python2.7/site-packages/nose/importer.py", line 86, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/Users/EmilyChen/.virtualenvs/pandas/localrepo/pandas/pandas/init.py", line 6, in
from . import hashtable, tslib, lib
File "numpy.pxd", line 156, in init pandas.hashtable (pandas/hashtable.c:20354)
ValueError: numpy.dtype has the wrong size, try recompiling
Ran 1 test in 0.001s
FAILED (errors=1)
Someone on the PyData mailing list said:
It looks like you have NumPy installed someplace else on your machine and AnacondaCE is not playing nicely in the virtualenv. The error you are getting is a Cython error message which occurs when the NumPy version it built against doesn't match the installed version on your system-- I had thought that 1.7.x was supposed to be ABI compatible with 1.6.x (so this would not happen) but I guess not. Sigh
The numpy version in Anaconda CE library is 1.7.0b2 and my system numpy installation is version 1.5.1. Setup.py linked to the numpy in the Anaconda distribution's libraries when it built pandas but my guess is it's linking to my system version when nosetests runs /pandas/init.py
Next, I repeated the steps outside a virtualenv, but got the same error. Finally, I decided to install all the dependencies in a new virtualenv instead of using the Anaconda distribution to build pandas. This way, I can see that dependencies like numpy reside in the lib directory of the virtualenv python installation, which takes precedent when pandas.init runs import statements. This is what I did:
1) installed numpy, dateutil, pytz, cython, scipy, matplotlib and openpyxl using pip
2) built c extensions in place
3) pandas install output here: http://pastebin.com/3CKf1f9i
4) pandas did not install correctly
(pandas)ems ~/.virtualenvs/pandas/localrepo/pandas> python
Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53)
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
cannot import name hashtable
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pandas/__init__.py", line 6, in <module>
from . import hashtable, tslib, lib
ImportError: cannot import name hashtable
I took a look at this question but cython installed in my case, and I am trying to build successfully from source rather than using pip like the answer recommended..
(pandas)ems ~/.virtualenvs/pandas/localrepo/pandas> which cython
/Users/EmilyChen/.virtualenvs/pandas/bin/cython

I've received the same error (ImportError: cannot import name hashtable) when trying to import pandas from the source code directory. Try starting the python interpreter from a different directory and import pandas again.