I am trying to install pygraphviz on Windows 10. There are many solutions to this problem online, but none have yet worked for me. The precise problem I'm having is with this via jupyter notebook-->
[1] import networkx as nx
import pylab as plt
from networkx.drawing.nx_agraph import graphviz_layout
[2]G = nx.DiGraph()
G.add_node(1,level=1)
G.add_node(2,level=2)
G.add_node(3,level=2)
G.add_node(4,level=3)
G.add_edge(1,2)
G.add_edge(1,3)
G.add_edge(2,4)
nx.draw(G, pos=graphviz_layout(G), node_size=1600, cmap=plt.cm.Blues,
node_color=range(len(G)),
prog='dot')
plt.show()
I get the following errors after [2]:
ModuleNotFoundError Traceback (most recent call last)
C:\Users\name\Anaconda3\lib\site-packages\networkx\drawing\nx_agraph.py
in
pygraphviz_layout(G, prog, root, args)
254 try:
--> 255 import pygraphviz
256 except ImportError:
ModuleNotFoundError: No module named 'pygraphviz'
and
ImportError Traceback (most recent call last)
<ipython-input-2-86a15892f0f0> in <module>()
9 G.add_edge(2,4)
10
---> 11 nx.draw(G, pos=graphviz_layout(G), node_size=1600, cmap=plt.cm.Blues,
12 node_color=range(len(G)),
13 prog='dot')
C:\Users\name\Anaconda3\lib\site-packages\networkx\drawing\nx_agraph.py in graphviz_layout(G, prog, root, args)
226
227 """
--> 228 return pygraphviz_layout(G,prog=prog,root=root,args=args)
229
230 def pygraphviz_layout(G,prog='neato',root=None, args=''):
C:\Users\name\Anaconda3\lib\site-packages\networkx\drawing\nx_agraph.py in pygraphviz_layout(G, prog, root, args)
256 except ImportError:
257 raise ImportError('requires pygraphviz ',
--> 258 'http://pygraphviz.github.io/')
259 if root is not None:
260 args+="-Groot=%s"%root
ImportError: ('requires pygraphviz ', 'http://pygraphviz.github.io/')
Here's what I've tried to resolve this
(1) Regular pip install: "pip install pygraphviz" This is the error I get at the end. EDIT I get the same error even if I run cmd as admin.
Command "C:\Users\name\Anaconda3\python.exe -u -c "import setuptools,
tokenize;__file__='C:\\Users\\name~1\\AppData\\Local\\Temp\\pip-build-
n81lykqs\\pygraphviz\\setup.py';f=getattr(tokenize, 'open', open)
(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code,
__file__, 'exec'))" install --record C:\Users\name~1\AppData\Local\Temp\pip-
b3jz1lk5-record\install-record.txt --single-version-externally-managed --
compile" failed with error code 1 in C:\Users\name~1\AppData\Local\Temp\pip-
build-n81lykqs\pygraphviz\
(2) Downloading and installing graphviz-2.38.msi, and then downloading both the 64-bit versions of the wheel. This is the result.
C:\Users\name\Anaconda3>pip install pygraphviz-1.3.1-cp34-none-
win_amd64.whl
pygraphviz-1.3.1-cp34-none-win_amd64.whl is not a supported wheel on this
platform.
C:\Users\name\Anaconda3>pip install pygraphviz-1.3.1-cp27-none-
win_amd64.whl
pygraphviz-1.3.1-cp27-none-win_amd64.whl is not a supported wheel on this
platform.
What I would like to try, but am not sure how to do properly:
Edit setup.py. I have read a lot about people finding solutions in changing the paths, but I'm not really sure how to do this. This method looks very complex.
Thank you for any help/insight!
Updated the repo: [GitHub]: CristiFati/Prebuilt-Binaries - (master) Prebuilt-Binaries/PyGraphviz/v1.5:
Using official Graphviz 2.42.2 sources
.whls (win_amd64, win32) for currently supported Python versions
Newer versions might be added (check one level up)
For Python 2.7, they are already built: [UCI.LFD]: Unofficial Windows Binaries for Python Extension Packages - PyGraphviz, an interface to the Graphviz graph layout and visualization package..
Notes:
In some (I guess, most of the) cases, a Graphviz installation will be required on the system where PyGraphviz runs on, because PyGraphviz uses some of Graphviz's tools (executables). They can be downloaded, or built (they don't have to match PyGraphviz architecture (032bit (PC032), 064bit (PC064)), as they are invoked). Update: I also added Graphviz 2.42.2 build (PC032 - as it works on both PC064 and PC032 Win) in the above repository
Check it for newer software versions
Also, a bug (present in previous versions) was fixed. Check [SO]: pygraphviz 1.5 default edge no arrow? (#CristiFati's answer) for more details
Anyone who wants to know more details about the build process, read on!
1. Intro
Almost 2 years later, and the problem (well, not exactly as in the question) still persists.
I want to start by emphasizing the difference between the 2 packages:
[PyPI]: pygraphviz - Download files: the one in question
[PyPI]: graphviz - Download files: (a simpler) one with a similar name
In Anaconda environment, [SO]: Installing PyGraphviz on Windows 10 64-bit, Python 3.6 (#TomHanks's answer) works perfectly.
PyGraphwiz only has available for download an archive (.zip, in this case) file, meaning that it contains (C / C++) sources.
A couple of words about packages (.whls) whose names contain things like cp34-none-win_amd64 (check [SO]: What does version name 'cp27' or 'cp35' mean in Python? (#WayneWerner's answer) for details):
They contain binaries (.so or .pyd (.dll)), which are linked against a specific Python library
They are meant to work only with that Python version (so 34 is not meant to work with Python 3.6)
Even is one somehow "outsmarts" PIP and manages to install such a package (it's not that hard, actually), it will fail at import time, or worse, it has a high probability of crashing Python
Now, many packages have prebuilt binaries for most common Python versions running on various OSes (e.g. [PyPI]: mysql-connector-python - Download files), but just as many don't, and those only contain sources. Unfortunately, PyGraphviz is in the 2nd category. For the latter ones, pip install will:
Download the sources
Build the sources locally
A C (C++) compiler is required, typically:
GCC on Nix
VStudio on Win
They might have other dependencies
Install the built artefacts (binaries and .py(c) files)
As a side note: pip -v ... enables verbose mode for the current command, which comes in extremely handy when experiencing install errors.
Back to our problem: Python 3.6 needs VStudio 2015 ([Python.Wiki]: WindowsCompilers).
This is a very vast topic, I covered some parts in:
[SO]: Simstring (python) installation in windows (#CristiFati's answer)
[SO]: How to build a DLL version of libjpeg 9b? (#CristiFati's answer)
You should check them before proceeding and also keep them open, as you will definitely need them in the next steps.
I have VStudio 2015 Community (among many other versions) installed, you should install it too, it's free ([MS.VStudio]: Still want an older version?).
PyGraphviz depends on [Graphviz]: Graph Visualization Software. So, at build time it will need (parts of) Graphviz (which also has other dependencies of its own) to be already built. Unfortunately, I couldn't find prebuilt binaries (there is [Graphviz]: Windows Packages - graphviz-2.38.zip, but that's not helping), so it will have to be built manually.
Before going further:
[cfati#CFATI-5510-0:e:\Work\Dev\StackOverflow\q045093811]> "e:\Work\Dev\VEnvs\py_064_03.06.08_test0\Scripts\python.exe" -c "import pygraphviz"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pygraphviz'
[cfati#CFATI-5510-0:e:\Work\Dev\StackOverflow\q045093811]> dir /b
other
src
This is my top dir, any sources are downloaded in the src dir, binaries will be placed in the bin dir.
2. Build Graphviz
Before starting, I want to mention that I heavily rely on Cygwin (you don't have to, there are also other variants (MSYS2)), and some of my tools are installed there, so I'll be alternating between Cygwin and Cmd terminals (which might be confusing).
[Graphviz]: Graphviz Build Instructions for Windows states:
For building on Windows:
(Graphviz versions ≥ 2.41)
First, in the root of the repository, perform git submodule update --init. This will download all submodules, which are mostly the dependencies for the Windows build. Next, add the windows\dependencies\graphviz-build-utilities directory to your PATH (and restart Visual Studio or the prompt with which you execute msbuild after that). This folder contains the tools Bison, Flex and SED (and future additions) with versions that are tested. If all went right, the dependencies are now set up and you can build Graphviz.
First, we need to download everything:
[cfati#cfati-5510-0:/cygdrive/e/Work/Dev/StackOverflow/q045093811/src/graphviz]> ~/sopr.sh
### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ###
[064bit prompt]> git clone https://gitlab.com/graphviz/graphviz.git .
Cloning into '.'...
remote: Enumerating objects: 71728, done.
remote: Counting objects: 100% (71728/71728), done.
remote: Compressing objects: 100% (19331/19331), done.
remote: Total 71728 (delta 52200), reused 71681 (delta 52157)
Receiving objects: 100% (71728/71728), 163.79 MiB | 480.00 KiB/s, done.
Resolving deltas: 100% (52200/52200), done.
Checking out files: 100% (3870/3870), done.
[064bit prompt]>
[064bit prompt]> git submodule update --init
Submodule 'dependencies/criterion' (https://github.com/Snaipe/Criterion.git) registered for path 'dependencies/criterion'
Submodule 'windows/dependencies/graphviz-build-utilities' (https://github.com/ErwinJanssen/graphviz-build-utilities.git) registered for path 'windows/dependencies/graphviz-build-utilities'
Submodule 'windows/dependencies/libraries' (https://github.com/ErwinJanssen/graphviz-windows-dependencies.git) registered for path 'windows/dependencies/libraries'
Cloning into '/cygdrive/e/Work/Dev/StackOverflow/q045093811/src/graphviz/dependencies/criterion'...
Cloning into '/cygdrive/e/Work/Dev/StackOverflow/q045093811/src/graphviz/windows/dependencies/graphviz-build-utilities'...
Cloning into '/cygdrive/e/Work/Dev/StackOverflow/q045093811/src/graphviz/windows/dependencies/libraries'...
Submodule path 'dependencies/criterion': checked out '301d143ea42c024f22b673b69c72a4cb3c8d151f'
Submodule path 'windows/dependencies/graphviz-build-utilities': checked out '050fff84ce195e0740878748760fd801eeb07b23'
Submodule path 'windows/dependencies/libraries': checked out '141d3a21be904fa8dc2ae3ed01d36684db07a35d'
[064bit prompt]>
[064bit prompt]> git show head
commit 89292b5945933b1501293c04894ed9cf886241be (HEAD -> master, origin/master, origin/HEAD)
Merge: 429d43615 97811bd35
Author: Stephen C North <scnorth#gmail.com>
Date: Mon Feb 4 08:09:40 2019 -0500
Merge branch 'wasbridge/graphviz-master' into HEAD
[064bit prompt]> git status
On branch master
Your branch is up to date with 'origin/master'.
nothing to commit, working tree clean
You'll end up with a dir that contains ~320 MiB of stuff. The dir contains a graphviz.sln file, which is a VStudio (2015) solution file which contains 63 projects.
Looking at the Anaconda or Python 2.7 PyGraphviz (built) package, it only depends on cgraph.dll, which in turn depends on cdt.dll, so only the 2 projects are relevant to us. Note that these 2 projects might not need all the Git submodules (so the dir might be trimmed down), but I didn't investigate further.
Unfortunately, the projects are only configured for 032bit (Win32 platform (pc032)). The 064bit one must be manually added (I did it from VStudio IDE - and also described the process in one of my answers that I referenced). After saving the projects, they will be shown as modified by Git:
[064bit prompt]> git status
On branch master
Your branch is up to date with 'origin/master'.
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
(commit or discard the untracked or modified content in submodules)
modified: lib/cdt/cdt.vcxproj
modified: lib/cgraph/cgraph.vcxproj
modified: windows/dependencies/graphviz-build-utilities (modified content)
no changes added to commit (use "git add" and/or "git commit -a")
The 3rd item is because I needed to reset some security permissions on 2 executables (used when building cgraph):
bison.exe
flex.exe
which were not set properly (most likely, because of Cygwin).
You can build the 2 projects from IDE, but I chose command line ([MS.Docs]: MSBuild command-line reference) since I find it more flexible:
[cfati#CFATI-5510-0:e:\Work\Dev\StackOverflow\q045093811]> sopr.bat
### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ###
[prompt]> "c:\Install\x86\Microsoft\Visual Studio Community\2015\vc\vcvarsall.bat" x64
[prompt]> set PATH=%PATH%;%CD%\src\graphviz\windows\dependencies\graphviz-build-utilities
[prompt]> msbuild src\graphviz\lib\cdt\cdt.vcxproj /t:Rebuild /p:Platform=x64;Configuration=Release;SolutionDir=%CD%\src\graphviz\;OutDir=%CD%\bin\Win\dynamic\064\UCRTv140\md\Release\graphviz\ >build_cdt_064.txt 2>&1
[prompt]> echo %errorlevel%
0
[prompt]> dir /b
bin
build_cdt.txt
other
src
[prompt]> msbuild src\graphviz\lib\cgraph\cgraph.vcxproj /t:Rebuild /p:Platform=x64;Configuration=Release;SolutionDir=%CD%\src\graphviz\;OutDir=%CD%\bin\Win\dynamic\064\UCRTv140\md\Release\graphviz\ >build_cgraph_064.txt 2>&1
[prompt]> echo %errorlevel%
0
[prompt]> dir /b "bin\Win\dynamic\064\UCRTv140\md\Release\graphviz"
cdt.dll
cdt.dll.lastcodeanalysissucceeded
cdt.exp
cdt.lib
cgraph.dll
cgraph.dll.lastcodeanalysissucceeded
cgraph.exp
cgraph.lib
So, we have everything needed (2 .lib and 2 .dll files) in order to go on.
3. Build PyGraphviz
PyGraphviz sources are (downloaded from [GitHub]: pygraphviz/pygraphviz - (pygraphviz-1.5) pygraphviz-pygraphviz-1.5.zip and) unpacked in src/pygraphviz/pygraphviz-pygraphviz-1.5.
One more adjustment is needed to Graphviz (probably it's done as part of another project - an install step): preparing the header files:
[prompt]> mkdir include\graphviz
[prompt]> copy src\graphviz\lib\cdt\cdt.h include\graphviz
1 file(s) copied.
[prompt]> copy src\graphviz\lib\cgraph\cgraph.h include\graphviz
1 file(s) copied.
Unfortunately, PyGraphviz does not build OOTB, because of [GitHub]: pygraphviz/pygraphviz - Python 3 support. To fix that, [GitHub]: eendebakpt/pygraphviz - Workaround for PyIOBase_Type for Python2 on win must be applied. I adapted it to work with the current sources (as it doesn't work OOTB, as well :X( ) for graphviz_wrap.cpp only:
pygraphviz-1.5-all-pyiobase_b85d12ac22d39063f7dbcc396e825c563431e352.patch:
--- pygraphviz/graphviz_wrap.c.orig 2018-09-10 16:07:12.000000000 +0300
+++ pygraphviz/graphviz_wrap.c 2019-02-26 18:05:20.281741400 +0200
## -2988,7 +2988,18 ##
#if PY_VERSION_HEX >= 0x03000000
-extern PyTypeObject PyIOBase_Type;
+static PyObject *PyIOBase_TypeObj;
+
+static int init_file_emulator(void)
+{
+ PyObject *io = PyImport_ImportModule("_io");
+ if (io == NULL)
+ return -1;
+ PyIOBase_TypeObj = PyObject_GetAttrString(io, "_IOBase");
+ if (PyIOBase_TypeObj == NULL)
+ return -1;
+ return 0;
+}
#endif
## -3449,7 +3460,7 ##
{
#if PY_VERSION_HEX >= 0x03000000 || defined(PYPY_VERSION)
#if !defined(PYPY_VERSION)
- if (!PyObject_IsInstance(obj0, (PyObject *)&PyIOBase_Type)) {
+ if (!PyObject_IsInstance(obj0, PyIOBase_TypeObj)) {
PyErr_SetString(PyExc_TypeError, "not a file handle");
return NULL;
}
## -3523,7 +3534,7 ##
{
#if PY_VERSION_HEX >= 0x03000000 || defined(PYPY_VERSION)
#if !defined(PYPY_VERSION)
- if (!PyObject_IsInstance(obj1, (PyObject *)&PyIOBase_Type)) {
+ if (!PyObject_IsInstance(obj1, PyIOBase_TypeObj)) {
PyErr_SetString(PyExc_TypeError, "not a file handle");
return NULL;
}
## -6051,6 +6062,12 ##
SWIG_InstallConstants(d,swig_const_table);
+#if PY_VERSION_HEX >= 0x03000000
+ if (init_file_emulator() < 0) {
+ return NULL;
+ }
+#endif
+
PyDict_SetItemString(md,(char*)"cvar", SWIG_globals());
SWIG_addvarlink(SWIG_globals(),(char*)"Agdirected",Swig_var_Agdirected_get, Swig_var_Agdirected_set);
SWIG_addvarlink(SWIG_globals(),(char*)"Agstrictdirected",Swig_var_Agstrictdirected_get, Swig_var_Agstrictdirected_set);
That is a diff (patch). See [SO]: Run/Debug a Django application's UnitTests from the mouse right click context menu in PyCharm Community Edition? (#CristiFati's answer) (Patching UTRunner section) for how to apply patches on Win (basically, every line that starts with one "+" sign goes in, and every line that starts with one "-" sign goes out).
[prompt]> :: Restore the original prompt as cwd is important
[prompt]> exit
[cfati#CFATI-5510-0:e:\Work\Dev\StackOverflow\q045093811]> set _TOP_DIR=%CD%
[cfati#CFATI-5510-0:e:\Work\Dev\StackOverflow\q045093811]> pushd src\pygraphviz\pygraphviz-pygraphviz-1.5
[cfati#CFATI-5510-0:e:\Work\Dev\StackOverflow\q045093811\src\pygraphviz\pygraphviz-pygraphviz-1.5]> pushd pygraphviz && "c:\Install\x64\Cygwin\Cygwin\AllVers\bin\patch.exe" -p 1 -buNi ..\pygraphviz-1.5-all-pyiobase_b85d12ac22d39063f7dbcc396e825c563431e352.patch && popd
patching file graphviz_wrap.c
[cfati#CFATI-5510-0:e:\Work\Dev\StackOverflow\q045093811\src\pygraphviz\pygraphviz-pygraphviz-1.5]> echo %errorlevel%
0
[cfati#CFATI-5510-0:e:\Work\Dev\StackOverflow\q045093811\src\pygraphviz\pygraphviz-pygraphviz-1.5]> "e:\Work\Dev\VEnvs\py_064_03.06.08_test0\Scripts\python.exe" setup.py install --include-path=%_TOP_DIR%\include --library-path=%_TOP_DIR%\bin\Win\dynamic\064\UCRTv140\md\Release\graphviz >%_TOP_DIR%\install_pygraphviz_064.txt 2>&1
[cfati#CFATI-5510-0:e:\Work\Dev\StackOverflow\q045093811\src\pygraphviz\pygraphviz-pygraphviz-1.5]> echo %errorlevel%
0
[cfati#CFATI-5510-0:e:\Work\Dev\StackOverflow\q045093811\src\pygraphviz\pygraphviz-pygraphviz-1.5]> popd
[cfati#CFATI-5510-0:e:\Work\Dev\StackOverflow\q045093811]> set PATH=%PATH%;%CD%\bin\Win\dynamic\064\UCRTv140\md\Release\graphviz
[cfati#CFATI-5510-0:e:\Work\Dev\StackOverflow\q045093811]> "e:\Work\Dev\VEnvs\py_064_03.06.08_test0\Scripts\python.exe" -c "import pygraphviz;print(dir(pygraphviz), \"\n\", pygraphviz.graphviz._graphviz)"
['AGraph', 'Attribute', 'DotError', 'Edge', 'ItemAttribute', 'Node', '__all__', '__author__', '__builtins__', '__cached__', '__date__', '__doc__', '__file__', '__license__', '__loader__', '__name__', '__package__', '__path__', '__revision__', '__spec__', '__version__', 'absolute_import', 'agraph', 'division', 'graphviz', 'print_function', 'release', 'test', 'tests', 'version']
<module '_graphviz' (e:\Work\Dev\VEnvs\py_064_03.06.08_test0\lib\site-packages\pygraphviz\_graphviz.cp36-win_amd64.pyd)>
As seen, the module was successfully imported.
As a remark, the 2 .dll dependencies (from previous section) must be available when the module is imported, so their dir is added to %PATH%.
Of course this is only a (lame) workaround (gainarie), this shouldn't happen every time one has to work with the package.
I don't know (yet) how to instruct setup.py to also copy them in the package build / install dir, so as an alternative (also workaround) one has to manually copy them in PyGraphviz install dir (next to _graphviz.cp36-win_amd64.pyd, which is (in my case): "e:\Work\Dev\VEnvs\py_064_03.06.08_test0\lib\site-packages\pygraphviz").
4. Shortcut
Since the whole process is complex and requires lots of manual interventions and hacks, I've managed to build (with minor setup.py modifications) the (.whl) package.
I am not aware of a simple way to make it publicly available, so (although maybe it's not the best practice,) I uploaded it at [GitHub]: CristiFati/Prebuilt-Binaries - (master) Prebuilt-Binaries/PyGraphviz/v1.5/pygraphviz-1.5-cp36-cp36m-win_amd64.whl.
Notes:
The following is regarding PyGraphviz, but it can be generalized and applied to any (.whl) package
Also, the GitHub repository will be used as an example, but (again) it can apply to (.whls in) any repository, including [PyPI]: The Python Package Index (I'm not using the PyGraphviz one as it has no binaries, but for example [PyPI]: numpy 1.17.0 - Download files)
You can download it locally (e.g. in C:\Path\to\downloaded), then install it like (this is one way, check [SO]: How to install a package for a specific Python version on Windows 10? (#CristiFati's answer) for alternatives):
"C:\Path\to\Python-3.6-amd64\python.exe" -m pip install "C:\Path\to\downloaded\pygraphviz-1.5-cp36-cp36m-win_amd64.whl"
The above .whl is located next to several others. Those are for different Python versions (and that happens when the .whl contains .dlls (.pyds) or .sos (on Nix) that link to a specific Python library). To match your Python version search the .whl name for cp${PYTHON_VERSION_MAJOR}${PYTHON_VERSION_MINOR}. In the current situation, it's cp36 corresponding to Python 3.6, but the same principle applies for other Python versions (cp37, cp38, cp39, cp310, cp311, ...). Check [SO]: What does version name 'cp27' or 'cp35' mean in Python? for more details.
Same thing for the architecture (win_amd64 / win32 (x86_64 / i686 on Nix)).
Note: It works for Anaconda environments as well!
The most voted answers seemed to be installing graphviz, rather than pygraphviz.
If you are using a conda environment, you may try using this channel:
conda install graphviz pygraphviz -c alubbock
I tried it out with networkx 2.1, it worked fine.
Here's what worked for me:
Win 7 AMD64
Install MSFT C++ compiler.
Install Anaconda for Win AMD64, Python3.
Install graphviz for Win.
Add C:\Program Files (x86)\Graphviz2.38\bin to your PATH environment variable.
Download pygraphviz-1.3.1-cp34-none-win_amd64.whl.
Create a Conda environment with Python version 3.4: conda create --name digraphs python=3.4 anaconda.
Enter the environment: activate digraphs.
Install pygraphviz using pip: pip install pygraphviz-1.3.1-cp34-none-win_amd64.whl.
Run example: python ./gviz_simple.py.
Exit the environment: deactivate
I put some stuff up on github about it. It's messy, use at your own risk:
https://github.com/darkhipo/Easy-Digraph-Draw
Solved it on Windows 10 64-bits and Python 3.6.
Steps:
Download Graphviz for windows from the graphiviz site.
Add the Graphviz bin path C:\Program Files (x86)\Graphviz2.38\bin in your Windows path.
Close and reopen your terminals so the path changes is recognized.
Download the graphviz python 3.6 wheel.
Install the graphviz wheel.
pip install graphviz-0.8.3-py2.py3-none-any.whl
Done!
What I currently try to achieve is building a python mapping of my C++ classes through Boost.Python.
After this I want to use the resulting shared library in a blender add-on to be able to take advantage of already existing functionality coming from the mapped C++ classes.
I can already build my shared library and write sample scripts in python, which are using my library as well.
Everything fine here but the problem is that as soon as I try to use it in an add-on, Blender 2.74 keeps crashing all the time as soon as I add an import statement with this little hint in the crash report:
6 libboost_python.dylib 0x000000010aa7cc3e boost::python::detail::init_module(PyModuleDef&, void (*)()) + 30 (module.cpp:44)
In module.cpp inside of boost line 41-46:
BOOST_PYTHON_DECL PyObject* init_module(PyModuleDef& moduledef, void(*init_function)())
{
return init_module_in_scope(
PyModule_Create(&moduledef),
init_function);
}
My boost 1_58 is compiled using Python 3.4.2:
otool -L /usr/local/lib/libboost_python.dylib
/usr/local/lib/libboost_python.dylib:
libboost_python.dylib (compatibility version 0.0.0, current version 0.0.0)
/Library/Frameworks/Python.framework/Versions/3.4/Python (compatibility version 3.4.0, current version 3.4.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
Running the python3 bin from that directory gives me:
python3
Python 3.4.2 (v3.4.2:ab2c023a9432, Oct 5 2014, 20:42:22)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
The Blender Python console gives me:
PYTHON INTERACTIVE CONSOLE 3.4.2 (default, Nov 25 2014, 12:01:44) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.54)]
Command History: Up/Down Arrow
Cursor: Left/Right Home/End
Remove: Backspace/Delete
Execute: Enter
Autocomplete: Ctrl-Space
Zoom: Ctrl +/-, Ctrl-Wheel
Builtin Modules: bpy, bpy.data, bpy.ops, bpy.props, bpy.types, bpy.context, bpy.utils, bgl, blf, mathutils
Convenience Imports: from mathutils import *; from math import *
Convenience Variables: C = bpy.context, D = bpy.data
>>>
My own project uses this python version as well.
Also Blender uses this Python version.
I really don't know what to try next here since in standalone mode EVERYTHING works as expected.
Also the fact that the crash even occurs as soon as I run a new script inside Blenders Text Editor having the import statement.
Anybody having experience with Boost.Python and Blender?
Thanks in advance
UPDATE:
After building and running blender in debug mode, I got the following state:
This doesn't look right and describes the crash in the first place.
Still searching for the cause...
Blender itself (as well as oiio and osl libs) are built using boost, ensure that the version you use to build boost-python matches the one used by blender. As blender has already loaded boost libs into ram, when you import boost-python it will use the already loaded boost libs not the ones you built boost-python with, if each is a different version it can cause problems, if each is built using different options it can also cause issues.
If you are building blender yourself also build OpenImageIO and OpenShadingLanguage so they use the same boost version. By building them yourself you can ensure they all load the version that you want to use. Check your cmake or sconscript settings point to the same boost lib files.
% ldd blender | grep boost
libboost_filesystem.so.1.55.0 => /usr/local/lib/libboost_filesystem.so.1.55.0 (0x806009000)
libboost_regex.so.1.55.0 => /usr/local/lib/libboost_regex.so.1.55.0 (0x80621e000)
libboost_system.so.1.55.0 => /usr/local/lib/libboost_system.so.1.55.0 (0x806525000)
libboost_thread.so.1.55.0 => /usr/local/lib/libboost_thread.so.1.55.0 (0x806728000)
libboost_date_time.so.1.55.0 => /usr/local/lib/libboost_date_time.so.1.55.0 (0x806942000)
libboost_wave.so.1.55.0 => /usr/local/lib/libboost_wave.so.1.55.0 (0x806b4f000)
libboost_locale.so.1.55.0 => /usr/local/lib/libboost_locale.so.1.55.0 (0x806e94000)
libboost_chrono.so.1.55.0 => /usr/local/lib/libboost_chrono.so.1.55.0 (0x814ee3000)
After a lot of trial and error I got it working!
I'll try to give a step by step tutorial on how to do that for OSX.
In the next couple of weeks I'll also try to get it working on windows and will update this answer as I succeed!
Step 1
Install python 3.4.2 from https://www.python.org/downloads/release/python-342/
Step 2
Checkout the blender git repo and the svn repo from blender (found at http://wiki.blender.org/index.php/Dev:Doc/Building_Blender/Mac):
mkdir ~/blender-build
cd ~/blender-build
git clone http://git.blender.org/blender.git
cd blender
git submodule update --init --recursive
git submodule foreach git checkout master
git submodule foreach git pull --rebase origin master
Download external libs:
cd ~/blender-build
mkdir lib
cd lib
svn checkout https://svn.blender.org/svnroot/bf-blender/trunk/lib/darwin-9.x.universal
Step 3 Use CMake to create the XCode project for building blender
Make sure you check the following settings for python:
This will automatically configure to build blender using the installed python framework on your mac.
Step 4 Download and build boost with the Python 3.4.2
Here you first have to check which version blender uses.
In the external libraries svn repository boost is contained as a pre build library at:
blender-build/lib/darwin-9.x.universal/README
For me this is boost 1.51: http://www.boost.org/users/history/version_1_51_0.html
Step 5 Build your shared library with Boost 1.51 and Python 3.14
Step 6 Build blender and make sure it is using the installed Python 3.4.2
Step 7 Copy your shared library into the addons folder of the blender.app
Here a tiny small little hint: In order to let python recognize your shared library it has to have the extension .so rather than .dylib (for windows this is .pyd rather than .dll)!
Step 8 Recheck
Check that your boost is compiled agains Python 3.4.2:
otool -L /usr/local/lib/libboost_python.dylib
/usr/local/lib/libboost_python3.dylib:
libboost_python3.dylib (compatibility version 0.0.0, current version 0.0.0)
/Library/Frameworks/Python.framework/Versions/3.4/Python (compatibility version 3.4.0, current version 3.4.0) <- IMPORTANT
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
You can also check if your library can be used in a simple python script like test.py by executing the following script in a folder were your .so is contained:
import mylib
print('hello world!')
Run:
python3 test.py
Basically now you should also be able to execute this simple script inside of blender, which is the case for me.
I am confused in trying to understand which python version is used when I run them in interpreter?
That is, how the shell decides which python version to load when I have more than one versions.
I have snapshot of my /usr/bin. It has atleast two different python versions2.5 and 2.6.
I have not specified anything in my .bashrc to choose which one from.
when I execute python command on bash shell, python interpreter opens up, but loads with version 2.6.
How is this detected by shell?
If I want to change versions, how will I do that? That is, I want to change the default version
lastly what kind of files are these in /usr/bin/python. I tried to open them in editor and I see some ###. I have read many posts here but some makes me confusing.
ls /usr/bin/python
python python2.5 python2.6 pythonw pythonw2.6
python-config python2.5-config python2.6-config pythonw2.5
python
Python 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
To know which version is used by default, type
which python
To use another one, use the full path when invoking them:
/usr/bin/python/python2.5 myfile.py
/usr/bin/python/python2.6 myfile.py
etc.
If you want to change the default, you can create an alias in ~/.bashrc for example:
my_python='/usr/bin/python/python2.7'
or better change the link direction of /usr/bin/python (or whatever comes from which python). It is a link, so link it to the exact version you need.
ln -s /prefered/path/of/python /usr/bin/python
The files you see in /usr/bin/python are executables. You can do the following to see this:
file /usr/bin/python/python2.5
In my case:
$ file /usr/bin/python
/usr/bin/python: symbolic link to `python2.7'
$ file /usr/bin/python2.7
/usr/bin/python2.7: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0xf66a10f2c444b2329b25ab6790abb7fbb4fe3f78, stripped
python --version
Then head on over to your .bashrc (should be in your home directory) and add:
alias python='[your path]'
for me I have:
alias python='/usr/bin/python2.7'
Just make sure you say:
source ./bashrc
to apply your changes.
I'm working on several Python projects who run on various versions of Python. I'm hoping to set up my vim environment to use ropevim, pyflakes, and pylint but I've run into some issues caused by using a single vim (compiled for a specific version of Python which doesn't match the project's Python version).
I'm hoping to build vim into each of my virtualenv directories but I've run into an issue and I can't get it to work. When I try to build vim from source, despite specifying the Python config folder in my virtualenv, the system-wide Python interpreter is always used.
Currently, I have Python 2.6.2 and Python 2.7.1 installed with several virtualenvs created from each version. I'm using Ubuntu 10.04 where the system-default Python is 2.6.5. Every time I compile vim and call :python import sys; print(sys.version) it returns Python 2.6.5.
configure --prefix=/virtualenv/project --enable-pythoninterp=yes --with-python-config-dir=/virtualenv/project/lib/python2.6/config
Results in the following in config.log:
...
configure:5151: checking --enable-pythoninterp argument
configure:5160: result: yes
configure:5165: checking for python
configure:5195: result: /usr/bin/python
...
It should be /virtualenv/project/bin/python. Is there any way to specify the Python interpreter for vim to use?
NOTE: I can confirm that /virtualenv/project/bin appears at the front of PATH environment variable.
I'd recommend building vim against the 2 interpreters, then invoking it using the shell script I provided below to point it to a particular virtualenv.
I was able to build vim against Python 2.7 using the following command (2.7 is installed under $HOME/root):
% LD_LIBRARY_PATH=$HOME/root/lib PATH=$HOME/root/bin:$PATH \
./configure --enable-pythoninterp \
--with-python-config-dir=$HOME/root/lib/python2.7/config \
--prefix=$HOME/vim27
% make install
% $HOME/bin/vim27
:python import sys; print sys.path[:2]
['/home/pat/root/lib/python27.zip', '/home/pat/root/lib/python2.7']
Your virtualenv is actually a thin wrapper around the Python interpreter it was created with -- $HOME/foobar/lib/python2.6/config is a symlink to /usr/lib/python2.6/config.
So if you created it with the system interpreter, VIM will probe for this and ultimately link against the real interpreter, using the system sys.path by default, even though configure will show the virtualenv's path:
% PATH=$HOME/foobar/bin:$PATH ./configure --enable-pythoninterp \
--with-python-config-dir=$HOME/foobar/lib/python2.6/config \
--prefix=$HOME/foobar
..
checking for python... /home/pat/foobar/bin/python
checking Python's configuration directory... (cached) /home/pat/foobar/lib/python2.6/config
..
% make install
% $HOME/foobar/bin/vim
:python import sys; print sys.path[:1]
['/usr/lib/python2.6']
The workaround: Since your system vim is most likely compiled against your system python, you don't need to rebuild vim for each virtualenv: you can just drop a shell script named vim in your virtualenv's bin directory, which extends the PYTHONPATH before calling system vim:
Contents of ~/HOME/foobar/bin/vim:
#!/bin/sh
ROOT=`cd \`dirname $0\`; cd ..; pwd`
PYTHONPATH=$ROOT/lib/python2.6/site-packages /usr/bin/vim $*
When that is invoked, the virtualenv's sys.path is inserted:
% $HOME/foobar/bin/vim
:python import sys; print sys.path[:2]
['/home/pat/foobar/lib/python2.6/site-packages', '/usr/lib/python2.6']
For what it's worth, and no one seems to have answered this here, I had some luck using a command line like the following:
vi_cv_path_python=/usr/bin/python26 ./configure --includedir=/usr/include/python2.6/ --prefix=/home/bcrowder/local --with-features=huge --enable-rubyinterp --enable-pythoninterp --disable-selinux --with-python-config-dir=/usr/lib64/python2.6/config
I would like to give a similar solution to crowder's that works quite well for me.
Imagine you have Python installed in /opt/Python-2.7.5 and that the structure of that folder is
$ tree -d -L 1 /opt/Python-2.7.5/
/opt/Python-2.7.5/
├── bin
├── include
├── lib
└── share
and you would like to build vim with that version of Python. All you need to do is
$ vi_cv_path_python=/opt/Python-2.7.5/bin/python ./configure --enable-pythoninterp --prefix=/SOME/FOLDER
Thus, just by explicitly giving vi_cv_path_python variable to configure the script will deduce everything on it's own (even the config-dir).
This was tested multiple times on vim 7.4+ and lately with vim-7-4-324.
I was having this same issue with 3 different versions of python on my system.
for me the easiest thing was to change my $PATH env variable so that the folder that has the version of python I wanted was (in my case /usr/local/bin) was found before another.
During my compiling vim80, the system python is 2.6, I have another python 2.7 under ~/local/bin, I find that, to make the compiling work:
update $PATH to place my python path ahead
add a soft link, ln -s python python2 ( the configure file would try to locate python config by probing python2 )
make distclean before re-run ./configure to make sure no cached wrong value is picked.
I can't get scipy to function in 32 bit mode when compiled as a i386/x86_64 universal binary, and executed on my 64 bit 10.6.2 MacPro1,1.
My python setup
With the help of this answer, I built a 32/64 bit intel universal binary of python 2.6.4 with the intention of using the arch command to select between the architectures. (I managed to make some universal binaries of a few libraries I wanted using lipo.) That all works. I then installed scipy according to the instructions on hyperjeff's article, only with more up-to-date numpy (1.4.0) and skipping the bit about moving numpy aside briefly during the installation of scipy.
Now, everything except scipy seems to be working as far as I can tell, and I can indeed select between 32 and 64 bit mode using arch -i386 python and arch -x86_64 python.
The error
Scipy complains in 32 bit mode:
$ arch -x86_64 python -c "import scipy.interpolate; print 'success'"
success
$ arch -i386 python -c "import scipy.interpolate; print 'success'"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/interpolate/__init__.py", line 7, in <module>
from interpolate import *
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/interpolate/interpolate.py", line 13, in <module>
import scipy.special as spec
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/__init__.py", line 8, in <module>
from basic import *
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/basic.py", line 8, in <module>
from _cephes import *
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/_cephes.so, 2): Symbol not found: _aswfa_
Referenced from: /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/_cephes.so
Expected in: flat namespace
in /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/_cephes.so
Attempt at tracking down the problem
It looks like scipy.interpolate imports something called _cephes, which looks for a symbol called _aswfa_ but can't find it in 32 bit mode. Browsing through scipy's source, I find an ASWFA subroutine in specfun.f. The only scipy product file with a similar name is specfun.so, but both that and _cephes.so appear to be universal binaries:
$ cd /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/
$ file _cephes.so specfun.so
_cephes.so: Mach-O universal binary with 2 architectures
_cephes.so (for architecture i386): Mach-O bundle i386
_cephes.so (for architecture x86_64): Mach-O 64-bit bundle x86_64
specfun.so: Mach-O universal binary with 2 architectures
specfun.so (for architecture i386): Mach-O bundle i386
specfun.so (for architecture x86_64): Mach-O 64-bit bundle x86_64
Ho hum. I'm stuck. Things I may try but haven't figured out how yet include compiling specfun.so myself manually, somehow.
I would imagine that scipy isn't broken for all 32 bit machines, so I guess something is wrong with the way I've installed it, but I can't figure out what.
I don't really expect a full answer given my fairly unique (?) setup, but if anyone has any clues that might point me in the right direction, they'd be greatly appreciated.
(edit) More details to address questions:
I'm using gfortran (GNU Fortran from GCC 4.2.1 Apple Inc. build 5646).
Python 2.6.4 was installed more-or-less like so:
cd /tmp
curl -O http://www.python.org/ftp/python/2.6.4/Python-2.6.4.tar.bz2
tar xf Python-2.6.4.tar.bz2
cd Python-2.6.4
# Now replace buggy pythonw.c file with one that supports the "arch" command:
curl http://bugs.python.org/file14949/pythonw.c | sed s/2.7/2.6/ > Mac/Tools/pythonw.c
./configure --enable-framework=/Library/Frameworks --enable-universalsdk=/ --with-universal-archs=intel
make -j4
sudo make frameworkinstall
Scipy 0.7.1 was installed pretty much as described as here, but it boils down to a simple sudo python setup.py install.
It would indeed appear that the symbol is undefined in the i386 architecture if you look at the _cephes library with nm, as suggested by David Cournapeau:
$ nm -arch x86_64 /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/_cephes.so | grep _aswfa_
00000000000d4950 T _aswfa_
000000000011e4b0 d _oblate_aswfa_data
000000000011e510 d _oblate_aswfa_nocv_data
(snip)
$ nm -arch i386 /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/scipy/special/_cephes.so | grep _aswfa_
U _aswfa_
0002e96c d _oblate_aswfa_data
0002e99c d _oblate_aswfa_nocv_data
(snip)
however, I can't yet explain its absence.
Have you tried using scipy compiled using macports?
sudo port install scipy +universal
(of course you must have the rest of the chain, python, py26-numpycompiled with the same option)
I get:
$ arch -x86_64 /opt/local/bin/python -c "import scipy.interpolate; print 'success'"
success
$ arch -i386 /opt/local/bin/python -c "import scipy.interpolate; print 'success'"
success
you may then use the setting and knowledge that the macports maintainers used to make your own compilation.
How did you install scipy, for which python version, and with which fortran compiler ?
You may also want to check that the missing symbol is indeed in both archs (I don't remember off-hand where the function is, but you should be able to find ti by yourself pretty easily using a combination of nm/otool).