Develop Reference

MacOS, c++ python extension cannot find a dynamic library

I'm building a c++ python extension. So far, I created versions for Linux and Windows. Currently, I'm struggling with the MacOS version. CMake produces 2 libraries:
43898860 Aug 29 13:40 libslide_io.dylib
214876 Aug 29 13:40 slideio.cpython-35m-darwin.so
I pack them to a whl file. After installation with pip, when I try to import the package, I'm getting the following error:
(sld-35)dist % python -c "import slideio"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: dlopen(/Users/.../opt/anaconda3/envs/sld-35/lib/python3.5/site-packages/slideio.cpython-35m-darwin.so, 2): Library not loaded: libslide_io.dylib
Referenced from: /Users/.../opt/anaconda3/envs/sld-35/lib/python3.5/site-packages/slideio.cpython-35m-darwin.so
Reason: image not found
Both libraries are correctly placed to the python environment directory:
/Users/.../opt/anaconda3/envs/sld-35/lib/python3.5/site-packages/
Moreover, if I unzip the whl file and import the package from the current directory, it works.
I suspect that it is something with rpath settings in my cmake files, but cannot find the correct solution. All my attempts are failed.
I would appreciate any help.

If somebody is interested. I solved the problem by adding a post-build with the execution of install_name utility with the #loader_path parameter. The command instructs the system looking for the library at the loader (in my case - the python package) folder.
if (APPLE)
add_custom_command(TARGET ${BINDLIB_NAME}
POST_BUILD COMMAND
${CMAKE_INSTALL_NAME_TOOL} -change libslide_io.dylib #loader_path/libslide_io.dylib
$<TARGET_FILE:${BINDLIB_NAME}>)
endif()

ImportError _versionhelper when setting up Bloomberg Python API

I am trying to get set up with the Bloomberg Python API. I have been following along with this tutorial. I am able to successfully python -m pip install — index-url=https://bloomberg.bintray.com/pip/simple blpapi, but then when I try to run import blpapi in the Python interpreter, I get the following error. How do I resolve this error (wondering if it has something to due with me using Python 3.8)?
ImportError:
---------------------------- ENVIRONMENT -----------------------------
Platform: Windows-10-10.0.17763-SP0
Architecture: ('32bit', 'WindowsPE')
Python: 3.8.0 (tags/v3.8.0:fa919fd, Oct 14 2019, 19:21:23) [MSC v.1916 32 bit (Intel)]
Python implementation: CPython
blpapi 64-bit will be loaded from: "C:\blp\API\blpapi_cpp_3.13.1.1\lib\blpapi3_64.dll"
blpapi 32-bit will be loaded from: "C:\blp\API\blpapi_cpp_3.13.1.1\lib\blpapi3_32.dll"
System PATH: (* marks locations where blpapi was found)
"C:\Windows\system32"
"C:\Windows"
"C:\Windows\System32\Wbem"
"C:\Windows\System32\WindowsPowerShell\v1.0\"
"C:\Windows\System32\OpenSSH\"
"C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL"
"C:\Program Files\Intel\Intel(R) Management Engine Components\DAL"
"c:\nx_dll"
"C:\Program Files\nodejs\"
* "C:\blp\API\blpapi_cpp_3.13.1.1\lib"
"C:\Program Files\MiKTeX 2.9\miktex\bin\x64\"
"C:\Users\cpage\AppData\Local\Programs\Python\Python38-32\Scripts\"
"C:\Users\cpage\AppData\Local\Programs\Python\Python38-32\"
"C:\Users\cpage\AppData\Local\Microsoft\WindowsApps"
* "C:\blp\DAPI"
"C:\blp\DAPI\DDE"
"C:\Users\cpage\AppData\Roaming\npm"
"C:\Program Files\Sublime Text 3"
"C:\Users\cpage\AppData\Local\hyper\app-3.0.2\resources\bin"
"C:\Users\cpage\AppData\Local\Programs\Microsoft VS Code\bin"
* "C:\blp\API\blpapi_cpp_3.13.1.1\lib"
blpapi package at: "C:\Users\cpage\AppData\Local\Programs\Python\Python38-32\lib\site-packages"
Current directory: "C:\Users\cpage"
----------------------------------------------------------------------
No module named '_versionhelper'
Could not open the C++ SDK library.
Download and install the latest C++ SDK from:
http://www.bloomberg.com/professional/api-library
If the C++ SDK is already installed, please ensure that the path to the library
was added to PATH before entering the interpreter.

You need to download the python blpapi package having the version matching your C++ lib.
In your case, since your C++ lib is 3.13.1.1, you can download the 3.13.1 python package:
pip install --index-url=https://bloomberg.bintray.com/pip/simple blpapi==3.13.1

Just managed to make it run using version blpapi-3.15.2 using C++ API blpapi_cpp_3.15.0.1
To install run:
python -m pip install --index-url=https://bloomberg.bintray.com/pip/simple blpapi
to import use:
import os
with os.add_dll_directory('<...>\blpapi_cpp_3.15.0.1\lib'):
import blpapi
or add '<...>\blpapi_cpp_3.15.0.1\lib' to PATH

The following procedure works out of the box for CPython 2.7 and 3.7. A Bloomberg rep claims he verified it for CPython 3.5 as well:
Download the C++ BLP API from the official Bloomberg repository. If this link breaks, use Bloomberg API Support homepage to find the up-to-date ones.
Unpack the archive to C:\Programs\blpapi_cpp_3.14.3.1
Exit from the shell.
Set the environment variable BLPAPI_ROOT=C:\Programs\blpapi_cpp_3.14.3.1\. Note that the trailing slash is important.
Start a new shell.
Execute python -m pip install --index-url=https://bloomberg.bintray.com/pip/simple blpapi==3.14.0
Check the installation by executing python -c "import blpapi", which should pass without displaying messages.

Any pyinstaller detailed example about hidden import for psutil?

I want to compile my python code to binary by using pyinstaller, but the hidden import block me. For example, the following code import psutil and print the CPU count:
# example.py
import psutil
print psutil.cpu_count()
And I compile the code:
$ pyinstaller -F example.py --hidden-import=psutil
When I run the output under dist:
ImportError: cannot import name _psutil_linux
Then I tried:
$ pyinstaller -F example.py --hidden-import=_psutil_linux
Still the same error. I have read the pyinstall manual, but I still don't know how to use the hidden import. Is there a detailed example for this? Or at least a example to compile and run my example.py?
ENVs:
OS: Ubuntu 14.04
Python: 2.7.6
pyinstaller: 2.1

Hi hope you're still looking for an answer. Here is how I solved it:
add a file called hook-psutil.py
from PyInstaller.hooks.hookutils import (collect_data_files, collect_submodules)
datas = [('./venv/lib/python2.7/site-packages/psutil/_psutil_linux.so', 'psutil'),
('./venv/lib/python2.7/site-packages/psutil/_psutil_posix.so', 'psutil')]
hiddenimports = collect_submodules('psutil')
And then call pyinstaller --additional-hooks-dir=(the dir contain the above script) script.py

pyinstall is hard to configure, the cx_freeze maybe better, both support windows (you can download the exe directly) and linux. Provide the example.py, In windows, suppose you have install python in the default path (C:\\Python27):
$ python c:\\Python27\\Scripts\\cxfreeze example.py -s --target-dir some_path
the cxfreeze is a python script, you should run it with python, then the build files are under some_path (with a lot of xxx.pyd and xxx.dll).
In Linux, just run:
$ cxfreeze example.py -s --target-dir some_path
and also output a lot of files(xxx.so) under some_path.
The defect of cx_freeze is it would not wrap all libraries to target dir, this means you have to test your build under different environments. If any library missing, just copy them to target dir. A exception case is, for example, if your build your python under Centos 6, but when running under Centos 7, the missing of libc.so.6 will throw, you should compile your python both under Centos 7 and Centos 6.

What worked for me is as follows:
Install python-psutil: sudo apt-get install python-psutil. If you
have a previous installation of the psutil module from other
method, for example through source or easy_install, remove it first.
Run pyinstaller as you do, without the hidden-import option.

still facing the error
Implementation:
1.python program with modules like platform , os , shutil and psutil
when i run the script directly using python its working fine.
2.if i build a binary using pyinstaller. The binary is build successfully. But if i run the binary iam getting the No module named psutil found.I had tried several methods like adding the hidden import and other things. None is working. I trying it almost 2 to 3 days.
Error:
ModuleNotFoundError: No module named 'psutil'
Command used for the creating binary
pyinstaller --hidden-import=['_psutil_linux'] --onefile --clean serverHW.py
i tried --additional-hooks-dir= also not working. When i run the binary im getting module not found error.

Couldn't create working virtual environment for Python 3.4

I installed Python 3.4.2 and Virtualenv 12.0.5 in my Linux Mint 17.1
Then I tried creating:
$ virtualenv venv
And also using --clear and/or -p /usr/bin/python3.4, always getting the messages:
Using base prefix '/usr'
New python executable in venv/bin/python3
Also creating executable in venv/bin/python
ERROR: The executable venv/bin/python3 could not be run: [Errno 13] Permission denied
Another try was:
$ pyvenv-3.4 venv
It gave no errors on creation, but in the venv/bin file the python3.4 is a symbolic link to /usr/local/bin/python3.4. Then when I activate and install any lib using pip or pip3, then try to import it, I get the error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named 'anymoduledownloaded'
I always used virtualenv in Python 2.X and never got this kind of errors. Any thoughts on what am I doing wrong?
Thanks!!
=======EDITED=======
This is the output of my partitions (fdisk -l):
Device Boot Start End Blocks Id System
/dev/sda1 2048 98707455 49352704 83 Linux
/dev/sda2 303507456 3890644991 1793568768 5 Extended
/dev/sda3 * 98707456 303507455 102400000 7 HPFS/NTFS/exFAT
/dev/sda4 3890644992 3907028991 8192000 82 Linux swap / Solaris
/dev/sda5 303509504 3890644991 1793567744 7 HPFS/NTFS/exFAT`
And also my fstab:
<file system> <mount point> <type> <options> <dump> <pass>
-> was on /dev/sda1 during installation
UUID=a38f9c6d-3cd9-4486-b896-acbc6182ec61 / ext4 errors=remount-ro 0 1
-> swap was on /dev/sda4 during installation
UUID=efad7b53-79a8-4230-8226-9ca90c68ea9d none swap sw 0 0`

Is that a shared partition that you have mounted? Does the shared partition have a different filesystem then the non-shared one you tried upon? If yes, then IMO, that will definitely cause an error since you are making and compiling binaries for python on one filesystem, and so it will not work on another filesystem.
As mentioned in this answer, adding to your /etc/fstab with an entry with exec flag might make it work for you, i.e., you might need to add another entry for the NTFS disk here to make it automount:
<file system> <mount point> <type> <options> <dump> <pass>
-> was on /dev/sdaX during installation
UUID=<uid_of_NTFS> / ntfs auto,user,exec,nodev,rw,errors=remount-ro 0 1

I struggled with this as well so I wrote an ugly bash script to help me with this. The only salient difference between what you do and what I do is on line 133:
/path/to/python/bin/python3.4 /path/to/python/bin/pyvenv /path/to/venv
That is, explicitly name the instance of python and the venv tool. Then
/path/to/venv/bin/pip install django # or whatever
Edit
I installed Linux Mint in a VM to try and build a Python 3.4 virtual environment. Based on the error messages I saw and this answer, I learned that I must do the following to get a complete Python 3.4 build:
apt-get install build-essential libssl-dev openssl
Without this, my Python 3.4 build did not contain pip. Note that you probably want to install readline and other development packages.
Unsolicited Advice
Do not do this as root, create a user dedicated to running your venv
Create a script to create your environment
Check that script into your source code repo
I deleted my python binaries and venvs multiple times and then recreated all of with this script to make sure that my script reproduced my environment and then stripped the identifying information and saved that on github to share it. I should really be using a more formal tool for this like docker/puppet/chef.

Mac OS X 10.6 Python 2.7 pytidylib utidylib could not find libtidy

I am answering my own question, but because I stayed up all night figuring it out, I will hopefully save others some pain. If you are getting either of the following messages after properly installing pytidylib or utidylib, this answer may help.
Learning Python on Snow Leopard, I had installed the 32-bit version of Python 2.7 so that I could use the IDLE interpreter/editor. Stackoverflow has a great explanation why I had to do this.
When I installed utidylib, I got the following error from 'import tidy':
'OSError: Couldn't find libtidy, please make sure it is installed.'
With pytidylib, I got this error when I tried 'from tidylib import tidy_document':
'OSError: Could not load libtidy using any of these names: libtidy,libtidy.so,libtidy-0.99.so.0,cygtidy-0-99-0,tidylib,libtidy.dylib,tidy.'
If you are getting these errors, please read this answer. I hope it helps you.

I had a similar issue (see stacktrace below) on Linux.
With pytidylib==0.2.1 installed with pip in a virtualenv on both Ubuntu 11.10 (Python 2.7) and Gentoo (Python 2.6).
Installing apt-get install libtidy-dev on Ubuntu and emerge app-text/htmltidy on Gentoo, solved it.
----------------------------------------------------------------------
File "/home/andre/Progz/some_site/djcode/apps/default/tests.py", line ?, in default.tests.__test__.doctest
Failed example:
from tidylib import tidy_document
Exception raised:
Traceback (most recent call last):
File "/home/andre/Progz/some_site/lib/python2.6/site-packages/django/test/_doctest.py", line 1267, in __run
compileflags, 1) in test.globs
File "", line 1, in
from tidylib import tidy_document
File "/home/andre/Progz/some_site/lib/python2.6/site-packages/tidylib/__init__.py", line 71, in
raise OSError("Could not load libtidy using any of these names: %s" % (",".join(LIB_NAMES)))
OSError: Could not load libtidy using any of these names: libtidy,libtidy.so,libtidy-0.99.so.0,cygtidy-0-99-0,tidylib,libtidy.dylib,tidy
----------------------------------------------------------------------

Although it looked like the Python packages could not find the files, what was really happening was that I compiled libtidy for the wrong architecture: x86_64 instead of i386. Using the 32-bit Python installer made them incompatible.
For neophytes like me, lipo will tell you the architecture of your files:
$ lipo -info /Library/Frameworks/Python.framework/Versions/2.7/bin/python
Architectures in the fat file: /Library/Frameworks/Python.framework/Versions/2.7/bin/python are: ppc i386
$ lipo -info /usr/lib/libtidy.dylib
Non-fat file: /usr/lib/libtidy.dylib is architecture: x86_64
You may have to locate your libtidy.dylib and python files to compare. To fix the problem, you have to re-compile libtidy for i386 (or link to an i386-compiled libtidy already on your system). To re-compile for i386, use these instructions with the following modification:
cvs -z3 -d:pserver:anonymous#tidy.cvs.sourceforge.net:/cvsroot/tidy co -P tidy
cd tidy
sh build/gnuauto/setup.sh
./configure --prefix=/usr CFLAGS="-arch i386"
make && sudo make install
This worked for me. I hope it also works for you.

I had a same issue.I installed tidylib,but still come out error as followed.
raise OSError("Could not load libtidy using any of these names: %s" % (",".join(LIB_NAMES)))
OSError: Could not load libtidy using any of these names: libtidy,libtidy.so,libtidy-0.99.so.0,cygtidy-0-99-0,tidylib,libtidy.dylib,tidy
then I copy /usr/bin/tidy to my project folder ,and it got solved.