Develop Reference

How to explictly tell python install commands which python to use?

I'm having a mess of problems trying to install python modules with pip using a local version of python/pip installed in my home directory. First, this should prove my PATH variable is set correctly:
command -v python
/home/myuserid/PYTHON4/bin/python
command -v python
/home/myuserid/PYTHON4/bin/pip
I then try to install python modules with pip. Some modules install successfully. But one named matlabplot is giving me problems. Specifically, I run
pip install matplotlib --target=/home/myuserid/PYTHON4/PYTHON
The errors are
Complete output from command /usr/local/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-VwF2LG/subprocess32/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-pj2g8u-record/install-record.txt --single-version-externally-managed --compile --home=/tmp/tmplDhrIG:
....
gcc -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/usr/include/python2.7 -c _posixsubprocess.c -o build/temp.linux-x86_64-2.7/_posixsubprocess.o
_posixsubprocess.c:16:10: fatal error: Python.h: No such file or directory
#include "Python.h"
There are two problems with that error message:
First, it is using /usr/local/bin/python. This is the wrong python (I previously installed python there). Where is that coming from? It's not on my path. If I print my entire environment with printenv, it's not listed anywhere.
Second, the gcc compile command explictly includes "-I/usr/include/python2.7" on the command line. That should not happen because I'm trying to use a local python that is installed in/p/home/myuserid/PYTHON4.
I'm guessing the cause is I'm somehow not specifying where python is installed. How can I fix these problems?

Avoid pip install, prefer python -m pip install. This way you're telling explicitly you want to use the python program for which you checked the path. And you can play many variations around it:
python3 -m pip install
/usr/bin/python3 -m pip install
python3.9 -m pip install
...
About the numpy issue, it's because pip don't find an already compiled numpy for your platform. It may be because you're running an old version or pip, so you can first try a python -m pip install --upgrade pip, and try installing numpy again.
You can also take a look at https://pypi.org/simple/numpy, you'll see a list of files like:
numpy-1.21.2-cp38-cp38-win_amd64.whl
numpy-1.21.2-cp39-cp39-macosx_11_0_arm64.whl
numpy-1.21.2-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl
the whl files are pre-compiled ones, but for specific platforms and specific Python versions, here the 3 wheels I listed are for numpy 2.21.2, for cpython 3.9, for windows, macosx, and many linuxes.
Maybe your specific set of Python version / numpy version / platform is not pre-compiled here?
If you really need to let pip compile numpy you'll need some more dependencies, the missing Python.h one means you're missing the python3-dev package from your distrib.
Last hint: you should use a venv: a venv is created by a specific version of Python, and all installations are done inside the venv (so easily removable: just remove the directory, in case it goes badly).

PyAudio installation always fails on Mac

Firstly, I tried
pip install pyaudio
But that didn't work... I found that I need PortAudio, so installed it with brew:
brew install portaudio
That worked and then I restarted my Mac.
But pip install pyaudio still doesn't work and I get the below error message:
ERROR: Command errored out with exit status 1:
command: /Applications/Xcode.app/Contents/Developer/usr/bin/python3 -u -c
'import sys, setuptools, tokenize; sys.argv[0] =
'"'"'/private/var/folders/12/44t6pktj77xbqmyp7x368tdr0000gn/T/pip-install-gvxfbjmp/pyaudio/setup.py'"'"';
file='"'"'/private/var/folders/12/44t6pktj77xbqmyp7x368tdr0000gn/T/pip-install-gvxfbjmp/pyaudio/setup.py'"'"';f=getattr(tokenize,
'"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"',
'"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))'
bdist_wheel -d
/private/var/folders/12/44t6pktj77xbqmyp7x368tdr0000gn/T/pip-wheel-gvmpobvy
cwd: /private/var/folders/12/44t6pktj77xbqmyp7x368tdr0000gn/T/pip-install-gvxfbjmp/pyaudio/
Complete output (16 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-10.14-x86_64-3.7
copying src/pyaudio.py -> build/lib.macosx-10.14-x86_64-3.7
running build_ext
building '_portaudio' extension
creating build/temp.macosx-10.14-x86_64-3.7
creating build/temp.macosx-10.14-x86_64-3.7/src
xcrun -sdk macosx clang -arch x86_64 -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders
-iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/Headers
-DMACOSX=1 -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/include/python3.7m
-c src/_portaudiomodule.c -o build/temp.macosx-10.14-x86_64-3.7/src/_portaudiomodule.o
src/_portaudiomodule.c:29:10: fatal error: 'portaudio.h' file not found
#include "portaudio.h"
^~~~~~~~~~~~~
1 error generated.
error: command 'xcrun' failed with exit status 1
----------------------------------------
ERROR: Failed building wheel for pyaudio
I also tried pip3 install pyaudio without any luck.
BTW I am just doing this because I want to use the Python Speech Recognition API and I think pyaudio is necessary... is it?

Very likely you need to specify the directory path where the compiler can find the source programs like portaudio.h.
Under assumption that the headers are in /usr/local/include directory, try to run the below command in the Terminal:
pip install --global-option='build_ext' --global-option='-I/usr/local/include' --global-option='-L/usr/local/lib' pyaudio

your problem is that with the Xcode command-line. Reinstall using this
xcode-select --install
after xcode is fixed the command('xcrun') should be able to run so just reuse
pip3 install pyaudio

you may need to upgrade pip
python3 -m pip install --upgrade pip
-- Successfully installed pip-21.2.4
and/or invoke pip3
pip3 install ...

error install mysqlclient with pip, library not found for -lssl

I am running mysql#8.x version on my MacOS, I have installed mysql and mysql-connector-o with brew. Currently brew link with mysql.
Running Django project on my python3 virtual env I am receiving following error -
django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module: No module named 'MySQLdb'.
Did you install mysqlclient or MySQL-python?
and on installing the mysqlclient using pip, I am reciving following error
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -Dversion_info=(1,3,13,'final',0) -D__version__=1.3.13 -I/usr/local/Cellar/mysql/8.0.12/include/mysql -I/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/include/python3.6m -c _mysql.c -o build/temp.macosx-10.13-x86_64-3.6/_mysql.o
clang -bundle -undefined dynamic_lookup -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk build/temp.macosx-10.13-x86_64-3.6/_mysql.o -L/usr/local/Cellar/mysql/8.0.12/lib -lmysqlclient -lssl -lcrypto -o build/lib.macosx-10.13-x86_64-3.6/_mysql.cpython-36m-darwin.so
ld: library not found for -lssl
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'clang' failed with exit status 1
----------------------------------------
Command "/Volumes/Samsung_T5/Work/python/repos/venv3/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/tmp/pip-install-kf2fkhtc/mysqlclient/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/tmp/pip-record-qwpjjw3e/install-record.txt --single-version-externally-managed --compile --install-headers /Volumes/Samsung_T5/Work/python/repos/venv3/bin/../include/site/python3.6/mysqlclient" failed with error code 1 in /private/tmp/pip-install-kf2fkhtc/mysqlclient/

As jordanm pointed it out, the issue was exactly with the missing openssl libraries, for which I followed these steps to fix my issue -
Installing openssl
brew install openssl
Now
pip install mysqlclient should work.
If it doesn't and still shows the same error library not found for -lssl, you could also try to link against brew's openssl:
env LDFLAGS="-I/usr/local/opt/openssl/include -L/usr/local/opt/openssl/lib" pip install mysqlclient
if this still does not work, you might need to use the --no-cache option of pip, e.g.
env LDFLAGS="-I/usr/local/opt/openssl/include -L/usr/local/opt/openssl/lib" pip --no-cache install mysqlclient

After installing openssl via Homebrew I placed
export PATH="/usr/local/opt/openssl/bin:$PATH"
export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"
in my .bash_profile and ran
source .bash_profile
and then
pip install mysqlclient

Compilation error when pip3 install scipy

I need to install scipy. However, after about 5-7minutes of compilation time I am getting the follwing error on Ubuntu 14.04.3 LTS:
sudo pip3 install scipy
...
error: Command "x86_64-linux-gnu-g++ -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -D__STDC_FORMAT_MACROS=1 -Iscipy/sparse/sparsetools -I/usr/local/lib/python3.4/dist-packages/numpy/core/include -I/usr/include/python3.4m -c scipy/sparse/sparsetools/csc.cxx -o build/temp.linux-x86_64-3.4/scipy/sparse/sparsetools/csc.o" failed with exit status 4
In file included from /usr/local/lib/python3.4/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1777:0,
from /usr/local/lib/python3.4/dist-packages/numpy/core/include/numpy/ndarrayobject.h:18,
from scipy/sparse/sparsetools/sparsetools.h:5,
from scipy/sparse/sparsetools/csc.cxx:4:
/usr/local/lib/python3.4/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by " \
^
x86_64-linux-gnu-g++: internal compiler error: Killed (program cc1plus)
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-4.8/README.Bugs> for instructions.
----------------------------------------
Cleaning up...
Command /usr/bin/python3 -c "import setuptools, tokenize;__file__='/tmp/pip_build_root/scipy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-e3q7ek8f-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip_build_root/scipy
Storing debug log for failure in /home/ubuntu/.pip/pip.log
Does anybody have an idea how to fix this?

It seems your numpy was installed using the distro package manager (i.e. by running apt-get install python3-numpy) since the numpy package is in the dist-packages directory of your python libs. Why don't you install scipy this way too? You can just run
sudo apt-get install python3-scipy
If you want to use pip, I suggest installing packages in virtualenv instances (or, even better, virtualenvwrapper) so that you avoid such conflicts. Otherwise, you should be using a systemwide version of pip, rather than a user installed one (the output says your pip belongs to user ubuntu), so you could first install it using apt-get and then make sure to use this new pip to install scipy.

Installing setuptools made it work for me on ubuntu 14.04

Install cartopy using pip on mac os and macports

I am trying to install cartopy on OS X 10.10 (Yosemite). My python is installed using macports and when I run:
sudo pip install cartopy
I get the following error:
/usr/bin/clang -Wno-unused-result -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -pipe -Os -I/opt/local/Library/Frameworks/Python.framework/Versions/3.4/include -I./lib/cartopy -I/opt/local/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m -c lib/cartopy/trace.cpp -o build/temp.macosx-10.10-x86_64-3.4/lib/cartopy/trace.o
lib/cartopy/trace.cpp:236:10: fatal error: 'geos_c.h' file not found
#include "geos_c.h"
^
1 error generated.
error: command '/usr/bin/clang' failed with exit status 1
----------------------------------------
Command "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/bin/python3.4 -c "import setuptools, tokenize;__file__='/tmp/pip-build-hwrcv0a1/cartopy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-ytar95bq-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-hwrcv0a1/cartopy
Note that I installed the dependencies as noted here and geos_c.h can be found in /opt/local/include. It seems like pip fails to find this dependency in the macports tree.
Did anyone experienced this kind of problem? How can I fix it?
This question was also raised as an issue at https://github.com/SciTools/cartopy/issues/535.

There was recently a bit of work to improve the pip install cartopy experience. Would you mind trying it again with cartopy v0.12.0.
See https://github.com/SciTools/cartopy/issues/610 for reference.

This is to answer the last question from Dror, since I am using macports and "pip install cartopy" failed to find "proj_api.h". To solve this:
You need to make sure that the proj port is installed: sudo port install proj
Help pip to locate the header in "/opt/local/include": CFLAGS="-I/opt/local/include" pip install cartopy
Cheers and HTH