Develop Reference

how to install pandas on m1 Mac

clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -DNPY_NO_DEPRECATED_API=0 -Ipandas/_libs/src/ujson/python -Ipandas/_libs/src/ujson/lib -Ipandas/_libs/src/datetime -I/Users/jesse/Desktop/ssc/pythonProject3/venv/lib/python3.8/site-packages/numpy/core/include -I/Users/jesse/Desktop/ssc/pythonProject3/venv/include -I/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c pandas/_libs/src/ujson/python/ujson.c -o build/temp.macosx-10.14.6-arm64-3.8/pandas/_libs/src/ujson/python/ujson.o -D_GNU_SOURCE -Wno-unused-function
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -DNPY_NO_DEPRECATED_API=0 -Ipandas/_libs/src/ujson/python -Ipandas/_libs/src/ujson/lib -Ipandas/_libs/src/datetime -I/Users/jesse/Desktop/ssc/pythonProject3/venv/lib/python3.8/site-packages/numpy/core/include -I/Users/jesse/Desktop/ssc/pythonProject3/venv/include -I/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c pandas/_libs/src/ujson/python/objToJSON.c -o build/temp.macosx-10.14.6-arm64-3.8/pandas/_libs/src/ujson/python/objToJSON.o -D_GNU_SOURCE -Wno-unused-function
pandas/_libs/src/ujson/python/objToJSON.c:181:12: error: use of undeclared identifier 'NUMPY_IMPORT_ARRAY_RETVAL'
return NUMPY_IMPORT_ARRAY_RETVAL;
^
1 error generated.
error: command 'clang' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /Users/jesse/Desktop/ssc/pythonProject3/venv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/ky/s4qqls154f9c4j5lfcfspbpm0000gn/T/pip-install-t5zlzk09/pandas_67e63708ec954c589e697c9f10e3730c/setup.py'"'"'; __file__='"'"'/private/var/folders/ky/s4qqls154f9c4j5lfcfspbpm0000gn/T/pip-install-t5zlzk09/pandas_67e63708ec954c589e697c9f10e3730c/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/ky/s4qqls154f9c4j5lfcfspbpm0000gn/T/pip-record-cnus0d48/install-record.txt --single-version-externally-managed --compile --install-headers /Users/jesse/Desktop/ssc/pythonProject3/venv/include/site/python3.8/pandas Check the logs for full command output.
(base)
I want to download pandas on Mac mini using PyCharm.
What should I do?

This was answered on:
Trouble installing Pandas on new MacBook Air M1
python3 -m pip install virtualenv
virtualenv -p python3.8 venv
source venv/bin/activate
pip install --upgrade pip
pip install numpy cython
git clone https://github.com/pandas-dev/pandas.git
cd pandas
python3 setup.py install

You have to have numpy installed in your environment to build pandas from source for all versions of pandas that come without a pyproject.toml. Please install numpy first and try again.
If numpy is installed on your machine you may need to update the version using pip install numpy -U or an equivalent command executed via PyCharm GUI.

While caeneb's trick worked like a charm, I found that upgrading your python to a later version and updating pip worked as well.
for me these where the commands that did the trick (I manage my python installation with pyenv)
pyenv install 3.9.9
pyenv global 3.9.9 # use this python version as the default
pip install pandas # just works
I guess that without pyenv you can simply run the following commands (untested)
brew install python#3.9.9
$(brew --prefix)/bin/python3.9 -m pip install pandas # makes sure to use the right pip

From manually reviewing the numpy git log, downgrading numpy using
pip install numpy==1.18.5 and then
OPENBLAS="$(brew --prefix openblas)" MACOSX_DEPLOYMENT_TARGET=12.4 pip3 install pandas==0.25.3 --no-use-pep517
After installation of pandas, I was able to upgrade numpy again

I recommend you to use conda, so that you can install virtual environments, also with different python version and eventually with different R (and also node.js). Much simpler and programs are pre-compiled (and conda was created for such cases).
Else, are you sure you have xcode installed? Check on Apple store. Xcode is the "SDK" (developer toolkit) of Apple with the compiler: the clang: you doesn't have it, so the error. It gives you also other tools which are generally needed for pip. Note: you should also execute manually also one xcode command, to be able to accept the xcode condition (else you may have problem on scripts which uses xcode).

This works for me
pip install git+git://github.com/pandas-dev/pandas.git

$pip install pandas
For installing pip in Mac :
$curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
OR
Install python with homebrew:
brew install python
Make sure of correct path:
$which python
Now install pandas:
$pip install pandas

"Failed building wheel for regex" while installing pip package

I am having issues installing a pip package on Windows 10 WSL running Debian Stretch.
While running sudo pip install invoice2data, with python3-pip installed I run into following errors.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Failed building wheel for regex
Running setup.py clean for regex
Failed to build regex
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c regex_2/_regex.c -o build/temp.linux-x86_64-2.7/regex_2/_regex.o
regex_2/_regex.c:46:20: fatal error: Python.h: No such file or directory
#include "Python.h"
^
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, okenize;__file__='/tmp/pip-install-D9zG6P/regex/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-0dvlsB/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-D9zG6P/regex/

WSL is unrelated to the issue, this is a fairly standard error.
Ensure the following packages are installed. Install them using apt-get install packagename. The issue this particular time was resolved by installing python-dev.
python3
python3-pip
ipython3
build-essential
python-dev
python3-dev
As a single command:
sudo apt-get install python3 python3-pip ipython3 build-essential python-dev python3-dev

Had a similar error while running pip3 install pyinquirer
ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-a7ojseph/regex/setup.py'"'"'; __file__='"'"'/tmp/pip-install-a7ojseph/regex/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-bodowot9/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.7/regex Check the logs for full command output.
Installing python3-dev didn't work for me, as I already had it installed. However installing build-essential did the trick and the regex module installed successfully.

I'm running Python 3.5 on Ubuntu. How I was able to install the regex Python package is:
$ sudo apt-get install libpython3.5-dev
$ pip3 install regex --no-use-wheel
Background research details:
I determined the package name that provides the missing Python.h file by using apt-file to locate it.
# install the apt-file package in case you don't have it
$ sudo apt-get install apt-file
# populate/refresh the local apt-file package data
$ sudo apt-file update
# search for /Python.h. Since it's a C header file,
# I also grep for /include to limit the results.
$ sudo apt-file search /Python.h | grep /include
libpython2.7-dbg: /usr/include/python2.7_d/Python.h
libpython2.7-dev: /usr/include/python2.7/Python.h
libpython3.5-dbg: /usr/include/python3.5dm/Python.h
libpython3.5-dev: /usr/include/python3.5m/Python.h
pypy-dev: /usr/lib/pypy/include/Python.h
Then made an educated guess as to which package I needed. Ignore Python2, ignore debug (dbg), ignore pypy, thus leaving libpython3.5-dev.

I landed on this page when I had a similar error on Alpine so I will leave this solution here for the alternate me when he has a similar issue :)
apk add build-base --no-cache
https://wiki.alpinelinux.org/wiki/GCC

restart the bash. I had the same error, but restarting helped

If your interpreter is pypy, then install following packages: using sudo apt-get install 'packagename'
pypy3
pypy3-pip
ipypy3
build-essential
pypy-dev
pypy3-dev
Works perfectly for installing nltk and regex

error install mysqlclient with pip, library not found for -lssl

I am running mysql#8.x version on my MacOS, I have installed mysql and mysql-connector-o with brew. Currently brew link with mysql.
Running Django project on my python3 virtual env I am receiving following error -
django.core.exceptions.ImproperlyConfigured: Error loading MySQLdb module: No module named 'MySQLdb'.
Did you install mysqlclient or MySQL-python?
and on installing the mysqlclient using pip, I am reciving following error
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -Dversion_info=(1,3,13,'final',0) -D__version__=1.3.13 -I/usr/local/Cellar/mysql/8.0.12/include/mysql -I/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/include/python3.6m -c _mysql.c -o build/temp.macosx-10.13-x86_64-3.6/_mysql.o
clang -bundle -undefined dynamic_lookup -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk build/temp.macosx-10.13-x86_64-3.6/_mysql.o -L/usr/local/Cellar/mysql/8.0.12/lib -lmysqlclient -lssl -lcrypto -o build/lib.macosx-10.13-x86_64-3.6/_mysql.cpython-36m-darwin.so
ld: library not found for -lssl
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'clang' failed with exit status 1
----------------------------------------
Command "/Volumes/Samsung_T5/Work/python/repos/venv3/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/private/tmp/pip-install-kf2fkhtc/mysqlclient/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /private/tmp/pip-record-qwpjjw3e/install-record.txt --single-version-externally-managed --compile --install-headers /Volumes/Samsung_T5/Work/python/repos/venv3/bin/../include/site/python3.6/mysqlclient" failed with error code 1 in /private/tmp/pip-install-kf2fkhtc/mysqlclient/

As jordanm pointed it out, the issue was exactly with the missing openssl libraries, for which I followed these steps to fix my issue -
Installing openssl
brew install openssl
Now
pip install mysqlclient should work.
If it doesn't and still shows the same error library not found for -lssl, you could also try to link against brew's openssl:
env LDFLAGS="-I/usr/local/opt/openssl/include -L/usr/local/opt/openssl/lib" pip install mysqlclient
if this still does not work, you might need to use the --no-cache option of pip, e.g.
env LDFLAGS="-I/usr/local/opt/openssl/include -L/usr/local/opt/openssl/lib" pip --no-cache install mysqlclient

After installing openssl via Homebrew I placed
export PATH="/usr/local/opt/openssl/bin:$PATH"
export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"
in my .bash_profile and ran
source .bash_profile
and then
pip install mysqlclient

Install cartopy using pip on mac os and macports

I am trying to install cartopy on OS X 10.10 (Yosemite). My python is installed using macports and when I run:
sudo pip install cartopy
I get the following error:
/usr/bin/clang -Wno-unused-result -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -pipe -Os -I/opt/local/Library/Frameworks/Python.framework/Versions/3.4/include -I./lib/cartopy -I/opt/local/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m -c lib/cartopy/trace.cpp -o build/temp.macosx-10.10-x86_64-3.4/lib/cartopy/trace.o
lib/cartopy/trace.cpp:236:10: fatal error: 'geos_c.h' file not found
#include "geos_c.h"
^
1 error generated.
error: command '/usr/bin/clang' failed with exit status 1
----------------------------------------
Command "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/bin/python3.4 -c "import setuptools, tokenize;__file__='/tmp/pip-build-hwrcv0a1/cartopy/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-ytar95bq-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-hwrcv0a1/cartopy
Note that I installed the dependencies as noted here and geos_c.h can be found in /opt/local/include. It seems like pip fails to find this dependency in the macports tree.
Did anyone experienced this kind of problem? How can I fix it?
This question was also raised as an issue at https://github.com/SciTools/cartopy/issues/535.

There was recently a bit of work to improve the pip install cartopy experience. Would you mind trying it again with cartopy v0.12.0.
See https://github.com/SciTools/cartopy/issues/610 for reference.

This is to answer the last question from Dror, since I am using macports and "pip install cartopy" failed to find "proj_api.h". To solve this:
You need to make sure that the proj port is installed: sudo port install proj
Help pip to locate the header in "/opt/local/include": CFLAGS="-I/opt/local/include" pip install cartopy
Cheers and HTH

Can't install lxml in python2.7

Im trying to install lxml within a virtualenv with sudo pip install lxml and also sudo pip install --upgrade lxml but getting the following in both cases:
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,
relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes
-D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat
-Werror=format-security build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o -lxslt
-lexslt -lxml2 -lz -lm -o build/lib.linux-x86_64-2.7/lxml/etree.so
/usr/bin/ld: cannot find -lz
collect2: error: ld returned 1 exit status
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Cleaning up...
Command /usr/bin/python -c "import setuptools,
tokenize;__file__='/tmp/pip_build_root/lxml/setup.py';
exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'),
__file__, 'exec'))" install --record /tmp/pip-nmFOYf-record/install-record.txt
--single-version-externally-managed --compile failed with error code 1 in
/tmp/pip_build_root/lxml
Storing debug log for failure in /root/.pip/pip.log
I have tried all the posted solutions here, which implies that I have libxml2-dev, libxslt-dev and python-dev installed and I also installed build-essential
I'm currently running Linux Mint 17 Debian Based which uses apt-get as package manager.
python-lxml was already preinstalled.

lxml depends on various C libraries, and you have to have those C libraries installed—including their development files (headers, .so or .a libraries, etc.)—to build lxml. The installation docs explain what prerequisites you need to build on your particular platform.
This error:
/usr/bin/ld: cannot find -lz
… means that the prerequisite you're missing is libz, aka zlib.
Since you're not on Windows, it's incredibly unlikely that you actually don't have zlib at all… but pretty plausible that you don't have the development files for zlib. On some platforms, most notably many linux distros, packages are typically split into separate pieces. For example, the parts of zlib needed at runtime may be in a package named zlib, while the parts needed for building other programs that need zlib in a package named zlib-dev or zlib-devel. The exact details depend on your exact platform.
That being said, if you don't have the zlib development files, you probably don't have the libxml2 or libxslt development files either, because I don't know of any platform where installing those wouldn't have pulled in the zlib files as well.
At any rate, since you haven't told us what platform you're one (and distro, if linux), I don't know what package manager to use, what the packages are named, etc., but do whatever's appropriate for your platform.
Also:
I already have python-lxml installed
You really shouldn't install the same package both with your distro's package manager and with pip; that's a great way to confuse yourself.
But at any rate, most likely you installed python-lxml from a binary package, not from source, which means you didn't need the build requirements. Now you're trying to build it from source, which means you do.

On an ubuntu based linux distro you can use:
sudo apt-get install zlib1g-dev