Develop Reference

Failure to install psycopg2 in Pycharm

I'm trying to install psycopg2 to use a postgresql db in the backend of my django project, but every time I run the pip install psycopg2 command I get an error saying:
Error: pg_config executable not found.
pg_config is required to build psycopg2 from source. Please add the directory
containing pg_config to the $PATH or specify the full executable path with the
option:
python setup.py build_ext --pg-config /path/to/pg_config build ...
or with the pg_config option in 'setup.cfg'.
I have added the \bin\ folder containing the file to my path, I've ran the pip install psycopg2-binary command as well, and I've uninstalled and reinstalled posgresql on my computer, I've also ran the python setup.py build_ext command mentioned in the error message, and this error still pops up. I'm operating on a windows 10 os and using posgresql 13.1. Any suggestions?
EDIT: This is the error I'm getting in my terminal:
Error in PyCharm terminal

You've probably done this already, but the steps should be:
Install PostgreSQL.
Add the PostgreSQL bin/ folder to your path.
Restart your command line to make sure your path is set correctly.
Run pip install psycopg2
I have a clean install of Windows 10 over here and it seems to work for me. Running pip install psycopg2-binary should definitely work (even without PostgreSQL) because that doesn't compile anything, so I'm interested if you also have errors there.

python3 setup.py install is failing (RHEL 8)

Issue:
#python3 setup.py install
is failing
Environment: RHEL 8 UBI container
I have a setup.py that looks like this: (any pointers or updates on this setup.py would be appreciated)
import setuptools
print('python/setup.py')
setuptools.setup(
name="process_data",
version="1.0",
description="desc",
author="FirstName LastName",
author_email="dude#abides.abide",
url="https://some.com",
packages=setuptools.find_packages(),
)
I have a directory structure like so (<some>.py in every directory, along with __init__.py):
python/
setup.py
<dir>/__init__.py
<dir>/__init__.py
<dir4>/__init__.py
<dir4>/<sub-dir1>/__init__.py
<dir4>/<sub-dir1>/<sub-dir>__init__.py
I want code in each of these directories to be part of a module package so I can import, nested however deep.
I'm doing this:
#cd python
#python3 setup.py install
I get this error:
python/setup.py
running install
error: can't create or remove files in install directory
The following error occurred while trying to add or remove files in the
installation directory:
/usr/local/lib/python3.6/site-packages/test-easy-install-1625.write-test
[Errno 2] No such file or directory: '/usr/local/lib/python3.6/site-packages/test-easy-install-1625.write-test'
The installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
/usr/local/lib/python3.6/site-packages/
This directory does not currently exist. Please create it and try again, or
choose a different installation directory (using the -d or --install-dir
option).
Questions:
Where is this coming from: test-easy-install-1625.write-test ? (so confused as to what this is)
On my system, indeed this directory does not exist (why is python3 setup.py install looking there? what told it to look there?):
/usr/local/lib/python3.6
There is nothing in my setup.py to indicate to look there.
I looked all over, and it looks like my /site-packages/ directory is here (where other modules are installed, when I ran pip)
`/usr/local/lib64/python3.6/site-packages/`
How do I fix this?
Thank you, I so badly need to get this working right away. I have been working on Windows, and >python setup.py install simply works there.
Are there any env variables I should have setup?
I simply installed python with: dnf -y install python36 that's it. (the latest version available in any RHEL 8 package repo for installation).

I ran into this same error and I was able to fix it by creating a symbolic link. I accept someone else maybe able to offer a more elegant solution and I welcome it.
sudo mkdir /usr/local/lib/python3.6
sudo ln -s /usr/lib/python3.6/site-packages /usr/local/lib/python3.6/

I assume you had enter python command mode, if you don't, the following commands are needed, notice to set variable Path and PythonPath of yours:
setlocal
set env_name=python3.6
set Path=C:\Users\s41167\Miniconda3\envs\%env_name%;C:\Users\s41167\Miniconda3\envs\%env_name%\Scripts;%Path%
set PythonPath=C:\Users\s41167\Miniconda3\envs\%env_name%\Lib\site-packages
cd C:\Users\s41167\Documents\%env_name%
cmd.exe /K activate %env_name%
if you enter python command mode, you can use pip to insall the package:
pip install <package location> --target <install location> --upgrade
below is my install command and result:
(python3.6) C:\Users\s41167\Documents\python3.6>pip install ./python_package --t
arget ./target --upgrade
Processing c:\users\s41167\documents\python3.6\python_package
Building wheels for collected packages: process-data
Building wheel for process-data (setup.py) ... done
Created wheel for process-data: filename=process_data-1.0-cp36-none-any.whl si
ze=1721 sha256=ef5eaf061000f30d472e5dde268694733e40a3f8a4a29fa78faec69f125443c9
Stored in directory: C:\Users\s41167\AppData\Local\Temp\pip-ephem-wheel-cache-
v6lt_ndp\wheels\61\01\f5\07e0760baa10d63e1c43b37eadbb55b79828f4fe337209026a
Successfully built process-data
Installing collected packages: process-data
Successfully installed process-data-1.0
(python3.6) C:\Users\s41167\Documents\python3.6>
and the file arrangement:
.\python_package
__init__.py
setup.py
\directory
__init__.py
\directory4
__init__.py
\sub_dirctory1
__init__.py

in Linux use
sudo python3 setup.py install
in windows use
python setup.py install
or
python3 setup.py install

Mac + virtualenv + pip + postgresql = Error: pg_config executable not found

I was trying to install postgres for a tutorial, but pip gives me error:
pip install psycopg
A snip of error I get:
Error: pg_config executable not found.
Please add the directory containing pg_config to the PATH
or specify the full executable path with the option:
python setup.py build_ext --pg-config /path/to/pg_config build ...
or with the pg_config option in 'setup.cfg'.
Where is pg_config in my virtualenv? How to configure it? I'm using virtualenv because I do not want a system-wide installation of postgres.

On the Mac, if you're using Postgres.app, the pg_config file is in your /Applications/Postgres.app/Contents/Versions/<current_version>/bin directory. That'll need to be added to your system path to fix this error, like this:
export PATH=$PATH:/Applications/Postgres.app/Contents/Versions/<current_version>/bin
So for example, if the current Postgres.app version is 9.5, this export line would be:
export PATH=$PATH:/Applications/Postgres.app/Contents/Versions/9.5/bin
With more recent versions of the Postgres.app (> 9.5?), you can simply add "latest" in place of the version number, like so:
export PATH=$PATH:/Applications/Postgres.app/Contents/Versions/latest/bin

On Mac, the solution is to install postgresql:
brew install postgresql
On CentOS, the solution is to install postgresql-devel:
sudo yum install postgresql-devel
pg_config is in postgresql-devel package

I totally agree with john hight that most of posted answers are totally offtopic assuming the OP exactly specified need of using virtualenv.
For me the answer was runing following command in prompt while having activated virtualenv:
export PATH="/Applications/Postgres.app/Contents/Versions/9.4/bin:$PATH"
(notice that part 9.4 stands for version and may vary)
or if you want to use the latest installed version of Postgres:
export PATH="/Applications/Postgres.app/Contents/Versions/latest/bin:$PATH"
and then:
pip install psycopg2
goes sucesfully assuming you have installed postgres. And if not, then remember that the best and recomended solution is to use: Postgres.app

Don't forget that your $PATH variable in the virtual environment != your global $PATH variable. You can confirm this with 'echo $PATH' in your virtualenv and also in a new shell. So, unless you want to install PostgreSQL as a unique instance inside your virtual environment (not a thing worth doing, imo), you'll need to modify the $PATH variable within the virtualenv to include the path to your global installation (which will solve your missing pg_config error).
Here are the steps:
1.) In a new shell, type 'which pg_config'. This will return the path. Copy it. In my case, the path looked like this: /Applications/Postgres.app/Contents/Versions/9.3/bin
2.) Back in your virtualenv shell, type 'export PATH=/your-path-to-pg_config:$PATH'
3.) Then, still within the virtualenv, 'pip install psycopg2'
If all goes according to plan, this will install psycopg2 within the virtual environment, but the installation will refer to your Global PostgreSQL installation. In my case, this Global installation was installed via Postgres.App, hence the path. I prefer this method of working with psycopg2 as it means I can use the database easily within any virtualenv rather than only within the defined virtual environment.
Hope this helps anyone who arrives here. For Google juice, here's the explicit (and vague) error language returned when you run into this problem:
Command python setup.py egg_info failed with error code 1

Here's how I was able to solve this problem on my Mac (OSX 10.9):
brew update
brew install --force ossp-uuid
brew install postgresql
pip install psycopg
I got a CLANG error when I tried pip install psycopg (an LLVM 5.1 issue), so I had to install psycopg with this command instead:
ARCHFLAGS=-Wno-error=unused-command-line-argument-hard-error-in-future pip install psycopg
It's similar to Mingyu's solution, but there are enough differences that I thought it was worth sharing.

you must configure path postgresql:
export PATH=$PATH:/Library/PostgreSQL/11/bin
after, you must install requirements:
pip3 install -r requirements

For OS X El Capitan (10.11.6) + brew + virtualenv + PostgreSQL 9.5:
After installing PostgreSQL 9.5:
brew install postgresql#9.5
Then, open your terminal and execute:
export PATH=$PATH:/usr/local/opt/postgresql\#9.5/bin/
pip install psycopg2

This error is caused when the build tools can't find the Postgresql libraries.
Often it's required to instruct psycopg2 how to find the pg_config binary, you can:
add the path to pg_config in your shell path (/usr/local/pgsql/bin/)
or edit the setup.cfg file in the psycopg2 source folder and provide
the full path to pg_config on the line that starts with pg_config=
pg_config=/usr/local/pgsql/bin/pg_config
the above is an example, you can do locate pg_config to find out where it resides, or simply type which pg_config and it should tell you the path.
Less often the error comes from not having postgresql installed on your system. If so, download and build postgres, or download a pre-built psycopg2 binary for OS X.

virtualenv is for python packages. I don't think you'll be able to contain postgres inside a virtualenv. The error message you're seeing is presumably because you haven't yet installed postgres. The psycopg2 install script is looking for postgres files (in this case pg_config) and not finding them because it is not installed. postgres can't be installed using pip or virtualenv.

On Windows I installed postgres manually from http://www.enterprisedb.com/products-services-training/pgdownload#windows.
After that the same command works.

In addition to the answers provided by #bkev and #andi, according to the documentation on Postgres.app, you should add the following to your bash_profile on Mac:
export PATH=$PATH:/Applications/Postgres.app/Contents/Versions/latest/bin
Note that, there is no hard-coded version number. I wanted to add this as a comment to above answers, but I don't have enough rep for this.

If you're using postgresql 9.4, the file is located in
/usr/pgsql-9.4/bin/pg_config
The name of the package is
postgresql94-9.4.9-1PGDG.rhel6.x86_64
to add pg_config to your PATH, do the following
PATH=$PATH:/usr/pgsql-9.4/bin/

If you don't have to use the psycopg driver specifically, switch to the pg8000 driver. It's pure Python and less finicky.

On Ubuntu I just needed the postgres dev package:
sudo apt-get install postgresql-server-dev-all

Mine was located in /Library/PostgreSQL/9.4/bin
export PATH=$PATH:/Library/PostgreSQL/9.4/bin

pip install psycopg2-binary
Binary install will work in virtual env, it Will have lib files necessary to work in virtual environment

I added to $PATH postgres, but had similar issue. After made a change in Dockerfile and installed through bin/sh, an install passed. It could be also links in with python alpine version. Not alpine version had not any problem during installation.
# syntax=docker/dockerfile:1
FROM python:3.11-rc-alpine
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
WORKDIR /code
COPY requirements.txt /code/
RUN #pip install --upgrade pip
RUN /bin/sh -c pip install -r requirements.txt
RUN #pip install -r requirements.txt
COPY . /code/

python pip specify a library directory and an include directory

I am using pip and trying to install a python module called pyodbc which has some dependencies on non-python libraries like unixodbc-dev, unixodbc-bin, unixodbc. I cannot install these dependencies system wide at the moment, as I am only playing, so I have installed them in a non-standard location. How do I tell pip where to look for these dependencies ? More exactly, how do I pass information through pip of include dirs (gcc -I) and library dirs (gcc -L -l) to be used when building the pyodbc extension ?

pip has a --global-option flag
You can use it to pass additional flags to build_ext.
For instance, to add a --library-dirs (-L) flag:
pip install --global-option=build_ext --global-option="-L/path/to/local" pyodbc
gcc supports also environment variables:
http://gcc.gnu.org/onlinedocs/gcc/Environment-Variables.html
I couldn't find any build_ext documentation, so here is the command line help
Options for 'build_ext' command:
--build-lib (-b) directory for compiled extension modules
--build-temp (-t) directory for temporary files (build by-products)
--plat-name (-p) platform name to cross-compile for, if supported
(default: linux-x86_64)
--inplace (-i) ignore build-lib and put compiled extensions into the
source directory alongside your pure Python modules
--include-dirs (-I) list of directories to search for header files
(separated by ':')
--define (-D) C preprocessor macros to define
--undef (-U) C preprocessor macros to undefine
--libraries (-l) external C libraries to link with
--library-dirs (-L) directories to search for external C libraries
(separated by ':')
--rpath (-R) directories to search for shared C libraries at runtime
--link-objects (-O) extra explicit link objects to include in the link
--debug (-g) compile/link with debugging information
--force (-f) forcibly build everything (ignore file timestamps)
--compiler (-c) specify the compiler type
--swig-cpp make SWIG create C++ files (default is C)
--swig-opts list of SWIG command line options
--swig path to the SWIG executable
--user add user include, library and rpath
--help-compiler list available compilers

Building on Thorfin's answer and assuming that your desired include and library locations are in /usr/local, you can pass both in like so:
sudo pip install --global-option=build_ext --global-option="-I/usr/local/include/" --global-option="-L/usr/local/lib" <you package name>

Another way to indicate the location of include files and libraries are set relevant environment variables before running pip e.g.
export LDFLAGS=-L/usr/local/opt/openssl/lib
export CPPFLAGS=-I/usr/local/opt/openssl/include
pip install cryptography

Just FYI... If you are having trouble installing a package with pip, then you can use the
--no-clean option to see what is exactly going on (that is, why the build did not work). For instance, if numpy is not installing properly, you could try
pip install --no-clean numpy
then look at the Temporary folder to see how far the build got. On a Windows machine, this should be located at something like:
C:\Users\Bob\AppData\Local\Temp\pip_build_Bob\numpy
Just to be clear, the --no-clean option tries to install the package, but does not clean up after itself, letting you see what pip was trying to do.
Otherwise, if you just want to download the source code, then I would use the -d flag. For instance, to download the Numpy source code .tar file to the current directory, use:
pip install -d %cd% numpy

I was also helped by Thorfin's answer; I was building GTK3+ on windows and installing pygobject, I was having difficulties on how to include multiple folders with pip install.
I tried creating pip config file as per pip documentation. but failed.
the one working is with the command line:
pip install --global-option=build_ext --global-option="-IlistOfDirectories"
# and/or with: --global-option="-LlistofDirectories"
the separator that works with multiple folders in windows is ';' semicolon, NOT colon ':' it might be different in other OS.
sample working command line:
pip install --global-option=build_ext --global-option="-Ic:/gtk-build/gtk/x64/release/include;d:/gtk-build/gtk/x64/release/include/gobject-introspection-1.0" --global-option="-Lc:\gtk-build\gtk\x64\release\lib" pygobject==3.27.1
you can use '' or '/' for path, but make sure do not type backslash next to "
this below will fail because there is backslash next to double quote
pip install --global-option=build_ext --global-option="-Ic:\willFail\" --global-option="-Lc:\willFail\" pygobject==3.27.1

Have you ever used virtualenv? It's Python package that let's you create and maintain multiple isolated environments on one machine. Each can use different modules independent of one another without screwing up dependencies in your system library or a separate virtual environment.
If you don't have root privileges, you can download and use the virtualenv package from source:
$ curl -O https://pypi.python.org/packages/source/v/virtualenv/virtualenv-X.X.tar.gz
$ tar xvfz virtualenv-X.X.tar.gz
$ cd virtualenv-X.X
$ python virtualenv.py myVE
I followed the above steps this weekend on Ubuntu Server 12.0.4 and it worked perfectly. Each new virtual environment you create comes with PIP by default so installing packages into your new environment is easy.

Just in case it's of help to somebody, I still could not find a way to do it through pip, so ended up simply downloading the package and doing through its 'setup.py'. Also switched to what seems an easier to install API called 'pymssql'.

MySQLdb within python2.5 virtualenv

I have a Fedora 11 box with MySQL server. Fedora 11 uses python 2.6 internally and python 2.6 is automatically installed on the box. I have created a python virtual-env for version 2.5.5, so that I can run turbogears 1.x application. I have MySQLdb rpm installed on the box (and it works fine with python 2.6).
When I import MySQLdb from within python version 2.6 it imports is successfully. When I import MySQLdb from within the python 2.5.5 virtual-env the import fails (because I have installed virtual-env with --no-site-packages). So, I have to install MySQLdb python as a local package (local to virtual-env).
'easy_install MySQL-python' within the virtual env fails. It downloads the MySQL-python-1.2.3.c1.tar.gz/download, but the 'python setup.py build' fails with error. The same problem occurs when building the MySQL outside of virtual-env.
Is the 'python setup.py build' for MySQL-python trying to link to a library (and am I missing some library)? Or is the downloaded code missing some header files (unlikely)?
Thanks.
S.Mark,
If I were to install MySQL header files, would they mess with the existing rpms?
[Sorry, for being redundant. In essence, MySQL is functional on the machine, MySQL-python is functional for python 2.6, but MySQL-python is not functional from virtualenv for python 2.5.5.]
Thank you for trying to help.
# rpm -qa | grep -i mysql
MySQL-python-1.2.3-0.4.c1.fc11.x86_64
perl-DBD-MySQL-4.010-1.fc11.x86_64
mysql-libs-5.1.42-7.fc11.x86_64
mysql-5.1.42-7.fc11.x86_64
php-mysql-5.2.12-1.fc11.x86_64
mysql-server-5.1.42-7.fc11.x86_64

_mysql.c:36:23: error: my_config.h: No such file or directory
_mysql.c:38:19: error: mysql.h: No such file or directory
_mysql.c:39:26: error: mysqld_error.h: No such file or directory
_mysql.c:40:20: error: errmsg.h: No such file or directory
Please install MySQL header files, probably here

Works! Thanks for all your help.
I installed mysql-devel package (yum install mysql-devel) on the box.
Then did easy_install MySQL-python from within virtual-env (python 2.5.5), and the compilation/installation was successful (with some warnings).
Thanks.

Just to mention for the others that might end up here too,
I solved this error in installing mysql-devel
$ yum install mysql-devel.x86_64
Hope this helps

In case it does not work even after installing mysql-devel package, check the contents of "/usr/include/mysql".
If is empty then its possible that mysql-devel has installed the header files at some pther place, run following command to find
find /usr/include/ -maxdepth 1 -name mysql*
If found, create a symlink like following to fix the problem.
ln -s /usr/include/mysql51 /usr/include/mysql