Develop Reference

How to find out where the Python include directory is?

I know that after installing Python via Homebrew my include directory is here:
/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/include/python2.7
Is there a way I can make Python tell me where its include/lib directories are? Something along the lines of:
python -c "import sys; print '\n'.join(sys.path)"

There must be an easier way to do this from Python, I thought, and there is, in the standard library of course. Use get_paths from sysconfig :
from sysconfig import get_paths
from pprint import pprint
info = get_paths() # a dictionary of key-paths
# pretty print it for now
pprint(info)
{'data': '/usr/local',
'include': '/usr/local/include/python2.7',
'platinclude': '/usr/local/include/python2.7',
'platlib': '/usr/local/lib/python2.7/dist-packages',
'platstdlib': '/usr/lib/python2.7',
'purelib': '/usr/local/lib/python2.7/dist-packages',
'scripts': '/usr/local/bin',
'stdlib': '/usr/lib/python2.7'}
You could also use the -m switch with sysconfig to get the full output of all configuration values.
This should be OS/Python version agnostic, use it anywhere. :-)

On my PC, the command is python-config --includes. Make sure you use the python-config that homebrew installed, not the default one.

My one-line solution is
python -c "from sysconfig import get_paths as gp; print(gp()['include'])"
If you would like to embed the code within a Unix shell (such as bash), you have to use escaped double quotations.
python -c "from sysconfig import get_paths as gp; print(gp()[\"include\"])"

Not getting LD_LIBRARY_PATH

I am amending an existing script in which I want to check the set of libraries used in an executable with the shared libraries called at the run time. I have the list of libraries which I need to compare with the shared libraries. For getting shared libraries I am trying to get LD_LIBRARY_PATH by giving below code but I had no luck. I tried checking the variable on command line by giving
echo $LD_LIBRARY_PATH
and it returned /opt/cray/csa/3.0.0-1_2.0501.47112.1.91.ari/lib64:/opt/cray/job/1.5.5-0.1_2.0501.48066.2.43.ari/lib64
the things that I have already tried are (this is a python script)
#! /usr/bin/python -E
import os
ld_lib_path = os.environ.get('LD_LIBRARY_PATH')
#ld_lib_path = os.environ["LD_LIBRARY_PATH"]

I think you are just missing a print in your script? This works for me from the command line:
python -c 'import os; temp=os.environ.get("LD_LIBRARY_PATH"); print temp'
script:
#! /usr/bin/python -E
import os
ld_lib_path = os.environ.get('LD_LIBRARY_PATH')
print ld_lib_path

os.environ['PATH'] does not work when running Python code from command-line

Why does the following not work?
$ python -c 'import os; print os.environ['PATH']'
Traceback (most recent call last):
File "<string>", line 1, in <module>
NameError: name 'PATH' is not defined
whereas
$ python
>>> import os
>>> print os.environ['PATH']
works?

Quotes. Try:
python -c 'import os; print os.environ["PATH"]'
The shell is stripping out the single quotes before the command gets to Python, as you can see with the following command:
echo 'import os; print os.environ['PATH']'
(In fact, you can see by the syntax coloring here that something weird is going to happen.)
Now, why does this happen? The single quotes around PATH end a string and start a new one. Essentially, the shell is parsing it as the following three strings (although without the line breaks):
import os; print os.environ[
PATH
]
And that's what gets passed to Python, which sees PATH as a variable name, and rightly complains that you haven't defined it.
For more information on using your shell, see man bash. ("How do I use my shell?" is not, strictly speaking, a Python question, or even a programming question.)

Eggs in path before PYTHONPATH environment variable

If I have packages installed from easy_install, the eggs are prepended to sys.path before the items in the PYTHONPATH variable.
For example, if I have an egg package called foo installed as well as a package called foo in the current directory, and then do this:
PYTHONPATH="." python
>>> import foo
This will use the egg version of foo instead of the local directory. Inspecting sys.path shows that eggs are placed before items from PYTHONPATH. This seems broken. Is there any way to override this behavior?

Unfortunately this is done with a hard-coded template deep inside setuptools/command/easy_install.py. You could create a patched setuptools with an edited template, but I've found no clean way to extend easy_install from the outside.
Each time easy_install runs it will regenerate the file easy_install.pth. Here is a quick script which you can run after easy_install, to remove the header and footer from easy_install.pth. You could create a wrapper shell script to run this immediately after easy_install:
#!/usr/bin/env python
import sys
path = sys.argv[1]
lines = open(path, 'rb').readlines()
if lines and 'import sys' in lines[0]:
open(path, 'wb').write(''.join(lines[1:-1]) + '\n')
Example:
% easy_install gdata
% PYTHONPATH=xyz python -c 'import sys; print sys.path[:2]'
['', '/Users/pat/virt/lib/python2.6/site-packages/gdata-2.0.14-py2.6.egg']
% ./fix_path ~/virt/lib/python2.6/site-packages/easy_install.pth
% PYTHONPATH=xyz python -c 'import sys; print sys.path[:2]'
['', '/Users/pat/xyz']
For more clarification, here is the format of easy-install.pth:
import sys; sys.__plen = len(sys.path)
./gdata-2.0.14-py2.6.egg
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)
The two import sys lines are the culprit causing the eggs to appear at the start of the path. My script just removes those sys.path-munging lines.

Consider using the -S command-line option to suppress *.pth processing:
python -c 'import sys; print("\n".join(sys.path))'
python -S -c 'import sys; print("\n".join(sys.path))'
https://docs.python.org/3/library/site.html#site.main
You can also use -S with site.main() to delay *.pth processing until runtime, say to capture the original sys.path for appending:
export PYTHONPATH=$(
PYTHONPATH='' \
python -c 'import sys; \
sys.path.extend(sys.argv[1:]); old=list(sys.path); \
import site; site.main(); \
[ old.append(p) for p in sys.path if p not in old ]; \
sys.path=old; \
print ":".join(sys.path)' \
$EXTRA_PATH $ANOTHER_PATH)
python -S ... # using explicit PYTHONPATH
Start from explicit empty PYTHONPATH
Append to sys.path explicitly with extend
Import site and call site.main()
Append new paths to old path and then install it in sys.path
Print with ":" for PYTHONPATH
python -S is desirable for later runs only using $PYTHONPATH
python -S may or may not be desirable while setting PYTHONPATH (depending on if you need sys.path expanded before extending)

I have done something like the following to prepend to the system path when running a top-level python executable file:
import sys
sys.path = ["<your python path>"] + sys.path
Often, the "<your python path>" for me involves use of the __file__ attribute to do relative look up for a path that includes the top-level module for my project. This is not recommended for use in producing, eggs, though I don't seem to mind the consequences. There may be another alternative to __file__.

Check for existence of Python dev files from bash script

I am creating a simple bash script to download and install a python Nagios plugin. On some older servers the script may need to install the subprocess module and as a result I need to make sure the correct python-devel files are installed.
What is an appropriate and cross platform method of checking for these files. Would like to stay away from rpm or apt.
If you can tell me how to do the check from within python that would work. Thanks!
Update:
This is the best I have come up with. Anyone know a better or more conclusive method?
if [ ! -e $(python -c 'from distutils.sysconfig import get_makefile_filename as m; print m()') ]; then echo "Sorry"; fi

That would be pretty much how I would go about doing it. Seems reasonable simple.
However, if I need to be really sure that python-devel files are installed for the current version of Python, I would look for the relevant Python.h file. Something along the lines of:
# first, makes sure distutils.sysconfig usable
if ! $(python -c "import distutils.sysconfig.get_config_vars" &> /dev/null); then
echo "ERROR: distutils.sysconfig not usable" >&2
exit 2
fi
# get include path for this python version
INCLUDE_PY=$(python -c "from distutils import sysconfig as s; print s.get_config_vars()['INCLUDEPY']")
if [ ! -f "${INCLUDE_PY}/Python.h" ]; then
echo "ERROR: python-devel not installed" >&2
exit 3
fi
Note: distutils.sysconfig may not be supported on all platforms so not the most portable solution, but still better than trying to cater for variations in apt, rpm and the likes.
If you really need to support all platforms, it might be worth exploring what is done in the AX_PYTHON_DEVEL m4 module. This module can be used in a configure.ac script to incorporate checks for python-devel during the ./configure stage of an autotools-based build.

Imho your solutions works well.
Otherwise, a more "elegant" solution would be to use a tiny script like:
testimport.py
#!/usr/bin/env python2
import sys
try:
__import__(sys.argv[1])
print "Sucessfully import", sys.argv[1]
except:
print "Error!"
sys.exit(4)
sys.exit(0)
And call it with testimport.sh distutils.sysconfig
You can adapt it to check for internal function if needed...

For those looking for a pure python solution that also works for python3:
python3 -c 'from distutils.sysconfig import get_makefile_filename as m; from os.path import isfile; import sys ; sys.exit(not isfile(m()))')
Or as a file script check-py-dev.py:
from distutils.sysconfig import get_makefile_filename as m
from os.path import isfile
import sys
sys.exit(not isfile(m()))
To get a string in bash, just use the exit output:
python3 check-py-dev.py && echo "Ok" || echo "Error: Python header files NOT found"