Develop Reference

What is the correct way (if any) to use Python 2 and 3 libraries in the same program?

I wish to write a python script for that needs to do task 'A' and task 'B'. Luckily there are existing Python modules for both tasks, but unfortunately the library that can do task 'A' is Python 2 only, and the library that can do task 'B' is Python 3 only.
In my case the libraries are small and permissively-licensed enough that I could probably convert them both to Python 3 without much difficulty. But I'm wondering what is the "right" thing to do in this situation - is there some special way in which a module written in Python 2 can be imported directly into a Python 3 program, for example?

The "right" way is to translate the Py2-only module to Py3 and offer the translation upstream with a pull request (or equivalent approach for non-git upstream repos). Seriously. Horrible hacks to make py2 and py3 packages work together are not worth the effort.

I presume you know of tools such as 2to3, that aim to make the job of porting code to py3k easier, just repeating it here for others' reference.
In situations where I have to use libraries from python3 and python2, I've been able to work around it using the subprocess module. Alternatively, I've gotten around this issue with shell scripts that pipes output from the python2 script to the python3 script and vice-versa. This of course covers only a tiny fraction of use cases, but if you're transferring text (or maybe even picklable objects) between 2 & 3, it (or a more thought out variant) should work.
To the best of my knowledge, there isn't a best practice when it comes to mixing versions of python.
I present to you an ugly hack
Consider the following simple toy example, involving three files:
# py2.py
# file uses python2, here illustrated by the print statement
def hello_world():
print 'hello world'
if __name__ == '__main__':
hello_world()
# py3.py
# there's nothing py3 about this, but lets assume that there is,
# and that this is a library that will work only on python3
def count_words(phrase):
return len(phrase.split())
# controller.py
# main script that coordinates the work, written in python3
# calls the python2 library through subprocess module
# the limitation here is that every function needed has to have a script
# associated with it that accepts command line arguments.
import subprocess
import py3
if __name__ == '__main__':
phrase = subprocess.check_output('python py2.py', shell=True)
num_words = py3.count_words(phrase)
print(num_words)
# If I run the following in bash, it outputs `2`
hals-halbook: toy hal$ python3 controller.py
2

os.path.islink on windows with python

On Windows 7 with Python 2.7 how can I detect if a path is a symbolic link?
This does not work os.path.islink(), it says it returns false if false or not supported and the path I'm providing is definitely a symbolic link so I'm assuming it's not supported on windows? What can I do?

The root problem is that you're using too old a version of Python. If you want to stick to 2.x, you will not be able to take advantage of new features added after early 2010.
One of those features is handling NTFS symlinks. That functionality was added in 3.2 in late 2010. (See the 3.2, 3.1, and 2.7 source for details.)
The reason Python didn't handle NTFS symlinks before then is that there was no such thing until late 2009. (IIRC, support was included in the 6.0 kernel, but userland support requires a service pack on Vista/2008; only 7/2008R2 and newer come with it built in. Plus, you need a new-enough MSVCRT to be able to access that userland support, and Python has an explicit policy of not upgrading to new Visual Studio versions within a minor release.)
The reason the code wasn't ported back to 2.x is that there will never be a 2.8, and bug fix releases like 2.7.3 (or 2.7.4) don't get new features, only bug fixes.
This has been reported as issue 13143, and the intended fix is to change the 2.7 docs to clarify that islink always returns False on Windows.
So, if you want to read NTFS symlinks under Windows, either upgrade to Python 3.2+, or you have to use win32api, ctypes, etc. to do it yourself.
Or, as Martijn Pieters suggests, instead of doing it yourself, use a third-party library like jaraco.windows that does it and/or borrow their code.
Or, if you really want, borrow the code from the 3.2 source and build a C extension module around it. If you trace down from ntpath to os to nt (which is actually posixmodule.c), I believe the guts of it are in win32_xstat_impl and win32_xstat_impl_w.

This is what I ended up using to determine if a file or a directory is a link in Windows 7:
from subprocess import check_output, CalledProcessError
import os.path
import ctypes
def isLink(path):
if os.path.exists(path):
if os.path.isdir(path):
FILE_ATTRIBUTE_REPARSE_POINT = 0x0400
attributes = ctypes.windll.kernel32.GetFileAttributesW(unicode(path))
return (attributes & FILE_ATTRIBUTE_REPARSE_POINT) > 0
else:
command = ['dir', path]
try:
with open(os.devnull, 'w') as NULL_FILE:
o0 = check_output(command, stderr=NULL_FILE, shell=True)
except CalledProcessError as e:
print e.output
return False
o1 = [s.strip() for s in o0.split('\n')]
if len(o1) < 6:
return False
else:
return 'SYMLINK' in o1[5]
else:
return False
EDIT: Modified code as per suggestions of Zitrax and Annan
EDIT: Added include statements as per the suggestion of shioko

For directories:
import os, ctypes
def IsSymlink(path):
FILE_ATTRIBUTE_REPARSE_POINT = 0x0400
return os.path.isdir(path) and (ctypes.windll.kernel32.GetFileAttributesW(unicode(path)) & FILE_ATTRIBUTE_REPARSE_POINT):
Source

You can also use the pywin32 module: GetFileAttributes is available in the win32api sub-module and FILE_ATTRIBUTE_REPARSE_POINT in the win32con module. For instance, to test if a given path is a symlink to a directory, the code becomes:
import os
import win32api
import win32con
def is_directory_symlink(path):
return bool(os.path.isdir(path)
and (win32api.GetFileAttributes(path) &
win32con.FILE_ATTRIBUTE_REPARSE_POINT))
If using Python 2 and the path may contain non-ascii characters, GetFileAttributes requires a unicode string. However, simply using unicode(path) will generally fail: you should test if path is a str and, if so, use its decode method.

Just using if file[-4:len(file)] != ".lnk": works for me

Tab-completion in Python interpreter in OS X Terminal

Several months ago, I wrote a blog post detailing how to achieve tab-completion in the standard Python interactive interpreter--a feature I once thought only available in IPython. I've found it tremendously handy given that I sometimes have to switch to the standard interpreter due to IPython unicode issues.
Recently I've done some work in OS X. To my discontent, the script doesn't seem to work for OS X's Terminal application. I'm hoping some of you with experience in OS X might be able to help me trouble-shoot it so it can work in Terminal, as well.
I am reproducing the code below
import atexit
import os.path
try:
import readline
except ImportError:
pass
else:
import rlcompleter
class IrlCompleter(rlcompleter.Completer):
"""
This class enables a "tab" insertion if there's no text for
completion.
The default "tab" is four spaces. You can initialize with '\t' as
the tab if you wish to use a genuine tab.
"""
def __init__(self, tab=' '):
self.tab = tab
rlcompleter.Completer.__init__(self)
def complete(self, text, state):
if text == '':
readline.insert_text(self.tab)
return None
else:
return rlcompleter.Completer.complete(self,text,state)
#you could change this line to bind another key instead tab.
readline.parse_and_bind('tab: complete')
readline.set_completer(IrlCompleter('\t').complete)
# Restore our command-line history, and save it when Python exits.
history_path = os.path.expanduser('~/.pyhistory')
if os.path.isfile(history_path):
readline.read_history_file(history_path)
atexit.register(lambda x=history_path: readline.write_history_file(x))
Note that I have slightly edited it from the version on my blog post so that the IrlCompleter is initialized with a true tab, which seems to be what is output by the Tab key in Terminal.

This should work under Leopard's python:
import rlcompleter
import readline
readline.parse_and_bind ("bind ^I rl_complete")
Whereas this one does not:
import readline, rlcompleter
readline.parse_and_bind("tab: complete")
Save it in ~/.pythonrc.py and execute in .bash_profile
export PYTHONSTARTUP=$HOME/.pythonrc.py

here is a full cross platform version of loading tab completion for Windows/OS X/Linux in one shot:
#Code UUID = '9301d536-860d-11de-81c8-0023dfaa9e40'
import sys
try:
import readline
except ImportError:
try:
import pyreadline as readline
# throw open a browser if we fail both readline and pyreadline
except ImportError:
import webbrowser
webbrowser.open("http://ipython.scipy.org/moin/PyReadline/Intro#line-36")
# throw open a browser
#pass
else:
import rlcompleter
if(sys.platform == 'darwin'):
readline.parse_and_bind ("bind ^I rl_complete")
else:
readline.parse_and_bind("tab: complete")
From http://www.farmckon.net/?p=181

To avoid having to use more GPL code, Apple doesn't include a real readline. Instead it uses the BSD-licensed libedit, which is only mostly-readline-compatible. Build your own Python (or use Fink or MacPorts) if you want completion.

This works for me on both Linux bash and OS X 10.4
import readline
import rlcompleter
readline.parse_and_bind('tab: complete')

If after trying the above, it still doesn't work, then try to execute in the shell:
sudo easy_install readline
Then, create ~/.profile file with the content:
export PYTHONSTARTUP=$HOME/.pythonrc.py
and a ~/.pythonrc.py file with the content:
try:
import readline
except:
print ("Module readline is not available.")
else:
import rlcompleter
readline.parse_and_bind("tab: complete")
Thanks to Steven Bamford for the easy_install tip, and Nicolas for the file content.

The documented way to tell libedit (the Mac OS semi-readline) from the real one is:
if "libedit" in readline.doc:
pass # Mac case
else:
pass # GNU readline case

After crashing into many issues dealing with Python (2 and 3) on FreeBSD, I finally got a proper extension to work using libedit directly as the completer for Python.
The basic issue with libedit/readline is that Python's completion and input was heavily bent towards GNU readline... Sadly, this is actually not a particularly good interface. It requires a giant number of globals in C and does not work well on an "instance" basis.
Solution:
https://github.com/mark-nicholson/python-editline
This is a true separate python extension which directly links to libedit using the actual "libedit" interface -- not the readline glue on the side.
I have tested it pretty thoroughly on Ubuntu, FreeBSD, OpenBSD, NetBSD and MacOS -- results are posted in the readme. The c-code is very clean and has virtually no platform dependent bits -- unlike the readline.c module in Python.
Notes:
It works on Python3 > 3.2.
It is NOT a drop-in replacement for 'import readline' in other scripts, but those scripts can be adjusted easily.
It can co-exist with readline.so -- there is code for a sitecustomize.py file which enables the selection.
It can use a distribution 'libedit.so', a custom built one or libedit built into the extension itself.

How to detect that Python code is being executed through the debugger?

Is there a simple way to detect, within Python code, that this code is being executed through the Python debugger?
I have a small Python application that uses Java code (thanks to JPype). When I'm debugging the Python part, I'd like the embedded JVM to be passed debug options too.

Python debuggers (as well as profilers and coverage tools) use the sys.settrace function (in the sys module) to register a callback that gets called when interesting events happen.
If you're using Python 2.6, you can call sys.gettrace() to get the current trace callback function. If it's not None then you can assume you should be passing debug parameters to the JVM.
It's not clear how you could do this pre 2.6.

Other alternative if you're using Pydev that also works in a multithreading is:
try:
import pydevd
DEBUGGING = True
except ImportError:
DEBUGGING = False

A solution working with Python 2.4 (it should work with any version superior to 2.1) and Pydev:
import inspect
def isdebugging():
for frame in inspect.stack():
if frame[1].endswith("pydevd.py"):
return True
return False
The same should work with pdb by simply replacing pydevd.py with pdb.py. As do3cc suggested, it tries to find the debugger within the stack of the caller.
Useful links:
The Python Debugger
The interpreter stack

Another way to do it hinges on how your python interpreter is started. It requires you start Python using -O for production and with no -O for debugging. So it does require an external discipline that might be hard to maintain .. but then again it might fit your processes perfectly.
From the python docs (see "Built-in Constants" here or here):
__debug__
This constant is true if Python was not started with an -O option.
Usage would be something like:
if __debug__:
print 'Python started without optimization'

If you're using Pydev, you can detect it in such way:
import sys
if 'pydevd' in sys.modules:
print "Debugger"
else:
print "commandline"

From taking a quick look at the pdb docs and source code, it doesn't look like there is a built in way to do this. I suggest that you set an environment variable that indicates debugging is in progress and have your application respond to that.
$ USING_PDB=1 pdb yourprog.py
Then in yourprog.py:
import os
if os.environ.get('USING_PDB'):
# debugging actions
pass

You can try to peek into your stacktrace.
https://docs.python.org/library/inspect.html#the-interpreter-stack
when you try this in a debugger session:
import inspect
inspect.getouterframes(inspect.currentframe()
you will get a list of framerecords and can peek for any frames that refer to the pdb file.

I found a cleaner way to do it,
Just add the following line in your manage.py
#!/usr/bin/env python
import os
import sys
if __debug__:
sys.path.append('/path/to/views.py')
if __name__ == "__main__":
....
Then it would automatically add it when you are debugging.

Since the original question doesn't specifically call out Python2 - This is to confirm #babbageclunk's suggested usage of sys also works in python3:
from sys import gettrace as sys_gettrace
DEBUG = sys_gettrace() is not None
print("debugger? %s" % DEBUG)

In my perllib, I use this check:
if 'pdb' in sys.modules:
# We are being debugged
It assumes the user doesn't otherwise import pdb

Cross-platform space remaining on volume using python

I need a way to determine the space remaining on a disk volume using python on linux, Windows and OS X. I'm currently parsing the output of the various system calls (df, dir) to accomplish this - is there a better way?

import ctypes
import os
import platform
import sys
def get_free_space_mb(dirname):
"""Return folder/drive free space (in megabytes)."""
if platform.system() == 'Windows':
free_bytes = ctypes.c_ulonglong(0)
ctypes.windll.kernel32.GetDiskFreeSpaceExW(ctypes.c_wchar_p(dirname), None, None, ctypes.pointer(free_bytes))
return free_bytes.value / 1024 / 1024
else:
st = os.statvfs(dirname)
return st.f_bavail * st.f_frsize / 1024 / 1024
Note that you must pass a directory name for GetDiskFreeSpaceEx() to work
(statvfs() works on both files and directories). You can get a directory name
from a file with os.path.dirname().
Also see the documentation for os.statvfs() and GetDiskFreeSpaceEx.

Install psutil using pip install psutil. Then you can get the amount of free space in bytes using:
import psutil
print(psutil.disk_usage(".").free)

You could use the wmi module for windows and os.statvfs for unix
for window
import wmi
c = wmi.WMI ()
for d in c.Win32_LogicalDisk():
print( d.Caption, d.FreeSpace, d.Size, d.DriveType)
for unix or linux
from os import statvfs
statvfs(path)

If you're running python3:
Using shutil.disk_usage()with os.path.realpath('/') name-regularization works:
from os import path
from shutil import disk_usage
print([i / 1000000 for i in disk_usage(path.realpath('/'))])
Or
total_bytes, used_bytes, free_bytes = disk_usage(path.realpath('D:\\Users\\phannypack'))
print(total_bytes / 1000000) # for Mb
print(used_bytes / 1000000)
print(free_bytes / 1000000)
giving you the total, used, & free space in MB.

If you dont like to add another dependency you can for windows use ctypes to call the win32 function call directly.
import ctypes
free_bytes = ctypes.c_ulonglong(0)
ctypes.windll.kernel32.GetDiskFreeSpaceExW(ctypes.c_wchar_p(u'c:\\'), None, None, ctypes.pointer(free_bytes))
if free_bytes.value == 0:
print 'dont panic'

From Python 3.3 you can use shutil.disk_usage("/").free from standard library for both Windows and UNIX :)

A good cross-platform way is using psutil: http://pythonhosted.org/psutil/#disks
(Note that you'll need psutil 0.3.0 or above).

You can use df as a cross-platform way. It is a part of GNU core utilities. These are the core utilities which are expected to exist on every operating system. However, they are not installed on Windows by default (Here, GetGnuWin32 comes in handy).
df is a command-line utility, therefore a wrapper required for scripting purposes.
For example:
from subprocess import PIPE, Popen
def free_volume(filename):
"""Find amount of disk space available to the current user (in bytes)
on the file system containing filename."""
stats = Popen(["df", "-Pk", filename], stdout=PIPE).communicate()[0]
return int(stats.splitlines()[1].split()[3]) * 1024

Below code returns correct value on windows
import win32file
def get_free_space(dirname):
secsPerClus, bytesPerSec, nFreeClus, totClus = win32file.GetDiskFreeSpace(dirname)
return secsPerClus * bytesPerSec * nFreeClus

The os.statvfs() function is a better way to get that information for Unix-like platforms (including OS X). The Python documentation says "Availability: Unix" but it's worth checking whether it works on Windows too in your build of Python (ie. the docs might not be up to date).
Otherwise, you can use the pywin32 library to directly call the GetDiskFreeSpaceEx function.

I Don't know of any cross-platform way to achieve this, but maybe a good workaround for you would be to write a wrapper class that checks the operating system and uses the best method for each.
For Windows, there's the GetDiskFreeSpaceEx method in the win32 extensions.

Most previous answers are correct, I'm using Python 3.10 and shutil.
My use case was Windows and C drive only ( but you should be able to extend this for you Linux and Mac as well (here is the documentation)
Here is the example for Windows:
import shutil
total, used, free = shutil.disk_usage("C:/")
print("Total: %d GiB" % (total // (2**30)))
print("Used: %d GiB" % (used // (2**30)))
print("Free: %d GiB" % (free // (2**30)))