I need to setup environment with the path to a binary. In the shell, I can use which to find the path. Is there an equivalent in python?
This is my code.
cmd = ["which","abc"]
p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
res = p.stdout.readlines()
if len(res) == 0: return False
return True
There is distutils.spawn.find_executable().
I know this is an older question, but if you happen to be using Python 3.3+ you can use shutil.which(cmd). You can find the documentation here. It has the advantage of being in the standard library.
An example would be like so:
>>> import shutil
>>> shutil.which("bash")
'/usr/bin/bash'
There's not a command to do that, but you can iterate over environ["PATH"] and look if the file exists, which is actually what which does.
import os
def which(file):
for path in os.environ["PATH"].split(os.pathsep):
if os.path.exists(os.path.join(path, file)):
return os.path.join(path, file)
return None
Good luck!
You could try something like the following:
import os
import os.path
def which(filename):
"""docstring for which"""
locations = os.environ.get("PATH").split(os.pathsep)
candidates = []
for location in locations:
candidate = os.path.join(location, filename)
if os.path.isfile(candidate):
candidates.append(candidate)
return candidates
If you use shell=True, then your command will be run through the system shell, which will automatically find the binary on the path:
p = subprocess.Popen("abc", stdout=subprocess.PIPE, shell=True)
This is the equivalent of the which command, which not only checks if the file exists, but also whether it is executable:
import os
def which(file_name):
for path in os.environ["PATH"].split(os.pathsep):
full_path = os.path.join(path, file_name)
if os.path.exists(full_path) and os.access(full_path, os.X_OK):
return full_path
return None
Here's a one-line version of earlier answers:
import os
which = lambda y: next(filter(lambda x: os.path.isfile(x) and os.access(x,os.X_OK),[x+os.path.sep+y for x in os.getenv("PATH").split(os.pathsep)]),None)
used like so:
>>> which("ls")
'/bin/ls'
Related
I am playing around with creating modules.I have two python scripts.The first (the module) has:
def abspath(relpath):
import os
absdir = os.path.realpath('__file__')
absdir = absdir.split('_')[0].replace('\\', '/')
filename = str(absdir + relpath )
print (filename)
return filename;
The second file (main) has:
import file_tools as ft
filename = ft.abspath('some/path/')
When I run Main, filename appears empty (Type:None). I have run the filename = abspath(etc) within the 'module', and it works. Clearly, I am missing something here!!
and doing this, so any help would be useful.
Thank's all.
MT
The problem lies in how you're finding the working directory; the preferred method being os.getcwd() (or os.getcwdb for Unix compatibility). Using that, we can see that your function boils down to:
def abspath(relpath):
return os.path.join(os.getcwd(), relpath)
I need to setup environment with the path to a binary. In the shell, I can use which to find the path. Is there an equivalent in python?
This is my code.
cmd = ["which","abc"]
p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
res = p.stdout.readlines()
if len(res) == 0: return False
return True
There is distutils.spawn.find_executable().
I know this is an older question, but if you happen to be using Python 3.3+ you can use shutil.which(cmd). You can find the documentation here. It has the advantage of being in the standard library.
An example would be like so:
>>> import shutil
>>> shutil.which("bash")
'/usr/bin/bash'
There's not a command to do that, but you can iterate over environ["PATH"] and look if the file exists, which is actually what which does.
import os
def which(file):
for path in os.environ["PATH"].split(os.pathsep):
if os.path.exists(os.path.join(path, file)):
return os.path.join(path, file)
return None
Good luck!
You could try something like the following:
import os
import os.path
def which(filename):
"""docstring for which"""
locations = os.environ.get("PATH").split(os.pathsep)
candidates = []
for location in locations:
candidate = os.path.join(location, filename)
if os.path.isfile(candidate):
candidates.append(candidate)
return candidates
If you use shell=True, then your command will be run through the system shell, which will automatically find the binary on the path:
p = subprocess.Popen("abc", stdout=subprocess.PIPE, shell=True)
This is the equivalent of the which command, which not only checks if the file exists, but also whether it is executable:
import os
def which(file_name):
for path in os.environ["PATH"].split(os.pathsep):
full_path = os.path.join(path, file_name)
if os.path.exists(full_path) and os.access(full_path, os.X_OK):
return full_path
return None
Here's a one-line version of earlier answers:
import os
which = lambda y: next(filter(lambda x: os.path.isfile(x) and os.access(x,os.X_OK),[x+os.path.sep+y for x in os.getenv("PATH").split(os.pathsep)]),None)
used like so:
>>> which("ls")
'/bin/ls'
I have a file that may be in a different place on each user's machine. Is there a way to implement a search for the file? A way that I can pass the file's name and the directory tree to search in?
os.walk is the answer, this will find the first match:
import os
def find(name, path):
for root, dirs, files in os.walk(path):
if name in files:
return os.path.join(root, name)
And this will find all matches:
def find_all(name, path):
result = []
for root, dirs, files in os.walk(path):
if name in files:
result.append(os.path.join(root, name))
return result
And this will match a pattern:
import os, fnmatch
def find(pattern, path):
result = []
for root, dirs, files in os.walk(path):
for name in files:
if fnmatch.fnmatch(name, pattern):
result.append(os.path.join(root, name))
return result
find('*.txt', '/path/to/dir')
In Python 3.4 or newer you can use pathlib to do recursive globbing:
>>> import pathlib
>>> sorted(pathlib.Path('.').glob('**/*.py'))
[PosixPath('build/lib/pathlib.py'),
PosixPath('docs/conf.py'),
PosixPath('pathlib.py'),
PosixPath('setup.py'),
PosixPath('test_pathlib.py')]
Reference: https://docs.python.org/3/library/pathlib.html#pathlib.Path.glob
In Python 3.5 or newer you can also do recursive globbing like this:
>>> import glob
>>> glob.glob('**/*.txt', recursive=True)
['2.txt', 'sub/3.txt']
Reference: https://docs.python.org/3/library/glob.html#glob.glob
I used a version of os.walk and on a larger directory got times around 3.5 sec. I tried two random solutions with no great improvement, then just did:
paths = [line[2:] for line in subprocess.check_output("find . -iname '*.txt'", shell=True).splitlines()]
While it's POSIX-only, I got 0.25 sec.
From this, I believe it's entirely possible to optimise whole searching a lot in a platform-independent way, but this is where I stopped the research.
If you are using Python on Ubuntu and you only want it to work on Ubuntu a substantially faster way is the use the terminal's locate program like this.
import subprocess
def find_files(file_name):
command = ['locate', file_name]
output = subprocess.Popen(command, stdout=subprocess.PIPE).communicate()[0]
output = output.decode()
search_results = output.split('\n')
return search_results
search_results is a list of the absolute file paths. This is 10,000's of times faster than the methods above and for one search I've done it was ~72,000 times faster.
If you are working with Python 2 you have a problem with infinite recursion on windows caused by self-referring symlinks.
This script will avoid following those. Note that this is windows-specific!
import os
from scandir import scandir
import ctypes
def is_sym_link(path):
# http://stackoverflow.com/a/35915819
FILE_ATTRIBUTE_REPARSE_POINT = 0x0400
return os.path.isdir(path) and (ctypes.windll.kernel32.GetFileAttributesW(unicode(path)) & FILE_ATTRIBUTE_REPARSE_POINT)
def find(base, filenames):
hits = []
def find_in_dir_subdir(direc):
content = scandir(direc)
for entry in content:
if entry.name in filenames:
hits.append(os.path.join(direc, entry.name))
elif entry.is_dir() and not is_sym_link(os.path.join(direc, entry.name)):
try:
find_in_dir_subdir(os.path.join(direc, entry.name))
except UnicodeDecodeError:
print "Could not resolve " + os.path.join(direc, entry.name)
continue
if not os.path.exists(base):
return
else:
find_in_dir_subdir(base)
return hits
It returns a list with all paths that point to files in the filenames list.
Usage:
find("C:\\", ["file1.abc", "file2.abc", "file3.abc", "file4.abc", "file5.abc"])
Below we use a boolean "first" argument to switch between first match and all matches (a default which is equivalent to "find . -name file"):
import os
def find(root, file, first=False):
for d, subD, f in os.walk(root):
if file in f:
print("{0} : {1}".format(file, d))
if first == True:
break
The answer is very similar to existing ones, but slightly optimized.
So you can find any files or folders by pattern:
def iter_all(pattern, path):
return (
os.path.join(root, entry)
for root, dirs, files in os.walk(path)
for entry in dirs + files
if pattern.match(entry)
)
either by substring:
def iter_all(substring, path):
return (
os.path.join(root, entry)
for root, dirs, files in os.walk(path)
for entry in dirs + files
if substring in entry
)
or using a predicate:
def iter_all(predicate, path):
return (
os.path.join(root, entry)
for root, dirs, files in os.walk(path)
for entry in dirs + files
if predicate(entry)
)
to search only files or only folders - replace “dirs + files”, for example, with only “dirs” or only “files”, depending on what you need.
Regards.
SARose's answer worked for me until I updated from Ubuntu 20.04 LTS. The slight change I made to his code makes it work on the latest Ubuntu release.
import subprocess
def find_files(file_name):
command = ['locate'+ ' ' + file_name]
output = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True).communicate()[0]
output = output.decode()
search_results = output.split('\n')
return search_results
#F.M.F's answers has a few problems in this version, so I made a few adjustments to make it work.
import os
from os import scandir
import ctypes
def is_sym_link(path):
# http://stackoverflow.com/a/35915819
FILE_ATTRIBUTE_REPARSE_POINT = 0x0400
return os.path.isdir(path) and (ctypes.windll.kernel32.GetFileAttributesW(str(path)) & FILE_ATTRIBUTE_REPARSE_POINT)
def find(base, filenames):
hits = []
def find_in_dir_subdir(direc):
content = scandir(direc)
for entry in content:
if entry.name in filenames:
hits.append(os.path.join(direc, entry.name))
elif entry.is_dir() and not is_sym_link(os.path.join(direc, entry.name)):
try:
find_in_dir_subdir(os.path.join(direc, entry.name))
except UnicodeDecodeError:
print("Could not resolve " + os.path.join(direc, entry.name))
continue
except PermissionError:
print("Skipped " + os.path.join(direc, entry.name) + ". I lacked permission to navigate")
continue
if not os.path.exists(base):
return
else:
find_in_dir_subdir(base)
return hits
unicode() was changed to str() in Python 3, so I made that adjustment (line 8)
I also added (in line 25) and exception to PermissionError. This way, the program won't stop if it finds a folder it can't access.
Finally, I would like to give a little warning. When running the program, even if you are looking for a single file/directory, make sure you pass it as a list. Otherwise, you will get a lot of answers that not necessarily match your search.
example of use:
find("C:\", ["Python", "Homework"])
or
find("C:\\", ["Homework"])
but, for example: find("C:\\", "Homework") will give un-wanted answers.
I would be lying if I said I know why this happens. Again, this is not my code and I just made the adjustments I needed to make it work. All credit should go to #F.M.F.
i want a python program that given a directory, it will return all directories within that directory that have 775 (rwxrwxr-x) permissions
thanks!
Neither answer recurses, though it's not entirely clear that that's what the OP wants. Here's a recursive approach (untested, but you get the idea):
import os
import stat
import sys
MODE = "775"
def mode_matches(mode, file):
"""Return True if 'file' matches 'mode'.
'mode' should be an integer representing an octal mode (eg
int("755", 8) -> 493).
"""
# Extract the permissions bits from the file's (or
# directory's) stat info.
filemode = stat.S_IMODE(os.stat(file).st_mode)
return filemode == mode
try:
top = sys.argv[1]
except IndexError:
top = '.'
try:
mode = int(sys.argv[2], 8)
except IndexError:
mode = MODE
# Convert mode to octal.
mode = int(mode, 8)
for dirpath, dirnames, filenames in os.walk(top):
dirs = [os.path.join(dirpath, x) for x in dirnames]
for dirname in dirs:
if mode_matches(mode, dirname):
print dirname
Something similar is described in the stdlib documentation for
stat.
Take a look at the os module. In particular os.stat to look at the permission bits.
import os
for filename in os.listdir(dirname):
path=os.path.join(dirname, filename)
if os.path.isdir(path):
if (os.stat(path).st_mode & 0777) == 0775:
print path
Compact generator based on Brian's answer:
import os
(fpath for fpath
in (os.path.join(dirname,fname) for fname in os.listdir(dirname))
if (os.path.isdir(fpath) and (os.stat(fpath).st_mode & 0777) == 0775))
Does it have to be python?
You can also use find to do that :
"find . -perm 775"
You can check 775 permission for files and directories using below command
path="location of file or directory"
if (oct(os.stat(path).st_mode)[-3:]=="775"):
print(path)
In Python, is there a portable and simple way to test if an executable program exists?
By simple I mean something like the which command which would be just perfect. I don't want to search PATH manually or something involving trying to execute it with Popen & al and see if it fails (that's what I'm doing now, but imagine it's launchmissiles)
I know this is an ancient question, but you can use distutils.spawn.find_executable. This has been documented since python 2.4 and has existed since python 1.6.
import distutils.spawn
distutils.spawn.find_executable("notepad.exe")
Also, Python 3.3 now offers shutil.which().
Easiest way I can think of:
def which(program):
import os
def is_exe(fpath):
return os.path.isfile(fpath) and os.access(fpath, os.X_OK)
fpath, fname = os.path.split(program)
if fpath:
if is_exe(program):
return program
else:
for path in os.environ["PATH"].split(os.pathsep):
exe_file = os.path.join(path, program)
if is_exe(exe_file):
return exe_file
return None
Edit: Updated code sample to include logic for handling case where provided argument is already a full path to the executable, i.e. "which /bin/ls". This mimics the behavior of the UNIX 'which' command.
Edit: Updated to use os.path.isfile() instead of os.path.exists() per comments.
Edit: path.strip('"') seems like the wrong thing to do here. Neither Windows nor POSIX appear to encourage quoted PATH items.
Use shutil.which() from Python's wonderful standard library.
Batteries included!
For python 3.3 and later:
import shutil
command = 'ls'
shutil.which(command) is not None
As a one-liner of Jan-Philip Gehrcke Answer:
cmd_exists = lambda x: shutil.which(x) is not None
As a def:
def cmd_exists(cmd):
return shutil.which(cmd) is not None
For python 3.2 and earlier:
my_command = 'ls'
any(
(
os.access(os.path.join(path, my_command), os.X_OK)
and os.path.isfile(os.path.join(path, my_command)
)
for path in os.environ["PATH"].split(os.pathsep)
)
This is a one-liner of Jay's Answer, Also here as a lambda func:
cmd_exists = lambda x: any((os.access(os.path.join(path, x), os.X_OK) and os.path.isfile(os.path.join(path, x))) for path in os.environ["PATH"].split(os.pathsep))
cmd_exists('ls')
Or lastly, indented as a function:
def cmd_exists(cmd, path=None):
""" test if path contains an executable file with name
"""
if path is None:
path = os.environ["PATH"].split(os.pathsep)
for prefix in path:
filename = os.path.join(prefix, cmd)
executable = os.access(filename, os.X_OK)
is_not_directory = os.path.isfile(filename)
if executable and is_not_directory:
return True
return False
Just remember to specify the file extension on windows. Otherwise, you have to write a much complicated is_exe for windows using PATHEXT environment variable. You may just want to use FindPath.
OTOH, why are you even bothering to search for the executable? The operating system will do it for you as part of popen call & will raise an exception if the executable is not found. All you need to do is catch the correct exception for given OS. Note that on Windows, subprocess.Popen(exe, shell=True) will fail silently if exe is not found.
Incorporating PATHEXT into the above implementation of which (in Jay's answer):
def which(program):
def is_exe(fpath):
return os.path.exists(fpath) and os.access(fpath, os.X_OK) and os.path.isfile(fpath)
def ext_candidates(fpath):
yield fpath
for ext in os.environ.get("PATHEXT", "").split(os.pathsep):
yield fpath + ext
fpath, fname = os.path.split(program)
if fpath:
if is_exe(program):
return program
else:
for path in os.environ["PATH"].split(os.pathsep):
exe_file = os.path.join(path, program)
for candidate in ext_candidates(exe_file):
if is_exe(candidate):
return candidate
return None
For *nix platforms (Linux and OS X)
This seems to be working for me:
Edited to work on Linux, thanks to Mestreion
def cmd_exists(cmd):
return subprocess.call("type " + cmd, shell=True,
stdout=subprocess.PIPE, stderr=subprocess.PIPE) == 0
What we're doing here is using the builtin command type and checking the exit code. If there's no such command, type will exit with 1 (or a non-zero status code anyway).
The bit about stdout and stderr is just to silence the output of the type command, since we're only interested in the exit status code.
Example usage:
>>> cmd_exists("jsmin")
True
>>> cmd_exists("cssmin")
False
>>> cmd_exists("ls")
True
>>> cmd_exists("dir")
False
>>> cmd_exists("node")
True
>>> cmd_exists("steam")
False
On the basis that it is easier to ask forgiveness than permission (and, importantly, that the command is safe to run) I would just try to use it and catch the error (OSError in this case - I checked for file does not exist and file is not executable and they both give OSError).
It helps if the executable has something like a --version or --help flag that is a quick no-op.
import subprocess
myexec = "python2.8"
try:
subprocess.call([myexec, '--version']
except OSError:
print "%s not found on path" % myexec
This is not a general solution, but will be the easiest way for a lot of use cases - those where the code needs to look for a single well known executable which is safe to run, or at least safe to run with a given flag.
See os.path module for some useful functions on pathnames. To check if an existing file is executable, use os.access(path, mode), with the os.X_OK mode.
os.X_OK
Value to include in the mode parameter of access() to determine if path can be executed.
EDIT: The suggested which() implementations are missing one clue - using os.path.join() to build full file names.
I know that I'm being a bit of a necromancer here, but I stumbled across this question and the accepted solution didn't work for me for all cases Thought it might be useful to submit anyway. In particular, the "executable" mode detection, and the requirement of supplying the file extension. Furthermore, both python3.3's shutil.which (uses PATHEXT) and python2.4+'s distutils.spawn.find_executable (just tries adding '.exe') only work in a subset of cases.
So I wrote a "super" version (based on the accepted answer, and the PATHEXT suggestion from Suraj). This version of which does the task a bit more thoroughly, and tries a series of "broadphase" breadth-first techniques first, and eventually tries more fine-grained searches over the PATH space:
import os
import sys
import stat
import tempfile
def is_case_sensitive_filesystem():
tmphandle, tmppath = tempfile.mkstemp()
is_insensitive = os.path.exists(tmppath.upper())
os.close(tmphandle)
os.remove(tmppath)
return not is_insensitive
_IS_CASE_SENSITIVE_FILESYSTEM = is_case_sensitive_filesystem()
def which(program, case_sensitive=_IS_CASE_SENSITIVE_FILESYSTEM):
""" Simulates unix `which` command. Returns absolute path if program found """
def is_exe(fpath):
""" Return true if fpath is a file we have access to that is executable """
accessmode = os.F_OK | os.X_OK
if os.path.exists(fpath) and os.access(fpath, accessmode) and not os.path.isdir(fpath):
filemode = os.stat(fpath).st_mode
ret = bool(filemode & stat.S_IXUSR or filemode & stat.S_IXGRP or filemode & stat.S_IXOTH)
return ret
def list_file_exts(directory, search_filename=None, ignore_case=True):
""" Return list of (filename, extension) tuples which match the search_filename"""
if ignore_case:
search_filename = search_filename.lower()
for root, dirs, files in os.walk(path):
for f in files:
filename, extension = os.path.splitext(f)
if ignore_case:
filename = filename.lower()
if not search_filename or filename == search_filename:
yield (filename, extension)
break
fpath, fname = os.path.split(program)
# is a path: try direct program path
if fpath:
if is_exe(program):
return program
elif "win" in sys.platform:
# isnt a path: try fname in current directory on windows
if is_exe(fname):
return program
paths = [path.strip('"') for path in os.environ.get("PATH", "").split(os.pathsep)]
exe_exts = [ext for ext in os.environ.get("PATHEXT", "").split(os.pathsep)]
if not case_sensitive:
exe_exts = map(str.lower, exe_exts)
# try append program path per directory
for path in paths:
exe_file = os.path.join(path, program)
if is_exe(exe_file):
return exe_file
# try with known executable extensions per program path per directory
for path in paths:
filepath = os.path.join(path, program)
for extension in exe_exts:
exe_file = filepath+extension
if is_exe(exe_file):
return exe_file
# try search program name with "soft" extension search
if len(os.path.splitext(fname)[1]) == 0:
for path in paths:
file_exts = list_file_exts(path, fname, not case_sensitive)
for file_ext in file_exts:
filename = "".join(file_ext)
exe_file = os.path.join(path, filename)
if is_exe(exe_file):
return exe_file
return None
Usage looks like this:
>>> which.which("meld")
'C:\\Program Files (x86)\\Meld\\meld\\meld.exe'
The accepted solution did not work for me in this case, since there were files like meld.1, meld.ico, meld.doap, etc also in the directory, one of which were returned instead (presumably since lexicographically first) because the executable test in the accepted answer was incomplete and giving false positives.
This seems simple enough and works both in python 2 and 3
try: subprocess.check_output('which executable',shell=True)
except: sys.exit('ERROR: executable not found')
Added windows support
def which(program):
path_ext = [""];
ext_list = None
if sys.platform == "win32":
ext_list = [ext.lower() for ext in os.environ["PATHEXT"].split(";")]
def is_exe(fpath):
exe = os.path.isfile(fpath) and os.access(fpath, os.X_OK)
# search for executable under windows
if not exe:
if ext_list:
for ext in ext_list:
exe_path = "%s%s" % (fpath,ext)
if os.path.isfile(exe_path) and os.access(exe_path, os.X_OK):
path_ext[0] = ext
return True
return False
return exe
fpath, fname = os.path.split(program)
if fpath:
if is_exe(program):
return "%s%s" % (program, path_ext[0])
else:
for path in os.environ["PATH"].split(os.pathsep):
path = path.strip('"')
exe_file = os.path.join(path, program)
if is_exe(exe_file):
return "%s%s" % (exe_file, path_ext[0])
return None
An important question is "Why do you need to test if executable exist?" Maybe you don't? ;-)
Recently I needed this functionality to launch viewer for PNG file. I wanted to iterate over some predefined viewers and run the first that exists. Fortunately, I came across os.startfile. It's much better! Simple, portable and uses the default viewer on the system:
>>> os.startfile('yourfile.png')
Update: I was wrong about os.startfile being portable... It's Windows only. On Mac you have to run open command. And xdg_open on Unix. There's a Python issue on adding Mac and Unix support for os.startfile.
You can try the external lib called "sh" (http://amoffat.github.io/sh/).
import sh
print sh.which('ls') # prints '/bin/ls' depending on your setup
print sh.which('xxx') # prints None
So basically you want to find a file in mounted filesystem (not necessarily in PATH directories only) and check if it is executable. This translates to following plan:
enumerate all files in locally mounted filesystems
match results with name pattern
for each file found check if it is executable
I'd say, doing this in a portable way will require lots of computing power and time. Is it really what you need?
There is a which.py script in a standard Python distribution (e.g. on Windows '\PythonXX\Tools\Scripts\which.py').
EDIT: which.py depends on ls therefore it is not cross-platform.