I'm writing a script on python 2.7.12 on linux with a bunch of functions that run external programs...
at some point, I have a function that runs a program, which generates an external file. The next step of the function is to read the external file and do some processing.
The function is
def RunProgram(input, options)
input_file=str(input)
options=str(options)
cmd=str('program + ' input_file + '--flag ' + options + ' --out temp.log &> /dev/null')
#print(cmd)
#print(cmd)
os.system(cmd)
with open('path_to_file/temp.log') as fp:
for i, line in enumerate(fp):
if i == 2:
#print(line)
solution=str(line) #stores 3rd line of log file
elif i > 2:
break
return solution
Somehow, although the external program runs and I see temp.log created from a bash shell, the function exits with the error
IOError: [Errno 2] No such file or directory: 'path_to_file/temp.log'
if in the function after os.system(cmd) I place
print(os.path.exists('path_to_file/temp.log'))
print(os.path.isfile('path_to_file/temp.log'))
I get 2 false, but if I run those commands after running the function and getting the error i get True for both and I see the file in the directory using ls.
Once I run the function again with temp.log already existing, the function works.
I have checked with os.getcwd() and the session is running in the correct dirrectory. I also checked I have read permissions on temp.log
..any ideas?
This is probably a race condition: When your code wants to read the output from the file, the program isn't finished yet and the output file doesn't exist yet. Try a loop with "sleep" waiting for the file you want to read to become available. Another possibility is that that the file permissions don't match.
I need to be able to open a document using its default application in Windows and Mac OS. Basically, I want to do the same thing that happens when you double-click on the document icon in Explorer or Finder. What is the best way to do this in Python?
Use the subprocess module available on Python 2.4+, not os.system(), so you don't have to deal with shell escaping.
import subprocess, os, platform
if platform.system() == 'Darwin': # macOS
subprocess.call(('open', filepath))
elif platform.system() == 'Windows': # Windows
os.startfile(filepath)
else: # linux variants
subprocess.call(('xdg-open', filepath))
The double parentheses are because subprocess.call() wants a sequence as its first argument, so we're using a tuple here. On Linux systems with Gnome there is also a gnome-open command that does the same thing, but xdg-open is the Free Desktop Foundation standard and works across Linux desktop environments.
open and start are command-interpreter things for Mac OS/X and Windows respectively, to do this.
To call them from Python, you can either use subprocess module or os.system().
Here are considerations on which package to use:
You can call them via os.system, which works, but...
Escaping: os.system only works with filenames that don't have any spaces or other shell metacharacters in the pathname (e.g. A:\abc\def\a.txt), or else these need to be escaped. There is shlex.quote for Unix-like systems, but nothing really standard for Windows. Maybe see also python, windows : parsing command lines with shlex
MacOS/X: os.system("open " + shlex.quote(filename))
Windows: os.system("start " + filename) where properly speaking filename should be escaped, too.
You can also call them via subprocess module, but...
For Python 2.7 and newer, simply use
subprocess.check_call(['open', filename])
In Python 3.5+ you can equivalently use the slightly more complex but also somewhat more versatile
subprocess.run(['open', filename], check=True)
If you need to be compatible all the way back to Python 2.4, you can use subprocess.call() and implement your own error checking:
try:
retcode = subprocess.call("open " + filename, shell=True)
if retcode < 0:
print >>sys.stderr, "Child was terminated by signal", -retcode
else:
print >>sys.stderr, "Child returned", retcode
except OSError, e:
print >>sys.stderr, "Execution failed:", e
Now, what are the advantages of using subprocess?
Security: In theory, this is more secure, but in fact we're needing to execute a command line one way or the other; in either environment, we need the environment and services to interpret, get paths, and so forth. In neither case are we executing arbitrary text, so it doesn't have an inherent "but you can type 'filename ; rm -rf /'" problem, and if the file name can be corrupted, using subprocess.call gives us little additional protection.
Error handling: It doesn't actually give us any more error detection, we're still depending on the retcode in either case; but the behavior to explicitly raise an exception in the case of an error will certainly help you notice if there is a failure (though in some scenarios, a traceback might not at all be more helpful than simply ignoring the error).
Spawns a (non-blocking) subprocess: We don't need to wait for the child process, since we're by problem statement starting a separate process.
To the objection "But subprocess is preferred." However, os.system() is not deprecated, and it's in some sense the simplest tool for this particular job. Conclusion: using os.system() is therefore also a correct answer.
A marked disadvantage is that the Windows start command requires you to pass in shell=True which negates most of the benefits of using subprocess.
I prefer:
os.startfile(path, 'open')
Note that this module supports filenames that have spaces in their folders and files e.g.
A:\abc\folder with spaces\file with-spaces.txt
(python docs) 'open' does not have to be added (it is the default). The docs specifically mention that this is like double-clicking on a file's icon in Windows Explorer.
This solution is windows only.
Just for completeness (it wasn't in the question), xdg-open will do the same on Linux.
import os
import subprocess
def click_on_file(filename):
'''Open document with default application in Python.'''
try:
os.startfile(filename)
except AttributeError:
subprocess.call(['open', filename])
If you have to use an heuristic method, you may consider webbrowser.
It's standard library and despite of its name it would also try to open files:
Note that on some platforms, trying to open a filename using this
function, may work and start the operating system’s associated
program. However, this is neither supported nor portable.
(Reference)
I tried this code and it worked fine in Windows 7 and Ubuntu Natty:
import webbrowser
webbrowser.open("path_to_file")
This code also works fine in Windows XP Professional, using Internet Explorer 8.
If you want to go the subprocess.call() way, it should look like this on Windows:
import subprocess
subprocess.call(('cmd', '/C', 'start', '', FILE_NAME))
You can't just use:
subprocess.call(('start', FILE_NAME))
because start is not an executable but a command of the cmd.exe program. This works:
subprocess.call(('cmd', '/C', 'start', FILE_NAME))
but only if there are no spaces in the FILE_NAME.
While subprocess.call method enquotes the parameters properly, the start command has a rather strange syntax, where:
start notes.txt
does something else than:
start "notes.txt"
The first quoted string should set the title of the window. To make it work with spaces, we have to do:
start "" "my notes.txt"
which is what the code on top does.
Start does not support long path names and white spaces. You have to convert it to 8.3 compatible paths.
import subprocess
import win32api
filename = "C:\\Documents and Settings\\user\\Desktop\file.avi"
filename_short = win32api.GetShortPathName(filename)
subprocess.Popen('start ' + filename_short, shell=True )
The file has to exist in order to work with the API call.
os.startfile(path, 'open') under Windows is good because when spaces exist in the directory, os.system('start', path_name) can't open the app correctly and when the i18n exist in the directory, os.system needs to change the unicode to the codec of the console in Windows.
I am pretty late to the lot, but here is a solution using the windows api. This always opens the associated application.
import ctypes
shell32 = ctypes.windll.shell32
file = 'somedocument.doc'
shell32.ShellExecuteA(0,"open",file,0,0,5)
A lot of magic constants. The first zero is the hwnd of the current program. Can be zero. The other two zeros are optional parameters (parameters and directory). 5 == SW_SHOW, it specifies how to execute the app.
Read the
ShellExecute API docs for more info.
Here is the answer from Nick, adjusted slightly for WSL:
import os
import sys
import logging
import subprocess
def get_platform():
if sys.platform == 'linux':
try:
proc_version = open('/proc/version').read()
if 'Microsoft' in proc_version:
return 'wsl'
except:
pass
return sys.platform
def open_with_default_app(filename):
platform = get_platform()
if platform == 'darwin':
subprocess.call(('open', filename))
elif platform in ['win64', 'win32']:
os.startfile(filename.replace('/','\\'))
elif platform == 'wsl':
subprocess.call('cmd.exe /C start'.split() + [filename])
else: # linux variants
subprocess.call(('xdg-open', filename))
If you want to specify the app to open the file with on Mac OS X, use this:
os.system("open -a [app name] [file name]")
On windows 8.1, below have worked while other given ways with subprocess.call fails with path has spaces in it.
subprocess.call('cmd /c start "" "any file path with spaces"')
By utilizing this and other's answers before, here's an inline code which works on multiple platforms.
import sys, os, subprocess
subprocess.call(('cmd /c start "" "'+ filepath +'"') if os.name is 'nt' else ('open' if sys.platform.startswith('darwin') else 'xdg-open', filepath))
On mac os you can call open:
import os
os.open("open myfile.txt")
This would open the file with TextEdit, or whatever app is set as default for this filetype.
I think you might want to open file in editor.
For Windows
subprocess.Popen(["notepad", filename])
For Linux
subprocess.Popen(["text-editor", filename])
I built a small library combining the best answers here for cross-platform support:
$ pip install universal-startfile
then launch a file or URL:
from startfile import startfile
startfile("~/Downloads/example.png")
startfile("http://example.com")
I was getting an error when calling my open file() function. I was following along with a guide but the guide was written in windows while I'm on Linux. So the os.statrfile method wasn't working for me. I was able to alleviate this problem by doing the following:
Import libraries
import sys, os, subprocess
import tkinter
import tkinter.filedioalog as fd
import tkinter.messagebox as mb
After the lib imports I then called the subprocess method for opening a file in unix based OS which is "xdg-open" and the file that will be opened.
def open_file():
file = fd.askopenfilename(title='Choose a file of any type', filetypes=[('All files', "*.*")])
subprocess.call(['xdg-open', file])
I would like to execute a specific version of mysqld through Python for unit testing. The idea is to execute the server on a thread, test, and kill the server when it's done. (Similar to testing.mysqld, which sadly doesn't work on Windows.). This is the current code:
#Create a temporary folder.
base_path = tempfile.mkdtemp()
#Extract the default files
zipfile.ZipFile(r"O:\Tools\mysql\min_mysql.zip").extractall(base_path)
#Setup my_ini file
my_ini_path = os.path.join(base_path, "my.ini").replace("\\", "/")
unix_base_path = posixpath.normpath(base_path).replace("\\", "/")
with open(my_ini_path, 'r') as my_ini:
filedata = my_ini.read()
filedata = filedata.replace("{{basedir}}", unix_base_path)
with open(my_ini_path, 'w', 0) as my_ini:
my_ini.write(filedata)
#Open mysqld
args = r"O:/Tools/mysql/bin/mysqld.exe --defaults-file=\"%s\"" % (my_ini_path)
args = shlex.split(args)
mysqld_process = subprocess.Popen(args, shell=True)
mysqld_process.wait()
But if I execute it through Python, I get this error:
Could not open required defaults file:
"c:\users\pelfeli1\appdata\local\temp\tmp2vct38\my.ini"
Fatal error in defaults handling. Program aborted
So far I have verified that the file exists before starting the process. If I print the command verbatim and execute it, the server runs fine.
There seems to be a difference between Popen and just executing in shell. What am I missing?
I'll copy my comment here, if you want to accept it as an answer:
I don't think this is the problem, but the args string shouldn't be
defined as raw (with the r). Instead, do this:
'O:/Tools/mysql/bin/mysqld.exe --defaults-file="%s"' (ie. use single
quotes). Unless you intend to pass the backslashes to the command line
Now, take into account that the following two strings
"foo\"bar\""
r"foo\"bar\""
Are not the same. The first one renders foo"bar", while the second gives you foo\"bar\".
So, what was happening is that the shell sees this as the file name: "c:\users\pelfeli1\appdata\local\temp\tmp2vct38\my.ini", including the quotes, because of the backquotes (\). You could have just written this:
args = 'O:/Tools/mysql/bin/mysqld.exe --defaults-file="%s"' % (my_ini_path)
just in case of spaces in my_ini_path, without problems.
Well, the problem was in the quotes. Just changed this line:
args = r"O:/Tools/mysql/bin/mysqld.exe --defaults-file=\"%s\"" % (my_ini_path)
to this line
args = "O:/Tools/mysql/bin/mysqld.exe --defaults-file=%s" % (my_ini_path)
I still have no idea why this changes anything, because printing args gives a valid (and working) command in both cases.
I want to open .wav file in default program. But it doesn´t work. This is my code:
audiofile=(myFile[index]+".wav") # I have all files in array (without ".wav")
try:
try:
os.system('xdg-open audiofile')
except:
os.system('start audiofile')
except:
print "error"
I don´t get any error, but it doesn´t work. How can I solve it? Thank you.
You aren't substituting the name of the audio file into your OS commands, so it can't possibly work.
You'd need something like:
os.system('xdg-open ' + audiofile)
This assumes that you have a default application associated with .wav files, which of course you can test by trying your command manually.
You might also want to check the return value of os.system for an error code, rather than relying on exceptions.
First of all, you should fill the variable audiofile into the command, not the string 'audiofile' itself
os.system('xdg-open %s' % audiofile)
Second,
os.system will NOT throw an exception when xdg-open or start doesn't exist in system.
Determine the type of system first by platform.system
>>> import platform
>>> platform.system()
'Linux'
I've figured out how to use call() to get my python script to run a command:
import subprocess
mycommandline = ['lumberjack', '-sleep all night', '-work all day']
subprocess.call(mycommandline)
This works but there's a problem, what if users don't have lumberjack in their command path? It would work if lumberjack was put in the same directory as the python script, but how does the script know it should look for lumberjack? I figured if there was a command-not-found error then lumberjack wouldn't be in the command path, the script could try to figure out what its directory is and look for lumberjack there and finally warn the user to copy lumberjack into one of those two places if it wasn't found in either one. How do I find out what the error message is? I read that check_call() can return an error message and something about a returncode attribute. I couldn't find examples on how to use check_call() and returncode, what the message would be or how I could tell if the message is command-not-found.
Am I even going about this the right way?
A simple snippet:
try:
subprocess.check_call(['executable'])
except subprocess.CalledProcessError:
pass # handle errors in the called executable
except OSError:
pass # executable not found
subprocess will raise an exception, OSError, when a command is not found.
When the command is found, and subprocess runs the command for you, the result code is returned from the command. The standard is that code 0 means success, and any failure is some non-zero error code (which varies; check the documentation for the specific command you are running).
So, if you catch OSError you can handle the non-existent command, and if you check the result code you can find out whether the command succeeded or not.
The great thing about subprocess is that you can make it collect all the text from stdout and stderr, and you can then discard it or return it or log it or display it as you like. I often use a wrapper that discards all output from a command, unless the command fails in which case the text from stderr is output.
I agree that you shouldn't be asking users to copy executables around. Programs should be in a directory listed in the PATH variable; if a program is missing it should be installed, or if it is installed in a directory not on the PATH the user should update the PATH to include that directory.
Note that you do have the option of trying subprocess multiple times with various hard-coded paths to executables:
import os
import subprocess as sp
def _run_cmd(s_cmd, tup_args):
lst_cmd = [s_cmd]
lst_cmd.extend(tup_args)
result = sp.call(lst_cmd)
return result
def run_lumberjack(*tup_args):
try:
# try to run from /usr/local/bin
return _run_cmd("/usr/local/bin/lumberjack", tup_args)
except OSError:
pass
try:
# try to run from /opt/forest/bin
return _run_cmd("/opt/forest/bin/lumberjack", tup_args)
except OSError:
pass
try:
# try to run from "bin" directory in user's home directory
home = os.getenv("HOME", ".")
s_cmd = home + "/bin/lumberjack"
return _run_cmd(s_cmd, tup_args)
except OSError:
pass
# Python 3.x syntax for raising an exception
# for Python 2.x, use: raise OSError, "could not find lumberjack in the standard places"
raise OSError("could not find lumberjack in the standard places")
run_lumberjack("-j")
EDIT: After thinking about it a little bit, I decided to completely rewrite the above. It's much cleaner to just pass a list of locations, and have a loop try the alternative locations until one works. But I didn't want to build the string for the user's home directory if it wasn't needed, so I just made it legal to put a callable into the list of alternatives. If you have any questions about this, just ask.
import os
import subprocess as sp
def try_alternatives(cmd, locations, args):
"""
Try to run a command that might be in any one of multiple locations.
Takes a single string argument for the command to run, a sequence
of locations, and a sequence of arguments to the command. Tries
to run the command in each location, in order, until the command
is found (does not raise OSError on the attempt).
"""
# build a list to pass to subprocess
lst_cmd = [None] # dummy arg to reserve position 0 in the list
lst_cmd.extend(args) # arguments come after position 0
for path in locations:
# It's legal to put a callable in the list of locations.
# When this happens, we should call it and use its return
# value for the path. It should always return a string.
if callable(path):
path = path()
# put full pathname of cmd into position 0 of list
lst_cmd[0] = os.path.join(path, cmd)
try:
return sp.call(lst_cmd)
except OSError:
pass
raise OSError('command "{}" not found in locations list'.format(cmd))
def _home_bin():
home = os.getenv("HOME", ".")
return os.path.join(home, "bin")
def run_lumberjack(*args):
locations = [
"/usr/local/bin",
"/opt/forest/bin",
_home_bin, # specify callable that returns user's home directory
]
return try_alternatives("lumberjack", locations, args)
run_lumberjack("-j")
Wow, that was fast! I combined Theodros Zelleke's simple example and steveha's use of functions with abarnert comment about OSError and Lattyware's comment about moving files:
import os, sys, subprocess
def nameandpath():
try:
subprocess.call([os.getcwd() + '/lumberjack'])
# change the word lumberjack on the line above to get an error
except OSError:
print('\nCould not find lumberjack, please reinstall.\n')
# if you're using python 2.x, change the () to spaces on the line above
try:
subprocess.call(['lumberjack'])
# change the word lumberjack on the line above to get an error
except OSError:
nameandpath()
I tested it on Mac OS-X (6.8/Snow Leopard), Debian (Squeeze) and Windows (7). It seemed to work the way I wanted it to on all three operating systems. I tried using check_call and CalledProcessError but no matter what I did, I seemed to get an error every time and I couldn't get the script to handle the errors. To test the script I changed the name from 'lumberjack' to 'deadparrot', since I had lumberjack in the directory with my script.
Do you see any problems with this script the way it's written?