printbob.py:
import sys
for arg in sys.argv:
print arg
getbob.py
import subprocess
#printbob.py will always be in root of getbob.py
#a sample of sending commands to printbob.py is:
#printboby.py arg1 arg2 arg3 (commands are seperated by spaces)
print subprocess.Popen(['printbob.py', 'arg1 arg2 arg3 arg4']).wait()
x = raw_input('done')
I get:
File "C:\Python27\lib\subprocess.py", line 672, in __init__
errread, errwrite)
File "C:\Python27\lib\subprocess.py", line 882, in _execute_child
startupinfo)
WindowsError: [Error 193] %1 is not a valid Win32 application
What am I doing wrong here?
I just want to get the output of another python script inside of another python script.
Do I need to call cmd.exe or can I just run printbob.py and send commands to it?
proc = subprocess.Popen(['python', 'printbob.py', 'arg1 arg2 arg3 arg4'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
print proc.communicate()[0]
There must be a better way of doing it though, since the script is also in Python. It's better to find some way to leverage that than what you're doing.
This is the wrong approach.
You should refactor printbob.py so that it can be imported by other python modules. This version can be imported and called from the command-line:
#!/usr/bin/env python
import sys
def main(args):
for arg in args:
print(arg)
if __name__ == '__main__':
main(sys.argv)
Here it is called from the command-line:
python printbob.py one two three four five
printbob.py
one
two
three
four
five
Now we can import it in getbob.py:
#!/usr/bin/env python
import printbob
printbob.main('arg1 arg2 arg3 arg4'.split(' '))
Here it is running:
python getbob.py
arg1
arg2
arg3
arg4
The shell argument (which defaults to False) specifies whether to use
the shell as the program to execute. If shell is True, it is
recommended to pass args as a string rather than as a sequence
Just wrap all arguments in a string and give shell=True
proc = subprocess.Popen("python myScript.py --alpha=arg1 -b arg2 arg3" ,stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
print proc.communicate()[0]
The subprocess.run() function was added in Python 3.5.
import subprocess
cmd = subprocess.run(["ls", "-ashl"], capture_output=True)
stdout = cmd.stdout.decode() # bytes => str
refer: PEP 324 – PEP proposing the subprocess module
Related
I want to execute a script inside a subdirectory/superdirectory (I need to be inside this sub/super-directory first). I can't get subprocess to enter my subdirectory:
tducin#localhost:~/Projekty/tests/ve$ python
Python 2.7.4 (default, Sep 26 2013, 03:20:26)
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> import os
>>> os.getcwd()
'/home/tducin/Projekty/tests/ve'
>>> subprocess.call(['cd ..'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1308, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Python throws OSError and I don't know why. It doesn't matter whether I try to go into an existing subdir or go one directory up (as above) - I always end up with the same error.
What your code tries to do is call a program named cd ... What you want is call a command named cd.
But cd is a shell internal. So you can only call it as
subprocess.call('cd ..', shell=True) # pointless code! See text below.
But it is pointless to do so. As no process can change another process's working directory (again, at least on a UNIX-like OS, but as well on Windows), this call will have the subshell change its dir and exit immediately.
What you want can be achieved with os.chdir() or with the subprocess named parameter cwd which changes the working directory immediately before executing a subprocess.
For example, to execute ls in the root directory, you either can do
wd = os.getcwd()
os.chdir("/")
subprocess.Popen("ls")
os.chdir(wd)
or simply
subprocess.Popen("ls", cwd="/")
To run your_command as a subprocess in a different directory, pass cwd parameter, as suggested in #wim's answer:
import subprocess
subprocess.check_call(['your_command', 'arg 1', 'arg 2'], cwd=working_dir)
A child process can't change its parent's working directory (normally). Running cd .. in a child shell process using subprocess won't change your parent Python script's working directory i.e., the code example in #glglgl's answer is wrong. cd is a shell builtin (not a separate executable), it can change the directory only in the same process.
subprocess.call and other methods in the subprocess module have a cwd parameter.
This parameter determines the working directory where you want to execute your process.
So you can do something like this:
subprocess.call('ls', shell=True, cwd='path/to/wanted/dir/')
Check out docs subprocess.popen-constructor
You want to use an absolute path to the executable, and use the cwd kwarg of Popen to set the working directory. See the docs.
If cwd is not None, the child’s current directory will be changed to
cwd before it is executed. Note that this directory is not considered
when searching the executable, so you can’t specify the program’s path
relative to cwd.
I guess these days you would do:
import subprocess
subprocess.run(["pwd"], cwd="sub-dir")
Another option based on this answer: https://stackoverflow.com/a/29269316/451710
This allows you to execute multiple commands (e.g cd) in the same process.
import subprocess
commands = '''
pwd
cd some-directory
pwd
cd another-directory
pwd
'''
process = subprocess.Popen('/bin/bash', stdin=subprocess.PIPE, stdout=subprocess.PIPE)
out, err = process.communicate(commands.encode('utf-8'))
print(out.decode('utf-8'))
just use os.chdir
Example:
>>> import os
>>> import subprocess
>>> # Lets Just Say WE want To List The User Folders
>>> os.chdir("/home/")
>>> subprocess.run("ls")
user1 user2 user3 user4
If you want to have cd functionality (assuming shell=True) and still want to change the directory in terms of the Python script, this code will allow 'cd' commands to work.
import subprocess
import os
def cd(cmd):
#cmd is expected to be something like "cd [place]"
cmd = cmd + " && pwd" # add the pwd command to run after, this will get our directory after running cd
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True) # run our new command
out = p.stdout.read()
err = p.stderr.read()
# read our output
if out != "":
print(out)
os.chdir(out[0:len(out) - 1]) # if we did get a directory, go to there while ignoring the newline
if err != "":
print(err) # if that directory doesn't exist, bash/sh/whatever env will complain for us, so we can just use that
return
If you need to change directory, run a command and get the std output as well:
import os
import logging as log
from subprocess import check_output, CalledProcessError, STDOUT
log.basicConfig(level=log.DEBUG)
def cmd_std_output(cd_dir_path, cmd):
cmd_to_list = cmd.split(" ")
try:
if cd_dir_path:
os.chdir(os.path.abspath(cd_dir_path))
output = check_output(cmd_to_list, stderr=STDOUT).decode()
return output
except CalledProcessError as e:
log.error('e: {}'.format(e))
def get_last_commit_cc_cluster():
cd_dir_path = "/repos/cc_manager/cc_cluster"
cmd = "git log --name-status HEAD^..HEAD --date=iso"
result = cmd_std_output(cd_dir_path, cmd)
return result
log.debug("Output: {}".format(get_last_commit_cc_cluster()))
Output: "commit 3b3daaaaaaaa2bb0fc4f1953af149fa3921e\nAuthor: user1<user1#email.com>\nDate: 2020-04-23 09:58:49 +0200\n\n
I want to execute a script inside a subdirectory/superdirectory (I need to be inside this sub/super-directory first). I can't get subprocess to enter my subdirectory:
tducin#localhost:~/Projekty/tests/ve$ python
Python 2.7.4 (default, Sep 26 2013, 03:20:26)
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> import os
>>> os.getcwd()
'/home/tducin/Projekty/tests/ve'
>>> subprocess.call(['cd ..'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1308, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Python throws OSError and I don't know why. It doesn't matter whether I try to go into an existing subdir or go one directory up (as above) - I always end up with the same error.
What your code tries to do is call a program named cd ... What you want is call a command named cd.
But cd is a shell internal. So you can only call it as
subprocess.call('cd ..', shell=True) # pointless code! See text below.
But it is pointless to do so. As no process can change another process's working directory (again, at least on a UNIX-like OS, but as well on Windows), this call will have the subshell change its dir and exit immediately.
What you want can be achieved with os.chdir() or with the subprocess named parameter cwd which changes the working directory immediately before executing a subprocess.
For example, to execute ls in the root directory, you either can do
wd = os.getcwd()
os.chdir("/")
subprocess.Popen("ls")
os.chdir(wd)
or simply
subprocess.Popen("ls", cwd="/")
To run your_command as a subprocess in a different directory, pass cwd parameter, as suggested in #wim's answer:
import subprocess
subprocess.check_call(['your_command', 'arg 1', 'arg 2'], cwd=working_dir)
A child process can't change its parent's working directory (normally). Running cd .. in a child shell process using subprocess won't change your parent Python script's working directory i.e., the code example in #glglgl's answer is wrong. cd is a shell builtin (not a separate executable), it can change the directory only in the same process.
subprocess.call and other methods in the subprocess module have a cwd parameter.
This parameter determines the working directory where you want to execute your process.
So you can do something like this:
subprocess.call('ls', shell=True, cwd='path/to/wanted/dir/')
Check out docs subprocess.popen-constructor
You want to use an absolute path to the executable, and use the cwd kwarg of Popen to set the working directory. See the docs.
If cwd is not None, the child’s current directory will be changed to
cwd before it is executed. Note that this directory is not considered
when searching the executable, so you can’t specify the program’s path
relative to cwd.
I guess these days you would do:
import subprocess
subprocess.run(["pwd"], cwd="sub-dir")
Another option based on this answer: https://stackoverflow.com/a/29269316/451710
This allows you to execute multiple commands (e.g cd) in the same process.
import subprocess
commands = '''
pwd
cd some-directory
pwd
cd another-directory
pwd
'''
process = subprocess.Popen('/bin/bash', stdin=subprocess.PIPE, stdout=subprocess.PIPE)
out, err = process.communicate(commands.encode('utf-8'))
print(out.decode('utf-8'))
just use os.chdir
Example:
>>> import os
>>> import subprocess
>>> # Lets Just Say WE want To List The User Folders
>>> os.chdir("/home/")
>>> subprocess.run("ls")
user1 user2 user3 user4
If you want to have cd functionality (assuming shell=True) and still want to change the directory in terms of the Python script, this code will allow 'cd' commands to work.
import subprocess
import os
def cd(cmd):
#cmd is expected to be something like "cd [place]"
cmd = cmd + " && pwd" # add the pwd command to run after, this will get our directory after running cd
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True) # run our new command
out = p.stdout.read()
err = p.stderr.read()
# read our output
if out != "":
print(out)
os.chdir(out[0:len(out) - 1]) # if we did get a directory, go to there while ignoring the newline
if err != "":
print(err) # if that directory doesn't exist, bash/sh/whatever env will complain for us, so we can just use that
return
If you need to change directory, run a command and get the std output as well:
import os
import logging as log
from subprocess import check_output, CalledProcessError, STDOUT
log.basicConfig(level=log.DEBUG)
def cmd_std_output(cd_dir_path, cmd):
cmd_to_list = cmd.split(" ")
try:
if cd_dir_path:
os.chdir(os.path.abspath(cd_dir_path))
output = check_output(cmd_to_list, stderr=STDOUT).decode()
return output
except CalledProcessError as e:
log.error('e: {}'.format(e))
def get_last_commit_cc_cluster():
cd_dir_path = "/repos/cc_manager/cc_cluster"
cmd = "git log --name-status HEAD^..HEAD --date=iso"
result = cmd_std_output(cd_dir_path, cmd)
return result
log.debug("Output: {}".format(get_last_commit_cc_cluster()))
Output: "commit 3b3daaaaaaaa2bb0fc4f1953af149fa3921e\nAuthor: user1<user1#email.com>\nDate: 2020-04-23 09:58:49 +0200\n\n
I've read numerous questions about this subject, and even 2 which have accepted answers, which then have in the comment the same issue as i'm experiencing.
So what I want to do is catch the output of this command (which works in the command line)
sudo /usr/bin/atq
in my Python program.
This is my code (which is an accepted answer in another question)
from subprocess import Popen, PIPE
output = Popen(['sudo /usr/bin/atq', ''], stdout=PIPE)
print output.stdout.read()
and this is the result:
File "try2.py", line 3, in <module>
output = Popen(['sudo /usr/bin/atq', ''], stdout=PIPE)
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Why oh why is this the result (in Python 2.7, on a debian Raspbarry Wheezy install)?
I believe all you need to do is change,
output = Popen(['sudo /usr/bin/atq'], stdout=PIPE)
to
output = Popen(['sudo', '/usr/bin/atq'], stdout=PIPE)
I get the same error when I include multiple arguments as a single string in the args list.
The arguments to Popen need to be a list, you could use shlex to handle this automatically for you
import shlex
args = shlex.split('sudo /usr/bin/atq')
print args
produces
['sudo', '/usr/bin/atq']
which you can then pass to Popen. Then you'll need to communicate with the process you've created.
Have a go with .communicate() (note arguments to Popen here are a list!) ie
prc = Popen(['sudo', '/usr/bin/atq'], stdout=PIPE, stderr=PIPE)
output, stderr = prc.communicate()
print output
Popen returns the subprocess handle, which you need to communicate with to get the output. Note - adding stderr=PIPE will give you access to STDERR as well as STDOUT.
You can use subprocess.check_output():
subprocess.check_output(['sudo', '/usr/bin/atq'])
example:
In [11]: print subprocess.check_output(["ps"])
PID TTY TIME CMD
4547 pts/0 00:00:00 bash
4599 pts/0 00:00:00 python
4607 pts/0 00:00:00 python
4697 pts/0 00:00:00 ps
help():
In [12]: subprocess.check_output?
Type: function
String Form:<function check_output at 0xa0e9a74>
File: /usr/lib/python2.7/subprocess.py
Definition: subprocess.check_output(*popenargs, **kwargs)
Docstring:
Run command with arguments and return its output as a byte string.
If the exit code was non-zero it raises a CalledProcessError. The
CalledProcessError object will have the return code in the returncode
attribute and output in the output attribute.
The arguments are the same as for the Popen constructor.
Sorry if this question is dumb. I am using python subprocess statement to call a .bat file in Ubuntu (Natty 11.04), however, I got error messages:
Traceback (most recent call last):
File "pfam_picloud.py", line 40, in <module>
a=subprocess.Popen(src2, shell=0)
File "/usr/lib/python2.7/subprocess.py", line 672, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1213, in _execute_child
raise child_exception
run this python file
$python pfam_picloud.py
Python code (pfam_picloud.py)
#!/usr/bin/python
#
met="wTest.dvf"
run="run_pfam.bat"
inp="pfam_input.PFA"
import os
import stat
import shutil
import subprocess
import string
import random
# Generate a random ID for file save
def id_generator(size=6, chars=string.ascii_uppercase + string.digits):
return ''.join(random.choice(chars) for x in range(size))
name_temp=id_generator()
cwd=os.getcwd()
src=cwd
src1=cwd+'/'+name_temp
if not os.path.exists(src1):
os.makedirs(src1)
else:
shutil.rmtree(src1)
os.makedirs(src1)
##
shutil.copy(src+"/"+run,src1)
shutil.copy(src+"/"+met,src1)
shutil.copy(cwd+"/pfam_pi.exe",src1)
shutil.copy(src+"/"+inp,src1)
#
src2=src1+"/run_pfam.bat"
os.chdir(src1)
a=subprocess.Popen(src2, shell=0)
a.wait()
bash file (run_pfam.bat)
#!/bin/sh
./pfam_pi.exe pfam_input.PFA
I can successfully run this bash file in Ubuntu. So I guess, I messed up something in my Python script. Could anyone give me some suggestions? Thanks for any inputs.
EDIT
the file pfam_pi.exe is a Linux executable. I compiled it in Ubuntu. Sorry for the confusion.
update
Well, I got different types of error now.
1. With #!/bin/sh, it said No such file or directory.
2. With /bin/sh, it said exec format error.
3. If I sent everything as arguments a=subprocess.Popen(['./pfam_pi.exe', 'inp', 'src1'], shell=0), it said end of line symbol error
Since feature requests to mark a comment as an answer remain declined, I copy the above solution here.
#Ellioh: Thanks for your comments. I found once I changed the shell=1, problem is solved. – tao.hong
Try running wine (you should have it installed) and pass pfam_pi.exe to it as a parameter. Maybe pfam_pi.exe is not a Linux executable. :-) Certainly, executable file extensions are not meaningful on Linux, but probably it really is a Windows program, otherwise I hardly can imagine it named pfam_pi.exe.
However, if it is a Linux executable, note subprocess.Popen accepts a list of args (the first element is the program itself), not a command line:
>>> import shlex, subprocess
>>> command_line = raw_input()
/bin/vikings -input eggs.txt -output "spam spam.txt" -cmd "echo '$MONEY'"
>>> args = shlex.split(command_line)
>>> print args
['/bin/vikings', '-input', 'eggs.txt', '-output', 'spam spam.txt', '-cmd', "echo '$MONEY'"]
>>> p = subprocess.Popen(args) # Success!
Can anyone spot why the following script is not printing the passed arguments?
import sys, getopt
def usage():
print 'Unknown arguments'
def main(argv):
try:
opts, args = getopt.getopt(argv,'fdmse:d',['files=','data-source=','mode=','start','end'])
except getopt.GetoptError:
usage()
sys.exit(999)
for opt, arg in opts:
# print opt,arg
if opt in('-f','--files'):
print 'files: ', arg #
if __name__ == "__main__":
main(sys.argv[1:])
When I run the script at the command line and pass the arguments -f=dummy.csv, usage() seems to be invoked instead - WHY?
BTW, I find the logic of the program flow a bit weird (I copied it from here). Normally, I would have thought that the logic will be implemented in the try branch, and then AFTER that comes the exception handler.
Is this (as pasted in the code above) the 'Pythonic' way to write try/catch blocks?
Did you get your answers?
One way to debug python exceptions is to move (or copy temporarily for debugging) the code out of the try block. You'll get a full trace.
And of course another way is to reduce the test case. Here I've reduced the problems to three lines, and tried the solution hinted at by #s.lott (using 'f:' in the getopts call), and also show at the end how calling with some different test data behaves:
$ cat x1.py
import sys, getopt
opts, args = getopt.getopt(sys.argv[1:],'fdmse:d',['files=','data-source=','mode=','start','end'])
print "opts=", opts, "args=", args
$ python x1.py -f=dummy.csv argblah
Traceback (most recent call last):
File "x1.py", line 2, in <module>
opts, args = getopt.getopt(sys.argv[1:],'fdmse:d',['files=','data-source=','mode=','start','end'])
File "/usr/lib/python2.6/getopt.py", line 91, in getopt
opts, args = do_shorts(opts, args[0][1:], shortopts, args[1:])
File "/usr/lib/python2.6/getopt.py", line 191, in do_shorts
if short_has_arg(opt, shortopts):
File "/usr/lib/python2.6/getopt.py", line 207, in short_has_arg
raise GetoptError('option -%s not recognized' % opt, opt)
getopt.GetoptError: option -= not recognized
$ sed 's/fdm/f:dm/' <x1.py >x2.py
$ diff x1.py x2.py
2c2
< opts, args = getopt.getopt(sys.argv[1:],'fdmse:d',['files=','data-source=','mode=','start','end'])
---
> opts, args = getopt.getopt(sys.argv[1:],'f:dmse:d',['files=','data-source=','mode=','start','end'])
$ python x2.py -f=dummy.csv argblah
opts= [('-f', '=dummy.csv')] args= ['argblah']
$ python x1.py -f dummy.csv argblah
opts= [('-f', '')] args= ['dummy.csv', 'argblah']
Normally, I would have thought that the logic will be implemented in the try branch
"Normally"? What does normally mean?
What is the program supposed to do? What exceptions make sense? What does the program do in response to the exceptions.
There's no "normally". Any more than there's a normal assignment statement or a normal function definition.
Your program does what makes sense to achieve the required end-state. There's no "normally".
Import and use argparse instead of getopt. It's much easier to use and has almost all you need for running from the command line, built into it.
An example,
parser = argparse.ArgumentParser(
description='Description of what the module does when run.')
parser.add_argument("-o", "--output", help='Path of log file.')
args = parser.parse_args()
As easy as that. You need to import argparse at the top of your file for this to work, of course.