This question already has answers here:
subprocess.Popen stdin read file
(3 answers)
Closed 6 years ago.
for a little project of my own, I am trying to write a program that prints out the contents of a file on the computers default printer.
I know theres alot of similar questions around, but none of them works on my pc (Linux mint 17.3)
here is one that I tried, it got the closest to what i needed:
from subprocess import Popen
from cStringIO import StringIO
# place the output in a file like object
sio = StringIO("test.txt")
# call the system's lpr command
p = Popen(["lpr"], stdin=sio, shell=True)
output = p.communicate()[0]
this gives me the following error:
Traceback (most recent call last):
File "/home/vandeventer/x.py", line 8, in <module>
p = Popen(["lpr"], stdin=sio, shell=True)
File "/usr/lib/python2.7/subprocess.py", line 702, in __init__
errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
File "/usr/lib/python2.7/subprocess.py", line 1117, in _get_handles
p2cread = stdin.fileno()
AttributeError: 'cStringIO.StringI' object has no attribute 'fileno'
Doe anyone out there know hoe one could implement this in python? it really does not have to work on windows
Regards
Cid-El
you don't have to use StringIO for this. Just use the pipe feature of subprocess and write your data to p.stdin:
from subprocess import Popen
# call the system's lpr command
p = Popen(["lpr"], stdin=subprocess.PIPE, shell=True) # not sure you need shell=True for a simple command
p.stdin.write("test.txt")
output = p.communicate()[0]
as a bonus, this is Python 3 compliant (StringIO has been renamed since :))
BUT: that would just print a big white page with one line: test.txt. lpr reads standard input and prints it (that's still an interesting piece of code :))
To print the contents of your file you have to read it, and in that case it's even simpler since pipe & files work right away together:
from subprocess import Popen
with open("test.txt") as f:
# call the system's lpr command
p = Popen(["lpr"], stdin=f, shell=True) # not sure you need shell=True for a simple command
output = p.communicate()[0]
Related
This question already has answers here:
Store output of subprocess.Popen call in a string [duplicate]
(15 answers)
Closed 4 years ago.
How can I get the output of a process run using subprocess.call()?
Passing a StringIO.StringIO object to stdout gives this error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 444, in call
return Popen(*popenargs, **kwargs).wait()
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 588, in __init__
errread, errwrite) = self._get_handles(stdin, stdout, stderr)
File "/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/subprocess.py", line 945, in _get_handles
c2pwrite = stdout.fileno()
AttributeError: StringIO instance has no attribute 'fileno'
>>>
If you have Python version >= 2.7, you can use subprocess.check_output which basically does exactly what you want (it returns standard output as string).
Simple example (linux version, see note):
import subprocess
print subprocess.check_output(["ping", "-c", "1", "8.8.8.8"])
Note that the ping command is using linux notation (-c for count). If you try this on Windows remember to change it to -n for same result.
As commented below you can find a more detailed explanation in this other answer.
Output from subprocess.call() should only be redirected to files.
You should use subprocess.Popen() instead. Then you can pass subprocess.PIPE for the stderr, stdout, and/or stdin parameters and read from the pipes by using the communicate() method:
from subprocess import Popen, PIPE
p = Popen(['program', 'arg1'], stdin=PIPE, stdout=PIPE, stderr=PIPE)
output, err = p.communicate(b"input data that is passed to subprocess' stdin")
rc = p.returncode
The reasoning is that the file-like object used by subprocess.call() must have a real file descriptor, and thus implement the fileno() method. Just using any file-like object won't do the trick.
See here for more info.
For python 3.5+ it is recommended that you use the run function from the subprocess module. This returns a CompletedProcess object, from which you can easily obtain the output as well as return code.
from subprocess import PIPE, run
command = ['echo', 'hello']
result = run(command, stdout=PIPE, stderr=PIPE, universal_newlines=True)
print(result.returncode, result.stdout, result.stderr)
I have the following solution. It captures the exit code, the stdout, and the stderr too of the executed external command:
import shlex
from subprocess import Popen, PIPE
def get_exitcode_stdout_stderr(cmd):
"""
Execute the external command and get its exitcode, stdout and stderr.
"""
args = shlex.split(cmd)
proc = Popen(args, stdout=PIPE, stderr=PIPE)
out, err = proc.communicate()
exitcode = proc.returncode
#
return exitcode, out, err
cmd = "..." # arbitrary external command, e.g. "python mytest.py"
exitcode, out, err = get_exitcode_stdout_stderr(cmd)
I also have a blog post on it here.
Edit: the solution was updated to a newer one that doesn't need to write to temp. files.
I recently just figured out how to do this, and here's some example code from a current project of mine:
#Getting the random picture.
#First find all pictures:
import shlex, subprocess
cmd = 'find ../Pictures/ -regex ".*\(JPG\|NEF\|jpg\)" '
#cmd = raw_input("shell:")
args = shlex.split(cmd)
output,error = subprocess.Popen(args,stdout = subprocess.PIPE, stderr= subprocess.PIPE).communicate()
#Another way to get output
#output = subprocess.Popen(args,stdout = subprocess.PIPE).stdout
ber = raw_input("search complete, display results?")
print output
#... and on to the selection process ...
You now have the output of the command stored in the variable "output". "stdout = subprocess.PIPE" tells the class to create a file object named 'stdout' from within Popen. The communicate() method, from what I can tell, just acts as a convenient way to return a tuple of the output and the errors from the process you've run. Also, the process is run when instantiating Popen.
The key is to use the function subprocess.check_output
For example, the following function captures stdout and stderr of the process and returns that as well as whether or not the call succeeded. It is Python 2 and 3 compatible:
from subprocess import check_output, CalledProcessError, STDOUT
def system_call(command):
"""
params:
command: list of strings, ex. `["ls", "-l"]`
returns: output, success
"""
try:
output = check_output(command, stderr=STDOUT).decode()
success = True
except CalledProcessError as e:
output = e.output.decode()
success = False
return output, success
output, success = system_call(["ls", "-l"])
If you want to pass commands as strings rather than arrays, use this version:
from subprocess import check_output, CalledProcessError, STDOUT
import shlex
def system_call(command):
"""
params:
command: string, ex. `"ls -l"`
returns: output, success
"""
command = shlex.split(command)
try:
output = check_output(command, stderr=STDOUT).decode()
success = True
except CalledProcessError as e:
output = e.output.decode()
success = False
return output, success
output, success = system_call("ls -l")
In Ipython shell:
In [8]: import subprocess
In [9]: s=subprocess.check_output(["echo", "Hello World!"])
In [10]: s
Out[10]: 'Hello World!\n'
Based on sargue's answer. Credit to sargue.
I have a script in Python3 and if I use subprocess.Popen.wait() I have problem — my script iterates some Linux command many times and it looks to me like my app is not responding. When I use subprocess.Popen.communicate() my application correctly completes its work in a second.
What is the right way to solve this problem using Linux?
I think the solution must be somewhere in manipulating with buffer's variable, but I searched through the entire Internet and could not find anything suitable. May be I don't know enough structure and operation of Linux as a whole.
My question can be reformulated as follows: What's happened exactly when I use .wait() method? And that leads to failure of it? What is the cause of the so long waiting? When I aborting running task I see the next log:
Traceback (most recent call last):
File "./test.py", line 6, in <module>
proc.wait()
File "/usr/lib/python3.5/subprocess.py", line 1658, in wait
(pid, sts) = self._try_wait(0)
File "/usr/lib/python3.5/subprocess.py", line 1608, in _try_wait
(pid, sts) = os.waitpid(self.pid, wait_flags)
KeyboardInterrupt
My files looks approximately like the next things:
script.py:
#!/usr/bin/python3
# -*-coding: utf-8 -*-
import subprocess
proc = subprocess.Popen(['./1.py', '1000000'], stdin=subprocess.PIPE, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
proc.wait()
out = proc.stdout.read()
# out = proc.communicate()[0]
print(len(out))
1.py:
#!/usr/bin/python3
# -*-coding: utf-8 -*-
import sys
x = sys.argv[-1]
# print(x, type(x))
for i in range(int(x)):
print(i)
UPD: As we understand, the problem is a buffer overflow. It turns out the last version of question is, how to use the Linux possibility to expand buffer or redirect buffer to a file before running the script?
UPD2: I also tried run the script as: $ python3 -u ./script.py, but, unfortunally, unbufferring doesn't work as I would like and script is hangs.
Your script is sending output to its stdout or stderr pipes. The operating system will buffer some data then block the process forever when the pipe fills. Suppose I have a long winded command like
longwinded.py:
for i in range(100000):
print('a'*1000)
The following hangs because the stdout pipe fills
import sys
import subprocess as subp
p = subp.Popen([sys.executable, 'longwinded.py'], stdout=subp.PIPE,
stderr=subp.PIPE)
p.wait()
The next one doesn't hang because communicate reads the stdout and stderr pipes into memory
p = subp.Popen([sys.executable, 'longwinded.py'], stdout=subp.PIPE,
stderr=subp.PIPE)
p.communicate()
If you don't care what stdout and err are, you can redirect them to the null device
p = subp.Popen([sys.executable, 'longwinded.py'],
stdout=open(os.devnull, 'w'),
stderr=open(os.devnull, 'w'))
p.wait()
or save them to a file
p = subp.Popen([sys.executable, 'longwinded.py'],
stdout=open('mystdout', 'w'),
stderr=open('mystderr', 'w'))
p.wait()
i'm trying to catch the output of airodump-ng, that has a continuous output, and process every line searching for a string. but that doesn't work. so i try the same thing with "htop" command that has the same kind of output, and it still doesn't work.
i'm trying this with python 3.4 and python 2.7, both on arch linux and osx mavericks. here's the code (not every import is necessary but nevermind):
import subprocess
import sys
import os
import time
command = ["htop"]
proc = subprocess.Popen(command, stdout = subprocess.PIPE)
outs, errs = proc.communicate(timeout=3)
proc.kill()
and it gives me:
Traceback (most recent call last):
File "/Users/andrei/Dropbox/python/file_prova.py", line 8, in <module>
outs, errs = proc.communicate(timeout=3)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/subprocess.py", line 960, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/subprocess.py", line 1618, in _communicate
self._check_timeout(endtime, orig_timeout)
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/subprocess.py", line 986, in _check_timeout
raise TimeoutExpired(self.args, orig_timeout)
subprocess.TimeoutExpired: Command '['htop']' timed out after 3 seconds
seems like it crashes at proc.communicate() and doesn't execute the lines under that. i also tried to handle the exception but no way to make it work...
[EDIT]
ok so it's for 4 am, i learned the try - exception handling, and after a looong time a managed to make it work with htop, following the tips hardly found here (the 2nd solution doesn't seem to work):
this is how it looks
from subprocess import Popen, PIPE
from time import sleep
from fcntl import fcntl, F_GETFL, F_SETFL
from os import O_NONBLOCK, read
# run the shell as a subprocess:
p = Popen(['htop'], stdout = PIPE)
# set the O_NONBLOCK flag of p.stdout file descriptor:
flags = fcntl(p.stdout, F_GETFL) # get current p.stdout flags
fcntl(p.stdout, F_SETFL, flags | O_NONBLOCK)
# let the shell output the result:
# get the output
while True:
sleep(1)
try:
print (read(p.stdout.fileno(), 1024).decode("utf-8")),
except OSError:
# the os throws an exception if there is no data
print ('[No more data]')
continue
it works flawlessly. with htop.
but not with airodump-ng. it prints on the terminal its output and every 1 second (the sleep() in the while loop) prints [No more data], like the stream is going elsewhere...
EDIT 2:
solved! the thing was just that airodump-ng dumps data to stderr, not stdout. pretty straight forward try ahah :D
From the documentation:
The timeout argument is passed to Popen.wait(). If the timeout
expires, the child process will be killed and then waited for again.
The TimeoutExpired exception will be re-raised after the child process
has terminated.
That seems to describe exactly the behavior you are seeing. You will need to learn about exception handling using try/except.
Sorry if this question is dumb. I am using python subprocess statement to call a .bat file in Ubuntu (Natty 11.04), however, I got error messages:
Traceback (most recent call last):
File "pfam_picloud.py", line 40, in <module>
a=subprocess.Popen(src2, shell=0)
File "/usr/lib/python2.7/subprocess.py", line 672, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1213, in _execute_child
raise child_exception
run this python file
$python pfam_picloud.py
Python code (pfam_picloud.py)
#!/usr/bin/python
#
met="wTest.dvf"
run="run_pfam.bat"
inp="pfam_input.PFA"
import os
import stat
import shutil
import subprocess
import string
import random
# Generate a random ID for file save
def id_generator(size=6, chars=string.ascii_uppercase + string.digits):
return ''.join(random.choice(chars) for x in range(size))
name_temp=id_generator()
cwd=os.getcwd()
src=cwd
src1=cwd+'/'+name_temp
if not os.path.exists(src1):
os.makedirs(src1)
else:
shutil.rmtree(src1)
os.makedirs(src1)
##
shutil.copy(src+"/"+run,src1)
shutil.copy(src+"/"+met,src1)
shutil.copy(cwd+"/pfam_pi.exe",src1)
shutil.copy(src+"/"+inp,src1)
#
src2=src1+"/run_pfam.bat"
os.chdir(src1)
a=subprocess.Popen(src2, shell=0)
a.wait()
bash file (run_pfam.bat)
#!/bin/sh
./pfam_pi.exe pfam_input.PFA
I can successfully run this bash file in Ubuntu. So I guess, I messed up something in my Python script. Could anyone give me some suggestions? Thanks for any inputs.
EDIT
the file pfam_pi.exe is a Linux executable. I compiled it in Ubuntu. Sorry for the confusion.
update
Well, I got different types of error now.
1. With #!/bin/sh, it said No such file or directory.
2. With /bin/sh, it said exec format error.
3. If I sent everything as arguments a=subprocess.Popen(['./pfam_pi.exe', 'inp', 'src1'], shell=0), it said end of line symbol error
Since feature requests to mark a comment as an answer remain declined, I copy the above solution here.
#Ellioh: Thanks for your comments. I found once I changed the shell=1, problem is solved. – tao.hong
Try running wine (you should have it installed) and pass pfam_pi.exe to it as a parameter. Maybe pfam_pi.exe is not a Linux executable. :-) Certainly, executable file extensions are not meaningful on Linux, but probably it really is a Windows program, otherwise I hardly can imagine it named pfam_pi.exe.
However, if it is a Linux executable, note subprocess.Popen accepts a list of args (the first element is the program itself), not a command line:
>>> import shlex, subprocess
>>> command_line = raw_input()
/bin/vikings -input eggs.txt -output "spam spam.txt" -cmd "echo '$MONEY'"
>>> args = shlex.split(command_line)
>>> print args
['/bin/vikings', '-input', 'eggs.txt', '-output', 'spam spam.txt', '-cmd', "echo '$MONEY'"]
>>> p = subprocess.Popen(args) # Success!
This question already has answers here:
Store output of subprocess.Popen call in a string [duplicate]
(15 answers)
Closed 4 years ago.
I want to run a command in pythong, using the subprocess module, and store the output in a variable. However, I do not want the command's output to be printed to the terminal.
For this code:
def storels():
a = subprocess.Popen("ls",shell=True)
storels()
I get the directory listing in the terminal, instead of having it stored in a. I've also tried:
def storels():
subprocess.Popen("ls > tmp",shell=True)
a = open("./tmp")
[Rest of Code]
storels()
This also prints the output of ls to my terminal. I've even tried this command with the somewhat dated os.system method, since running ls > tmp in the terminal doesn't print ls to the terminal at all, but stores it in tmp. However, the same thing happens.
Edit:
I get the following error after following marcog's advice, but only when running a more complex command. cdrecord --help. Python spits this out:
Traceback (most recent call last):
File "./install.py", line 52, in <module>
burntrack2("hi")
File "./install.py", line 46, in burntrack2
a = subprocess.Popen("cdrecord --help",stdout = subprocess.PIPE)
File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1139, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
To get the output of ls, use stdout=subprocess.PIPE.
>>> proc = subprocess.Popen('ls', stdout=subprocess.PIPE)
>>> output = proc.stdout.read()
>>> print output
bar
baz
foo
The command cdrecord --help outputs to stderr, so you need to pipe that indstead. You should also break up the command into a list of tokens as I've done below, or the alternative is to pass the shell=True argument but this fires up a fully-blown shell which can be dangerous if you don't control the contents of the command string.
>>> proc = subprocess.Popen(['cdrecord', '--help'], stderr=subprocess.PIPE)
>>> output = proc.stderr.read()
>>> print output
Usage: wodim [options] track1...trackn
Options:
-version print version information and exit
dev=target SCSI target to use as CD/DVD-Recorder
gracetime=# set the grace time before starting to write to #.
...
If you have a command that outputs to both stdout and stderr and you want to merge them, you can do that by piping stderr to stdout and then catching stdout.
subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
As mentioned by Chris Morgan, you should be using proc.communicate() instead of proc.read().
>>> proc = subprocess.Popen(['cdrecord', '--help'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> out, err = proc.communicate()
>>> print 'stdout:', out
stdout:
>>> print 'stderr:', err
stderr:Usage: wodim [options] track1...trackn
Options:
-version print version information and exit
dev=target SCSI target to use as CD/DVD-Recorder
gracetime=# set the grace time before starting to write to #.
...
If you are using python 2.7 or later, the easiest way to do this is to use the subprocess.check_output() command. Here is an example:
output = subprocess.check_output('ls')
To also redirect stderr you can use the following:
output = subprocess.check_output('ls', stderr=subprocess.STDOUT)
In the case that you want to pass parameters to the command, you can either use a list or use invoke a shell and use a single string.
output = subprocess.check_output(['ls', '-a'])
output = subprocess.check_output('ls -a', shell=True)
With a = subprocess.Popen("cdrecord --help",stdout = subprocess.PIPE)
, you need to either use a list or use shell=True;
Either of these will work. The former is preferable.
a = subprocess.Popen(['cdrecord', '--help'], stdout=subprocess.PIPE)
a = subprocess.Popen('cdrecord --help', shell=True, stdout=subprocess.PIPE)
Also, instead of using Popen.stdout.read/Popen.stderr.read, you should use .communicate() (refer to the subprocess documentation for why).
proc = subprocess.Popen(['cdrecord', '--help'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = proc.communicate()