I want to use both multiline input for command 'cat -' and single line input in 'pwd'.
For that i am trying to get input directly in process.communicate() , I am getting broken pipe error. What shall i replace in the argument passed?
command = 'cat -'
stdout_value=''
process = subprocess.Popen(shlex.split(command),
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
shell=True)
while True:
if process.poll() is not None:
break
stdout_value = stdout_value + process.communicate(process.stdin)[0]
print(repr(stdout_value))
This code gets error:
Traceback (most recent call last):
File "try.py", line 67, in <module>
stdout_value = stdout_value + process.communicate(process.stdin)[0]
File "/usr/lib64/python3.5/subprocess.py", line 1068, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/usr/lib64/python3.5/subprocess.py", line 1687, in _communicate
input_view = memoryview(self._input)
TypeError: memoryview: a bytes-like object is required, not '_io.BufferedWriter'
[vibhcool#localhost try]$ Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe
process.communicate() expects string which you have encoded() to bytes and it returns bytes which you have to decode() to string.
You use process.stdin which is not string. It even doesn't have method read() to do process.communicate(process.stdin.read()) (besides using process.stdin with the same process makes no sense). It could be rather process.communicate(other_process.stdout.read()) (if you have other process)
Working example - I send text with numbers to command sort and it returns text with sorted numbers.
input: 3\n2\n5\n-1
output: -1\n2\n3\n5\n
Code
import subprocess
import shlex
command = 'sort'
process = subprocess.Popen(shlex.split(command),
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
shell=True)
stdout_value = ''
while True:
if process.poll() is not None:
break
result = process.communicate("3\n2\n5\n-1".encode('utf-8'))[0]
stdout_value += result.decode('utf-8')
print(repr(stdout_value))
Related
I'm trying to run multiple UNIX commands in a python script like this
import subprocess
cmds = ['sleep 3', 'uptime','time ls -l /']
p = subprocess.Popen(cmds,stdout=subprocess.PIPE,shell=True)
while p.poll() is None:
time.sleep(0.5)
tempdata = p.stdout.read()
print(tempdata)
However my output does not contain all output and doesn't seem to run all the commands. Setting shell=False also causes an error.
Traceback (most recent call last):
File "task1.py", line 32, in ?
p = subprocess.Popen(commands,stdout=subprocess.PIPE,stderr=subprocess.PIPE,shell=False)
File "/usr/lib64/python36/subprocess.py", line 550, in __init__
errread, errwrite)
File "/usr/lib64/python36/subprocess.py", line 996, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
When you create a new process, you don't pass it a list of commands to run; rather, you pass it a single command -- either as a string (with shell=True) or as a list of args (with shell=False).
import subprocess
cmds = ['sleep 1', 'uptime', 'ls -l /']
for cmd in cmds:
stdout = subprocess.check_output(cmd, shell=True)
print('\n# {}'.format(cmd))
print(stdout)
If you just want to collect stdout, subprocess.check_output() might be simpler than Popen() -- but either approach will work, depending on what you need to do with the process.
Your problem is 'sleep 3' causes the error you get from the traceback, when I removed that it worked.
To run all for these:
cmds = ['sleep 3', 'uptime','time ls -l /']
You have to call popen for each of them:
for cmd in cmds:
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
while p.poll() is None:
time.sleep(0.5)
output = p.stdout.read()
Or simpler:
for cmd in cmds:
output = subprocess.check_output(cmd, stdout=subprocess.PIPE, shell=True)
Second question: This captures all output written to stdout. To capture also stderr, redirect that into subprocess.PIPE as well.
I am using below function to read my python scripts output line by line and save in parallel. But getting Traceback error in end.
Code:
def myrun(cmd):
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
stdout = []
while True:
line = p.stdout.readline()
stdout.append(line)
print (line),
if line == '' and p.poll() != None:
break
return ''.join(stdout)
When call function as :
myrun(os.system("./tests_run.py"))
I get below error:
Error:
Traceback (most recent call last):
File "./Portability_Tests.py", line 38, in <module>
myrun(os.system("./tests_run.py"))
File "./Tests.py", line 11, in myrun
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
File "/usr/local/lib/python3.5/subprocess.py", line 676, in __init__
restore_signals, start_new_session)
File "/usr/local/lib/python3.5/subprocess.py", line 1171, in _execute_child
args = list(args)
TypeError: 'int' object is not iterable
Anyone know how can i fix this error ?
The subprocess.Popen function receives a "sequence of program arguments or else a single string" as its args argument.
What you are passing in the args argument is the output of the os.system() call which according to the documentation is the "exit status of the process", thus an int number. Instead in the cmd variable you should directly pass the string (or an other iterator) of your file /tests_run.py. If you want this path to be relative to your current project you can use the os.path module.
As I started asking on a previous question, I'm extracting a tarball using the tarfile module of python. I don't want the extracted files to be written on the disk, but rather get piped directly to another program, specifically bgzip.
#!/usr/bin/env python
import tarfile, subprocess, re
mov = []
def clean(s):
s = re.sub('[^0-9a-zA-Z_]', '', s)
s = re.sub('^[^a-zA-Z_]+', '', s)
return s
with tarfile.open("SomeTarballHere.tar.gz", "r:gz") as tar:
for file in tar.getmembers():
if file.isreg():
mov = file.name
proc = subprocess.Popen(tar.extractfile(file).read(), stdout = subprocess.PIPE)
proc2 = subprocess.Popen('bgzip -c > ' + clean(mov), stdin = proc, stdout = subprocess.PIPE)
mov = None
But now I get stuck on this:
Traceback (most recent call last):
File "preformat.py", line 12, in <module>
proc = subprocess.Popen(tar.extractfile(file).read(), stdout = subprocess.PIPE)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1335, in _execute_child
raise child_exception
OSError: [Errno 36] File name too long
Is there any workaround for this? I have been using the LightTableLinux.tar.gz (it contains the files for a text editor program) as a tarball to test the script on it.
The exception is raised in the forked-off child process when trying to execute the target program from this invocation:
proc = subprocess.Popen(tar.extractfile(file).read(), stdout = subprocess.PIPE)
This
reads the contents of an entry in the tar file
tries to execute a program with the name of the contents of that entry.
Also your second invocation won't work, as you are trying to use shell redirection without using shell=True in Popen():
proc2 = subprocess.Popen('bgzip -c > ' + clean(mov), stdin = proc, stdout = subprocess.PIPE)
The redirect may also not be necessary, as you should be able to simply redirect the output from bgzip to a file from python directly.
Edit: Unfortunately, despite extractfile() returning a file-like object, Popen() expects a real file (with a fileno). Hence, a little wrapping is required:
with tar.extractfile(file) as tarfile, file(clean(mov), 'wb') as outfile:
proc = subprocess.Popen(
('bgzip', '-c'),
stdin=subprocess.PIPE,
stdout=outfile,
)
shutil.copyfileobj(tarfile, proc.stdin)
proc.stdin.close()
proc.wait()
How can I analyze the output of my command, which I pipe to file, in real-time while the file is written?
This is what I have so far:
with open('output.log', 'w') as out:
command = ['pg_dump', 'myDB']
p = subprocess.Popen(cmd, stdout=out, stderr=subprocess.STDOUT)
for line in iter(p.stdout.readline, b''):
sys.stdout.flush()
print(">>> " + line.rstrip())
But this generates the following error:
Traceback (most recent call last):
File "pipe-to-file.py", line 95, in <module>
for line in iter(p.stdout.readline, b''):
AttributeError: 'NoneType' object has no attribute 'readline'
Why is p.stdout equal to None here?
You have to use subprocess.PIPE for the stdout argument in order to get a file object, else it'll be None. That's why p.stdout equals to None in your code.
From the DOC
Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
If you want to write stdout to a file while analyzing the output then you can use something like this.
with open('log', 'ab+') as out:
p = subprocess.Popen('ls', stdout=subprocess.PIPE, stderr=subprocess.PIPE)
std_out, std_error = p.communicate()
# Do something with std_out
# ...
# Write to the file
out.write( std_out )
# You can use `splitlines()` to iterate over the lines.
for line in std_out.splitlines():
print line
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output,error=p.communicate()
Now you have your output,error.
I'm having a strange issue when using subprocess.Popen.communicate(). For background, I want to execute an application from my python script. When I run the program from the command line, I do it like this (UNIX):
$ echo "input text" | /path/to/myapp
From my script, I also want to pipe the input into the application. So, I tried the following. But I get a "broken pipe" error when I try to send the input with communicate():
>>> cmd = ['/path/to/myapp']
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
>>> out,err = p.communicate('input text')
Traceback (most recent call last):
File "", line 1, in
File "/usr/lib/python2.5/subprocess.py", line 670, in communicate
return self._communicate(input)
File "/usr/lib/python2.5/subprocess.py", line 1223, in _communicate
bytes_written = self._write_no_intr(self.stdin.fileno(), buffer(input, input_offset, 512))
File "/usr/lib/python2.5/subprocess.py", line 1003, in _write_no_intr
return os.write(fd, s)
OSError: [Errno 32] Broken pipe
To make matters stranger, if I leave out the input data, I don't get any errors. However, this isn't really a good workaround because the application needs input to work.
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
>>> out,err = p.communicate()
>>> print out
[error from myapp regarding lack of input]
Any idea what I'm missing?
Your observation suggests that myapp is terminating without reading (all of the) input. Not knowing anything about myapp, that's hard to confirm, but consider for example
$ echo 'hello world' | tr 'l' 'L'
heLLo worLd
now...:
>>> cmd = ['/usr/bin/tr']
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, stdin=subprocess.PIPE)
>>> out,err = p.communicate('hello world')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.5/subprocess.py", line 668, in communicate
return self._communicate(input)
File "/usr/lib/python2.5/subprocess.py", line 1218, in _communicate
bytes_written = self._write_no_intr(self.stdin.fileno(), buffer(input, input_offset, 512))
File "/usr/lib/python2.5/subprocess.py", line 997, in _write_no_intr
return os.write(fd, s)
OSError: [Errno 32] Broken pipe
because...:
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
>>> /usr/bin/tr: missing operand
Try `/usr/bin/tr --help' for more information.
and if we fix the bug:
>>> cmd = ['/usr/bin/tr', 'l', 'L']
>>> p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stdin=subprocess.PIPE)
>>> out,err = p.communicate('hello world')>>> print out
heLLo worLd
>>> print err
None
...it fixes everything. What happens if you omit the stderr redirection -- do you perchance see any error messages from myapp...?