I searched a lot here and googled also, trying to find why stderr from my first command is not seen in the final stderr. I know of other methods like "check_output" (python3) and "commands" (python2), but I want to write my own cross-platform one. Here is the problem:
import subprocess
p1 = subprocess.Popen('dirr', shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
p2 = subprocess.Popen('find "j"', shell=True, stdin=p1.stdout, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
p1.stdout.close()
output,error=p2.communicate()
print(output,'<----->',error)
I also tried redirecting stderr=subprocess.STDOUT, but this didn't change things.
Can you please tell how to redirect the stderr from the first command, so I can see it in the stdout or stderr?
Regards,
To see stderr of the first command in stdout/stderr of the second command:
the second command could read stderr1 e.g., from its stdin2
the second command could print this content to its stdout2/stderr2
Example
#!/usr/bin/env python
import sys
from subprocess import Popen, PIPE, STDOUT
p1 = Popen([sys.executable, '-c', """import sys; sys.stderr.write("stderr1")"""],
stderr=STDOUT, stdout=PIPE)
p2 = Popen([sys.executable, '-c', """import sys
print(sys.stdin.read() + " stdout2")
sys.stderr.write("stderr2")"""],
stdin=p1.stdout, stderr=PIPE, stdout=PIPE)
p1.stdout.close()
output, error = p2.communicate()
p1.wait()
print(output, '<----->', error)
Output
('stderr1 stdout2\n', '<----->', 'stderr2')
For p2, you shouldn't be using subprocess.PIPE for either of the outputs because you're not piping it to another program.
Related
When I execute below program, it list file correctly.
import subprocess
foo = subprocess.run("ls /home/my_home",
shell=True,
executable="/bin/bash",
stdout=subprocess.PIPE,
stdin=subprocess.PIPE,
stderr=subprocess.PIPE)
my_std_out = foo.stdout.decode("utf-8")
But when execute below program, there is nothing in stdout.
import subprocess
foo = subprocess.Popen(["ls /home/my_home"],
shell=True,
executable="/bin/bash",
stdout=subprocess.PIPE,
stdin=subprocess.PIPE,
stderr=subprocess.PIPE)
my_std_out = foo.stdout.read().decode("utf-8")
I wonder is there anything wrong with my second part program?
Thankyou in advance!
From python docs:
"communicate() returns a tuple (stdout_data, stderr_data). The data will be strings if streams were opened in text mode; otherwise, bytes."
Therefore, if you'd like to get output via Popen, you have to unpack the retruned tuple from communicate() like this:
out, err = foo.communicate()
In [150]: out
Out[150]: b''
In [151]: err
Out[151]: b"ls: cannot access '/home/my_home': No such file or directory\n"
I think the bash command and the path should be placed between quotes each when you use brackets like the following
import subprocess foo = subprocess.Popen(["ls", "/home/my_home"], shell=True, executable=/bin/bash, stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE) my_std_out = foo.stdout.read().decode("utf-8")
When I am executing an utility, blab, and it will ask yes or no for confirmation, what can I do? Thanks,
The code is as below:
proc = subprocess.Popen("blab delete {}".format(num), shell=True,
stderr=subprocess.STDOUT, stdin=subprocess.STDIN)
stdout_value = proc.communicate()[0]
Popen.communicate() documentation:
If you want to send data to process's stdin using python, create the Popen object with stdin=PIPE. Similarly, to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE too.
from subprocess import PIPE, Popen, STDOUT
process = Popen("blab delete {}".format(num), shell=True, stdin=PIPE, stdout=PIPE, stderr=STDOUT)
output = process.communicate(input=b'yes')[0]
output = output.decode('utf-8')
I have a python script search for logs, it continuously output the logs found and I want to use linux pipe to filter the desired output. example like that:
$python logsearch.py | grep timeout
The problem is the sort and wc are blocked until the logsearch.py finishes, while the logsearch.py will continuous output the result.
sample logsearch.py:
p = subprocess.Popen("ping google.com", shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
for line in p.stdout:
print(line)
UPDATE:
figured out, just change the stdout in subprocess to sys.stdout, python will handle the pipe for you.
p = subprocess.Popen("ping -c 5 google.com", shell=True, stdout=**sys.stdout**)
Thanks for all of you help!
And why use grep? Why don't do all the stuff in Python?
from subprocess import Popen, PIPE
p = Popen(['ping', 'google.com'], shell=False, stdin=PIPE, stdout=PIPE)
for line in p.stdout:
if 'timeout' in line.split():
# Process the error
print("Timeout error!!")
else:
print(line)
UPDATE:
I change the Popen line as recommended #triplee. Pros and cons in Actual meaning of 'shell=True' in subprocess
I'm using multiple commands to run:
e.g. cd foo/bar; ../../run_this -arg1 -arg2="yeah_ more arg1 arg2" arg3=/my/path finalarg
Running with:
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE)
(out, err) = p.communicate()
But this spits output on screen (Python 2.7.5)
And out is empty string.
You have shell=True, so you're basically reading the standard output of the shell spawned, not the standard output of the program you want to run.
I'm guessing you're using shell=True to accommodate the directory changing. Fortunately, subprocess can take care of that for you (by passing a directory via the cwd keyword argument):
import subprocess
import shlex
directory = 'foo/bar'
cmd = '../../run_this -arg1 -arg2="yeah_ more arg1 arg2" arg3=/my/path finalarg'
p = subprocess.Popen(shlex.split(cmd), cwd=directory, stdout=subprocess.PIPE)
(out, err) = p.communicate()
As per comment I added stderr too and that worked!:
p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
It seems that using shell=True in the first process of a chain somehow drops the stdout from downstream tasks:
p1 = Popen(['echo','hello'], stdout=PIPE)
p2 = Popen('cat', stdin=p1.stdout, stdout=PIPE)
p2.communicate()
# outputs correctly ('hello\n', None)
Making the first process use shell=True kills the output somehow...
p1 = Popen(['echo','hello'], stdout=PIPE, shell=True)
p2 = Popen('cat', stdin=p1.stdout, stdout=PIPE)
p2.communicate()
# outputs incorrectly ('\n', None)
shell=True on the second process doesn't seem to matter. Is this expected behavior?
When you pass shell=True, Popen expects a single string argument, not a list. So when you do this:
p1 = Popen(['echo','hello'], stdout=PIPE, shell=True)
What happens is this:
execve("/bin/sh", ["/bin/sh", "-c", "echo", "hello"], ...)
That is, it calls sh -c "echo", and hello is effectively ignored (technically it becomes a positional argument to the shell). So the shell runs echo, which prints \n, which is why you see that in your output.
If you use shell=True, you need to do this:
p1 = Popen('echo hello', stdout=PIPE, shell=True)