I was have to call a shell command
subprocess.Popen(command, stderr=subprocess.STDOUT, stdout=subprocess.PIPE)
I did it. and:
and, that command is prints lots of things like verbose is on, and then when its done it's job it prints (writes) blah blah : Ready
I have to call this command, wait for the 'Ready' text and leave it running on background, then let the rest of the code run
I tried this and things like this, didn't work
...
done=False
with subprocess.Popen(command, stderr=subprocess.STDOUT, stdout=subprocess.PIPE) as proc:
while not done:
x=proc.stdout.read()
x=x.find('Ready')
if x > -1:
done=True
print("YaaaY, this subprocess is done it's job and now running on background")
#rest of the code
i ran similar (edited) code on python terminal and I think I can't even access (read) the terminal of the subprocess. because...
I was expecting it will show every line that this subprocess print but. its just waiting.
Your problem is proc.stdout.read(). This reads the entire output of your subprocess, which is not known until it has terminated (usually). Try something like:
output = b''
while not done:
output += proc.stdout.read(1)
x = output.find(b'Ready')
if x > -1:
done = True
This reads the process's stdout one character at a time, so it doesn't have to wait for it to finish.
Note that using Popen in a context manager (with block) will cause your program to wait for the child process to terminate before it exits the with block, so it will not leave it running past that point. It's unclear if that is desired behaviour or not.
Related
I'm trying to send the cmd many commands according to the answers he sends me.
I'm getting a run time error message:
ValueError: I/O operation on closed file
When I'm running something like this:
import subprocess
process = subprocess.Popen("cmd.exe", stdout=subprocess.PIPE,stdin=subprocess.PIPE)
answer = process.communicate(input="some command\n" + '\n')[0]
"""
choosing another command according to answer
"""
print process.communicate(input=another_command + '\n')[0]
process.kill()
Any idea on how to solve the problem?
Thank you for your help!
Do not send your commands to cmd.exe. Call your commands directly like:
subprocess.Popen("dir", shell=True, stdout=subprocess.PIPE,stdin=subprocess.PIPE)
Perhaps you will not need the pipe for stdin if you use it this way.
The error is normal. communicate closes the standard input of the subprocess to indicate that no more input is pending so that the subprocess can flush its output. So you cannot chain multiple communicate calls on one single subprocess.
But if you commands are simple enough (not many kbytes of input data), and if you do not need to collect and process the output of one command before sending the next one, you should be able to write all the commands in sequence, reading as much output as possible between two of them. After last command, you could then close the subprocess standard input and wait for it to terminate, still collating the output:
process = subprocess.Popen("cmd.exe", stdout=subprocess.PIPE, stdin=subprocess.PIPE)
process.stdin.write("some command\n\n")
partial_answer = process.stdout.read() # all or part of the answer can still be buffered subprocess side
...
process.stdin.write("some other command\n\n")
...
# after last command, time to close subprocess
process.stdin.close()
retcode = None
while True:
end_of_answer += process.stdout.read()
if retcode is not None: break
child = subprocess.Popen(command,
shell=True,
env=environment,
close_fds=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
stdin=sys.stdin,
preexec_fn=os.setsid
)
child_interrupted = False
while child.poll() is None:
if Signal.isInterrupted():
child_interrupted = True
os.killpg(os.getpgid(child.pid), signal.SIGTERM)
break
time.sleep(0.1)
subout = child.communicate()[0]
logging.info(subout)
the above works for most command it executes (90%) but for some commands it hangs
for those command that repeatedly hangs, if i get rid of the below, it works fine:
child_interrupted = False
while child.poll() is None:
if Signal.isInterrupted():
child_interrupted = True
os.killpg(os.getpgid(child.pid), signal.SIGTERM)
break
time.sleep(0.1)
im assuming for those hanging commands, child.poll() is None even though the job is finished??
communicate() can tell the process is finished but poll() cant?
i've executed ps -ef on those processes
and they are defunct only when child.poll() code is in place
any idea why?
it looks like defunct means "That's a zombie process, it's finished but the parent hasn't wait()ed for it yet."
well, im polling to see if i can call wait/communitcate...
You've set the Popen object to receive the subprocess's stdout via pipe. Problem is, you're not reading from that pipe until the process exits. If the process produces enough output to fill the OS level pipe buffers, and you don't drain the pipe, then you're deadlocked; the subprocess wants you to read the output its writing so it can continue to write, then exit, while you're waiting for it to exit before you'll read the output.
If your explicit poll and interrupt checking is necessary, the easiest solution to this deadlock is probably to launch a thread that drains the pipe:
... launch the thread just after Popen called ...
draineddata = []
# Trivial thread just reads lines from stdout into the list
drainerthread = threading.Thread(target=draineddata.extend, args=(child.stdout,))
drainerthread.daemon = True
drainerthread.start()
... then where you had been doing communicate, change it to: ...
child.wait()
drainerthread.join()
subout = b''.join(draineddata) # Combine the data read back to a single output
I'm using the following command to run a shell command (creating a subprocess):
cmd = "ls"
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True)
Then, I want to get its return code when it's finished. I should use wait() or poll()? It seems to me wait() is equal to a poll() enclosed in a busy wait. Something like:
while process.poll() == None:
time.sleep(0.5)
I read that wait() could generate a deadlock if stdout/stderr buffer is filled. process.poll() used like above also could generate a deadlock? If this is true,
I should use process.comunicate() to solve the problem? Nowadays, I only use
process.comunicate() when I'm interested in the subprocess stdout/stderr.
Thanks in advance.
Yes. subprocess.poll, when used in a loop like that, will cause a deadlock if the stdout is piped into your process and you aren't reading from it. If you don't pipe stdout or you're consistently reading from it, neither poll nor wait will deadlock. subprocess.communicate will solve the deadlock in the cases it would occur. However, if you just want to run a command, check its return code, and get its output, use subprocess.check_output, which wraps all of that.
I am working on some scripts (in the company I work in) that are loaded/unloaded into hypervisors to fire a piece of code when an event occurs. The only way to actually unload a script is to hit Ctrl-C. I am writing a function in Python that automates the process
As soon as it sees the string "done" in the output of the program, it should kill the vprobe.
I am using subprocess.Popen to execute the command:
lineList = buff.readlines()
cmd = "vprobe /vprobe/myhello.emt"
p = subprocess.Popen(args = cmd, shell=True,stdout = buff, universal_newlines = True,preexec_fn=os.setsid)
while not re.search("done",lineList[-1]):
print "waiting"
os.kill(p.pid,signal.CTRL_C_EVENT)
As you can see, I am writing the output in buff file descriptor opened in read+write mode. I check the last line; if it has 'done', I kill it. Unfortunately, the CTRL_C_EVENT is only valid for Windows.
What can I do for Linux?
I think you can just send the Linux equivalent, signal.SIGINT (the interrupt signal).
(Edit: I used to have something here discouraging the use of this strategy for controlling subprocesses, but on more careful reading it sounds like you've already decided you need control-C in this specific case... So, SIGINT should do it.)
In Linux, Ctrl-C keyboard interrupt can be sent programmatically to a process using Popen.send_signal(signal.SIGINT) function. For example
import subprocess
import signal
..
process = subprocess.Popen(..)
..
process.send_signal(signal.SIGINT)
..
Don't use Popen.communicate() for blocking commands..
Maybe I misunderstand something, but the way you do it it is difficult to get the desired result.
Whatever buff is, you query it first, then use it in the context of Popen() and then you hope that by maciv lineList fills itself up.
What you probably want is something like
logfile = open("mylogfile", "a")
p = subprocess.Popen(['vprobe', '/vprobe/myhello.emt'], stdout=subprocess.PIPE, buff, universal_newlines=True, preexec_fn=os.setsid)
for line in p.stdout:
logfile.write(line)
if re.search("done", line):
break
print "waiting"
os.kill(p.pid, signal.CTRL_C_EVENT)
This gives you a pipe end fed by your vprobe script which you can read out linewise and act appropriately upon the found output.
I'm running multiple commands which may take some time, in parallel, on a Linux machine running Python 2.6.
So, I used subprocess.Popen class and process.communicate() method to parallelize execution of mulitple command groups and capture the output at once after execution.
def run_commands(commands, print_lock):
# this part runs in parallel.
outputs = []
for command in commands:
proc = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.STDOUT, close_fds=True)
output, unused_err = proc.communicate() # buffers the output
retcode = proc.poll() # ensures subprocess termination
outputs.append(output)
with print_lock: # print them at once (synchronized)
for output in outputs:
for line in output.splitlines():
print(line)
At somewhere else it's called like this:
processes = []
print_lock = Lock()
for ...:
commands = ... # a group of commands is generated, which takes some time.
processes.append(Thread(target=run_commands, args=(commands, print_lock)))
processes[-1].start()
for p in processes: p.join()
print('done.')
The expected result is that each output of a group of commands is displayed at once while execution of them is done in parallel.
But from the second output group (of course, the thread that become the second is changed due to scheduling indeterminism), it begins to print without newlines and adding spaces as many as the number of characters printed in each previous line and input echo is turned off -- the terminal state is "garbled" or "crashed". (If I issue reset shell command, it restores normal.)
At first, I tried to find the reason from handling of '\r', but it was not the reason. As you see in my code, I handled it properly using splitlines(), and I confirmed that with repr() function applied to the output.
I think the reason is concurrent use of pipes in Popen and communicate() for stdout/stderr. I tried check_output shortcut method in Python 2.7, but no success. Of course, the problem described above does not occur if I serialize all command executions and prints.
Is there any better way to handle Popen and communicate() in parallel?
A final result inspired by the comment from J.F.Sebastian.
http://bitbucket.org/daybreaker/kaist-cs443/src/247f9ecf3cee/tools/manage.py
It seems to be a Python bug.
I am not sure it is clear what run_commands needs to be actually doing, but it seems to be simply doing a poll on a subprocess, ignoring the return-code and continuing in the loop. When you get to the part where you are printing output, how could you know the sub-processes have completed?
In your example code I noticed your use of:
for line in output.splitlines():
to address partially the issue of " /r " ; use of
for line in output.splitlines(True):
would have been helpful.