I'm running a program via subprocess.Popen and running into an unexpected issue wherein the stdout does not print live, and instead waits for the program to finish. Oddly enough, this only occurs when the program being called is written in Python.
My control program (the one using subprocess) is as follows:
import subprocess
import os
print("Warming up")
pop = subprocess.Popen("python3 signaler.py", shell=True,stdout=subprocess.PIPE,stdin=subprocess.PIPE)
for line in pop.stdout:
print(line)
print("I'm done")
Signaler.py:
import time
print("Running")
time.sleep(5)
print("Done running")
When I run the control program, output is as follows. The code waits 5 seconds before printing Running, despite the fact that print("Running") occurs before the actual delay.
Warming up
*waits 5 seconds*
b'Running\n'
b'123\n'
I'm done
The strange thing is that when I modify the control program to instead run a Node program, the delay functions as expected and Running is printed 5 seconds before Done Running. The Node program is as follows:
const delay = require("delay")
async function run(){
console.log("Running")
await delay(5000)
console.log("Done running")
}
run()
This issue doesn't occur when I use os.system to call signaler.py, and still occurs when I run shell=False and modify the arguments as such. Any ideas what's causing this?
It sounds like python is buffering the output of the print function and javascript isn't. You can force print statements to be flushed to stdout by calling it with the flush keyword in signaler.py
print("Running", flush=True)
Related
This question already has answers here:
How to flush the input stream?
(4 answers)
Closed 1 year ago.
I'm making a game that runs on the console in python. When I exit the game, all the keys I pressed are automatically typed. How do I stop this from happening? Also, I still want to have user input when I use the input() function. This is on Windows by the way.
If you want the code, this has the same "problem":
for _ in range(100000):
print("Hello")
When the program finishes in the command prompt, this comes up:
C:\Users\User>awdsaawdsadwsaasdwaws
Basically, whatever keys were pressed while the code was running. This happens when other things run in the command prompt too, but I want to know how to disable it in python.
Edit: I kept digging and found that what I was looking for was flushing or clearing keyboard buffer. I marked my question as a duplicate of another which has a few answers, but this one worked best for me:
def flush_input():
try:
import msvcrt
while msvcrt.kbhit():
msvcrt.getch()
except ImportError:
import sys, termios #for linux/unix
termios.tcflush(sys.stdin, termios.TCIOFLUSH)
This happens because your computer registers the key strokes and on the console, those are made available on the stdin input stream.
If you save your script as test.py and run python test.py and start entering some keystrokes, like abc, those letters will be on standard input.
Your script doesn't read them, because it doesn't touch that stream, as you're not using input() or any other calls that would read that stream. So your script finishes, the characters are still on standard input, the prompt comes back and it reads those characters, with the given result:
Hello
Hello
Hello
PS C:\Users\username> abc
To avoid this, you can read / flush the input buffer at the end of your script. However, this is surprisingly hard if you need it to work across all operating systems and in different modes of running your script (directly from cmd, IDLE, in other IDEs, etc.)
The problem is there's no way to know if there's input on the standard input stream, until you try to read from it. But if you try to read from it, your script will pause until an 'end of line' or 'end of file' is received. And if the user is just hitting keys, that won't happen, so you'll end up reading until they hit something like Ctrl+Break or Ctrl+C.
Here's a way I think is relatively robust, but I recommend you test it in scenarios and environments you consider likely for use of your script:
import sys
import threading
import queue
import os
import signal
for _ in range(100000):
print("Hello")
timeout = 0.1 # sec
def no_input():
# stop main thread (which is probably blocked reading input) via an interrupt signal
# only available for windows in Python version 3.2 or higher
os.kill(os.getpid(), signal.SIGINT)
exit()
# if a sigint is received, exit the main thread (you could call another function to do work first and then exit()
signal.signal(signal.SIGINT, exit)
# input is stored here, until it's dealt with
input_queue = queue.Queue()
# read all available input until main thread exit
def get_input():
while True:
try:
# read input, not doing anything with it
_ = input_queue.get(timeout=timeout)
except queue.Empty:
no_input()
reading_thread = threading.Thread(target=get_input)
reading_thread.start()
# main loop: put any available input in the queue, will wait for input if there is none
for line in sys.stdin:
input_queue.put(line)
# wait for reading thread
reading_thread.join()
It basically reads the input from a second thread, allowing that the main thread to get the input and possibly do something with it until there's nothing left and then it just tells the main thread to exit. Note that this will result in your script exiting with an exit code of 2, which may not be what you want.
Also note that you'll still see the input on screen, but it will no longer be passed to the terminal:
Hello
Hello
Hello
abc
PS C:\Users\username>
I don't know if there's an easy way to avoid the echo, other than on Linux doing something like stty -echo. You could of course just call the system to clear the screen at the end of your script:
from subprocess import call
from os import name as os_name
call('clear' if os_name =='posix' else 'cls')
I'm having a problem with subprocess poll not returning the return code when the process has finished.
I found out how to set a timeout on subprocess.Popen and used that as the basis for my code. However, I have a call that uses Java that doesn't correctly report the return code so each call "times out" even though it is actually finished. I know the process has finished because when removing the poll timeout check, the call runs without issue returning a good exit code and within the time limit.
Here is the code I am testing with.
import subprocess
import time
def execute(command):
print('start command: {}'.format(command))
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print('wait')
wait = 10
while process.poll() is None and wait > 0:
time.sleep(1)
wait -= 1
print('done')
if wait == 0:
print('terminate')
process.terminate()
print('communicate')
stdout, stderr = process.communicate()
print('rc')
exit_code = process.returncode
if exit_code != 0:
print('got bad rc')
if __name__ == '__main__':
execute(['ping','-n','15','127.0.0.1']) # correctly times out
execute(['ping','-n','5','127.0.0.1']) # correctly runs within the time limit
# incorrectly times out
execute(['C:\\dev\\jdk8\\bin\\java.exe', '-jar', 'JMXQuery-0.1.8.jar', '-url', 'service:jmx:rmi:///jndi/rmi://localhost:18080/jmxrmi', '-json', '-q', 'java.lang:type=Runtime;java.lang:type=OperatingSystem'])
You can see that two examples are designed to time out and two are not to time out and they all work correctly. However, the final one (using jmxquery to get tomcat metrics) doesn't return the exit code and therefore "times out" and has to be terminated, which then causes it to return an error code of 1.
Is there something I am missing in the way subprocess poll is interacting with this Java process that is causing it to not return an exit code? Is there a way to get a timeout option to work with this?
This has the same cause as a number of existing questions, but the desire to impose a timeout requires a different answer.
The OS deliberately gives only a small amount of buffer space to each pipe. When a process writes to one that is full (because the reader has not yet consumed the previous output), it blocks. (The reason is that a producer that is faster than its consumer would otherwise be able to quickly use a great deal of memory for no gain.) Therefore, if you want to do more than one of the following with a subprocess, you have to interleave them rather than doing each in turn:
Read from standard output
Read from standard error (unless it’s merged via subprocess.STDOUT)
Wait for the process to exit, or for a timeout to elapse
Of course, the subprocess might close its streams before it exits, write useful output after you notice the timeout and before you kill it, and/or start additional processes that keep the pipe open indefinitely, so you might want to have multiple timeouts. Probably what’s most informative is the EOF on the pipe, so repeatedly use something like select to wait for (however much is left of) the timeout, issue single reads on the streams that are ready, and wait (with another timeout if you’re concerned about hangs after an early stream closure) on EOF. If the timeout occurs instead, (try to) kill the subprocess, and consider issuing non-blocking reads (or another timeout loop) to get any last available output before closing the pipes.
Using the other answer by #DavisHerring as the basis for more research, I came across a concept that worked for my original case. Here is the code that came out of that.
import subprocess
import threading
import time
def execute(command):
print('start command: {}'.format(command))
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
timer = threading.Timer(10, terminate_process, [process])
timer.start()
print('communicate')
stdout, stderr = process.communicate()
print('rc')
exit_code = process.returncode
timer.cancel()
if exit_code != 0:
print('got bad rc')
def terminate_process(p):
try:
p.terminate()
except OSError:
pass # ignore error
It uses the threading.Timer to make sure that the process doesn't go over the time limit and terminates the process if it does. It otherwise waits for a response back and cancels the timer once it finishes.
I'm trying to write some basic tests for a piece of code that normally accepts input endlessly through stdin until given a specific exit command.
I want to check if the program crashes on being given some input string (after some amount of time to account for processing), but can't seem to figure out how to send data and not be stuck waiting for output which I don't care about.
My current code looks like this (using cat as an example of the program):
myproc = subprocess.Popen(['cat'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
myproc.communicate(input=inputdata.encode("utf-8"))
time.sleep(0.1)
if myproc.poll() != None:
print("not running")
else:
print("still running")
How can I modify this to allow the program to proceed to the polling instead of hanging after the communicate() call?
You are using the wrong tool here with communicate which waits for the end of the program. You should simply feed the standard input of the subprocess:
myproc = subprocess.Popen(['cat'], stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
myproc.stdin.write(inputdata.encode("utf-8"))
time.sleep(0.1)
if myproc.poll() != None:
print("not running")
else:
print("still running")
But beware: you cannot be sure that the output pipes will contain anything before the end of the subprocess...
You could set a timeout in the Popen.communicate(input=None, timeout=None) function. After the timeout the process is still running and I think but you have to test it you can still send in input with communicate.
From the docs:
If the process does not terminate after timeout seconds, a TimeoutExpired exception will be raised. Catching this exception and retrying communication will not lose any output.
The child process is not killed if the timeout expires, so in order to
cleanup properly a well-behaved application should kill the child
process and finish communication:
I think I understand what you want here. If you know an existing command that will crash your program, you can use subprocess.Popen.wait() and it'll still block, but it'll return a tuple of the output message and the error associated with it, if any.
Then you can note the error and catch it in a try exception statement.
This was really helpful when I was working with sub processes:
https://docs.python.org/3/library/asyncio-subprocess.html
I'm executing a function as a thread in python. Now, the program will wait for the function to execute and then terminate after its completion.
My target is to starting the background thread and closing the program calling it.
how can we do it. As in below code, the thread will take 30 min to execute. I want to stop the main program after calling the thread and let the thread run in background.
thread = threading.Thread(target=function_that_runs_for_30_min)
thread.start()
print "Thread Started"
quit()
You cannot do that directly. A thread is just a part of a process. Once the process exits, all the threads are gone. You need to create a background process to achieve that.
You cannot use the multiprocessing module either because it is a package that supports spawning processes using an API similar to the threading module (emphasize mine). As such it has no provision to allow a process to run after the end of the calling one.
The only way I can imagine is to use the subprocess module to restart the script with a specific parameter. For a simple use case, adding a parameter is enough, for more complex command line parameters, the module argparse should be used. Example of code:
import subprocess
import sys
# only to wait some time...
import time
def f(name):
"Function that could run in background for a long time (30')"
time.sleep(5)
print 'hello', name
if __name__ == '__main__':
if (len(sys.argv) > 1) and (sys.argv[1] == 'SUB'):
# Should be an internal execution: start the lengthy function
f('bar')
else:
# normal execution: start a subprocess with same script to launch the function
p = subprocess.Popen("%s %s SUB" % (sys.executable, sys.argv[0]))
# other processing...
print 'END of normal process'
Execution:
C:\>python foo.py
END of normal process
C:\>
and five seconds later:
hello bar
Let's say that I have this simple line in python:
os.system("sudo apt-get update")
of course, apt-get will take some time untill it's finished, how can I check in python if the command had finished or not yet?
Edit: this is the code with Popen:
os.environ['packagename'] = entry.get_text()
process = Popen(['dpkg-repack', '$packagename'])
if process.poll() is None:
print "It still working.."
else:
print "It finished"
Now the problem is, it never print "It finished" even when it really finish.
As the documentation states it:
This is implemented by calling the Standard C function system(), and
has the same limitations
The C call to system simply runs the program until it exits. Calling os.system blocks your python code until the bash command has finished thus you'll know that it is finished when os.system returns. If you'd like to do other stuff while waiting for the call to finish, there are several possibilities. The preferred way is to use the subprocessing module.
from subprocess import Popen
...
# Runs the command in another process. Doesn't block
process = Popen(['ls', '-l'])
# Later
# Returns the return code of the command. None if it hasn't finished
if process.poll() is None:
# Still running
else:
# Has finished
Check the link above for more things you can do with Popen
For a more general approach at running code concurrently, you can run that in another thread or process. Here's example code:
from threading import Thread
...
thread = Thread(group=None, target=lambda:os.system("ls -l"))
thread.run()
# Later
if thread.is_alive():
# Still running
else:
# Has finished
Another option would be to use the concurrent.futures module.
os.system will actually wait for the command to finish and return the exit status (format dependent format).
os.system is blocking; it calls the command waits for its completion, and returns its return code.
So, it'll be finished once os.system returns.
If your code isn't working, I think that could be caused by one of sudo's quirks, it refuses to give rights on certain environments(I don't know the details tho.).