I was reading about std.flush() in python. And I found this example a lot.
import sys,time
for i in range(10):
print i,
#sys.stdout.flush()
time.sleep(1)
It is often said that it makes a difference with/without the "sys.stdout.flush()".
However, when I called this script from command prompt, it didn't make a difference in my case. Both printed numbers to the screen in real time.
I used python 2.7.5 in windows.
Why is that happening?
p.s. In another example which printed the output through subprocess.PIPE instead of to the screen directly, I did observe a difference of the buffering.
What am I missing?
Using flush will generally guarantee that flushing is done but assuming the reverse relationship is a logical fallacy, akin to:
Dogs are animals.
This is an animal.
Therefore this is a dog.
In other words, not using flush does not guarantee flushing will not happen.
Interestingly enough, using Python 2.7.8 under Cygwin in Win81, I see the opposite behaviour - everything is batched up until the end. It may be different with Windows-native Python, it may also be different from within IDLE.
See stdio buffering. In brief:
Default Buffering modes:
stdin is always buffered
stderr is always unbuffered
if stdout is a terminal then buffering is automatically set to line buffered, else it is set to buffered
For me, the example you gave prints:
In cmd:
all the numbers upon exit in Cygwin's python
one by one in Win32 python
In mintty:
both all upon exit
both one by one with -u option
sys.stdout.isatty() returns False!
So, it looks like msvcrt's stdout is unbuffered when it points to a terminal. A test with a simple C program shows the same behaviour.
Related
I am using a 3rd-party python module which is normally called through terminal commands. When called through terminal commands it has a verbose option which prints to terminal in real time.
I then have another python program which calls the 3rd-party program through subprocess. Unfortunately, when called through subprocess the terminal output no longer flushes, and is only returned on completion (the process takes many hours so I would like real-time progress).
I can see the source code of the 3rd-party module and it does not set printing to be flushed such as print('example', flush=True). Is there a way to force the flushing through my module without editing the 3rd-party source code? Furthermore, can I send this output to a log file (again in real time)?
Thanks for any help.
The issue is most likely that many programs work differently if run interactively in a terminal or as part of a pipe line (i.e. called using subprocess). It has very little to do with Python itself, but more with the Unix/Linux architecture.
As you have noted, it is possible to force a program to flush stdout even when run in a pipe line, but it requires changes to the source code, by manually applying stdout.flush calls.
Another way to print to screen, is to "trick" the program to think it is working with an interactive terminal, using a so called pseudo-terminal. There is a supporting module for this in the Python standard library, namely pty. Using, that, you will not explicitly call subprocess.run (or Popen or ...). Instead you have to use the pty.spawn call:
def prout(fd):
data = os.read(fd, 1024)
while(data):
print(data.decode(), end="")
data = os.read(fd, 1024)
pty.spawn("./callee.py", prout)
As can be seen, this requires a special function for handling stdout. Here above, I just print it to the terminal, but of course it is possible to do other thing with the text as well (such as log or parse...)
Another way to trick the program, is to use an external program, called unbuffer. Unbuffer will take your script as input, and make the program think (as for the pty call) that is called from a terminal. This is arguably simpler if unbuffer is installed or you are allowed to install it on your system (it is part of the expect package). All you have to do then, is to change your subprocess call as
p=subprocess.Popen(["unbuffer", "./callee.py"], stdout=subprocess.PIPE)
and then of course handle the output as usual, e.g. with some code like
for line in p.stdout:
print(line.decode(), end="")
print(p.communicate()[0].decode(), end="")
or similar. But this last part I think you have already covered, as you seem to be doing something with the output.
I am working on a French chatbot using python. For a first text-to-speech attempt, I am using espeak with mbrola. I call it with subprocess :
from subprocess import run, DEVNULL
def speak(text):
command = ["espeak", "-vmb-fr1", text]
run(command, stderr=DEVNULL, stdout=DEVNULL)
speak("Bonjour.")
As you see, I'm sending stderr and stdout to /dev/null
When I run the program, It seems to work, espeak is speaking, but I get this :
*** Error in `mbrola': free(): invalid pointer: 0x08e3af18 ***
*** Error in `mbrola': free(): invalid pointer: 0x0988af88 ***
I think it is a C error in mbrola. I think I can't fix it. But it works, so I just want to mute the error. How can I do ? Is there a way ?
Edit, in response to abarnert :
When I redirect stdout and stderr by the shell (python myscript.py 2>&1 >/dev/null), the message still show up.
distro : Debian 9.3
glibc version : 2.24
Run it with setsid (just add that string in front of the command and arguments). That will stop it from opening /dev/tty to report the malloc errors. It will also prevent terminal signals, including SIGHUP when the terminal is closed, from affecting the process, which may be a good or a bad thing.
Alternatively, set the environment variable LIBC_FATAL_STDERR_ to some nonempty string, with whose name I was able to find several similar questions.
The root problem is that mbrola/espeak has a serious bug with memory allocation. If you haven't checked for a new version, and reported the bug to them, that's the first thing you should do.
These warnings are emitted by glibc's malloc checker, which is described in the mallopt docs. If heap checking is enabled, every detected error with malloc (and free and related functions) will be printed out to stderr, but if it's disabled, nothing will be done. (Other possibilities are available as well, but that's not relevant here.)
According to the documentation, unless the program explicitly calls mallopt, either setting the environment variable MALLOC_CHECK_ to 0 or not setting it at all should mean no malloc debug output. However, most of the major distros (starting with Debian) have long shipped a glibc that's configured to default to 1 (meaning print the error message) instead of 0. You can still override this by explicitly setting MALLOC_CHECK_=0.
Also, the documentation implies that malloc errors go to stderr unless malloc_printerr is replaced. But again, many distros do replace it with an intentionally-harder-to-ignore function that logs to the current process's tty if pretend and stderr if not. This is why it shows up even if you pipe espeak's stderr to /dev/null, and your own program's as well.
So, to hide these errors, you can:
Set the environment variable MALLOC_CHECK_ to 0 in espeak, which will disable the checks.
Prevent espeak from opening a tty, which means the checks will still happen, but the output will have nowhere to go.
Using setsid, a tool that calls setsid at the start of the new process, is one way to do the latter. Whether that's a good idea or not depends on whether you want the process to lead its own process group. You really should read up on what that means and decide what you want, not choose between the options because typing setsid is shorter than typing MALLOC_CHECK_=0.
And again, you really should check for a new version first, and report this bug upstream if they haven't fixed it yet.
I have recently came across a few posts on stack overflow saying that subprocess is much better than os.system, however I am having difficulty finding the exact advantages.
Some examples of things I have run into:
https://docs.python.org/3/library/os.html#os.system
"The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function."
No idea in what ways it is more powerful though, I know it is easier in many ways to use subprocess but is it actually more powerful in some way?
Another example is:
https://stackoverflow.com/a/89243/3339122
The advantage of subprocess vs system is that it is more flexible (you can get the stdout, stderr, the "real" status code, better error handling, etc...).
This post which has 2600+ votes. Again could not find any elaboration on what was meant by better error handling or real status code.
Top comment on that post is:
Can't see why you'd use os.system even for quick/dirty/one-time. subprocess seems so much better.
Again, I understand it makes some things slightly easier, but I hardly can understand why for example:
subprocess.call("netsh interface set interface \"Wi-Fi\" enable", shell=True)
is any better than
os.system("netsh interface set interface \"Wi-Fi\" enabled")
Can anyone explain some reasons it is so much better?
First of all, you are cutting out the middleman; subprocess.call by default avoids spawning a shell that examines your command, and directly spawns the requested process. This is important because, besides the efficiency side of the matter, you don't have much control over the default shell behavior, and it actually typically works against you regarding escaping.
In particular, do not do this:
subprocess.call('netsh interface set interface "Wi-Fi" enable')
since
If passing a single string, either shell must be True (see below) or else the string must simply name the program to be executed without specifying any arguments.
Instead, you'll do:
subprocess.call(["netsh", "interface", "set", "interface", "Wi-Fi", "enable"])
Notice that here all the escaping nightmares are gone. subprocess handles escaping (if the OS wants arguments as a single string - such as Windows) or passes the separated arguments straight to the relevant syscall (execvp on UNIX).
Compare this with having to handle the escaping yourself, especially in a cross-platform way (cmd doesn't escape in the same way as POSIX sh), especially with the shell in the middle messing with your stuff (trust me, you don't want to know what unholy mess is to provide a 100% safe escaping for your command when calling cmd /k).
Also, when using subprocess without the shell in the middle you are sure you are getting correct return codes. If there's a failure launching the process you get a Python exception, if you get a return code it's actually the return code of the launched program. With os.system you have no way to know if the return code you get comes from the launched command (which is generally the default behavior if the shell manages to launch it) or it is some error from the shell (if it didn't manage to launch it).
Besides arguments splitting/escaping and return code, you have way better control over the launched process. Even with subprocess.call (which is the most basic utility function over subprocess functionalities) you can redirect stdin, stdout and stderr, possibly communicating with the launched process. check_call is similar and it avoids the risk of ignoring a failure exit code. check_output covers the common use case of check_call + capturing all the program output into a string variable.
Once you get past call & friends (which is blocking just as os.system), there are way more powerful functionalities - in particular, the Popen object allows you to work with the launched process asynchronously. You can start it, possibly talk with it through the redirected streams, check if it is running from time to time while doing other stuff, waiting for it to complete, sending signals to it and killing it - all stuff that is way besides the mere synchronous "start process with default stdin/stdout/stderr through the shell and wait it to finish" that os.system provides.
So, to sum it up, with subprocess:
even at the most basic level (call & friends), you:
avoid escaping problems by passing a Python list of arguments;
avoid the shell messing with your command line;
either you have an exception or the true exit code of the process you launched; no confusion about program/shell exit code;
have the possibility to capture stdout and in general redirect the standard streams;
when you use Popen:
you aren't restricted to a synchronous interface, but you can actually do other stuff while the subprocess run;
you can control the subprocess (check if it is running, communicate with it, kill it).
Given that subprocess does way more than os.system can do - and in a safer, more flexible (if you need it) way - there's just no reason to use system instead.
There are many reasons, but the main reason is mentioned directly in the docstring:
>>> os.system.__doc__
'Execute the command in a subshell.'
For almost all cases where you need a subprocess, it is undesirable to spawn a subshell. This is unnecessary and wasteful, it adds an extra layer of complexity, and introduces several new vulnerabilities and failure modes. Using subprocess module cuts out the middleman.
I'm writing a pygtk application in Python 2.7.5 that requires some heavy mathematical calculations, so I need to do these calculations in an external pypy (that don't support gtk) for efficiency and plot the results in the main program as they are produced.
Since the ouput of the calculations is potentially infinite and I want to show it as it is produced, I cannot use subprocess.Popen.communicate(input).
I am able to do non-blocking reads of the output (via fcntl), but I am not able to effectively send the input (or anyway something else that I don't see is going wrong). For ex, the following code:
import subprocess
# start pypy subprocess
pypy = subprocess.Popen(['pypy', '-u'], bufsize=0, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
# send input to pypy
pypy.stdin.write('import sys\nprint "hello"\nsys.stdout.flush()\n')
pypy.stdin.flush()
# read output from pypy
pypy.stdout.flush()
print pypy.stdout.readline()
Will get stuck on the last line. What is weird to me is that if I substitute 'pypy' with 'cat' it will work, and if I substitute the input-output lines with
print pypy.communicate(input='import sys\nprint "hello"\nsys.stdout.flush()\n')[0]
it will also work (but it does not fit with what I want to do). I thought it was a problem of buffering, but I tried several ways of avoiding it (including writing to stderr and so) with no luck. I also tried sending to pypy the command to print in a while True loop, also with no luck (that makes me think that is not a problem with output buffering but maybe with input buffering).
I need to monitor a screen session in real time using a Python script. It needs to know when the display changes. I believe this can be described as whenever stdout is flushed, or a character is entered to stdin. Is there some way to do this; perhaps with pipes?
I have some code found here that gets a character from stdin, and I assume works on a pipe (if I modify the code, or change sys.stdin)?
Does the flush function of a stream (like stdout) get called in a pipe, or is it just called explicitly? My understanding is that the display is only updated when stdout is flushed.
Probably you want to take a look at script, which already does pretty much everything you want.
Have you tried python curses? It is similar of Linux curses and provides a good way to handle terminal related i/o.