What does sys.stdout.flush() do?
Python's standard out is buffered (meaning that it collects some of the data "written" to standard out before it writes it to the terminal). Calling sys.stdout.flush() forces it to "flush" the buffer, meaning that it will write everything in the buffer to the terminal, even if normally it would wait before doing so.
Here's some good information about (un)buffered I/O and why it's useful:
http://en.wikipedia.org/wiki/Data_buffer
Buffered vs unbuffered IO
Consider the following simple Python script:
import time
import sys
for i in range(5):
print(i),
#sys.stdout.flush()
time.sleep(1)
This is designed to print one number every second for five seconds, but if you run it as it is now (depending on your default system buffering) you may not see any output until the script completes, and then all at once you will see 0 1 2 3 4 printed to the screen.
This is because the output is being buffered, and unless you flush sys.stdout after each print you won't see the output immediately. Remove the comment from the sys.stdout.flush() line to see the difference.
As per my understanding, When ever we execute print statements output will be written to buffer. And we will see the output on screen when buffer get flushed(cleared). By default buffer will be flushed when program exits. BUT WE CAN ALSO FLUSH THE BUFFER MANUALLY by using "sys.stdout.flush()" statement in the program. In the below code buffer will be flushed when value of i reaches 5.
You can understand by executing the below code.
chiru#online:~$ cat flush.py
import time
import sys
for i in range(10):
print i
if i == 5:
print "Flushing buffer"
sys.stdout.flush()
time.sleep(1)
for i in range(10):
print i,
if i == 5:
print "Flushing buffer"
sys.stdout.flush()
chiru#online:~$ python flush.py
0 1 2 3 4 5 Flushing buffer
6 7 8 9 0 1 2 3 4 5 Flushing buffer
6 7 8 9
import sys
for x in range(10000):
print "HAPPY >> %s <<\r" % str(x),
sys.stdout.flush()
As per my understanding sys.stdout.flush() pushes out all the data that has been buffered to that point to a file object.
While using stdout, data is stored in buffer memory (for some time or until the memory gets filled) before it gets written to terminal. Using flush() forces to empty the buffer and write to terminal even before buffer has empty space.
You can see the differences b/w these two
import sys
for i in range(1,10 ):
sys.stdout.write(str(i))
sys.stdout.flush()
for i in range(1,10 ):
print i
In the first case, the characters are output one by one after each is written, because of the flush. In the second case, the characters are buffered by Python until it thinks it's got something worth the effort to write, and then written all in a batch.
If you add, say, a time.sleep(0.2) in the loops, this becomes more obvious.
Imagine you have a toy box where you keep all your toys. When you want to play with a toy, you usually take it out of the box, right?
In a similar way, when you write something on the computer, it gets stored in a temporary place called a buffer. Think of it like a toy box for computer data. The computer waits until the toy box is full before it takes the data out of the box and shows it to you on the screen.
However, sometimes you might want to play with a toy right away, without waiting for the toy box to fill up with other toys. This is like when you call sys.stdout.flush() in Python. It tells the computer to take the data out of the buffer and show it to you on the screen right away, without waiting for the buffer to fill up.
Related
I have a c exectuable that I want to exploit.
The output of that file looks like this:
$ ./vuln_nostack
Enter some text:
enteringTEXT
You entered: enteringTEXT
You enter some text, and the program spits it back.
I want to run this prorgam (and later exploit it) with python and pwntools.
So far, the functioning part of my pwntools program looks like this:
from concurrent.futures import process
from sys import stdout
from pwn import *
import time
pty = process.PTY
p = process("./vuln_nostack", stdin=pty, stdout=pty)
ss = p.recv()
p.clean()
asstring = ss.decode("utf-8")
print(asstring)
This works fine, it gets the first line and then prints it.
What I want to do now is to send a message to the program and then get the final line.
I have tried something along these lines:
p.send(b"dong")
p.clean()
print(p.recv())
I'm not sure whether or not the send actually ever sends anything, but as soon as I add the recv function, the prorgam just hangs and never finishes.
My guess is that the input to the executable is never given properly, and therefore it's still just waiting.
How do I actually get a message delivered to the exectuable so that it can move on and srever me the last line?
You can also use p.sendline():
p.sendline("payload")
This automatically adds a breakline after your bytes.
Moreover, to know whether your exploit is sending/receiving messages to/from the program, you can use debug context by adding this assignment:
context.log_level = 'debug'
The answer was a lot more simple than formerly presumed.
I just needed a breakline in the send:
p.send("payload \n")
I am trying to make a simple program in komodo edit using python that when it runs will print out 10 seconds worth of time in the command output.
The code is as follows:
import time
seconds = 0
while seconds != 10:
time.sleep(1)
seconds += 1
print(">", seconds)
When I run this in komodo edit, it doesn't print out the numbers as wanted.
I want the number 1 to be printed after one second of time, the number 2 to be printed after two seconds of time, and etc.
Instead, it prints out all the numbers (1-10) after 10 seconds of time.
I have run this exact same program in the python IDLE and it works as it should, printing one number per second.
What am I doing wrong or what do I not understand/know about?
The program is likely running in an environment where it does not believe its output is connected to a terminal, so stdout defaults to block buffering, not line buffering. Since you're outputting very little data, the buffer never fills, and is only flushed just before the program exits.
The simplest fix is to add the flush=True argument to your print call, so the buffer is explicitly flushed after each print:
print(">", seconds, flush=True)
Consider this small python script odd-read-blocking.py:
#!/usr/bin/python
import signal
import sys
sig = None
def handler(signum, frame):
global sig
sig = signum
signal.signal(signal.SIGINT, handler)
signal.signal(signal.SIGTERM, handler)
x = sys.stdin.read(3)
print 'signal', sig
print 'read bytes', len(x)
exit(0)
I run this and feed it with two bytes of standard input data ('a' + '\n'):
> echo a | ./odd-read-blocking.py
signal None
read bytes 2
>
Fine.
Now I feed it with the same two bytes (by typing 'a' + '\n' into its standard input). Please note that standard input is then not at EOF yet and potentially has more data to come. So the read blocks, as it expects one more byte. I use Ctrl+C on the script.
> ./odd-read-blocking.py
a
^Csignal 2
read bytes 2
>
Fine. We see that two bytes have been read and signal 2 was received.
Now I open a standard input stream, but do not send any byte on it. The read blocks as expected. If I now use Ctrl+C on the script, it will keep sitting there and wait. The read will not be interrupted. The SIGINT will not be processed.
> ./odd-read-blocking.py
^C
Nothing here. Script still running (seemingly blocked at the read).
Now hitting return once, then Ctrl+C again:
^Csignal 2
read bytes 1
>
So, only after receiving at least some data (a single '\n' in this case) on its standard input will the script behave as I expect it and correctly interrupt the blocked read and tell me it has received signal 2 and read 1 byte.
Alternative 1: instead of using Ctrl+C as shown above, I have tried this same thing using kill pid from a separate terminal. The behaviour is the same.
Alternative 2: instead of using the shell standard input as described above, I have done this:
> sleep 2000 | ./odd-read-blocking.py
When using kill pid to send SIGTERM to the odd-read-blocking.py process I get the same behaviour. Here, the script process can only be killed using SIGKILL (9).
Why isn't the read interrupted, when it is blocking on an as yet empty but still active standard input stream?
I find this odd. Who doesn't? Who can explain?
The short version
If a Python signal handler throws an exception to abandon an ongoing file.read, any data already read is lost. (Any asynchronous exception, like the default KeyboardInterrupt, makes it basically impossible to prevent this sort of failure unless you have a way to mask it.)
To minimize the need for this, file.read returns early (i.e., with a shorter string than requested) when it is interrupted by a signal—note that this is in addition to the EOF and non-blocking I/O cases that are documented! However, it can't do this when it has no data yet, since it returns the empty string to indicate EOF.
Details
As always, the way to understand behavior like this is with strace.
read(2)
The actual read system call has a dilemma when a signal arrives while the process is blocked. First, the (C) signal handler gets invoked—but because that could happen between any two instructions, there's very little it can do beyond setting a flag (or writing to a self-pipe). Then what? If SA_RESTART is set, the call is resumed; otherwise…
If no data has been transferred yet, read can fail and the client can check its signal flag. It fails with the special EINTR to clarify that nothing actually went wrong with the I/O.
If some data has already been written into the (userspace) buffer, it can't just return "failure", because data would be lost—the client can't know how much (if any) data is in the buffer. So it just returns success (the number of bytes read so far)! Short reads like this are always a possibility: the client has to call read again to check that it has reached end of file. (Just like file.read, a short read of 0 bytes would be EOF.) The client therefore has to check their signal flag after every read, whether it succeeds or not. (Note that this is still not perfectly reliable, but it's good enough for many interactive use cases.)
file.read()
The system call isn't the whole story: after all, the normal configuration for a terminal has it return immediately after seeing a newline. Python 2's low-level file.read is a wrapper for fread, which will issue another read if one is short. But when a read fails with EINTR, fread returns early and file.read calls your (Python) signal handler. (If you add output to it, you'll see that it's called immediately for each signal you send, even if file.read doesn't return.)
Then it's faced with a dilemma similar to that for the system call: as discussed, a short read can't be empty because it means EOF. Unlike a C signal handler, however, a Python one can do arbitrary work (including raising an exception to abort the I/O immediately, at the cost of risking data loss as mentioned at the beginning), and it's considered a convenient simplification to the interface to hide the possibility EINTR. So the fread call is just silently repeated.
Python 3.5
The rules for retrying changed in 3.5. Now the io.IOBase.read resumes even if it has data in hand; this is more consistent, but it forces the use of exceptions to stop reading, which means that you can't opt to wait on some data in order not to risk losing any you already have. The very heavyweight solution is to switch to multiplexed I/O and use signal.set_wakeup_fd(); this has the added advantage of allowing SIGINT to affect the main thread without having to bother with masking it in all the others.
I'm using the Enthought Canopy IDE to write Python and the the output of print commands do not reach the ipython output window at the time when I would expect them to. The clearest way to explain is to give an example:
import time
print 1
print 2
time.sleep(1)
print 3
for i in range(10):
print i
time.sleep(0.5)
The initial 1, 2 and 3 are displayed simultaneously after a second, and then there is a half second delay before the next 0 is displayed.
(as opposed to the expected instantaneous and simultaneous display of 1 and 2, then a 1 second delay before a simultaneous display of 3 and 0, then half second delays)
This is causing me issues as I am looking to get readouts of variables before a time consuming portion of script runs, and I cannot get it to display these readouts without terminating the program after calling the print command.
Any ideas/solutions?
I'm on Windows 7, let me know if you need any other specs/config details...
Thanks!
You can start the Python interpreter with an additional -u parameter, which gives you unbuffered output (i.e. instant).
Or you can call sys.stdout.flush() after every print. Maybe even wrapping it in a function like this:
from __future__ import print_function
import sys
def print_flush(*args):
print(*args)
sys.stdout.flush()
Short version: I have a program that prints to the terminal successive integer numbers in an infinite loop. At some point, the terminal became black, and no output is visible, but I can still execute commands.
Details:
I read this answer in PCG and wanted to try it in Python. Here it is:
#!/bin/python2
class Dier(object):
def __init__(self):
global ct
ct += 1
print ct
def __del__(self):
Dier()
ct = 0
Dier()
This program loops indefinitely, printing the number of iterations on each step. Left overnight (we get to the tenths of millions in a matter of minutes) executed from an Ubuntu gnome terminal, the terminall just shows black. The program is still running, and new lines appear, but nothing is visible. I killed the program, but the terminal, including the command prompt, is black. I can input commands, and they work, but no output is visible.
Why is this happening?
Some information I provided in the comments:
The memory used by the program (reported by top) remains constant and low.
I can print absurdly large numbers to the terminal without getting this behaviour. The largest number I have printed has 10^10^6 digits (of course, the terminal has forgotten the beginning of it, and only shows the last digits, but its log10 is 1246124). My program couldn't have gone that far, that would take millions of times the age of the universe.
On an added note, if I try to print something ever bigger than that, it just seems to freeze, using the CPU but without any output (while the original output was there, but invisible).
Seems like the output is being buffered. You should try and change the print to
print ct, "\n"
or
print ct
sys.stdout.flush()
alternatively you can tell python to not buffer the output by calling it with -u parameter (see https://docs.python.org/2/using/cmdline.html#cmdoption-u) or setting PYTHONUNBUFFERED=1 env variable (see https://docs.python.org/2/using/cmdline.html#envvar-PYTHONUNBUFFERED)