Hi I'm very new to Perl and CGI.
I'm trying to convert a perl script to python.
It is mentioned $|=1 in the script. What I understood is it clears the buffer.
I am searching if there is any python equivalent to do the exact thing.
Any suggestions??
I'd consider not worrying about porting this line for the time being, as flushing stdout after every print will likely be the least of your porting worries.
But if it is, you have many options:
Simply add the flush=True keyword argument to your print function call.
Run Python in "unbuffered" mode with the -u switch.
Re-open stdout in unbuffered mode (e.g. the final 0 in):
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
Write a print wrapper function that shadows the builtin print and flushes stdout
Write a TextIOWrapper object that wraps sys.stdout and flushes
I'll try to find some links for the rest of the points and edit them in.
Related
I am including in my python code a function compiled in c via a cython wrapper. I have to take that function as given and cannot change it. Unfortunately, when I run that function, I see output that is bothering me.
I have tried a lot of tricks that are supposed to get rid of it, all of which play with sys.stdout or sys.stderr -- most noteably, the new contextlib.redirect_stdout. However, nothing I tried managed to silence the output.
At the most basic level, I simply tried setting
sys.stdout = open(os.devnull, 'w')
sys.stderr = open(os.devnull, 'w')
Which is not a safe, or practicable way of doing it, but it should shut the function up. Unfortunately, I can still see the output. What am I missing? Is there perhaps another "output type" besides stdout that this function might be using?
If it helps, I am inside a Pycharm debugging session and see this output in my debugging console.
Updated question to reflect that changing stderr did not help
A C function prints to a file descriptor (1 for stdout, 2 for stderr). If you want to prevent the printing, redirect that FD, that can be done also temporarily. Here is a litte demo:
import os
STDOUT = 1
saved_fd = os.dup(STDOUT)
null_fd = os.open(os.devnull, os.O_WRONLY)
os.dup2(null_fd, STDOUT)
os.system('echo TESTÂ 1') # redirected to /dev/null
os.dup2(saved_fd, STDOUT)
os.system('echo TEST 2') # normal
# note: close the null_fd, saved_fd when no longer needed
If the C code opens the terminal device itself, there is very little you can do to prevent it. But that would be very unusual (I would even say a bug) unless there is a specific reason to do so.
Is there perhaps another "output type" besides stdout that this
function might be using?
Yes, there exist stderr, which would be unaffected by stdout redirect, simple example, let printer.py content be
import sys
sys.stderr.write("printing to stderr")
then running in terminal
python printer.py > output.txt
lead to appearance of
printing to stderr
as > output.txt redirects only stdout.
Due to Apache gateway timeouts, and a desire to display more information to the end user, I'd like to be able to flush STDOUT on a python CGI script hosted on PCF, essentially giving updates on the status of the script.
I have tried enabling the -u tag in python (#!/usr/python -u at head of my script), sys.stdout.flush() command, and even using subprocess.call to execute a perl script that is set to flush to STDOUT that prints some progress text ($| = 1; at beginning of perl script). Furthermore, I've double checked that I'm not using any Apache modules that would require buffering (no mod_deflate, for example). Finally, I'll mention that executing a standard perl CGI rather than a python CGI allows for the STDOUT flushing, so I figure it must be something with my python/Apache/PCF configuration.
I'm fresh out of ideas here, and would like some advice.
With any of these above methods, I would have thought stdout would flush. But, none of them have worked!
Thanks in advance for any assisstance.
You can disable buffering using something like this in your Python2 code:
# set stdout as non-buffered
if hasattr(sys.stdout, 'fileno'):
fileno = sys.stdout.fileno()
tmp_fd = os.dup(fileno)
sys.stdout.close()
os.dup2(tmp_fd, fileno)
os.close(tmp_fd)
sys.stdout = os.fdopen(fileno, "w", 0)
That is reopening sys.stdout with no buffer (i.e. the 0 as third arg). After you do that, anything writing to sys.stdout should not be buffered.
I want to do an atomic write with Python's print function. I found this answer, already:
How can I do an atomic write to stdout in python?
But this uses sys.stdout.write. I want to be more flexible and use print instead. When I implemented the same code with print, apparently this matters, since my output turned out not to be correct.
lock = Lock()
def synced_out(*inp):
with lock:
print(*inp, file=args.out, sep=args.field_seperator)
Apparently it matters that I use print and not sys.stdout.write.
Full code here, if you expect that is not possible and I might be doing something else wrong:
https://termbin.com/s9ox
In the case of corruption the file was sys.stdout, but I was using redirection and send it to file anyway. I want to preserve however the --out flag so that people with less understanding of "> file" can also use it, or if it is used with a pipe, just maintaining this flexibility.
Python 3.5.2
Linux Ubuntu 16.04
Let's say I want to read RAM usage from /proc/meminfo. There are two basic ways to do this that I can think of.
Use a shell command
output = subprocess.check_output('cat /proc/meminfo', shell=True)
# or output = subprocess.check_output(['cat', '/proc/meminfo'])
lines = output.splitlines()
Use open()
with open('/proc/meminfo') as meminfo:
output = meminfo.read()
lines = output.splitlines()
My question is what is the difference between the two methods? Is there a significant performance difference? My assumption is that using open() is the preferred method, since using a shell command is a bit hackish and may be system dependent, but I can't find any information on this so I thought I'd ask.
...so, let's look at what output = subprocess.check_output('cat /proc/meminfo', shell=True) does:
Creates a FIFO pair with mkfifo(), and spawns a shell running sh -c 'cat /proc/meminfo' writing to the input end of the FIFO (while the Python interpreter itself watches for output on the other end, either using the select() call or blocking IO operations). This means opening /bin/sh, opening all the libraries it depends on, etc.
The shell parses those arguments as code. This can be dangerous if, instead of opening /proc/meminfo. you're instead opening /tmp/$(rm -rf ~)/pwned.txt.
The shell forks a subprocess (optionally; shells may have an implicit exec), which then uses the execve system call to invoke /bin/cat with an argv of ['cat', '/proc/meminfo'] -- meaning that /bin/cat is again loaded as an executable, with its dynamic libraries, with all the performance overhead that implies.
/bin/cat then opens /proc/meminfo, reads from it, and writes to its stdout
The shell, if it did not use the implicit-exec optimization, waits for the /bin/cat executable to finish and exit using a wait()-family syscall.
The Python interpreter reads from the remote end of the FIFO until it provides an EOF (which will not happen until after cat has closed its output pipeline, potentially by exiting), and then uses a wait()-family call to retrieve information on how the shell it spawned exited, checking that exit status to determine whether an error occurred.
Now, let's look at what open('/proc/meminfo').read() does:
Opens the file using the open() syscall.
Reads the file using the read() syscall.
Drops the reference count on the file, allowing it to be closed (either immediately or on a future garbage collection pass) with the close() syscall.
One of these things is much, much, much more efficient and generally sensible than the other.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I know this sounds like something I can google, but the truth is that I don't find or do not understand what the very few Python 3 sources explains.
So here are my questions:
Is input() the stdin function in Python 3? Does that mean that when you open your filename.py program, the stdin is what the user types?
Is print() the stdout function in Python 3, or do you have to write to a file?
For the Spotify puzzle, is says "Input is read from stdin". What should my file include of stdin and stdout?
Update:
Does that mean that i can use:
import sys
unfmtdDate = str(sys.stdin.read())
...instead of...
unfmtdDate = str(input())
?
stdin and stdout are file-like objects provided by the OS. In general, when a program is run in an interactive session, stdin is keyboard input and stdout is the user's tty, but the shell can be used to redirect them from normal files or piped output from and input to other programs.
input() is used to prompt the user for typed input. In the case of something like a programming puzzle, it's normally assumed that stdin is redirected from a data file, and when the input format is given it's usually best to use sys.stdin.read() rather than prompting for input with input(). input() is intended for interactive user input, it can display a prompt (on sys.stdout) and use the GNU readline library (if present) to allow line editing, etc.
print() is, indeed, the most common way of writing to stdout. There's no need to do anything special to specify the output stream. print() writes to sys.stdout if no alternate file is given to it as a file= parameter.
When you run your Python program, sys.stdin is the file object connected to standard input (STDIN), sys.stdout is the file object for standard output (STDOUT), and sys.stderr is the file object for standard error (STDERR).
Anywhere in the documentation you see references to standard input, standard output, or standard error, it is referring to these file handles. You can access them directly (sys.stdout.write(...), sys.stdin.read() etc.) or use convenience functions that use these streams, like input() and print().
For the Spotify puzzle, the easiest way to read the input would be something like this:
import sys
data = sys.stdin.read()
After these two lines the input for your program is now in the str data.
Is input() the stdin function in Python 3?
Yes.
Does that mean that when you open your filename.py program, the stdin is what the user types?
Yes.
Is print() the stdout function in Python 3, or do you have to write to a file?
You can use either.
For the Spotify puzzle, is says "Input is read from stdin". What should my file include of stdin and stdout?
Just use import sys to get to the sys.stdin and sys.stdout files. You don't need the import if you use the input and print built-in functions instead.
Is input() the stdin function in Python 3? Does that mean that when you open your filename.py program, the stdin is what the user types?
input() reads from the standard input (called stdin for short in some contexts), yes. No language that I know of has a function named stdin in the standard library. When you run the program in the usual way, what the user types is supplied to standard input, yes. That's what "standard input" means. There is a separate program that is feeding what the user types to standard input (and doing a couple of other convenient things like interpreting the backspace key). Your script is not what creates the window that text is typed into.
sys.stdin (i.e., the value stdin defined in the sys module) is an object that represents the standard input. This is provided so that you can use it in contexts where you're expected to specify "a file", to cause input to be read from the user instead of a file.
Is print() the stdout function in Python 3, or do you have to write to a file?
print() writes (by default; you can supply a file keyword argument to change this) to the standard output (called stdout for short in some contexts), yes. All the above caveats apply: stdout isn't a function in any language I've ever heard of, and a completely separate program is actually causing the output of your script to be displayed on screen in a pretty 80x24 white-text-on-black-background (or however you have it configured) box.
sys.stdout is an object that represents the standard output, used similarly to sys.stdin. You can use it explicitly with the print function, but there's no point: it's the default.
For the Spotify puzzle, is says "Input is read from stdin". What should my file include of stdin and stdout?
The problem specification means "use the input() function to receive input".
From the Python documentation for print:
The file argument must be an object with a write(string) method; if it is not present or None, sys.stdout will be used. Output buffering is determined by file. Use file.flush() to ensure, for instance, immediate appearance on a screen.