I would like to have my print output length limited to X characters.
I've been looking for some info and I found the command textwrap.fill which makes just what I was looking for by using something like:
print(textwrap.fill("Hello world", X))
However, I was wondering if there is a way to apply this length limitation to every print without having to write it (I do have plenty of them) by creating or setting a class or something at the very beginning of the script.
Monkeypatching print is doable, but it's not a good idea. For one thing, what do you want to happen if someone does a print(spam, eggs)? Or print(spam, end='')? Or print(spam, file=outfile)?
A better solution is probably replacing sys.stdout with a wrapper.
The normal sys.stdout is a plain old text file object, a TextIOWrapper just like the ones you get from open, except that when you write to it, it goes to the console instead of to a file on disk.
And you're allowed to replace it with anything else that meets the TextIOBase protocol.
And writing a TextIOBase is really simple. All you really need to implement is write and/or read and readline (depending on whether you're wrapping output, input, or both), and all our wrapper needs to do in write is to buffer up lines, fill them, and pass them to the real file object underneath.
Like this:
import io
import sys
import textwrap
class Filler(io.TextIOBase):
def __init__(self, file, width=70):
self.file = file
self.textwrapper = textwrap.TextWrapper(width=width)
self.buf = ''
def write(self, buf):
self.buf += buf
lines = self.buf.split('\n')
self.buf = lines.pop()
for line in lines:
self.file.write(self.textwrapper.fill(line) + '\n')
def close(self):
if self.buf:
self.file.write(self.textwrapper.fill(buf))
self.buf = ''
self.file.close()
sys.stdout = Filler(sys.stdout, 32)
print('Spam spam spam spammity ' * 10)
print('Spam', 'Eggs')
sys.stdout.textwrapper.width = 72
print('Spam ' + 'spam ' * 50, 'and eggs', sep='... ')
print('Spam', end=' ')
print('Eggs', end=' ')
print('Cheese')
Technically, I think I may be cheating in a few ways here:
The docs say the ABC TextIOBase wants detach, read, and readline, even if they don't make sense here. But the ABC doesn't seem to enforce them as abstract methods, so I didn't bother.
I think it's legal (and it works) to leave encoding and errors set to None, since we're just passing through to another TextIOBase and expecting it to do the encoding, but I can't find anything that says it's legal. And if some code were to test sys.stdout.encoding to see if it's UTF-8 or something, that might be trouble.
Similarly for newlines. And, since I haven't tested on Windows, I can't be as sure that it works.
Also, forwarding other methods to self.file might be a good idea, like fileno() and isatty(). But I'd worry that any app that wants to access stdout as a TTY probably need to know about the Filler that we stuck in front of it, not just transparently go through it.
This is of course all Python 3-specific. In Python 2:
sys.stdout is a file, not a TextIOWrapper. The API you need to wrap is a bit different, and not nearly as well defined.
Unless you __future__ up the 3.x-style print function, print is a statement, so you can't monkeypatch it. (I mean, you could write an import hook that bytecode-hacks out every PRINT_* bytecode, or maybe even inject a .so that replaces PyObject_Print… but who cares anyway? It's Python 2.)
Related
I'm wondering if there is a way to write to a file that was opened in a separate script in Python. For example if the following was run within main.py:
f = open(fname, "w")
writer.write()
Then, within a separate script called writer.py, we have a function write() with the form:
def write()
get_currently_open_file().write("message")
Without defining f within writer.py. This would be similar to how matplotlib has the method:
pyplot.gca()
Which returns the current axis that's open for plotting. This allows you to plot to an axis defined previously without redefining it within the script you're working in.
I'm trying to write to a file with inputs from many different scripts and it would help a lot to be able to write to a file without reading a file object or filename as an input to each script.
Yes. Python functions have local variables, but those are only the variables that are assigned in the function. Python will look to the containing scope for the others. If you use f, but don't try to assign f, python will find the one you created in the global scope.
def write():
f.write("text")
fname = "test"
f = open(fname, "w")
write()
This only works if the function is in the same module as the global variable (python "global" is really "module level").
UPDATE
Leveraging a function's global namespace, you could write a module that holds the writing function and a variable holding the file. Every script/module that imports this module could use the write function that gets its file handles from its own module. In this example, filewriter.py is the common place where test.py and somescript.py cooperate on file management.
filewriter.py
def opener(filename, mode="r"):
global f
f = open(filename, mode)
def write(text):
return f.write(text) # uses the `f` in filewriter namespace
test.py
from filewriter import write
def my_test():
write("THIS IS A TEST\n")
somescript.py
import filewriter
import test
filewriter.opener("test.txt", "w")
test.my_test()
# verify
filewriter.f.seek(0)
assert f.read() == "THIS IS A TEST\n"
Writing as a separate answer because it's essentially unrelated to my other answer, the other semi-reasonable solution here is to define a protocol in terms of the contextvars module. In the file containing write, you define:
import contextlib
import io
import sys
from contextvars import ContextVar
outputctx: ContextVar[io.TextIOBase] = ContextVar('outputctx', default=sys.stdout)
#contextlib.contextmanager
def using_output_file(file):
token = outputctx.set(file)
try:
yield
finally:
outputctx.reset(token)
Now, your write function gets written as:
def write():
outputctx.get().write("message")
and when you want to redirect it for a time, the code that wants to do so does:
with open(fname, "w") as f, using_output_file(f):
... do stuff where calling write implicitly uses the newly opened file ...
... original file is restored ...
The main differences between this and using sys.stdout with contextlib.redirect_stdout are:
It's opt-in, functions have to cooperate to use it (mild negative)
It's explicit, so no one gets confused when the code says print or sys.stdout.write and nothing ends up on stdout
You don't mess around with sys.stdout (temporarily cutting off sys.stdout from code that doesn't want to be redirected)
By using contextvars, it's like thread-local state (where changing it in one thread doesn't change it for other threads, which would cause all sorts of problems if multithreaded code), but moreso; even if you're writing asyncio code (cooperative multitasking of tasks that are all run in the same thread), the context changes won't leak outside the task that makes them, so there's no risk that task A (which wants to be redirected) changes how task B (which does not wish to be redirected) behaves. By contrast, contextlib.redirect_stdout is explicitly making global changes; all threads and tasks see the change, they can interfere with each other, etc. It's madness.
Obviously what you're asking for is hacky, but there are semi-standard ways to express the concept "The thing we're currently writing to". sys.stdout is one of those ways, but it's normally sent to the terminal or a specific file chosen outside the program by the user through piping syntax. That said, you can perform temporary replacement of sys.stdout so that it goes to an arbitrary location, and that might satisfy your needs. Specifically, you use contextlib.redirect_stdout in a with statement.
On entering the with, sys.stdout is saved and replaced with an arbitrary open file; while in the with all code (including code called from within the with, not just the code literally shown in the block) that writes to sys.stdout instead writes to the replacement file, and when the with statement ends, the original sys.stdout is restored. Such uses can be nested, effectively creating a stack of sys.stdouts where the top of the stack is the current target for any writes to sys.stdout.
So for your use case, you could write:
import sys
def write():
sys.stdout.write("message")
and it would, by default, write to sys.stdout. But if you called write() like so:
from contextlib import redirect_stdout
with open(fname, "w") as f, redirect_stdout(f): # Open a file and redirect stdout to it
write()
the output would seamlessly go to the file located wherever fname describes.
To be clear, I don't think this is a good idea. I think the correct solution is for the functions in the various scripts to just accept a file-like object as an argument which they will write to ("Explicit is better than implicit", per the Zen of Python). But it's an option.
I have a service running on a Linux box that creates a named pipe character device-special file, and I want to write a Python3 program that communicates with the service by writing text commands and reading text replies from the pipe device. I don't have source code for the service.
I can use os.open(named_pipe_pathname, os.O_RDWR), and I can use os.read(...) and os.write(...) to read and write it, but that's a pain because I have to write my own code to convert between bytes and strings, I have to write my own readline(...) function, etc.
I would much rather use a Python3 io object to read and write the pipe device, but every way I can think to create one returns the same error:
io.UnsupportedOperation: File or stream is not seekable.
For example, I get that message if I try open(pathname, "r+"), and I get that same message if I try fd=os.open(...) followed by os.fdopen(fd, "r+", ...).
Q: What is the preferred way for a Python3 program to write and read text to and from a named pipe character device?
Edit:
Oops! I assumed that I was dealing with a named pipe because documentation for the service describes it as a "pipe" and, because it doesn't appear in the file system until the user-mode service runs. But, the Linux file utility says it is in fact, a character device special file.
The problem occurs because attempting to use io.open in read-write mode implicitly tries to wrap the underlying file in io.BufferedRandom (which is then wrapped in io.TextIOWrapper if in text mode), which assumes the underlying file is not only read/write, but random access, and it takes liberties (seeking implicitly) based on this. There is a separate class, io.BufferedRWPair, intended for use with read/write pipes (the docstring specifically mentions it being used for sockets and two way pipes).
You can mimic the effects of io.open by manually wrapping layer by layer to produce the same end result. Specifically, for a text mode wrapper, you'd do something like:
rawf = io.FileIO(named_pipe_pathname, mode="rb+")
with io.TextIOWrapper(io.BufferedRWPair(rawf, rawf), encoding='utf-8', write_through=True) as txtf:
del rawf # Remove separate reference to rawf; txtf manages lifetime now
# Example use that works (but is terrible form, since communicating with
# oneself without threading, select module, etc., is highly likely to deadlock)
# It works for this super-simple case; presumably you have some parallel real code
txtf.write("abcé\n")
txtf.flush()
print(txtf.readline(), flush=True)
I believe this will close rawf twice when txtf is closed, but luckily, double-close is harmless here (the second close does nothing, realizing it's already closed).
Solution
You can use pexpect. Here is an example using two python modules:
caller.py
import pexpect
proc = pexpect.spawn('python3 backwards.py')
proc.expect(' > ')
while True:
n = proc.sendline(input('Feed me - '))
proc.expect(' > ')
print(proc.before[n+1:].decode())
backwards.py
x = ''
while True:
x = input(x[::-1] + ' > ')
Explanation
caller.py is using a "Pseudo-TTY device" to talk to backwards.py. We are providing input with sendline and capturing input with expect (and the before attribute).
It looks like you need to create separate handles for reading and for writing: to open read/write just requires a seek method. I couldn't figure out how to timeout reading, so it's nice to add an opener (see the docstring for io.open) that opens the reader in non-blocking mode. I set up a simple echo service on a named pipe called /tmp/test_pipe:
In [1]: import io
In [2]: import os
In [3]: nonblockingOpener = lambda name, flags:os.open(name, flags|os.O_NONBLOCK)
In [4]: reader = io.open('/tmp/test_pipe', 'r', opener = nonblockingOpener)
In [5]: writer = io.open('/tmp/test_pipe', 'w')
In [6]: writer.write('Hi have a line\n')
In [7]: writer.flush()
In [8]: reader.readline()
Out[8]: 'You said: Hi have a line\n'
In [9]: reader.readline()
''
I wanted to see if it was possible to define new keywords or, as they're called in WAT's Destroy All Software talk when discussing Ruby, bare words, in Python.
I came up with an answer that I couldn't find elsewhere, so I decided to share it Q&A style on StackOverflow.
I've only tried this in the REPL, outside any block, so far. It may be possible to make it work elsewhere, too.
I put this in my python startup file:
import sys, traceback
def bareWordsHandler(type_, value, traceback_):
if isinstance(value, SyntaxError):
import traceback
# You can probably modify this next line so that it'll work within blocks, as well as outside them:
bareWords = traceback.format_exception(type_, value, traceback_)[1].split()
# At this point we have the raw string that was entered.
# Use whatever logic you want on it to decide what to do.
if bareWords[0] == 'Awesome':
print(' '.join(bareWords[1:]).upper() + '!')
return
bareWordsHandler.originalExceptHookFunction(type_, value, traceback_)
bareWordsHandler.originalExceptHookFunction = sys.excepthook
sys.excepthook = bareWordsHandler
Quick REPL session demonstration afterwords:
>>> Awesome bare words
BARE WORDS!
Use responsibly.
Edit: Here's a more useful example. I added in a run keyword.
if bareWords[0] == 'from' and bareWords[2] == 'run':
atPrompt.autoRun = ['from ' + bareWords[1] + ' import ' + bareWords[3].split('(')[0],
' '.join(bareWords[3:])]
return
atPrompt.autoRun is a list of variables that, when my prompt is displayed, will automatically be checked and fed back. So, for example, I can do this:
>>> from loadBalanceTester run loadBalancerTest(runJar = False)
And this gets interpreted as:
from loadBalancerTest import loadBalancerTest
loadBalancerTest(runJar = False)
It's kind of like a macro - it's common for me to want to do this kind of thing, so I decided to add in a keyword that lets me do it in fewer keystrokes.
I'm in the process of learning how a large (356-file), convoluted Python program is set up. Besides manually reading through and parsing the code, are there any good methods for following program flow?
There are two methods which I think would be useful:
Something similar to Bash's "set -x"
Something that displays which file outputs each line of output
Are there any methods to do the above, or any other ways that you have found useful?
I don't know if this is actually a good idea, but since I actually wrote a hook to display the file and line before each line of output to stdout, I might as well give it to you…
import inspect, sys
class WrapStdout(object):
_stdout = sys.stdout
def write(self, buf):
frame = sys._getframe(1)
try:
f = inspect.getsourcefile(frame)
except TypeError:
f = 'unknown'
l = frame.f_lineno
self._stdout.write('{}:{}:{}'.format(f, l, buf))
def flush(self):
self._stdout.flush()
sys.stdout = WrapStdout()
Just save that as a module, and after you import it, every chunk of stdout will be prefixed with file and line number.
Of course this will get pretty ugly if:
Anyone tries to print partial lines (using stdout.write directly, or print magic comma in 2.x, or end='' in 3.x).
You mix Unicode and non-Unicode in 2.x.
Any of the source files have long pathnames.
etc.
But all the tricky deep-Python-magic bits are there; you can build on top of it pretty easily.
Could be very tedious, but using a debugger to trace the flow of execution, instruction by instruction could probably help you to some extent.
import pdb
pdb.set_trace()
You could look for a cross reference program. There is an old program called pyxr that does this. The aim of cross reference is to let you know how classes refer to each other. Some of the IDE's also do this sort of thing.
I'd recommend running the program inside an IDE like pydev or pycharm. Being able to stop the program and inspect its state can be very helpful.
It turns out that "with" is a funny word to search for on the internet.
Does anyone knows what the deal is with nesting with statements in python?
I've been tracking down a very slippery bug in a script I've been writing and I suspect that it's because I'm doing this:
with open(file1) as fsock1:
with open(file2, 'a') as fsock2:
fstring1 = fsock1.read()
fstring2 = fsock2.read()
Python throws up when I try to read() from fsock2. Upon inspection in the debugger, this is because it thinks the file is empty. This wouldn't be worrisome except for the fact that running the exact same code in the debugging interperter not in a with statement shows me that the file is, in fact, quite full of text...
I'm going to proceed on the assumption that for now nesting with statements is a no-no, but if anyone who knows more has a different opinion, I'd love to hear it.
I found the solution in python's doc. You may want to have a look at this (Python 3) or this (Python 2)
If you are running python 2.7+ you can use it like this:
with open(file1) as fsock1, open(file2, 'a') as fsock2:
fstring1 = fsock1.read()
fstring2 = fsock2.read()
This way you avoid unnecessary indentation.
AFAIK you can't read a file open with append mode 'a'.
Upon inspection in the debugger, this is because it thinks the file is empty.
I think that happens because it can't actually read anything. Even if it could, when you append to a file, the seek pointer is moved to the end of the file in preparation for writing to occur.
These with statements work just fine for me:
with open(file1) as f:
with open(file2, 'r') as g: # Read, not append.
fstring1 = f.read()
fstring2 = g.read()
Note that use of contextlib.nested, as another poster suggested, is potentially fraught with peril here. Let's say you do this:
with contextlib.nested(open(file1, "wt"), open(file2)) as (f_out, f_in):
...
The context managers here get created one at a time. That means that if the opening of file2 fails (say, because it doesn't exist), then you won't be able to properly finalize file1 and you'll have to leave it up to the garbage collector. That's potentially a Very Bad Thing.
There is no problem with nesting with statements -- rather, you're opening file2 for append, so you can't read from it.
If you do dislike nesting with statements, for whatever reason, you can often avoid that with the contextlib.nested function. However, it won't make broken code (e.g., code that opens a file for append and then tries to read it instead) work, nor will lexically nesting with statements break code that's otherwise good.
As of python 3.10 you can do it like this
with (
Something() as example1,
SomethingElse() as example2,
YetSomethingMore() as example3,
):
...
this can be helpful in pytests when you want to do nested patches in some autouse fixture like so
from unittest.mock import patch
import pytest
#pytest.fixture(scope="session", autouse=True)
def setup():
with (
patch("something.Slow", MagicMock()) as slow_mock,
patch("something.Expensive") as expensive_mock,
patch("other.ThirdParty", as third_party_mock,
):
yield
As for searching for "with", prefixing a word with '+' will prevent google from ignoring it.