Related
NB. I have seen Log output of multiprocessing.Process - unfortunately, it doesn't answer this question.
I am creating a child process (on windows) via multiprocessing. I want all of the child process's stdout and stderr output to be redirected to a log file, rather than appearing at the console. The only suggestion I have seen is for the child process to set sys.stdout to a file. However, this does not effectively redirect all stdout output, due to the behaviour of stdout redirection on Windows.
To illustrate the problem, build a Windows DLL with the following code
#include <iostream>
extern "C"
{
__declspec(dllexport) void writeToStdOut()
{
std::cout << "Writing to STDOUT from test DLL" << std::endl;
}
}
Then create and run a python script like the following, which imports this DLL and calls the function:
from ctypes import *
import sys
print
print "Writing to STDOUT from python, before redirect"
print
sys.stdout = open("stdout_redirect_log.txt", "w")
print "Writing to STDOUT from python, after redirect"
testdll = CDLL("Release/stdout_test.dll")
testdll.writeToStdOut()
In order to see the same behaviour as me, it is probably necessary for the DLL to be built against a different C runtime than than the one Python uses. In my case, python is built with Visual Studio 2010, but my DLL is built with VS 2005.
The behaviour I see is that the console shows:
> stdout_test.py
Writing to STDOUT from python, before redirect
Writing to STDOUT from test DLL
While the file stdout_redirect_log.txt ends up containing:
Writing to STDOUT from python, after redirect
In other words, setting sys.stdout failed to redirect the stdout output generated by the DLL. This is unsurprising given the nature of the underlying APIs for stdout redirection in Windows. I have encountered this problem at the native/C++ level before and never found a way to reliably redirect stdout from within a process. It has to be done externally.
This is actually the very reason I am launching a child process - it's so that I can connect externally to its pipes and thus guarantee that I am intercepting all of its output. I can definitely do this by launching the process manually with pywin32, but I would very much like to be able to use the facilities of multiprocessing, in particular the ability to communicate with the child process via a multiprocessing Pipe object, in order to get progress updates. The question is whether there is any way to both use multiprocessing for its IPC facilities and to reliably redirect all of the child's stdout and stderr output to a file.
UPDATE: Looking at the source code for multiprocessing.Processs, it has a static member, _Popen, which looks like it can be used to override the class used to create the process. If it's set to None (default), it uses a multiprocessing.forking._Popen, but it looks like by saying
multiprocessing.Process._Popen = MyPopenClass
I could override the process creation. However, although I could derive this from multiprocessing.forking._Popen, it looks like I would have to copy a bunch of internal stuff into my implementation, which sounds flaky and not very future-proof. If that's the only choice I think I'd probably plump for doing the whole thing manually with pywin32 instead.
The solution you suggest is a good one: create your processes manually such that you have explicit access to their stdout/stderr file handles. You can then create a socket to communicate with the sub-process and use multiprocessing.connection over that socket (multiprocessing.Pipe creates the same type of connection object, so this should give you all the same IPC functionality).
Here's a two-file example.
master.py:
import multiprocessing.connection
import subprocess
import socket
import sys, os
## Listen for connection from remote process (and find free port number)
port = 10000
while True:
try:
l = multiprocessing.connection.Listener(('localhost', int(port)), authkey="secret")
break
except socket.error as ex:
if ex.errno != 98:
raise
port += 1 ## if errno==98, then port is not available.
proc = subprocess.Popen((sys.executable, "subproc.py", str(port)), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
## open connection for remote process
conn = l.accept()
conn.send([1, "asd", None])
print(proc.stdout.readline())
subproc.py:
import multiprocessing.connection
import subprocess
import sys, os, time
port = int(sys.argv[1])
conn = multiprocessing.connection.Client(('localhost', port), authkey="secret")
while True:
try:
obj = conn.recv()
print("received: %s\n" % str(obj))
sys.stdout.flush()
except EOFError: ## connection closed
break
You may also want to see the first answer to this question to get non-blocking reads from the subprocess.
I don't think you have a better option than redirecting a subprocess to a file as you mentioned in your comment.
The way consoles stdin/out/err work in windows is each process when it's born has its std handles defined. You can change them with SetStdHandle. When you modify python's sys.stdout you only modify where python prints out stuff, not where other DLL's are printing stuff. Part of the CRT in your DLL is using GetStdHandle to find out where to print out to. If you want, you can do whatever piping you want in windows API in your DLL or in your python script with pywin32. Though I do think it'll be simpler with subprocess.
Alternatively - and I know this might be slightly off-topic, but helped in my case for the same problem - , this can be resolved with screen on Linux:
screen -L -Logfile './logfile_%Y-%m-%d.log' python my_multiproc_script.py
this way no need to implement all the master-child communication
I assume I'm off base and missing something, but for what it's worth here is what came to mind when I read your question.
If you can intercept all of the stdout and stderr (I got that impression from your question), then why not add or wrap that capture functionality around each of your processes? Then send what is captured through a queue to a consumer that can do whatever you want with all of the outputs?
In my situation I changed sys.stdout.write to write to a PySide QTextEdit. I couldn't read from sys.stdout and I didn't know how to change sys.stdout to be readable. I created two Pipes. One for stdout and the other for stderr. In the separate process I redirect sys.stdout and sys.stderr to the child connection of the multiprocessing pipe. On the main process I created two threads to read the stdout and stderr parent pipe and redirect the pipe data to sys.stdout and sys.stderr.
import sys
import contextlib
import threading
import multiprocessing as mp
import multiprocessing.queues
from queue import Empty
import time
class PipeProcess(mp.Process):
"""Process to pipe the output of the sub process and redirect it to this sys.stdout and sys.stderr.
Note:
The use_queue = True argument will pass data between processes using Queues instead of Pipes. Queues will
give you the full output and read all of the data from the Queue. A pipe is more efficient, but may not
redirect all of the output back to the main process.
"""
def __init__(self, group=None, target=None, name=None, args=tuple(), kwargs={}, *_, daemon=None,
use_pipe=None, use_queue=None):
self.read_out_th = None
self.read_err_th = None
self.pipe_target = target
self.pipe_alive = mp.Event()
if use_pipe or (use_pipe is None and not use_queue): # Default
self.parent_stdout, self.child_stdout = mp.Pipe(False)
self.parent_stderr, self.child_stderr = mp.Pipe(False)
else:
self.parent_stdout = self.child_stdout = mp.Queue()
self.parent_stderr = self.child_stderr = mp.Queue()
args = (self.child_stdout, self.child_stderr, target) + tuple(args)
target = self.run_pipe_out_target
super(PipeProcess, self).__init__(group=group, target=target, name=name, args=args, kwargs=kwargs,
daemon=daemon)
def start(self):
"""Start the multiprocess and reading thread."""
self.pipe_alive.set()
super(PipeProcess, self).start()
self.read_out_th = threading.Thread(target=self.read_pipe_out,
args=(self.pipe_alive, self.parent_stdout, sys.stdout))
self.read_err_th = threading.Thread(target=self.read_pipe_out,
args=(self.pipe_alive, self.parent_stderr, sys.stderr))
self.read_out_th.daemon = True
self.read_err_th.daemon = True
self.read_out_th.start()
self.read_err_th.start()
#classmethod
def run_pipe_out_target(cls, pipe_stdout, pipe_stderr, pipe_target, *args, **kwargs):
"""The real multiprocessing target to redirect stdout and stderr to a pipe or queue."""
sys.stdout.write = cls.redirect_write(pipe_stdout) # , sys.__stdout__) # Is redirected in main process
sys.stderr.write = cls.redirect_write(pipe_stderr) # , sys.__stderr__) # Is redirected in main process
pipe_target(*args, **kwargs)
#staticmethod
def redirect_write(child, out=None):
"""Create a function to write out a pipe and write out an additional out."""
if isinstance(child, mp.queues.Queue):
send = child.put
else:
send = child.send_bytes # No need to pickle with child_conn.send(data)
def write(data, *args):
try:
if isinstance(data, str):
data = data.encode('utf-8')
send(data)
if out is not None:
out.write(data)
except:
pass
return write
#classmethod
def read_pipe_out(cls, pipe_alive, pipe_out, out):
if isinstance(pipe_out, mp.queues.Queue):
# Queue has better functionality to get all of the data
def recv():
return pipe_out.get(timeout=0.5)
def is_alive():
return pipe_alive.is_set() or pipe_out.qsize() > 0
else:
# Pipe is more efficient
recv = pipe_out.recv_bytes # No need to unpickle with data = pipe_out.recv()
is_alive = pipe_alive.is_set
# Loop through reading and redirecting data
while is_alive():
try:
data = recv()
if isinstance(data, bytes):
data = data.decode('utf-8')
out.write(data)
except EOFError:
break
except Empty:
pass
except:
pass
def join(self, *args):
# Wait for process to finish (unless a timeout was given)
super(PipeProcess, self).join(*args)
# Trigger to stop the threads
self.pipe_alive.clear()
# Pipe must close to prevent blocking and waiting on recv forever
if not isinstance(self.parent_stdout, mp.queues.Queue):
with contextlib.suppress():
self.parent_stdout.close()
with contextlib.suppress():
self.parent_stderr.close()
# Close the pipes and threads
with contextlib.suppress():
self.read_out_th.join()
with contextlib.suppress():
self.read_err_th.join()
def run_long_print():
for i in range(1000):
print(i)
print(i, file=sys.stderr)
print('finished')
if __name__ == '__main__':
# Example test write (My case was a QTextEdit)
out = open('stdout.log', 'w')
err = open('stderr.log', 'w')
# Overwrite the write function and not the actual stdout object to prove this works
sys.stdout.write = out.write
sys.stderr.write = err.write
# Create a process that uses pipes to read multiprocess output back into sys.stdout.write
proc = PipeProcess(target=run_long_print, use_queue=True) # If use_pipe=True Pipe may not write out all values
# proc.daemon = True # If daemon and use_queue Not all output may be redirected to stdout
proc.start()
# time.sleep(5) # Not needed unless use_pipe or daemon and all of stdout/stderr is desired
# Close the process
proc.join() # For some odd reason this blocks forever when use_queue=False
# Close the output files for this test
out.close()
err.close()
Here is the simple and straightforward way for capturing stdout for multiprocessing.Process:
import app
import io
import sys
from multiprocessing import Process
def run_app(some_param):
sys.stdout = io.TextIOWrapper(open(sys.stdout.fileno(), 'wb', 0), write_through=True)
app.run()
app_process = Process(target=run_app, args=('some_param',))
app_process.start()
# Use app_process.termninate() for python <= 3.7.
app_process.kill()
Is there a way in python to programmatically determine the width of the console? I mean the number of characters that fits in one line without wrapping, not the pixel width of the window.
Edit
Looking for a solution that works on Linux
Not sure why it is in the module shutil, but it landed there in Python 3.3. See:
Querying the size of the output terminal
>>> import shutil
>>> shutil.get_terminal_size((80, 20)) # pass fallback
os.terminal_size(columns=87, lines=23) # returns a named-tuple
A low-level implementation is in the os module. Cross-platform—works under Linux, Mac OS, and Windows, probably other Unix-likes. There's a backport as well, though no longer relevant.
import os
rows, columns = os.popen('stty size', 'r').read().split()
uses the 'stty size' command which according to a thread on the python mailing list is reasonably universal on linux. It opens the 'stty size' command as a file, 'reads' from it, and uses a simple string split to separate the coordinates.
Unlike the os.environ["COLUMNS"] value (which I can't access in spite of using bash as my standard shell) the data will also be up-to-date whereas I believe the os.environ["COLUMNS"] value would only be valid for the time of the launch of the python interpreter (suppose the user resized the window since then).
(See answer by #GringoSuave on how to do this on python 3.3+)
use
import console
(width, height) = console.getTerminalSize()
print "Your terminal's width is: %d" % width
EDIT: oh, I'm sorry. That's not a python standard lib one, here's the source of console.py (I don't know where it's from).
The module seems to work like that: It checks if termcap is available, when yes. It uses that; if no it checks whether the terminal supports a special ioctl call and that does not work, too, it checks for the environment variables some shells export for that.
This will probably work on UNIX only.
def getTerminalSize():
import os
env = os.environ
def ioctl_GWINSZ(fd):
try:
import fcntl, termios, struct, os
cr = struct.unpack('hh', fcntl.ioctl(fd, termios.TIOCGWINSZ,
'1234'))
except:
return
return cr
cr = ioctl_GWINSZ(0) or ioctl_GWINSZ(1) or ioctl_GWINSZ(2)
if not cr:
try:
fd = os.open(os.ctermid(), os.O_RDONLY)
cr = ioctl_GWINSZ(fd)
os.close(fd)
except:
pass
if not cr:
cr = (env.get('LINES', 25), env.get('COLUMNS', 80))
### Use get(key[, default]) instead of a try/catch
#try:
# cr = (env['LINES'], env['COLUMNS'])
#except:
# cr = (25, 80)
return int(cr[1]), int(cr[0])
Code above didn't return correct result on my linux because winsize-struct has 4 unsigned shorts, not 2 signed shorts:
def terminal_size():
import fcntl, termios, struct
h, w, hp, wp = struct.unpack('HHHH',
fcntl.ioctl(0, termios.TIOCGWINSZ,
struct.pack('HHHH', 0, 0, 0, 0)))
return w, h
hp and hp should contain pixel width and height, but don't.
It's either:
import os
columns, rows = os.get_terminal_size(0)
# or
import shutil
columns, rows = shutil.get_terminal_size()
The shutil function is just a wrapper around os one that catches some errors and set up a fallback, however it has one huge caveat - it breaks when piping!, which is a pretty huge deal.
To get terminal size when piping use os.get_terminal_size(0) instead.
First argument 0 is an argument indicating that stdin file descriptor should be used instead of default stdout. We want to use stdin because stdout detaches itself when it is being piped which in this case raises an error.
I've tried to figure out when would it makes sense to use stdout instead of stdin argument and have no idea why it's a default here.
I searched around and found a solution for windows at :
http://code.activestate.com/recipes/440694-determine-size-of-console-window-on-windows/
and a solution for linux here.
So here is a version which works both on linux, os x and windows/cygwin :
""" getTerminalSize()
- get width and height of console
- works on linux,os x,windows,cygwin(windows)
"""
__all__=['getTerminalSize']
def getTerminalSize():
import platform
current_os = platform.system()
tuple_xy=None
if current_os == 'Windows':
tuple_xy = _getTerminalSize_windows()
if tuple_xy is None:
tuple_xy = _getTerminalSize_tput()
# needed for window's python in cygwin's xterm!
if current_os == 'Linux' or current_os == 'Darwin' or current_os.startswith('CYGWIN'):
tuple_xy = _getTerminalSize_linux()
if tuple_xy is None:
print "default"
tuple_xy = (80, 25) # default value
return tuple_xy
def _getTerminalSize_windows():
res=None
try:
from ctypes import windll, create_string_buffer
# stdin handle is -10
# stdout handle is -11
# stderr handle is -12
h = windll.kernel32.GetStdHandle(-12)
csbi = create_string_buffer(22)
res = windll.kernel32.GetConsoleScreenBufferInfo(h, csbi)
except:
return None
if res:
import struct
(bufx, bufy, curx, cury, wattr,
left, top, right, bottom, maxx, maxy) = struct.unpack("hhhhHhhhhhh", csbi.raw)
sizex = right - left + 1
sizey = bottom - top + 1
return sizex, sizey
else:
return None
def _getTerminalSize_tput():
# get terminal width
# src: http://stackoverflow.com/questions/263890/how-do-i-find-the-width-height-of-a-terminal-window
try:
import subprocess
proc=subprocess.Popen(["tput", "cols"],stdin=subprocess.PIPE,stdout=subprocess.PIPE)
output=proc.communicate(input=None)
cols=int(output[0])
proc=subprocess.Popen(["tput", "lines"],stdin=subprocess.PIPE,stdout=subprocess.PIPE)
output=proc.communicate(input=None)
rows=int(output[0])
return (cols,rows)
except:
return None
def _getTerminalSize_linux():
def ioctl_GWINSZ(fd):
try:
import fcntl, termios, struct, os
cr = struct.unpack('hh', fcntl.ioctl(fd, termios.TIOCGWINSZ,'1234'))
except:
return None
return cr
cr = ioctl_GWINSZ(0) or ioctl_GWINSZ(1) or ioctl_GWINSZ(2)
if not cr:
try:
fd = os.open(os.ctermid(), os.O_RDONLY)
cr = ioctl_GWINSZ(fd)
os.close(fd)
except:
pass
if not cr:
try:
cr = (env['LINES'], env['COLUMNS'])
except:
return None
return int(cr[1]), int(cr[0])
if __name__ == "__main__":
sizex,sizey=getTerminalSize()
print 'width =',sizex,'height =',sizey
Starting at Python 3.3 it is straight forward:
https://docs.python.org/3/library/os.html#querying-the-size-of-a-terminal
>>> import os
>>> ts = os.get_terminal_size()
>>> ts.lines
24
>>> ts.columns
80
It looks like there are some problems with that code, Johannes:
getTerminalSize needs to import os
what is env? looks like os.environ.
Also, why switch lines and cols before returning? If TIOCGWINSZ and stty both say lines then cols, I say leave it that way. This confused me for a good 10 minutes before I noticed the inconsistency.
Sridhar, I didn't get that error when I piped output. I'm pretty sure it's being caught properly in the try-except.
pascal, "HHHH" doesn't work on my machine, but "hh" does. I had trouble finding documentation for that function. It looks like it's platform dependent.
chochem, incorporated.
Here's my version:
def getTerminalSize():
"""
returns (lines:int, cols:int)
"""
import os, struct
def ioctl_GWINSZ(fd):
import fcntl, termios
return struct.unpack("hh", fcntl.ioctl(fd, termios.TIOCGWINSZ, "1234"))
# try stdin, stdout, stderr
for fd in (0, 1, 2):
try:
return ioctl_GWINSZ(fd)
except:
pass
# try os.ctermid()
try:
fd = os.open(os.ctermid(), os.O_RDONLY)
try:
return ioctl_GWINSZ(fd)
finally:
os.close(fd)
except:
pass
# try `stty size`
try:
return tuple(int(x) for x in os.popen("stty size", "r").read().split())
except:
pass
# try environment variables
try:
return tuple(int(os.getenv(var)) for var in ("LINES", "COLUMNS"))
except:
pass
# i give up. return default.
return (25, 80)
Many of the Python 2 implementations here will fail if there is no controlling terminal when you call this script. You can check sys.stdout.isatty() to determine if this is in fact a terminal, but that will exclude a bunch of cases, so I believe the most pythonic way to figure out the terminal size is to use the builtin curses package.
import curses
w = curses.initscr()
height, width = w.getmaxyx()
Try "blessings"
I was looking for the very same thing. It is very easy to use and offers tools for coloring, styling and positioning in the terminal. What you need is as easy as:
from blessings import Terminal
t = Terminal()
w = t.width
h = t.height
Works like a charm in Linux. (I'm not sure about MacOSX and Windows)
Download and documentation here
or you can install it with pip:
pip install blessings
I was trying the solution from here that calls out to stty size:
columns = int(subprocess.check_output(['stty', 'size']).split()[1])
However this failed for me because I was working on a script that expects redirected input on stdin, and stty would complain that "stdin isn't a terminal" in that case.
I was able to make it work like this:
with open('/dev/tty') as tty:
height, width = subprocess.check_output(['stty', 'size'], stdin=tty).split()
If you're using Python 3.3 or above, I'd recommend the built-in get_terminal_size() as already recommended. However if you are stuck with an older version and want a simple, cross-platform way of doing this, you could use asciimatics. This package supports versions of Python back to 2.7 and uses similar options to those suggested above to get the current terminal/console size.
Simply construct your Screen class and use the dimensions property to get the height and width. This has been proven to work on Linux, OSX and Windows.
Oh - and full disclosure here: I am the author, so please feel free to open a new issue if you have any problems getting this to work.
#reannual's answer works well, but there's an issue with it: os.popen is now deprecated. The subprocess module should be used instead, so here's a version of #reannual's code that uses subprocess and directly answers the question (by giving the column width directly as an int:
import subprocess
columns = int(subprocess.check_output(['stty', 'size']).split()[1])
Tested on OS X 10.9
Here is an version that should be Linux and Solaris compatible. Based on the posts and commments from madchine. Requires the subprocess module.
def termsize():
import shlex, subprocess, re
output = subprocess.check_output(shlex.split('/bin/stty -a'))
m = re.search('rows\D+(?P\d+); columns\D+(?P\d+);', output)
if m:
return m.group('rows'), m.group('columns')
raise OSError('Bad response: %s' % (output))
>>> termsize()
('40', '100')
I'm trying to read data from memory of a process by inputing the process name, then finding PID using psutil. So far I have this:
import ctypes
from ctypes import *
from ctypes.wintypes import *
import win32ui
import psutil # install, not a default module
import sys
# input process name
nameprocess = "notepad.exe"
# find pid
def getpid():
for proc in psutil.process_iter():
if proc.name() == nameprocess:
return proc.pid
PROCESS_ID = getpid()
if PROCESS_ID == None:
print "Process was not found"
sys.exit(1)
# read from addresses
STRLEN = 255
PROCESS_VM_READ = 0x0010
process = windll.kernel32.OpenProcess(PROCESS_VM_READ, 0, PROCESS_ID)
readProcMem = windll.kernel32.ReadProcessMemory
buf = ctypes.create_string_buffer(STRLEN)
for i in range(1,100):
if readProcMem(process, hex(i), buf, STRLEN, 0):
print buf.raw
The last for loop should read and print contents of the first 100 addresses in the process if I'm getting this right. Only thing is, the output looks like complete gibberish.
There are 2 problems for me here: first, am I really reading the addresses from the selected process this way? And second, how can I figure how long in the loop I should go, if there is maybe some kind of end address?
I didn't install psutil, but just pulled a process ID and valid virtual address using Task Manager and SysInternals VMMap. The numbers will vary of course.
Good practice with ctypes is to define the argument types and return value via .argtypes and .restype. Get your own instance of the kernel32 library because changing the attributes of the cached windll.kernel32 instance could cause issues with other modules using ctypes and kernel32.
You need a valid virtual address. In answer to your 2nd problem, I think VMMap proves there is a way to do it. Pick up a copy of Windows Internals to learn the techniques.
from ctypes import *
from ctypes.wintypes import *
PROCESS_ID = 9476 # From TaskManager for Notepad.exe
PROCESS_HEADER_ADDR = 0x7ff7b81e0000 # From SysInternals VMMap utility
# read from addresses
STRLEN = 255
PROCESS_VM_READ = 0x0010
k32 = WinDLL('kernel32')
k32.OpenProcess.argtypes = DWORD,BOOL,DWORD
k32.OpenProcess.restype = HANDLE
k32.ReadProcessMemory.argtypes = HANDLE,LPVOID,LPVOID,c_size_t,POINTER(c_size_t)
k32.ReadProcessMemory.restype = BOOL
process = k32.OpenProcess(PROCESS_VM_READ, 0, PROCESS_ID)
buf = create_string_buffer(STRLEN)
s = c_size_t()
if k32.ReadProcessMemory(process, PROCESS_HEADER_ADDR, buf, STRLEN, byref(s)):
print(s.value,buf.raw)
Output (Note 'MZ' is the start of a program header):
255 b'MZ\x90\x00\x03\x00\x00\x00\x04\x00\x00\x00\xff\xff\x00\x00\xb8\x00\x00\x00\x00\x00\x00\x00#\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xe8\x00\x00\x00\x0e\x1f\xba\x0e\x00\xb4\t\xcd!\xb8\x01L\xcd!This program cannot be run in DOS mode.\r\r\n$\x00\x00\x00\x00\x00\x00\x00\xd0\x92\xa7\xd1\x94\xf3\xc9\x82\x94\xf3\xc9\x82\x94\xf3\xc9\x82\x9d\x8bZ\x82\x8a\xf3\xc9\x82\xfb\x97\xca\x83\x97\xf3\xc9\x82\xfb\x97\xcd\x83\x83\xf3\xc9\x82\xfb\x97\xcc\x83\x91\xf3\xc9\x82\xfb\x97\xc8\x83\x8f\xf3\xc9\x82\x94\xf3\xc8\x82\x82\xf2\xc9\x82\xfb\x97\xc1\x83\x8d\xf3\xc9\x82\xfb\x976\x82\x95\xf3\xc9\x82\xfb\x97\xcb\x83\x95\xf3\xc9\x82Rich\x94\xf3\xc9\x82\x00\x00\x00\x00\x00\x00\x00\x00PE\x00\x00d\x86\x06\x00^\'\x0f\x84\x00\x00\x00\x00\x00\x00\x00\x00\xf0\x00"'
Here's a screenshot of VMMap indicating the header address of notepad.exe:
Here's a screenshot of a hexdump of the content of notepad.exe that matches the output of the program:
On Windows, the PyMem library can help you with that: https://pymem.readthedocs.io/
Is there a way in python to programmatically determine the width of the console? I mean the number of characters that fits in one line without wrapping, not the pixel width of the window.
Edit
Looking for a solution that works on Linux
Not sure why it is in the module shutil, but it landed there in Python 3.3. See:
Querying the size of the output terminal
>>> import shutil
>>> shutil.get_terminal_size((80, 20)) # pass fallback
os.terminal_size(columns=87, lines=23) # returns a named-tuple
A low-level implementation is in the os module. Cross-platform—works under Linux, Mac OS, and Windows, probably other Unix-likes. There's a backport as well, though no longer relevant.
import os
rows, columns = os.popen('stty size', 'r').read().split()
uses the 'stty size' command which according to a thread on the python mailing list is reasonably universal on linux. It opens the 'stty size' command as a file, 'reads' from it, and uses a simple string split to separate the coordinates.
Unlike the os.environ["COLUMNS"] value (which I can't access in spite of using bash as my standard shell) the data will also be up-to-date whereas I believe the os.environ["COLUMNS"] value would only be valid for the time of the launch of the python interpreter (suppose the user resized the window since then).
(See answer by #GringoSuave on how to do this on python 3.3+)
use
import console
(width, height) = console.getTerminalSize()
print "Your terminal's width is: %d" % width
EDIT: oh, I'm sorry. That's not a python standard lib one, here's the source of console.py (I don't know where it's from).
The module seems to work like that: It checks if termcap is available, when yes. It uses that; if no it checks whether the terminal supports a special ioctl call and that does not work, too, it checks for the environment variables some shells export for that.
This will probably work on UNIX only.
def getTerminalSize():
import os
env = os.environ
def ioctl_GWINSZ(fd):
try:
import fcntl, termios, struct, os
cr = struct.unpack('hh', fcntl.ioctl(fd, termios.TIOCGWINSZ,
'1234'))
except:
return
return cr
cr = ioctl_GWINSZ(0) or ioctl_GWINSZ(1) or ioctl_GWINSZ(2)
if not cr:
try:
fd = os.open(os.ctermid(), os.O_RDONLY)
cr = ioctl_GWINSZ(fd)
os.close(fd)
except:
pass
if not cr:
cr = (env.get('LINES', 25), env.get('COLUMNS', 80))
### Use get(key[, default]) instead of a try/catch
#try:
# cr = (env['LINES'], env['COLUMNS'])
#except:
# cr = (25, 80)
return int(cr[1]), int(cr[0])
Code above didn't return correct result on my linux because winsize-struct has 4 unsigned shorts, not 2 signed shorts:
def terminal_size():
import fcntl, termios, struct
h, w, hp, wp = struct.unpack('HHHH',
fcntl.ioctl(0, termios.TIOCGWINSZ,
struct.pack('HHHH', 0, 0, 0, 0)))
return w, h
hp and hp should contain pixel width and height, but don't.
It's either:
import os
columns, rows = os.get_terminal_size(0)
# or
import shutil
columns, rows = shutil.get_terminal_size()
The shutil function is just a wrapper around os one that catches some errors and set up a fallback, however it has one huge caveat - it breaks when piping!, which is a pretty huge deal.
To get terminal size when piping use os.get_terminal_size(0) instead.
First argument 0 is an argument indicating that stdin file descriptor should be used instead of default stdout. We want to use stdin because stdout detaches itself when it is being piped which in this case raises an error.
I've tried to figure out when would it makes sense to use stdout instead of stdin argument and have no idea why it's a default here.
I searched around and found a solution for windows at :
http://code.activestate.com/recipes/440694-determine-size-of-console-window-on-windows/
and a solution for linux here.
So here is a version which works both on linux, os x and windows/cygwin :
""" getTerminalSize()
- get width and height of console
- works on linux,os x,windows,cygwin(windows)
"""
__all__=['getTerminalSize']
def getTerminalSize():
import platform
current_os = platform.system()
tuple_xy=None
if current_os == 'Windows':
tuple_xy = _getTerminalSize_windows()
if tuple_xy is None:
tuple_xy = _getTerminalSize_tput()
# needed for window's python in cygwin's xterm!
if current_os == 'Linux' or current_os == 'Darwin' or current_os.startswith('CYGWIN'):
tuple_xy = _getTerminalSize_linux()
if tuple_xy is None:
print "default"
tuple_xy = (80, 25) # default value
return tuple_xy
def _getTerminalSize_windows():
res=None
try:
from ctypes import windll, create_string_buffer
# stdin handle is -10
# stdout handle is -11
# stderr handle is -12
h = windll.kernel32.GetStdHandle(-12)
csbi = create_string_buffer(22)
res = windll.kernel32.GetConsoleScreenBufferInfo(h, csbi)
except:
return None
if res:
import struct
(bufx, bufy, curx, cury, wattr,
left, top, right, bottom, maxx, maxy) = struct.unpack("hhhhHhhhhhh", csbi.raw)
sizex = right - left + 1
sizey = bottom - top + 1
return sizex, sizey
else:
return None
def _getTerminalSize_tput():
# get terminal width
# src: http://stackoverflow.com/questions/263890/how-do-i-find-the-width-height-of-a-terminal-window
try:
import subprocess
proc=subprocess.Popen(["tput", "cols"],stdin=subprocess.PIPE,stdout=subprocess.PIPE)
output=proc.communicate(input=None)
cols=int(output[0])
proc=subprocess.Popen(["tput", "lines"],stdin=subprocess.PIPE,stdout=subprocess.PIPE)
output=proc.communicate(input=None)
rows=int(output[0])
return (cols,rows)
except:
return None
def _getTerminalSize_linux():
def ioctl_GWINSZ(fd):
try:
import fcntl, termios, struct, os
cr = struct.unpack('hh', fcntl.ioctl(fd, termios.TIOCGWINSZ,'1234'))
except:
return None
return cr
cr = ioctl_GWINSZ(0) or ioctl_GWINSZ(1) or ioctl_GWINSZ(2)
if not cr:
try:
fd = os.open(os.ctermid(), os.O_RDONLY)
cr = ioctl_GWINSZ(fd)
os.close(fd)
except:
pass
if not cr:
try:
cr = (env['LINES'], env['COLUMNS'])
except:
return None
return int(cr[1]), int(cr[0])
if __name__ == "__main__":
sizex,sizey=getTerminalSize()
print 'width =',sizex,'height =',sizey
Starting at Python 3.3 it is straight forward:
https://docs.python.org/3/library/os.html#querying-the-size-of-a-terminal
>>> import os
>>> ts = os.get_terminal_size()
>>> ts.lines
24
>>> ts.columns
80
It looks like there are some problems with that code, Johannes:
getTerminalSize needs to import os
what is env? looks like os.environ.
Also, why switch lines and cols before returning? If TIOCGWINSZ and stty both say lines then cols, I say leave it that way. This confused me for a good 10 minutes before I noticed the inconsistency.
Sridhar, I didn't get that error when I piped output. I'm pretty sure it's being caught properly in the try-except.
pascal, "HHHH" doesn't work on my machine, but "hh" does. I had trouble finding documentation for that function. It looks like it's platform dependent.
chochem, incorporated.
Here's my version:
def getTerminalSize():
"""
returns (lines:int, cols:int)
"""
import os, struct
def ioctl_GWINSZ(fd):
import fcntl, termios
return struct.unpack("hh", fcntl.ioctl(fd, termios.TIOCGWINSZ, "1234"))
# try stdin, stdout, stderr
for fd in (0, 1, 2):
try:
return ioctl_GWINSZ(fd)
except:
pass
# try os.ctermid()
try:
fd = os.open(os.ctermid(), os.O_RDONLY)
try:
return ioctl_GWINSZ(fd)
finally:
os.close(fd)
except:
pass
# try `stty size`
try:
return tuple(int(x) for x in os.popen("stty size", "r").read().split())
except:
pass
# try environment variables
try:
return tuple(int(os.getenv(var)) for var in ("LINES", "COLUMNS"))
except:
pass
# i give up. return default.
return (25, 80)
Many of the Python 2 implementations here will fail if there is no controlling terminal when you call this script. You can check sys.stdout.isatty() to determine if this is in fact a terminal, but that will exclude a bunch of cases, so I believe the most pythonic way to figure out the terminal size is to use the builtin curses package.
import curses
w = curses.initscr()
height, width = w.getmaxyx()
Try "blessings"
I was looking for the very same thing. It is very easy to use and offers tools for coloring, styling and positioning in the terminal. What you need is as easy as:
from blessings import Terminal
t = Terminal()
w = t.width
h = t.height
Works like a charm in Linux. (I'm not sure about MacOSX and Windows)
Download and documentation here
or you can install it with pip:
pip install blessings
I was trying the solution from here that calls out to stty size:
columns = int(subprocess.check_output(['stty', 'size']).split()[1])
However this failed for me because I was working on a script that expects redirected input on stdin, and stty would complain that "stdin isn't a terminal" in that case.
I was able to make it work like this:
with open('/dev/tty') as tty:
height, width = subprocess.check_output(['stty', 'size'], stdin=tty).split()
If you're using Python 3.3 or above, I'd recommend the built-in get_terminal_size() as already recommended. However if you are stuck with an older version and want a simple, cross-platform way of doing this, you could use asciimatics. This package supports versions of Python back to 2.7 and uses similar options to those suggested above to get the current terminal/console size.
Simply construct your Screen class and use the dimensions property to get the height and width. This has been proven to work on Linux, OSX and Windows.
Oh - and full disclosure here: I am the author, so please feel free to open a new issue if you have any problems getting this to work.
#reannual's answer works well, but there's an issue with it: os.popen is now deprecated. The subprocess module should be used instead, so here's a version of #reannual's code that uses subprocess and directly answers the question (by giving the column width directly as an int:
import subprocess
columns = int(subprocess.check_output(['stty', 'size']).split()[1])
Tested on OS X 10.9
Here is an version that should be Linux and Solaris compatible. Based on the posts and commments from madchine. Requires the subprocess module.
def termsize():
import shlex, subprocess, re
output = subprocess.check_output(shlex.split('/bin/stty -a'))
m = re.search('rows\D+(?P\d+); columns\D+(?P\d+);', output)
if m:
return m.group('rows'), m.group('columns')
raise OSError('Bad response: %s' % (output))
>>> termsize()
('40', '100')
I'm trying to redirect the output of printf functions to a file on Windows. I'm using ctypes with python3 to invoke the functions. My code is:
import os, sys
from ctypes import *
if __name__ == '__main__':
print("begin")
saved_stdout=os.dup(1)
test_file=open("TEST.TXT", "w")
os.dup2(test_file.fileno(), 1)
test_file.close()
print("python print")
cdll.msvcrt.printf(b"Printf function 1\n")
cdll.msvcrt.printf(b"Printf function 2\n")
cdll.msvcrt.printf(b"Printf function 3\n")
os.dup2(saved_stdout, 1)
print("end")
But when I run the code from Eclipse I get the following on the screen:
begin
end
Printf function 1
Printf function 2
Printf function 3
...and the following in the TEST.txt
python print
When I run this from cmd, this is what is on the screen:
begin
end
..and this is in the TEST.txt:
python print
When I comment out the second dup2() statement e.g.
import os, sys
from ctypes import *
if __name__ == '__main__':
print("begin")
saved_stdout=os.dup(1)
test_file=open("TEST.TXT", "w")
os.dup2(test_file.fileno(), 1)
test_file.close()
print("python print")
cdll.msvcrt.printf(b"Printf function 1\n")
cdll.msvcrt.printf(b"Printf function 2\n")
cdll.msvcrt.printf(b"Printf function 3\n")
#os.dup2(saved_stdout, 1)
print("end")
From Eclipse, on the screen:
begin
...and in the TEST.txt file:
python print
end
Printf function 1
Printf function 2
Printf function 3
From cmd, on the screen:
begin
...and in the TEST.txt file:
python print
end
I'm totally confused now. I read all the redirection threads here on StackOverflow and I can't understand what's going on.
Anyway, what I've gathered is that C functions access the stdout that is bind directly to the file descriptor, while python uses a special object for that - stdout File Object. So the elementary sys.stdout=*something* doesn't work with ctypes.
I've even tried os.fdopen(1) on the dup2-ed output and then calling flush() after every printf statement but this isn't working again.
I'm totally out of ideas now and would appreciate if someone have a solution for this.
Use the same C runtime that CPython 3.x uses (e.g. msvcr100.dll for 3.3). Also include a call to fflush(NULL) before and after redirecting stdout. For good measure, redirect the Windows StandardOutput handle, in case a program uses the Windows API directly.
This can get complicated if the DLL uses a different C runtime, which has its own set of POSIX file descriptors. That said, it should be OK if it gets loaded after you've redirected Windows StandardOutput.
Edit:
I've modified the example to run in Python 3.5+. VC++ 14's new "Universal CRT" makes it much more difficult to use C standard I/O via ctypes.
import os
import sys
import ctypes, ctypes.util
kernel32 = ctypes.WinDLL('kernel32')
STD_OUTPUT_HANDLE = -11
if sys.version_info < (3, 5):
libc = ctypes.CDLL(ctypes.util.find_library('c'))
else:
if hasattr(sys, 'gettotalrefcount'): # debug build
libc = ctypes.CDLL('ucrtbased')
else:
libc = ctypes.CDLL('api-ms-win-crt-stdio-l1-1-0')
# VC 14.0 doesn't implement printf dynamically, just
# __stdio_common_vfprintf. This take a va_array arglist,
# which I won't implement, so I escape format specificiers.
class _FILE(ctypes.Structure):
"""opaque C FILE type"""
libc.__acrt_iob_func.restype = ctypes.POINTER(_FILE)
def _vprintf(format, arglist_ignored):
options = ctypes.c_longlong(0) # no legacy behavior
stdout = libc.__acrt_iob_func(1)
format = format.replace(b'%%', b'\0')
format = format.replace(b'%', b'%%')
format = format.replace(b'\0', b'%%')
arglist = locale = None
return libc.__stdio_common_vfprintf(
options, stdout, format, locale, arglist)
def _printf(format, *args):
return _vprintf(format, args)
libc.vprintf = _vprintf
libc.printf = _printf
def do_print(label):
print("%s: python print" % label)
s = ("%s: libc _write\n" % label).encode('ascii')
libc._write(1, s, len(s))
s = ("%s: libc printf\n" % label).encode('ascii')
libc.printf(s)
libc.fflush(None) # flush all C streams
if __name__ == '__main__':
# save POSIX stdout and Windows StandardOutput
fd_stdout = os.dup(1)
hStandardOutput = kernel32.GetStdHandle(STD_OUTPUT_HANDLE)
do_print("begin")
# redirect POSIX and Windows
with open("TEST.TXT", "w") as test:
os.dup2(test.fileno(), 1)
kernel32.SetStdHandle(STD_OUTPUT_HANDLE, libc._get_osfhandle(1))
do_print("redirected")
# restore POSIX and Windows
os.dup2(fd_stdout, 1)
kernel32.SetStdHandle(STD_OUTPUT_HANDLE, hStandardOutput)
do_print("end")