I have an issue with Pool.map in combination with the curses module of Python. Whenever I compute bigger workloads with Pool.map my curses UI breaks: it does not react upon the default screen's getch anymore. Instead of reading in any pressed key instantly (and continuing with parsing it) I can press any amount of keys until I hit enter. Sometimes (in addition to that) even the UI breaks (like showing a fraction of my normal shell).
Curses UI wrapper
This is a wrapper class (Screen) that handles the curses UI stuff for me:
# -*- coding: utf-8 -*-
import curses
class Screen(object):
def __init__(self):
# create a default screen
self.__mainscr = curses.initscr()
self.__stdscr = curses.newwin(curses.LINES - 2, curses.COLS - 2, 1, 1)
self.__max_height, self.__max_width = self.__stdscr.getmaxyx()
# start coloring
curses.start_color()
curses.use_default_colors()
# define colors
curses.init_pair(1, 197, -1) # red
curses.init_pair(2, 227, -1) # yellow
curses.init_pair(3, curses.COLOR_MAGENTA, -1)
curses.init_pair(4, curses.COLOR_GREEN, -1) # darkgreen
curses.init_pair(5, curses.COLOR_BLUE, -1)
curses.init_pair(6, curses.COLOR_BLACK, curses.COLOR_WHITE)
curses.init_pair(7, curses.COLOR_WHITE, -1)
curses.init_pair(8, curses.COLOR_CYAN, -1)
curses.init_pair(9, 209, -1) # orange
curses.init_pair(10, 47, -1) # green
def add_str_to_scr(self, add_str: str, colorpair: int = 7):
self.__stdscr.addstr(str(add_str), curses.color_pair(colorpair))
def linebreak(self):
self.__stdscr.addstr("\n")
def clear_screen(self):
self.__stdscr.clear()
def refresh_screen(self):
self.__stdscr.refresh()
def wait_for_enter_or_esc(self):
curses.noecho()
while True:
c = self.__stdscr.getch()
if c == 10 or c == 27: # 10: Enter, 27: ESC
break
curses.echo()
def get_user_input_chr(self) -> str:
return chr(self.__stdscr.getch())
def get_user_input_str(self) -> str:
return self.__stdscr.getstr().decode(encoding="utf-8")
Actual program
I've written a small example, since the mentioned failure always happen when I combine Pool.map within a curses UI and have high workload. The code just computes some useless mult and add stuff on a numpy array.
import curses
from screen import Screen
from multiprocessing import Pool, cpu_count
import numpy as np
s = Screen() # initializing my Screen wrapper
np.random.seed(1234) # setting the rng fixed to make results comparable
# worker function to simulate workload
def worker(arr):
return arr * 2 + 1
s.clear_screen() # cleans the screen
s.refresh_screen() # displays current buffer's content
s.add_str_to_scr("Start processing data...")
s.linebreak()
s.linebreak()
s.refresh_screen()
# data to feed worker function with (sliced by rows)
data_arr = np.random.rand(8, int(1e7)) # <-- big array for high workload
with Pool(cpu_count()) as p:
buffer = p.map(worker, [data_arr[row] for row in np.ndindex(data_arr.shape[0])])
s.add_str_to_scr("...finished processing:")
s.linebreak()
s.linebreak()
s.refresh_screen()
for row in buffer:
s.add_str_to_scr(row[0:3])
s.linebreak()
s.refresh_screen()
# *Here* the program should wait until the user presses *any* key
# and continue INSTANTLY when any key gets pressed.
# However, for big workloads, it does not react to single key presses,
# but wait for any amount of keys pressed until you hit 'Enter'
s.get_user_input_chr()
curses.endwin()
Now, when I execute the code with a high workload (i.e. crunching an array of shape (8, int(1e7) equals 8 rows with 10,000,000 columns) curse's getch breaks and I get this behavior:
As you can see, I can hit q (or any other key) as often as I want, but curse's getch does not react. I have to press the Enter key to make it recognize the input.
Moreover the first line gets overwritten with my original shell's output for some reason.
This behavior only happens, when the calculation of Pool.map roughly needs 1 second or longer.
When I set data_arr to a small array like np.random.rand(8, 100) everything works like a charm, but as soon as I feed big arrays where the computation takes like >= 1second, this weird bug appears and breaks my curses UI.
Any ideas?
Is Pool.map not joining the worker processes correctly somehow?
The program is doing what you told it to do:
you're calling initscr (and ignoring the fact that curses creates a top-level window),
then creating a subwindow which covers most of the screen,
printing several lines on the screen, refreshing the display after each line, and
finally at the end, waiting for input from the subwindow.
However, your program doesn't call cbreak, raw, etc., which would let you read an unbuffered (no "Enter" pressed) character. Also, the program doesn't turn off echo. If the load is light, you won't notice, since response is fast. But under heavy load, e.g., swapping or high memory/CPU usage, it'll still be recovering when it gets to the prompt. So you notice.
Regarding the screen size, perhaps you meant
self.__stdscr = curses.newwin(curses.LINES - 1, curses.COLS - 1, 0, 0)
but supposing that you intended to leave "empty" space around the window, you could improve things by doing
self.__mainscr.refresh()
immediately after initscr (which would erase the screen).
Related
I'm trying to use curses for a class project where we need to use python
The premise is a server/client chat. I was trying to implement it such that the incoming messages from the server would be displayed above the message prompt. I was able to circumvent the python input() function pausing the program with a second thread displaying the messages I decided to use curses to print things at different parts of the screen.
when I tried multi threading with curses I it had a lot of issues with the input cursor. It used displayed the characters that were being input to the program right next to the incoming chat text despite the cursor visually moving back on the screen to the input area
I've isolated the code that's supposed to make this happen:
import threading
from time import sleep
import curses
screen = curses.initscr()
screen.clear()
screen.refresh()
num_rows, num_cols = screen.getmaxyx()
#prints what would be chat to the top of the screen
def thread_delay(thread_name, delay):
i = 0
while i < 10:
#is supposed to save the cursor's location before moving it
global x_location, y_location
x_location, y_location = curses.getsyx()
screen.addstr(0+i, 0, "stuff")
screen.refresh()
#is supposed to restore the cursor to the input location
curses.setsyx(x_location, y_location)
curses.doupdate()
sleep(1)
i+=1
#gets and displays input to the bottom of the screen
def my_raw_input(stdscr, r, c, prompt_string):
curses.echo()
stdscr.addstr(r, c, prompt_string)
stdscr.refresh()
global input
input = stdscr.getstr(r + 1, c, 20)
return # ^^^^ reading input at next line
#threads
t1 = threading.Thread(target = thread_delay, args = ('t1', 1,))
t2 = threading.Thread(target = my_raw_input, args = (screen, num_rows - 4, 0, "type what you want"))
t1.start()
t2.start()
t1.join()
t2.join()
curses.napms(3000)
curses.endwin()
#for debugging
print(input.decode('utf-8'))
Picture of what is happening vs what's supposed to happen:
I guess your class is over, so it might not be useful anymore...
There are several problems....
first problem is that you are not accepting curses text input, and instead using standard input. The curses module, works by moving the cursor around and adding a character, so wherever your last addstr or addch command ends is where the cursor is positioned, and will next add text. So you could possibly move the cursor back to where you want the input to go, after writing to the screen or you could use curses to capture keyboard input.
Secondly using curses in two processes at the same time is probably not a good idea, since it bypasses the GIL. Instead, you could split your code threads into a server and a curses TUI, and share a global send queue and a receive queue from the multiprocessing.Manager to transfer messages.
Then your server thread can send messages from the send_queue with a .get() and receive messages and putting them in the receive_queue with a .put(). Conversely your curses thread can display messages from the recv_queue with a .get() and put a string composed from curses.getch() or curses.getkey() into the send_queue with a .put()
also consider using the curses.wrapper to simplify entering and exiting your curses features. I think you probably didn't use this, because you tried to use curses over multiple processes in which case a wrapper might not work so well.
I'm sure there are other and probably better ways to do it, but this is just one of many. But I hope this helps someone, or encourages someone more experienced to contribute a better answer.
I want to create a Gtk program that requires to continuously update a ListBox, i.e. add and remove rows. I am trying this approach. It could be cleaner but here it is just long enough to explain the problem.
gi.require_version('Gtk', '3.0')
from gi.repository import Gtk
from time import sleep
from threading import Thread
def add_widgets(listbox):
# add 5 rows
for _ in range(5):
for _ in range(5):
add_row(listbox)
window.show_all() # make the changes visible
sleep(1) # to make the change stay
# remove all the rows
for row in listbox.get_children():
listbox.remove(row)
window.show_all()
sleep(1)
# all the addition and removal done
print('finished')
def add_row(listbox):
listboxrow = Gtk.ListBoxRow()
label = Gtk.Label()
label.set_text("Hey There")
listboxrow.add(label)
listbox.add(listboxrow)
window = Gtk.Window()
window.connect('destroy', Gtk.main_quit)
listbox = Gtk.ListBox()
window.add(listbox)
window.show_all()
# Thread to add and remove rows
update_thread = Thread(target=add_widgets, args=(listbox, ))
update_thread.start()
Gtk.main()
This code runs fine like 50% of the time. For the rest, it gives me 3 types of errors, all of them randomly.
The good old SegFault after like 2 or 3 iteration of the parent loop in add_widgets
The following:
**
Gtk:ERROR:../gtk/gtk/gtkcssnode.c:319:lookup_in_global_parent_cache: assertion failed: (node->cache == NULL)
Bail out! Gtk:ERROR:../gtk/gtk/gtkcssnode.c:319:lookup_in_global_parent_cache: assertion failed: (node->cache == NULL)
Aborted
The last one doesn't crash the program but it adds random number of rows instead. i.e. it adds 3(random) rows instead of 5, as specified or maybe no rows at all.
I've tried adding more sleep statements at appropriate places because initially, I thought, it could be because the windows are not ready when updating widgets.
I wanna know why its happening and how can I fix it.
EDIT: It's probably something to do with the UI and the thread synchronization because, it(the crash) happens more frequently when interacting with the window. For instance, when you are dragging it, it has more chances of crashing.
You must not call GTK+ code from other threads. All interactions must be done in the main thread. This is documented in https://developer.gnome.org/gdk3/stable/gdk3-Threads.html.
If you want to do some "background" updates of your widgets, you can simulate that with functions such as g_idle_add() (https://developer.gnome.org/glib/stable/glib-The-Main-Event-Loop.html#g-idle-add), g_timeout_add() (https://developer.gnome.org/glib/stable/glib-The-Main-Event-Loop.html#g-timeout-add) and other related functions.
I've been working on a prototype application in Python for stereoscopic imaging of a 3D model using VTK, but I'm running into some issues on the interface end of things. The goal of the code below at the moment is to zoom in on both renderWindows when the middlemouse is pressed. However, upon calling the vtkRenderWindowInteractor.Start() function, my vtkRenderWindowInteractors are effectively stalling out the entire program as if they were being run in the same thread. Even more curious is that keyboard interrupts are not being thrown when I use CTRL-C (I'm working in UNIX shell) until I close the render windows manually using the 'x' button. If I just close the window manually without hitting CTRL-C, the program picks up directly after the Start() call (e.g. in the code below, the infinite while loop). I've provided a sequence of screen captures at the end of this post to visualize exactly what is happening in the case that my explanation is confusing.
I've tried multiple workarounds to remedy this but none so far have worked. Threading the renders into isolated threads made no difference even when I tried using ncurses for input, while forking them to a new process resulted in some OS issues that I'd rather not deal with. The most current interactor styles method (shown below) where I just use built-in VTK listeners works to a degree, allowing me to detect inputs when the windows are in focus and the interactors are active, but because of the lack of association between the camera and the MyInteractorStyle class, I can't really access the cameras without the inclusion a loop after the Start() call, which leads me right back to where I started.
Any thoughts? Am I just misunderstanding how VTK's render tools are supposed to be used?
from vtk import*
import os.path
#import thread
#import time
#import threading
#import curses
class MyInteractorStyle(vtk.vtkInteractorStyleTrackballCamera):
pos1 = [0, 0, 200]
foc1 = [0, 0, 0]
pos2 = [40, 0, 200]
foc2 = [0, 0, 0]
def __init__(self,parent=None):
self.AddObserver("MiddleButtonPressEvent", self.middleButtonPressEvent)
self.AddObserver("MiddleButtonReleaseEvent", self.middleButtonReleaseEvent)
def middleButtonPressEvent(self,obj,event):
print "Middle button pressed"
self.pos1[2] += 10
self.pos2[2] += 30
self.OnMiddleButtonDown()
return
def middleButtonReleaseEvent(self,obj,event):
print "Middle button released"
self.OnMiddleButtonUp()
return
def main():
# create two cameras
camera1 = vtkCamera()
camera1.SetPosition(0,0,200)
camera1.SetFocalPoint(0,0,0)
camera2 = vtkCamera()
camera2.SetPosition(40,0,200)
camera2.SetFocalPoint(0,0,0)
# create a rendering window and renderer
ren1 = vtkRenderer()
ren1.SetActiveCamera(camera1)
ren2 = vtkRenderer()
ren2.SetActiveCamera(camera2)
# create source
reader = vtkPolyDataReader()
path = "/home/compilezone/Documents/3DSlicer/SlicerScenes/LegoModel-6_25/Model_5_blood.vtk"
reader.SetFileName(path)
print(path)
reader.Update()
# create render window
renWin1 = vtkRenderWindow()
renWin1.AddRenderer(ren1)
renWin2 = vtkRenderWindow()
renWin2.AddRenderer(ren2)
# create a render window interactor
inputHandler = MyInteractorStyle()
iren1 = vtkRenderWindowInteractor()
iren1.SetRenderWindow(renWin1)
iren1.SetInteractorStyle(inputHandler)
iren2 = vtkRenderWindowInteractor()
iren2.SetRenderWindow(renWin2)
iren2.SetInteractorStyle(inputHandler)
# mapper
mapper = vtkPolyDataMapper()
mapper.SetInput(reader.GetOutput())
# actor
actor = vtkActor()
actor.SetMapper(mapper)
# assign actor to the renderer
ren1.AddActor(actor)
ren2.AddActor(actor)
# enable user interface interactor
iren1.Initialize()
iren2.Initialize()
renWin1.Render()
renWin2.Render()
iren1.Start()
iren2.Start()
print "Test"
while 1:
pos1 = iren1.GetInteractorStyle().pos1
foc1 = iren1.GetInteractorStyle().foc1
pos2 = iren2.GetInteractorStyle().pos2
foc2 = iren2.GetInteractorStyle().foc2
print
if __name__ == '__main__':
main()
Program running
KeyboardInterrupt (CTRL-C hit and echoed in terminal but nothing happens)
Render windows manually closed, KeyboardInterrupt thrown
Calling Start() on a RenderWindowInteractor starts the event loop necessary to execute render events, much as the event loop in a GUI. So what you're trying to do, starting two event loops, doesn't really make sense.
A conceptual workaround would be to not call Start on the RenderWindowInteractors but to write a small GUI with multiple toolkit-specific RenderWindowInteractors and use that GUI's event loop.
As an example, here's how this is done with GUI toolkit-specific code in tvtk's wxVtkRenderWindowInteractor class, which doesn't call start on the RenderWindowInteractor but instead uses the GUI's event loop to manage events:
def wxVTKRenderWindowInteractorConeExample():
"""Like it says, just a simple example
"""
# every wx app needs an app
app = wx.PySimpleApp()
# create the top-level frame, sizer and wxVTKRWI
frame = wx.Frame(None, -1, "wxVTKRenderWindowInteractor", size=(400,400))
widget = wxVTKRenderWindowInteractor(frame, -1)
sizer = wx.BoxSizer(wx.VERTICAL)
sizer.Add(widget, 1, wx.EXPAND)
frame.SetSizer(sizer)
frame.Layout()
# It would be more correct (API-wise) to call widget.Initialize() and
# widget.Start() here, but Initialize() calls RenderWindow.Render().
# That Render() call will get through before we can setup the
# RenderWindow() to render via the wxWidgets-created context; this
# causes flashing on some platforms and downright breaks things on
# other platforms. Instead, we call widget.Enable(). This means
# that the RWI::Initialized ivar is not set, but in THIS SPECIFIC CASE,
# that doesn't matter.
widget.Enable(1)
widget.AddObserver("ExitEvent", lambda o,e,f=frame: f.Close())
ren = vtk.vtkRenderer()
widget.GetRenderWindow().AddRenderer(ren)
cone = vtk.vtkConeSource()
cone.SetResolution(8)
coneMapper = vtk.vtkPolyDataMapper()
coneMapper.SetInput(cone.GetOutput())
coneActor = vtk.vtkActor()
coneActor.SetMapper(coneMapper)
ren.AddActor(coneActor)
# show the window
frame.Show()
app.MainLoop()
(Note that this code is not altered and has some clear differences with what you are trying to do.)
Also, the reason that ctrl+C doesn't work is because the VTK event loop doesn't do anything with this event. Some GUIs do respect this event, including wxpython. But if you aren't using a GUI that respects this event (for example, Qt) you can manually tell the python interpreter to intercept this event and crash instead of forwarding the event to the GUI event loop:
import signal
signal.signal(signal.SIGINT, signal.SIG_DFL)
For anyone who happens to stumble along this with the same problem of being unable to manipulate the camera from the vtkInteractorStyle classes, check out the Dolly(), Pan(), Spin(), Rotate(), Zoom(), and UniformScale(). All of these should let you access the camera from your whichever child class of vtkInteractorStyle you're using. Good luck!
EDIT: Even better, just attach your camera to your class that inherits from vtkInteractorStyle as a property, e.g.:
style = MyInteractorStyleClass()
style.camera = myCam
This way you can access it from anywhere within your custom class! Pretty basic, but it flew right past me.
For my class, I am creating a "Mandelbrot Explorer" program. There is one main issue: I lose control of the GUI (all written in Tkinter/Ttk, in Python 2.7) when actually drawing to the Canvas.
Here is my code:
# There is some code above and below, but only this is relevant
for real, imag in graph.PlaneIteration(self.graph.xMin, self.graph.xMax, resolution, self.graph.yMin, self.graph.yMax, resolution, master = self.graph, buffer_action = self.graph.flush):
# the above line iterates on the complex plane, updating the Canvas for every x value
c = complex(real, imag)
function, draw, z, current_iter = lambda z: z**2 + c, True, 0, 1
while current_iter <= iterations:
z = function(z)
if abs(z) > limit:
draw = False
break
current_iter += 1
self.progressbar.setValue(100 * (real + self.graph.xMax) / total)
color = self.scheme(c, current_iter, iterations, draw)
# returns a hex color value
self.graph.plot(c, color)
# self.graph is an instance of my custom class (ComplexGraph) which is a wrapper
# around the Canvas widget
# self.graph.plot just creates a line on the Canvas:
# self.create_line(xs,ys,xs+1,ys+1, fill=color)
My issue is that when run, the graphing takes a while - about 30 seconds. In this time, I cannot use the GUI. If I try to, the window freezes and only unfreezes once the drawing is done.
I tried using threading (I enclosed the entirety of the upper code in a function, thread_process):
thread.start_new_thread(thread_process, ())
However, the problem remains.
Is there a way to fix this? Thanks!
You can execute your loop "threaded" with Tkinter by implicitly returning to Tkinter's main loop execution after every point your draw. Do this by using widget.after to register the next function call:
plane = graph.PlaneIteration(...)
def plotNextPoint():
try:
real, imag = plane.next()
except StopIteration:
return
c = complex(real, imag)
...
self.graph.plot(c, color)
self.graph.after(0, plotNextPoint)
plotNextPoint()
This way, after each point you draw, the Tkinter mainloop will run again and update the display before calling your plotNextPoint function again. If this is too slow, try wrapping the body of plotNextPoint in a for _ in xrange(n) loop to draw n points between redraws.
You're right about the cause of the problem—the GUI event loop is not running while you're busy running this code.
And you're right about threading being a good solution. (The other major solution is to break the job up into smaller subtasks and have each one schedule the next. For a more detailed overview of the options and all of the wrinkles, see Why your GUI app freezes.)
But it's not quite as simple as putting the whole thing on a thread.
Unfortunately, Tkinter (like many GUI frameworks) is not free-threaded. You cannot call methods on any GUI objects from a background thread. If you do, different things happen on different platforms and versions, ranging from blocking the main thread to crashing the program to raising exceptions.
Also, remember that, even without Tkinter, you can't safely share mutable objects between threads without some kind of synchronization. And you're doing exactly that with the Tkinter objects, right?
The Tkinter wiki explains one way to get around both of these problems at once in Tkinter and Threads: Create a Queue, have the background thread put messages on it, and have the main thread check it every so often (e.g., by using after to schedule a nonblocking get every 100ms until the background thread is done).
If you don't want to come up with a "protocol" for passing data from the background thread to the main thread, remember that in Python, a bound method, or a tuple of a bound method and some arguments, it perfectly good, passable data. So, instead of calling self.graph.plot(c, color), you can just self.q.put((self.graph.plot, c, color)).
The library mtTkinter wraps this all up for you, making it look like Tkinter is free-threaded by using a Queue in the background. It isn't highly tested or frequently maintained, but even if it doesn't work in the future it still makes great sample code.
The code below lets you walk around a small grid on the screen using the arrow keys putting "." where you've explored or been next to. Even though I have my refresh before the first getch (to get a key-stroke) the screen doesn't first display anything until you've moved off your starting position. Shouldn't the addstr followed by refresh immediately show and then the getch waits after that? I even tried adding a stdscr.refresh(), but that didn't help either. How do I get the screen to refresh immediately before waiting for the first key-stroke?
import curses
def start(stdscr):
curses.curs_set(0)
movement = curses.newpad(10, 10)
cur_x, cur_y = 5, 5
while True:
movement.addstr(cur_y, cur_x, '#')
for (x_off, y_off) in [(-1,0),(1,0),(0,-1),(0,1)]:
movement.addstr(cur_y + y_off, cur_x + x_off, '.')
movement.refresh(1, 1, 0, 0, 7, 7) #Nothing is displayed until after the first key-stroke
key_stroke = stdscr.getch()
move_attempt = False
if 0 < key_stroke < 256:
key_stroke = chr(key_stroke)
elif key_stroke == curses.KEY_UP and cur_y > 1:
cur_y -= 1
elif key_stroke == curses.KEY_DOWN and cur_y < 8:
cur_y += 1
elif key_stroke == curses.KEY_LEFT and cur_x > 1:
cur_x -= 1
elif key_stroke == curses.KEY_RIGHT and cur_x < 8:
cur_x += 1
else:
pass
if __name__ == '__main__':
curses.wrapper(start)
The docs are broken. I'd used curses back in the day, but libncurses is new to me.
My first hint came from ncurses(3):
The ncurses library permits manipulation of data structures, called windows, which can be thought of as two-dimensional arrays of characters representing all or part of a CRT screen. A default window called stdscr, which is the size of the terminal screen, is supplied. Others may be created with newwin.
…
Special windows called pads may also be manipulated. These are windows which are not constrained to the size of the screen and whose contents need not be completely displayed.
But then refresh(3) got decidedly evasive:
The routine wrefresh works by first calling wnoutrefresh, which copies the named window to the virtual screen, and then calling doupdate, which compares the virtual screen to the physical screen and does the actual update. … The phrase "copies the named window to the virtual screen" above is ambiguous. What actually happens is that all touched (changed) lines in the window are copied to the virtual screen. This affects programs that use overlapping windows; it means that if two windows overlap, you can refresh them in either order and the overlap region will be modified only when it is explicitly changed. [emphasis mine]
which prompted me to try adding
stdscr.refresh()
after your pad.refresh() which worked. And then I moved it further up start() to see if it was really needed on every pad modification. I moved it all the way up to the first point there is a stdscr to work with yielding:
def start(stdscr):
stdscr.refresh()
curses.curs_set(0)
…
which smacks of voodoo programming, but I'm not going to look at the innards of a 20-year old library made to cope with glass ttys to try to grok it.
Add stdscr.refresh() sometime before the movement.refresh() to solve the issue.
By adding time.sleep(1) after the refresh statement, it does write to the screen, but then it disappears when stdscr.getch() is called, but only the first time. Probably has to do with some sort of delayed initialization of stdscr.
Calling stdscr.refresh() after the movement.refresh() has the same effect: The very first time through the loop stdscr.refresh() clears the screen, but not in subsequent times through the loop. By calling stdscr.refresh() early in the program it gets this weird first time refresh out of the way.
When using a pad, for some reason—I don't know why—you have to call curses.doupdate after invoking the pad's refresh.
Adding window.nodelay(1) before while solved issue for me.