I'm trying to plot live the output of a generator.
The following code works as expected (Ctrl-C terminates execution):
import numpy as np
import pylab as p
from Queue import Queue
from threading import Thread
import time
def dataGenerator():
while True:
yield np.random.random()
def populate():
f = dataGenerator()
while True:
x = f.next(); y = f.next()
q.put([x,y])
q = Queue()
p.figure(); p.hold(True); p.show(block=False)
populatorThread = Thread(target=populate)
populatorThread.daemon = True
populatorThread.start()
while True:
data = q.get()
x = data[0]
y = data[1]
p.plot(x,y,'o')
p.draw()
q.task_done()
populatorThread.join()
However, if instead I put the plotting in a thread, I get RuntimeError: main thread is not in main loop:
import numpy as np
import pylab as p
from Queue import Queue
from threading import Thread
import time
def dataGenerator():
while True:
yield np.random.random()
def plotter():
while True:
data = q.get()
x = data[0]
y = data[1]
p.plot(x,y,'o')
p.draw()
print x,y
q.task_done()
q = Queue()
p.figure(); p.hold(True); p.show(block=False)
plotThread = Thread(target=plotter)
plotThread.daemon = True
plotThread.start()
f = dataGenerator()
while True:
x = f.next()
y = f.next()
q.put([x,y])
plotThread.join()
Why does matplotlib care which thread does the plotting?
EDIT: I'm not asking how to solve this but rather why is this happening in the first place.
It's probably the GUI that you're using for backend. The GUI likely expects to find itself in the main thread, but it isn't when matplotlib calls get_current_fig_manager().canvas.draw().
For example, when I do this, I get the following traceback:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "tmp.py", line 18, in plotter
p.draw()
File "/usr/lib/pymodules/python2.7/matplotlib/pyplot.py", line 555, in draw
get_current_fig_manager().canvas.draw()
File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_tkagg.py", line 349, in draw
tkagg.blit(self._tkphoto, self.renderer._renderer, colormode=2)
File "/usr/lib/pymodules/python2.7/matplotlib/backends/tkagg.py", line 13, in blit
tk.call("PyAggImagePhoto", photoimage, id(aggimage), colormode, id(bbox_array))
RuntimeError: main thread is not in main loop
Note the tk.call(...) line. The exception you get is not raised from matplotlib, it's raised from TkInter.
Why does matplotlib care which thread does the plotting?
I'm not asking how to solve this but rather why is this happening in the first place.
#Evert is right, it's not matplotlib, it's your GUI toolkit (one of the backends that matplotlib uses to create a window with a plot for you). It happens because GUI toolkits are event-driven (you don't want blocking behavior for the user interface, right?) and they have internal event loop, that controls program execution. The idea is that events are monitored by the event loop and dispatched to the callbacks. To do this, event loop should be started in the main thread, while callbacks for long-running tasks are moved to separate threads.
Related
I am trying to create a shared memory for my Python application, which should be used in the parent process and in another process that is spawned from that parent process. In most cases that works fine, however, sometimes I get the following stacktrace:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/usr/lib/python3.8/multiprocessing/spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
File "/usr/lib/python3.8/multiprocessing/synchronize.py", line 110, in __setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory: '/psm_47f7f5d7'
I want to emphasize that our code/application works fine in 99% of the time. We are spawning these new processes with new shared memory for each such process on a regular basis in our application (which is a server process, so it's running 24/7). Nearly all the time this works fine, only from time to time this error above is thrown, which then kills the whole application.
Update: I noticed that this problem occurs mainly when the application was running for a while already. When I start it up the creation of shared memory and spawning new processes works fine without this error.
The shared memory is created like this:
# Spawn context for multiprocessing
_mp_spawn_ctxt = multiprocessing.get_context("spawn")
_mp_spawn_ctxt_pipe = _mp_spawn_ctxt.Pipe
# Create shared memory
mem_size = width * height * bpp
shared_mem = shared_memory.SharedMemory(create=True, size=mem_size)
image = np.ndarray((height, width, bpp), dtype=np.uint8, buffer=shared_mem.buf)
parent_pipe, child_pipe = _mp_spawn_ctxt_pipe()
time.sleep(0.1)
# Spawn new process
# _CameraProcess is a custom class derived from _mp_spawn_ctxt.Process
proc = _CameraProcess(shared_mem, child_pipe)
proc.start()
Any ideas what could be the issue here?
I had the similar issue in case, that more processes had access to the shared memory/object and one process did update the shared memory/object.
I solved these issues based on these steps:
I synchronized all operations with shared memory/object via mutexes (see sample for multiprocessing usage superfastpython or protect shared resources). Critical part of code are create, update, delete but also reading content of shared object/memory, because at the same time different process can do update of shared object/memory, etc.
I avoided libraries with only single thread execution support
See sample code with synchronization:
def increase(sharedObj, lock):
for i in range(100):
time.sleep(0.01)
lock.acquire()
sharedObj = sharedObj + 1
lock.release()
def decrease(sharedObj, lock):
for i in range(100):
time.sleep(0.001)
lock.acquire()
sharedObj = sharedObj - 1
lock.release()
if __name__ == '__main__':
sharedObj = multiprocessing.Value ('i',1000)
lock=multiprocessing.Lock()
p1=multiprocessing.Process(target=increase, args=(sharedObj, lock))
p2=multiprocessing.Process(target=decrease, args=(sharedObj, lock))
p1.start()
p2.start()
p1.join()
p2.join()
I'm currently working with Tkinter to create a GUI for my Script. Among other things there is a function which writes and save some Data in some Files. To visualize the progress for the User i got an label which shows the progress. When i press the button to execute the function
button_download_data = tk.Button(text="Get Data",command=gatherData)
the window freezes and beside the counter for the progress increasaes it isnt shown due to the window is frozen. My solution was to start the function in a new thread using the threading modul.
button_download_data = tk.Button(text="Get Data",command=threading.Thread(target=gatherData).start)
Now the progress is showed but i cant press the button again because i get an error:
Exception in Tkinter callback
Traceback (most recent call last):
File "C:\WPy64-31050\python-3.10.5.amd64\lib\tkinter\__init__.py", line 1921, in __call__
return self.func(*args)
File "C:\WPy64-31050\python-3.10.5.amd64\lib\threading.py", line 930, in start
raise RuntimeError("threads can only be started once")
RuntimeError: threads can only be started once
I tried to "kill" the thread when the function is done with raise Exception() and sys.Exit() but it doesn't work at all.
I figuerd out that i can outsource the thread start out of the tkinter button line with:
def gatherDate():
do something
def threadStart():
threading.Thread(target=gatherData).start
button_download_data = tk.Button(text="Get Data",command=threadStart)
and i think it might help to start a new thread on button press and not the same again but i cant imagine how.
You should be able to handle this by creating a separate function to spawn new worker threads as needed - here's a very basic example
import tkinter as tk
from threading import Thread
from time import sleep # for example - simulate a long-running process
def get_data():
print('foo') # do whatever you need to do here
sleep(2.0) # simulate the thread 'working' on something...for example
def spawn_thread():
t = Thread(target=get_data, daemon=True)
t.start()
root = tk.Tk()
button_download_data = tk.Button(root, text='Get Data', command=spawn_thread)
button_download_data.pack()
if __name__ == '__main__':
root.mainloop()
You could simplify spawn_thread a little by skipping the variable assignment t=... and doing Thread(target=get_data, daemon=True).start() instead (as long as you don't need access to the Thread object t for anything else)
I want to run 2 processes at the same time. 1 will keep printing 'a' every second and the other will ask for an input and when the input is 'Y', the first process will stop printing 'a'. I am fairly new to Python and I can't figure it out...
This is what I came up with so far:
from multiprocessing import Process
import time
go = True
def loop_a():
global go
while go == True:
time.sleep(1)
print("a")
def loop_b():
global go
text = input('Y/N?')
if text == 'Y':
go = False
if __name__ == '__main__':
Process(target=loop_a).start()
Process(target=loop_b).start()
This is the error message I'm getting:
Process Process-2:
Traceback (most recent call last):
File "C:\Users\Tip\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 249, in _bootstrap
self.run()
File "C:\Users\Tip\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "F:\ProgrammingTK\PROGproject\test.py", line 15, in loop_b
text = input('Y/N?')
EOFError: EOF when reading a line
Expanding upon jasonharper's comment as he is correct.
There are a couple issues
The go variable is not shared among the processes. As Jason suggested you can use something like Manager in multiprocessing in order to share a value among multiple processes. Technically, that go variable will be copied over into each process but it won't be shared between them so a change in one process won't be seen by the other.
Again, as he mentioned you need to pull the input(..) into the main thread of the program. Also, if you are on 2.7 you will need to use raw_input(..).
Also, if you are only checking the flag once and then exiting then you'll likely hit a BrokenPipeError.
Taking that in, you can try something like this:
from multiprocessing import Process, Manager
import time
def loop_a(go):
while True:
# run forever and print out the msg if the flag is set
time.sleep(1)
if go.value:
print("a")
if __name__ == '__main__':
# shared value flag
manager = Manager()
go_flag = manager.Value('flag', True)
# other process that is printing
Process(target=loop_a, args=(go_flag,)).start()
# normal main thread; toggle on and off the other process
while True:
text = input('Stop Y/N?')
if text == 'Y':
go_flag.value = False
print("Changed the flag {}".format(go_flag.value))
else:
go_flag.value = True
print("Changed the flag {}".format(go_flag.value))
Brief description on what I'm trying to achieve:
I'm working on a analytics software built using Python, wxPython, and matplotlib. I'm trying to implement a function where the program can plot the results after performing some analytical calculations. At the moment, the program freezes when the it's performing the calculations (and the calculation time takes up to 10 seconds depending on the amount of data) so I'm trying to use threading to create a non-blocking program to improve user experience.
Problem I'm getting
I keep getting this error :
(PyAssertionError: C++ assertion "hdcDst && hdcSrc" failed at ...... \src\msw\dc.cpp(2559) in AlphaBlt():AlphaBlt():invalid HDC)
and googling hasn't really help with identifying the cause.
I'll post the full traceback at the bottom of the post.
Here's my code:
import wx
import time
import matplotlib.pyplot as plt
from wx.lib.pubsub import Publisher as pub
from threading import Thread
def plotgraph(x,y,sleeptime):
plt.plot(x,y)
#Simulate long process using time.sleep
time.sleep(sleep time)
#Send out a message once process is completed
pub.sendMessage('PLOT','empty')
class listener():
def __init__(self,name):
self.name = name
#Listens to message
pub.subscribe(self.Plot,'PLOT')
pass
def Plot(self,message):
print self.name
plt.show()
print 'printed'
waiting = listener('Bob')
t1 = Thread(target=plotgraph,args=([1,2,3],[1,2,3],5))
t1.start()
t2 = Thread(target=plotgraph,args=([1,2,3],[1,2,3],3))
t2.start()
Basically, the user will be clicking an icon on the GUI and that will trigger a function to perform some analytical calculation simulated by 'plotgraph()' here. At the moment, without using threads, plotgraph() will block my entire program, so I'm trying to use threads to perform the calculations to free up my GUI.
However when I tried to plot my GUI within the thread, i.e. have plt.show() in plotgraph(), the plot appears then disappears again. When I click the button on the GUI to spawn the thread a second time, I get the same error.
So I've tried to work around it by sending a message after the thread's ended so that the plt.show() will happen outside the thread but I'm still getting the same error.
I can't seem to be able to find a similar error online, except for one thread posted in 2008. If anyone could help that would be awesome!
In a nutshell
I need a way to implement sort of a callback function that allows me to perform the analytic calculation in a thread, then plot the graph once the calculations are completed to free up my GUI. It'd be great if someone could explain to me what's wrong here, or could suggest an alternative method to do it. Thanks very much!!
Here's the full traceback:
File "C:\Users\chaishen\AppData\Local\Enthought\Canopy32\App\appdata\canopy-1.
.5.3123.win-x86\lib\threading.py", line 810, in __bootstrap_inner
self.run()
File "C:\Users\chaishen\AppData\Local\Enthought\Canopy32\App\appdata\canopy-1.
.5.3123.win-x86\lib\threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "<ipython-input-5-0cb01f87e97a>", line 13, in plotgraph
pub.sendMessage('PLOT','empty')
File "C:\Users\chaishen\AppData\Local\Enthought\Canopy32\User\lib\site-package
\wx\lib\pubsub.py", line 811, in sendMessage
self.__topicTree.sendMessage(aTopic, message, onTopicNeverCreated)
File "C:\Users\chaishen\AppData\Local\Enthought\Canopy32\User\lib\site-package
\wx\lib\pubsub.py", line 498, in sendMessage
deliveryCount += node.sendMessage(message)
File "C:\Users\chaishen\AppData\Local\Enthought\Canopy32\User\lib\site-package
\wx\lib\pubsub.py", line 336, in sendMessage
listener(message)
File "<ipython-input-5-0cb01f87e97a>", line 24, in Plot
plt.show()
File "C:\Users\chaishen\AppData\Local\Enthought\Canopy32\User\lib\site-package
\matplotlib\pyplot.py", line 155, in show
return _show(*args, **kw)
File "C:\Users\chaishen\AppData\Local\Enthought\Canopy32\User\lib\site-package
\matplotlib\backend_bases.py", line 154, in __call__
manager.show()
File "C:\Users\chaishen\AppData\Local\Enthought\Canopy32\User\lib\site-package
\matplotlib\backends\backend_wx.py", line 1414, in show
self.canvas.draw()
File "C:\Users\chaishen\AppData\Local\Enthought\Canopy32\User\lib\site-package
\matplotlib\backends\backend_wxagg.py", line 50, in draw
self.gui_repaint(drawDC=drawDC)
File "C:\Users\chaishen\AppData\Local\Enthought\Canopy32\User\lib\site-package
\matplotlib\backends\backend_wx.py", line 911, in gui_repaint
drawDC.DrawBitmap(self.bitmap, 0, 0)
File "C:\Users\chaishen\AppData\Local\Enthought\Canopy32\User\lib\site-package
\wx\_gdi.py", line 3460, in DrawBitmap
return _gdi_.DC_DrawBitmap(*args, **kwargs)
yAssertionError: C++ assertion "hdcDst && hdcSrc" failed at ..\..\src\msw\dc.cp
(2559) in AlphaBlt(): AlphaBlt(): invalid HDC
I think what you need is the wx.PyEventBinder.
It works like this:
anEVT_CALCULATED = wx.NewEventType()
EVT_CALCULATED = wx.PyEventBinder(anEVT_CALCULATED, 1)
def onCalculate(self, event): # this is your click
calc_thread = CalculatorThread(self, params)
calc_thread.start()
return
def onConnected(self, event):
''' this is where your thread comes back '''
self.doSomeThingLikePlotting(event.resultdata)
class CalcEvent(wx.PyCommandEvent):
''' Event to signal that the thread has calculated'''
def __init__(self, etype, eid, resultdata):
wx.PyCommandEvent.__init__(self, etype, eid)
self.resultdata = resultdata
class CalculatorThread(threading.Thread):
''' This is the thread doing your calculation and handing it back'''
def __init__(self, listener, params):
threading.Thread.__init__(self)
self.listener = listener
self.params = params
def run(self):
resultdata = calculate(params) # this is your calculation
event = CalcEvent(anEVT_CALCULATED, -1, resultdata=resultdata)
wx.PostEvent(self.listener, event)
return
And of course you need to add one line to your __init__
self.Bind(EVT_CONNECTED, self.onCalculated)
This is not very important, just a silly experiment. I would like to create my own message passing.
I would like to have a dictionary of queues, where each key is the PID of the process.
Because I'd like to have the processes (created by Process()) to exchange messages inserting them in the queue of the process they want to send it to (knowing its pid).
This is a silly code:
from multiprocessing import Process, Manager, Queue
from os import getpid
from time import sleep
def begin(dic, manager, parentQ):
parentQ.put(getpid())
dic[getpid()] = manager.Queue()
dic[getpid()].put("Something...")
if __name__== '__main__':
manager = Manager()
dic = manager.dict()
parentQ = Queue()
p = Process(target = begin, args=(dic, manager, parentQ))
p.start()
son = parentQ.get()
print son
sleep(2)
print dic[son].get()
dic[getpid()] = manager.Queue(), this works fine. But when I perform
dic[son].put()/get() I get this message:
Process Process-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "mps.py", line 8, in begin
dic[getpid()].put("Something...")
File "<string>", line 2, in __getitem__
File "/usr/lib/python2.7/multiprocessing/managers.py", line 773, in _callmethod
raise convert_to_error(kind, result)
RemoteError:
---------------------------------------------------------------------------
Unserializable message: ('#RETURN', <Queue.Queue instance at 0x8a92d0c>)
---------------------------------------------------------------------------
do you know what's the right way to do it?
I believe your code is failing because Queues are not serializable, just like the traceback says. The multiprocessing.Manager() object can create a shared dict for you without a problem, just as you've done here, but values stored in the dict still need to be serializable (or picklable in Pythonese). If you're okay with the subprocesses not having access to each other's queues, then this should work for you:
from multiprocessing import Process, Manager, Queue
from os import getpid
number_of_subprocesses_i_want = 5
def begin(myQ):
myQ.put("Something sentimental from your friend, PID {0}".format(getpid()))
return
if __name__== '__main__':
queue_dic = {}
queue_manager = Manager()
process_list = []
for i in xrange(number_of_subprocesses_i_want):
child_queue = queue_manager.Queue()
p = Process(target = begin, args=(child_queue,))
p.start()
queue_dic[p.pid] = child_queue
process_list.append(p)
for p in process_list:
print(queue_dic[p.pid].get())
p.join()
This leaves you with a dictionary whose keys are the child processes, and the values are their respective queues, which can be used from the main process.
I don't think your original goal is achievable with queues because queues that you want a subprocess to use must be passed to the processes when they are created, so as you launch more processes, you have no way to give an existing process access to a new queue.
One possible way to have inter-process communication would be to have everyone share a single queue to pass messages back to your main process bundled with some kind of header, such as in a tuple:
(destination_pid, sender_pid, message)
..and have main read the destination_pid and direct (sender_pid, message) to that subprocess' queue. Of course, this implies that you need a method of notifying existing processes when a new process is available to communicate with.