I have two classes, a consumer and a producer. To prevent a race condition between threads, I've tried using QWaitCondition and QMutex, but I keep running into timing issues. The producer object fills my home object's buffer and a buffer of its own. The consumer object copies this buffer and displays the data in the buffer. As it is displaying, I would like the producer object to fill the buffer again and then wait until the displaying is complete. Immediately, the consumer object would copy the buffer and continue the display. Rinse and repeat. However, I can not figure out the timing sequence with my code.
from PyQt5.QtCore import QThread, pyqtSignal
from PyQt5.QtWidgets import QWidget
from PyQt5.QtCore import QWaitCondition, QMutex
mutex = QMutex()
displayFramesDone = QWaitCondition()
copyFramesDone = QWaitCondition()
sequenceDone = QWaitCondition()
class Producer(QThread):
finished = pyqtSignal()
def __init__(self, home_obj):
self.parent = self
self.home = home_obj
self.prod_buffer = []
"""
functions
"""
def run(self):
self.runSequence()
sequenceDone.wakeAll()
while self.home.condition == True:
self.runSequence()
mutex.lock()
displayFramesDone.wait(mutex)
mutex.unlock()
sequenceDone.wakeAll()
mutex.lock()
copyFramesDone.wait(mutex)
mutex.unlock()
self.finished.emit()
class Consumer(QThread):
finished = pyqtSignal()
def __init__(self, home_obj):
self.parent = self
self.home = home_obj
self.con_buffer = []
"""
functions
"""
def run(self):
while self.home.condition == True:
mutex.lock()
sequenceDone.wait(mutex)
mutex.unlock()
self.copyFrames()
copyFramesDone.wakeAll()
self.displayFrames()
displayFramesDone.wakeAll()
self.finished.emit()
class Home(QWidget):
def __init__(self):
super().__init__()
self.condition == True
self.buffer = []
I've never used QWaitCondition or QMutex before, so if I'm using them appropriately is also another matter, however, this seems to work only occasionally when the timing manages to align with the required sequence.
Additionally, I'd like to know if anyone could answer the following:
Must a wake signal be inside of a mutex block?
Is QWaitCondition/QMutex a good tool for this type of timing problem?
Using python 3.8.5, i've try to create a class object which have a inside infinite loop which will update a value that i can have to read later... Unfortunaly, my knowledge on this field are a little poor.
The aims of this trial is to dissociate an app which has a state from the gui i've made...
Here is one not working trial i've made.
import asyncio
from time import sleep
class value_holder:
def __init__(self):
self.value = 0
asyncio.create_task(self.infinite_loop())
async def infinite_loop(self):
while True:
self.value += 1
sleep(3)
v = value_holder()
while True:
print(v.value)
sleep(1)
I'm actually clueless so if someone have any clue or keyword for helping me in this search of solution, i will be very thankful
Best regards
You could work with a timer instead :
import threading
from time import sleep
class value_holder:
value = 0
timer = None
delay = 1
def __init__(self, delay):
self.value = 0
self.delay = delay
def run(self):
self.value += 1
print(self.value)
if self.timer != None: self.start()
def start(self):
self.timer = threading.Timer(self.delay, self.run)
self.timer.start()
def stop(self):
self.timer.cancel()
v = value_holder(3.0)
v.start()
sleep(9) #==> 1, 2, 3
v.stop()
print("Stopped")
So I currently have a PYQT application utilizing a combination of pythons built-in threading, and QT Qthread/worker system to allow for asynchronous activities
The code for generating the threads/workers is process_batch which is contained in a class called Scraper, which is a QObject
def process_batch(self, batch):
for worker in batch:
self.host.threads.append(QThread())
th = self.host.threads[len(self.host.threads)-1]
worker.moveToThread(th)
th.started.connect(worker.process)
qApp.aboutToQuit.connect(th.quit)
worker.finished.connect(worker.deleteLater)
th.finished.connect(th.deleteLater)
for thread in self.host.threads:
thread.start()
self.activeThreads += 1
the the self.host.threads is a list variable helf by the ScraperHost object, the batch variable is a standard python list with 50 of the following class:
class ScraperWorker(QObject):
complete = pyqtSignal(str)
start = pyqtSignal(str)
finished = pyqtSignal()
def __init__(self, url, task, _id):
QObject.__init__(self)
self.url = url
self.task = task
self.id = _id
self.start.connect(self.process)
def __str__(self):
return self.url
#pyqtSlot()
def process(self):
time.sleep(float(random.random() * random.randint(1,3)))
#self.complete.emit(str(self.id))
self.finished.emit()
The time.sleep was added to see if spacing out the thread's completions randomly would fix the issue, it did not
Originally I was running into an issue of the threads getting deleted before finishing when the process_batch function completed, so I implemented the self.host.threads to keep them from getting garbage collected.
threads are contained below in :
class ScraperHost(threading.Thread):
def __init__(self, threadID, urls, task):
threading.Thread.__init__(self)
self.threadID = threadID
self.scraper = Scraper(urls, task, self)
self.threads = []
def run(self):
self.scraper.run()
The scraper host is created in my MainWindow by:
def selectFileList(self):
options = QFileDialog.Options()
fileName, _ = QFileDialog.getOpenFileName(self, "Select List","", "Excel File (*.xls *.xlsx)", options=options)
if fileName:
...
...
self.ScraperHost = ScraperHost(0, urls, "index")
self.ScraperHost.run()
As it is currently I am able to run the function/program with zero errors, but it only completes exactly half of the threads prior to finishing. It gives me no errors, messages, or any indication as to why they aren't finishing. If I increase the batch size to 100, it finishes 50. If I run it in debug mode in VS Code, i can see that all 50 threads are created correctly, and are all in the host. threads variable. If ** I step through the thread's creation 1 by 1 in debug mode they will all finish correctly**, but if I just run the program normally only exactly half will finish. The 25 threads that finish are different every time due to the random time they wait. The self.finished.emit() of the worker just prints the thread's ID, which is how I am seeing how many finish
EDIT: Reproducable Sample: Side Note: Gets through more, sometimes all, but with no consistency, of the threads when ran from idle as opposed to VS Code
from PyQt5.QtWidgets import *
from PyQt5.QtGui import *
from PyQt5.QtCore import *
import threading
import time
import random
scraperhost = None
class ScraperHost(threading.Thread):
def __init__(self, threadID, urls, task):
threading.Thread.__init__(self)
self.threadID = threadID
self.scraper = Scraper(urls, task, self)
self.threads = []
def run(self):
self.scraper.run()
class Scraper(QObject):
output = pyqtSignal(str)
complete = pyqtSignal()
def __init__(self, urls, task, host):
QObject.__init__(self)
self.urls = urls
self.batches = []
self.activeThreads = 0
self.task = task
self.host = host
def __del__(self):
self.wait()
def run(self):
batch_size = 50
batch_count = 1
for batch in range(0, batch_count):
self.batches.append([])
for i in range (0, batch_size):
try:
c_url = "www.website.com"
worker = ScraperWorker(c_url, self.task, i)
worker.complete.connect(self.worker_finished)
self.batches[batch].append(worker)
except IndexError:
break
if len(self.batches) == 0:
return
self.process_batch(self.batches.pop())
#pyqtSlot(str)
def worker_finished(self, ss):
self.activeThreads -= 1
print(self.activeThreads)
def process_batch(self, batch):
for worker in batch:
self.host.threads.append(QThread())
th = self.host.threads[len(self.host.threads)-1]
worker.moveToThread(th)
th.started.connect(worker.process)
qApp.aboutToQuit.connect(th.quit)
worker.finished.connect(worker.deleteLater)
th.finished.connect(th.deleteLater)
for thread in self.host.threads:
thread.start()
self.activeThreads += 1
class ScraperWorker(QObject):
complete = pyqtSignal(str)
start = pyqtSignal(str)
finished = pyqtSignal()
def __init__(self, url, task, _id):
QObject.__init__(self)
self.url = url
self.task = task
self.id = _id
self.start.connect(self.process)
def __str__(self):
return self.url
#pyqtSlot()
def process(self):
time.sleep(float(random.random() * random.randint(1,3)))
self.complete.emit(str(self.id))
self.finished.emit()
def main():
app = QApplication([])
ex = QWidget()
ex.show()
global scraperhost
scraperhost = ScraperHost(0, [], "index")
scraperhost.start()
app.exec_()
return
main()
I'm trying to understand how thread works, and i'm stuck with this problem. That's my program explained:
i made a simple GUI in pyqt that use a QObject as a worker class. When i press the botton start the gui read a random value from a list and pass it to the thread, that print the
next five number. When the thread finish the work, it pass the data to the gui. Now i want the GUI to restart automatically a new thread with a new start value. I can restart the thread by pressing start again, but i need to start it without human interaction. Are there
any method?
thanks in advance
from PyQt4.QtCore import *
from PyQt4.QtGui import *
import time
import sys
import numpy as np
class SomeObject(QObject):
finished = pyqtSignal(object)
valore = pyqtSignal(object)
vector = pyqtSignal(object)
def __init():
super(SomeObject, self).__init__()
def longRunning(self):
vec = []
end = self.count + 5
while self.count < end:
time.sleep(1)
vec.append(self.count)
self.valore.emit(self.count)
self.count += 1
self.finished.emit(vec)
#self.vector.emit()
def setCount(self, num):
self.count = num
class GUI(QDialog):
def __init__(self, parent = None):
super(GUI, self).__init__(parent)
#declare QThread object
self.objThread = QThread()
#declare SomeObject type, and move it to thread
self.obj = SomeObject()
self.obj.moveToThread(self.objThread)
#connect finished signal to nextVector method
self.obj.finished.connect(self.nextVector)
#connect valore to self.prova method
self.obj.valore.connect(self.prova)
#self.obj.vector.connect(self.nextVector)
#Connect thread.start to the method long running
self.objThread.started.connect(self.obj.longRunning)
botton = QPushButton("start")
self.connect(botton, SIGNAL("clicked()"), self.showcount)
box = QHBoxLayout()
box.addWidget(botton)
self.setLayout(box)
#a list of random number
a = np.random.randint(10, size = 5)
self.iter = iter(a)
def showcount(self):
"""
When botton clicked, read the next value from iter, pass it to
setCount and when start the thread
"""
try:
a = self.iter.next()
print a
self.obj.setCount(a)
self.objThread.start()
except StopIteration:
print "finito"
#self.obj.setCount(a)
#self.objThread.start()
#print self.objThread.currentThreadId()
def prova(self, value):
"""
Connected to signal valore, print the value
"""
print value
def nextVector(self, vec):
"""
Print the whole vector
"""
print vec
self.objThread.quit()
try:
a = self.iter.next()
print a
self.obj.setCount(a)
self.objThread.start()
except StopIteration:
print "finito"
app = QApplication(sys.argv)
form = GUI()
form.show()
app.exec_()
You already have it set up. When your thread is finished it emits the finished signal which calls the nextVector method, so just call the start method at the end of nextVector.
def nextVector(self, vec):
...
self.showcount()
# end nextVector
You may also want to change to the new signal connection for your QPushButton
button.clicked.connect(self.showcount)
I am totally confused by the QThread behavior. My idea is to acquire some audio signal in a qthread, save it in a python queue object and with a QTimer I read the queue and plot it using pyqtgraph. It works, however, only at around 6-7 fps. However, when I use .terminate() to terminate the thread, the thread does actually NOT terminate, but rather speeds up to > 100 fps, exactly what I actually wanted.
My issues:
why does the QThread not terminate/is aborted/closed...?
what is .terminate() actually doing?
what is slowing down the normal thread.start()?
On a side note, I know that I am not using a Signal/Slot for checking if it should still run or not, I just want to understand this strange behavior, and why the thread is not fast from the very beginning! Something is maybe blocking the proper function and is turned off (?!) by the .terminate() function...
My minimal working example (hope you guys have a soundcard/mic somewhere):
from PyQt5.QtWidgets import QApplication, QWidget, QGridLayout, QPushButton
from PyQt5.QtCore import QThread, QTimer
import sounddevice as sd
import queue
import pyqtgraph as pg
import numpy as np
import time
class Record(QThread):
def __init__(self):
super().__init__()
self.q = queue.Queue()
def callback(self, indata, frames, time, status):
self.q.put(indata.copy())
def run(self):
with sd.InputStream(samplerate=48000, device=1, channels=2, callback=self.callback, blocksize=4096):
print('Stream started...')
while True:
pass
print(self.isRunning(), 'Done?') # never called
class Main(QWidget):
def __init__(self):
super().__init__()
self.recording = False
self.r = None
self.x = 0
self.times = list(range(10))
self.setWindowTitle("Record Audio Tester")
self.l = QGridLayout()
self.setLayout(self.l)
self.pl = pg.PlotWidget(autoRange=False)
self.curve1 = self.pl.plot(np.zeros(8000))
self.curve2 = self.pl.plot(np.zeros(8000)-1, pen=pg.mkPen("y"))
self.l.addWidget(self.pl)
self.button_record = QPushButton("Start recording")
self.button_record.clicked.connect(self.record)
self.l.addWidget(self.button_record)
def record(self):
if self.recording and self.r is not None:
self.button_record.setText("Start recording")
self.recording = False
self.r.terminate()
else:
self.button_record.setText("Stop recording")
self.recording = True
self.r = Record()
self.r.start()
self.t = QTimer()
self.t.timeout.connect(self.plotData)
self.t.start(0)
def plotData(self):
self.times = self.times[1:]
self.times.append(time.time())
fps = 1 / (np.diff(np.array(self.times)).mean() + 1e-5)
self.setWindowTitle("{:d} fps...".format(int(fps)))
if self.r.q.empty():
return
d = self.r.q.get()
self.curve1.setData(d[:, 0])
self.curve2.setData(d[:, 1]-3)
if __name__ == '__main__':
app = QApplication([])
w = Main()
w.show()
app.exec_()
edit 1
The first suggestion #Dennis Jensen was to not subclass QThread, but use rather QObject/QThread/moveToThread. I did this, see code below, and one can see that the issue is gone with either using while and just app.processEvents() or while with time.sleep(0.1), but to make it response you have to use anyway app.processEvents(), so this is sufficient. The pass statement alone eats up alot of CPU processing power, resulting in 7-10 fps, but if you thread.terminate() this thread, everything still runs.
I added additionally a trace, what happens on which thread, and the callback is always on a separate thread, regardless which callback you use (outside any class, in QObject or in the main thread), indicating that the answer from #three_pineapples is correct.
from PyQt5.QtWidgets import QApplication, QWidget, QGridLayout, QPushButton, QCheckBox
from PyQt5.QtCore import QThread, QTimer, QObject, pyqtSignal, pyqtSlot
import threading
import sounddevice as sd
import queue
import pyqtgraph as pg
import numpy as np
import time
q = queue.Queue()
# It does not matter at all where the callback is,
# it is always on its own thread...
def callback(indata, frames, time, status):
print("callback", threading.get_ident())
# print()
q.put(indata.copy())
class Record(QObject):
start = pyqtSignal(str)
stop = pyqtSignal()
data = pyqtSignal(np.ndarray)
def __init__(self, do_pass=False, use_terminate=False):
super().__init__()
self.q = queue.Queue()
self.r = None
self.do_pass = do_pass
self.stop_while = False
self.use_terminate = use_terminate
print("QObject -> __init__", threading.get_ident())
def callback(self, indata, frames, time, status):
print("QObject -> callback", threading.get_ident())
self.q.put(indata.copy())
#pyqtSlot()
def stopWhileLoop(self):
self.stop_while = True
#pyqtSlot()
def run(self, m='sth'):
print('QObject -> run', threading.get_ident())
# Currently uses a callback outside this QObject
with sd.InputStream(device=1, channels=2, callback=callback) as stream:
# Test the while pass function
if self.do_pass:
while not self.stop_while:
if self.use_terminate: # see the effect of thread.terminate()...
pass # 7-10 fps
else:
app.processEvents() # makes it real time, and responsive
print("Exited while..")
stream.stop()
else:
while not self.stop_while:
app.processEvents() # makes it responsive to slots
time.sleep(.01) # makes it real time
stream.stop()
print('QObject -> run ended. Finally.')
class Main(QWidget):
def __init__(self):
super().__init__()
self.recording = False
self.r = None
self.x = 0
self.times = list(range(10))
self.q = queue.Queue()
self.setWindowTitle("Record Audio Tester")
self.l = QGridLayout()
self.setLayout(self.l)
self.pl = pg.PlotWidget(autoRange=False)
self.curve1 = self.pl.plot(np.zeros(8000))
self.curve2 = self.pl.plot(np.zeros(8000)-1, pen=pg.mkPen("y"))
self.l.addWidget(self.pl)
self.button_record = QPushButton("Start recording")
self.button_record.clicked.connect(self.record)
self.l.addWidget(self.button_record)
self.pass_or_sleep = QCheckBox("While True: pass")
self.l.addWidget(self.pass_or_sleep)
self.use_terminate = QCheckBox("Use QThread terminate")
self.l.addWidget(self.use_terminate)
print("Main thread", threading.get_ident())
def streamData(self):
self.r = sd.InputStream(device=1, channels=2, callback=self.callback)
def record(self):
if self.recording and self.r is not None:
self.button_record.setText("Start recording")
self.recording = False
self.r.stop.emit()
# And this is where the magic happens:
if self.use_terminate.isChecked():
self.thr.terminate()
else:
self.button_record.setText("Stop recording")
self.recording = True
self.t = QTimer()
self.t.timeout.connect(self.plotData)
self.t.start(0)
self.thr = QThread()
self.thr.start()
self.r = Record(self.pass_or_sleep.isChecked(), self.use_terminate.isChecked())
self.r.moveToThread(self.thr)
self.r.stop.connect(self.r.stopWhileLoop)
self.r.start.connect(self.r.run)
self.r.start.emit('go!')
def addData(self, data):
# print('got data...')
self.q.put(data)
def callback(self, indata, frames, time, status):
self.q.put(indata.copy())
print("Main thread -> callback", threading.get_ident())
def plotData(self):
self.times = self.times[1:]
self.times.append(time.time())
fps = 1 / (np.diff(np.array(self.times)).mean() + 1e-5)
self.setWindowTitle("{:d} fps...".format(int(fps)))
if q.empty():
return
d = q.get()
# print("got data ! ...")
self.curve1.setData(d[:, 0])
self.curve2.setData(d[:, 1]-1)
if __name__ == '__main__':
app = QApplication([])
w = Main()
w.show()
app.exec_()
edit 2
Here the code that uses no QThread environment, and this works as expected!
from PyQt5.QtWidgets import QApplication, QWidget, QGridLayout, QPushButton, QCheckBox
from PyQt5.QtCore import QTimer
import threading
import sounddevice as sd
import queue
import pyqtgraph as pg
import numpy as np
import time
class Main(QWidget):
def __init__(self):
super().__init__()
self.recording = False
self.r = None
self.x = 0
self.times = list(range(10))
self.q = queue.Queue()
self.setWindowTitle("Record Audio Tester")
self.l = QGridLayout()
self.setLayout(self.l)
self.pl = pg.PlotWidget(autoRange=False)
self.curve1 = self.pl.plot(np.zeros(8000))
self.curve2 = self.pl.plot(np.zeros(8000)-1, pen=pg.mkPen("y"))
self.l.addWidget(self.pl)
self.button_record = QPushButton("Start recording")
self.button_record.clicked.connect(self.record)
self.l.addWidget(self.button_record)
print("Main thread", threading.get_ident())
def streamData(self):
self.r = sd.InputStream(device=1, channels=2, callback=self.callback)
self.r.start()
def record(self):
if self.recording and self.r is not None:
self.button_record.setText("Start recording")
self.recording = False
self.r.stop()
else:
self.button_record.setText("Stop recording")
self.recording = True
self.t = QTimer()
self.t.timeout.connect(self.plotData)
self.t.start(0)
self.streamData()
def callback(self, indata, frames, time, status):
self.q.put(indata.copy())
print("Main thread -> callback", threading.get_ident())
def plotData(self):
self.times = self.times[1:]
self.times.append(time.time())
fps = 1 / (np.diff(np.array(self.times)).mean() + 1e-5)
self.setWindowTitle("{:d} fps...".format(int(fps)))
if self.q.empty():
return
d = self.q.get()
# print("got data ! ...")
self.curve1.setData(d[:, 0])
self.curve2.setData(d[:, 1]-1)
if __name__ == '__main__':
app = QApplication([])
w = Main()
w.show()
app.exec_()
The problem is due to the while True: pass line in your thread. To understand why, you need to understand how PortAudio (the library wrapped by sounddevice) works.
Anything passed a callback like you are doing with InputStream is likely calling the provided method from a separate thread (not the main thread or your QThread). Now from what I can tell, whether the callback is called from a separate thread or some sort of interrupt is platform dependent, but either way it is operating somewhat independently of your QThread even though the method exists inside that class.
The while True: pass is going to consume close to 100% of your CPU, limiting what any other thread can do. That is until you terminate it! Which frees up resources for whatever is actually calling the callback to work faster. While you might expect the audio capture to be killed along with your thread, chances are it hasn't been garbage collected yet (and garbage collection gets complicated when dealing with C/C++ wrapped libraries, nevermind when you have two of them! [PortAudio and Qt] - And there is a good chance that garbage collection in Python might not actually free the resources in your case anyway!)
So this explains why things get faster when you terminate the thread.
The solution is to change your loop to while True: time.sleep(.1) which will ensure it doesn't consume resources unnecessarily! You could also look into whether you actually need that thread at all (depending on how PortAudio works on your platform). If you move to the signal/slot architecture and do away with the with statement (managing the open/close of the resource in separate slots) that would also work as you wouldn't need the problematic loop at all.
There are some small issues in your code, all concurring to the "low" framerate, mostly due to the fact that you're using a blocksize of 4096 (which is too high if you want frequent updates) and are also trying to update the GUI too fast while also processing data.
On a slightly old computer as mine, your code completely hangs the interface as soon as I start recording, and stopping is almost impossible if not by killing the app at all.
What you see is not the thread "speeding" up, but rather the QTimer that has much more "time" (cycles) to call its timeout more frequently.
First of all, you shouldn't use terminate, but maybe use a Queue to send a "quit" command to the while loop, allowing it to exit gracefully.
Then, it's better to use signals/slots to retrieve and process the output data, as it's more intuitive and improves the overall cpu load.
Finally, if you want to get the fps of the received data, there's no use in a QTimer set to 0 (which will make it only run as fast as possible even if unnecessary, making the cpu spike needlessly).
class Record(QThread):
audioData = pyqtSignal(object)
def __init__(self):
super().__init__()
self.stopper = queue.Queue()
def callback(self, indata, frames, time, status):
self.audioData.emit(indata.copy())
def run(self):
with sd.InputStream(samplerate=48000, channels=2, callback=self.callback, blocksize=1024):
print('Stream started...')
while True:
try:
if self.stopper.get(timeout=.1):
break
except:
pass
print(self.isRunning(), 'Done?') # never called
def stop(self):
self.stopper.put(True)
class Main(QWidget):
# ...
def record(self):
if self.recording and self.r is not None:
self.button_record.setText("Start recording")
self.recording = False
self.r.stop()
else:
self.button_record.setText("Stop recording")
self.recording = True
self.r = Record()
self.r.audioData.connect(self.plotData)
self.r.start()
def plotData(self, data):
self.curve1.setData(data[:, 0])
self.curve2.setData(data[:, 1]-3)
self.times = self.times[1:]
self.times.append(time.time())
fps = 1 / (np.diff(np.array(self.times)).mean() + 1e-5)
self.setWindowTitle("{:d} fps...".format(int(fps)))