I have implemented a progress bar in my pythonGUI script and when the run button is clicked it executes another python script in which I have used multiprocessing for fetching query from database and genrating a output excel. Suppose there is about 10 query to execute and generate 10 excel outputs . In this case, how do I implement progress bar for multiprocessing.?
Thanks you in advance
Given that I had none of your code to go off of, I made a few assumptions about what you are using and the overall structure. Hopefully however, there is enough info here so that you can adapt this to your needs and your specific code.
import PyQt5
from PyQt5 import QtWidgets
import threading
class MyClass(object):
progressbar_progress = PyQt5.QtCore.pyqtSignal(int)
def __init__(self):
# progressBar refers to YOUR progress bar object in your interface
# Change that name to whatever you've named your actual progress bar
self.progressbar_progress.connect(self.progressbar.setValue)
#staticmethod
def start_thread(functionName, *args, **kwargs):
if len(args) == 0:
t = threading.Thread(target=functionName)
else:
try:
t = threading.Thread(target=functionName, args=args, kwargs=kwargs)
except:
try:
if args is None:
t = threading.Thread(target=functionName, kwargs=kwargs)
except:
t = threading.Thread(target=functionName)
t.daemon = True
t.start()
def button_run_clicked(self):
MyClass.start_thread(self.run_query)
# You would probably want to come up with a way to get a numerical value for your
# processes in your query. For example, if you know that you have to search through
# 10,000 entries, then that becomes your reference point. So the first query would
# be 1 out of 10,000, which would be .01% progress and the PyQt5 progress bar's
# display value is based on a percentage. So just send the current state's progress
# percentage to the signal that was created.
def run_query(self):
# All your querying code
# For example, let's assume my_query is a list
for i, q in enumerate(my_query):
# Send the value to display to the signal like this
self.progressBar.progress.emit(i / len(my_query))
print('Done!')
Related
Upon looking up event handler modules, I came across pydispatcher, which seemed beginner friendly. My use case for the library is that I want to send a signal if my queue size is over a threshold. The handler function can then start processing and removing items from the queue (and subsequently do a bulk insert into the database).
I would like the handler function to run in the background. I am aware that I can simply overwrite the queue.append() method checking for the queue size and calling the handler function asynchronously, but I would like to implement the listener-dispatcher model to keep the logic clean and separated.
Does pydispatcher do this out of the box? If not, is there another module that can help me do this? Would I need to manage the access to the queue, since there might be multiple threads processing and appending to the queue at the same time?
Note that in my use case there is only a single dispatcher and event handler.
I've recently released the Akuanduba module, that may help you with this task. There's a single example on the repository that may help you understand how it works and it seems similar to what you want.
Anyway, I'll try to explain here a way of implementing your code with Akuanduba:
First you could make a data frame that would hold your queue:
# Mandatory imports
from Akuanduba.core.messenger.macros import *
from Akuanduba.core.constants import *
from Akuanduba.core import NotSet, AkuandubaDataframe
# Your imports go here:
from queue import Queue
class MyQueue (AkuandubaDataframe):
def __init__(self, name):
# Mandatory stuff
AkuandubaDataframe.__init__(self, name)
self.__queue = Queue ()
def getQueue (self):
return self.__queue
def putQueue (self, val):
self.__queue.put(val)
def getQueueSize (self):
return self.__queue.qsize()
#
# "toRawObj" method is a mandatory method that delivers a dict with the desired data
# for file saving
#
def toRawObj(self):
d = {
"Queue" : self.getQueue(),
}
return d
Then you could make a TriggerCondition that would check the queue size:
from Akuanduba.core import StatusCode, NotSet, StatusTrigger
from Akuanduba.core.messenger.macros import *
from Akuanduba.core import TriggerCondition
import time
class CheckQueueSize (TriggerCondition):
def __init__(self, name, maxSize):
TriggerCondition.__init__(self, name)
self._name = name
self._maxSize = maxSize
def initialize(self):
return StatusCode.SUCCESS
def execute (self):
size = self.getContext().getHandler("MyQueue").getQueueSize()
if (size > SIZE_THRESHOLD):
return StatusTrigger.TRIGGERED
else:
return StatusTrigger.NOT_TRIGGERED
def finalize(self):
return StatusCode.SUCCESS
Make a tool that would be your handler function:
# Mandatory imports
from Akuanduba.core import AkuandubaTool, StatusCode, NotSet, retrieve_kw
# Your imports go here:
class SampleTool(AkuandubaTool):
def __init__(self, name, **kw):
# Mandatory stuff
AkuandubaTool.__init__(self, name)
def initialize(self):
# Lock the initialization. After that, this tool can not be initialized once again
self.init_lock()
return StatusCode.SUCCESS
def execute(self,context):
#
# DO SOMETHING HERE
#
# Always return SUCCESS
return StatusCode.SUCCESS
def finalize(self):
self.fina_lock()
return StatusCode.SUCCESS
And finally, make a main script in order to make it all work together:
# Akuanduba imports
from Akuanduba.core import Akuanduba, LoggingLevel, AkuandubaTrigger
from Akuanduba import ServiceManager, ToolManager, DataframeManager
# This sample's imports
import MyQueue, CheckQueueSize, SampleTool
# Creating your handler
your_handler = SampleTool ("Your Handler's name")
# Creating dataframes
queue = MyQueue ("MyQueue")
# Creating trigger
trigger = AkuandubaTrigger("Sample Trigger Name", triggerType = 'or')
# Append conditions and tools to trigger just adding them
# Tools appended to the trigger will only run when trigger is StatusTrigger.TRIGGERED,
# and will run in the order they've been appended
trigger += CheckQueueSize( "CheckQueueSize condition", MAX_QUEUE_SIZE )
trigger += your_handler
# Creating Akuanduba
manager = Akuanduba("Akuanduba", level=LoggingLevel.INFO)
# Appending tools
#
# ToolManager += TOOL_1
# ToolManager += TOOL_2
#
ToolManager += trigger
# Apprending dataframes
DataframeManager += sampleDataframe
# Initializing
manager.initialize()
manager.execute()
manager.finalize()
That way, you'd have clean and separated code.
I need a progressbar to show the user the progress of the svn updates, but I do not know how to do this simultaneously to the svn update, I can only do it afterwards and then only indirectly, but if I let the progressbar show after the update , then I need twice the time and that's not the point.
Please help me how to update the progressbar while the update is running which needs a variable time and please give me an example of what you mean
class test():
def __init__(self, dictionary_with_paths):
self.dictionary_with_paths = {}
self.progressbar = Progressbar(range_end)
def update_local_tag_folder(self):
os.system('svn update '+ self.dictionary_with_paths['local_tags'])
def update_progressbar(self, current_value):
"""
Progressbar() is a class that simply create an tk() frame with 1
label, an progressbar and a button and it need an start value and
an end point, the end point comes with an parameter in the class,
the start/current count are given from method
Progressbar().set_current_Value(current_value)
"""
self.progressbar.set_current_value(current_Value)
self.progressbar.update()
def get_update_progress_and_update_bar(self):
"""
here is the point i have no idea how can i can get an list of
files or any other useful things
"""
SVN_list = os.system('svn ls -v -R '+ \
self.dictionary_with_paths['repository_trunk'])
"""
Console shows an list with all files, but i cant read the lines to
count that
"""
I also thought about using subprocessor but I do not know exactly how to read the output of SVN_list so that I can count the lines
SOLVED:
Fixed problem, I have created new threads in which I can once run the svn update and in another I can simultaneously create the Progressbar and let it run ...
short snapshot what i mean...
from threading import Thread
from time import sleep
class Yeeeehaaa():
def mymethod1(self, arg):
for e in range(0,arg):
print('Yeeeee')
sleep(1)
def mymethod2(self, arg):
for e in range(0,arg):
print('haaaaa')
sleep(1)
def start_threads(self):
thread = Thread(target=mymethod1, args=(10,))
thread.start()
thread = Thread(target=mymethod2, args=(10,))
thread.start()
thread.join()
thread.join()
I have written the following code that parses a text file, breaks it into tokens and inserts these tokens into the database. I want to show the current status of the process using the progress bar but the following code isn't working.
I wrote the following code based on this How to connect pyqtSignal between classes in PyQT
yast_gui.py
class YastGui(QtGui.QMainWindow):
incrementTokenSignal = QtCore.pyqtSignal(int)
...
def __init__(self):
self.incrementTokenSignal.connect(self.increment_token_count)
...
def increment_token_count(self, val):
msg = "{}/{}".format(val, self.total_db_records)
self.ui.records_processed_value_label.setText(msg)
yast.py
class LogFile(object):
def __init__(self, file_path, YastGui_object):
super(LogFile, self).__init__()
# Gui object
self.gui = YastGui_object
self.total_db_records = 0
...
def tokenize(self):
for i, record in enumerate(myfile):
...
self.gui.incrementFilterSignal.emit(i + 1)
settings.session.commit()
According to this PYQT and progress Bar during Long Process, I should create QTheads to deal with the progress bar but I'm not sure on how to do it.
Here is the entire Gui file and main file.
I found the solution to my problem here Change the value of the progress bar from a class other than my GUI class PyQt4.
The trick is to pass the progressBar object to the function as a parameter and then use progressBar.setValue() inside that function.
My PyQt program has 2 widgets (selecting files etc) and then a Main Window which displays the results of the parsed files.
The program works great for small sample files, but when trying to parse larger files it will hang (display "Not Responding") and then show the results after about 30 seconds or so.
I would like to implement a QDialog before the Main Window opens. The QDialog will have a progress bar to let the user know when the Main Window will open.
This progress bar needs to be set to the length of time before the Main Window pops up.
What is the best way to implement this? I have seen some examples, but the progress bar is just set to a standardised time, not when the processing(parsing) is complete.
I currently have the following code which opens the Main Window.
def openWidgetMain(self):
self.WidgetMain = WidgetMain()
self.WidgetMain.show()
self.close()
All the processing for this window is done when it opens. So how do I connect the QProgressBar?
Put your long lasting process in some kind of thread. Read this: http://qt-project.org/doc/qt-5/threads-technologies.html
Emit a signal from that thread to update your progress bar. This way your application will not hang and the user sees the progress.
However it is up to your loading routine to decide which percentage to show in the progress bar. If you can't calculate an exact percentage try some kind of estimation (e.g. based on the size of the file vs. processed amount of the file).
First, best way to implement this, Your must estimate your load progress file. Next, implement it with QtCore.QThread to create background process. Last, put your call back progress into your QtGui.QMainWindow.
Little example;
import sys
import time
from PyQt4 import QtGui
from PyQt4 import QtCore
class QCustomThread (QtCore.QThread):
startLoad = QtCore.pyqtSignal(int)
progressLoad = QtCore.pyqtSignal(int)
statusLoad = QtCore.pyqtSignal(bool)
def __init__ (self, parentQWidget = None):
super(QCustomThread, self).__init__(parentQWidget)
self.wasCanceled = False
def run (self):
# Simulate data load estimation
numberOfprogress = 100
self.startLoad.emit(numberOfprogress)
for progress in range(numberOfprogress + 1):
# Delay
time.sleep(0.1)
if not self.wasCanceled:
self.progressLoad.emit(progress)
else:
break
self.statusLoad.emit(True if progress == numberOfprogress else False)
self.exit(0)
def cancel (self):
self.wasCanceled = True
class QCustomMainWindow (QtGui.QMainWindow):
def __init__ (self):
super(QCustomMainWindow, self).__init__()
# Create action with QPushButton
self.startQPushButton = QtGui.QPushButton('START')
self.startQPushButton.released.connect(self.startWork)
self.setCentralWidget(self.startQPushButton)
# Create QProgressDialog
self.loadingQProgressDialog = QtGui.QProgressDialog(self)
self.loadingQProgressDialog.setLabelText('Loading')
self.loadingQProgressDialog.setCancelButtonText('Cancel')
self.loadingQProgressDialog.setWindowModality(QtCore.Qt.WindowModal)
def startWork (self):
myQCustomThread = QCustomThread(self)
def startLoadCallBack (numberOfprogress):
self.loadingQProgressDialog.setMinimum(0)
self.loadingQProgressDialog.setMaximum(numberOfprogress)
self.loadingQProgressDialog.show()
def progressLoadCallBack (progress):
self.loadingQProgressDialog.setValue(progress)
def statusLoadCallBack (flag):
print 'SUCCESSFUL' if flag else 'FAILED'
myQCustomThread.startLoad.connect(startLoadCallBack)
myQCustomThread.progressLoad.connect(progressLoadCallBack)
myQCustomThread.statusLoad.connect(statusLoadCallBack)
self.loadingQProgressDialog.canceled.connect(myQCustomThread.cancel)
myQCustomThread.start()
myQApplication = QtGui.QApplication(sys.argv)
myQCustomMainWindow = QCustomMainWindow()
myQCustomMainWindow.show()
sys.exit(myQApplication.exec_())
More infomation of QtCore.QThread (Recommend read to understand behavior)
Here is the code sample:
class RunGui (QtGui.QMainWindow)
def __init__(self, parent=None):
...
QtCore.Qobject.connect(self.ui.actionNew, QtCore.SIGNAL("triggered()"), self.new_select)
...
def normal_output_written(self, qprocess):
self.ui.text_edit.append("caught outputReady signal") #works
self.ui.text_edit.append(str(qprocess.readAllStandardOutput())) # doesn't work
def new_select(self):
...
dialog_np = NewProjectDialog()
dialog_np.exec_()
if dialog_np.is_OK:
section = dialog_np.get_section()
project = dialog_np.get_project()
...
np = NewProject()
np.outputReady.connect(lambda: self.normal_output_written(np.qprocess))
np.errorReady.connect(lambda: self.error_output_written(np.qprocess))
np.inputNeeded.connect(lambda: self.input_from_line_edit(np.qprocess))
np.params = partial(np.create_new_project, section, project, otherargs)
np.start()
class NewProject(QtCore.QThread):
outputReady = QtCore.pyqtSignal(object)
errorReady = QtCore.pyqtSignal(object)
inputNeeded = QtCore.pyqtSignal(object)
params = None
message = ""
def __init__(self):
super(NewProject, self).__init__()
self.qprocess = QtCore.QProcess()
self.qprocess.moveToThread(self)
self._inputQueue = Queue()
def run(self):
self.params()
def create_new_project(self, section, project, otherargs):
...
# PyDev for some reason skips the breakpoints inside the thread
self.qprocess.start(command)
self.qprocess.waitForReadyRead()
self.outputReady.emit(self.qprocess) # works - I'm getting signal in RunGui.normal_output_written()
print(str(self.qprocess.readAllStandardOutput())) # prints empty line
.... # other actions inside the method requiring "command" to finish properly.
The idea is beaten to death - get the GUI to run scripts and communicate with the processes. The challenge in this particular example is that the script started in QProcess as command runs an app, that requires user input (confirmation) along the way. Therefore I have to be able to start the script, get all output and parse it, wait for the question to appear in the output and then communicate back the answer, allow it to finish and only then to proceed further with other actions inside create_new_project()
I don't know if this will fix your overall issue, but there are a few design issues I see here.
You are passing around the qprocess between threads instead of just emitting your custom signals with the results of the qprocess
You are using class-level attributes that should probably be instance attributes
Technically you don't even need the QProcess, since you are running it in your thread and actively using blocking calls. It could easily be a subprocess.Popen...but anyways, I might suggest changes like this:
class RunGui (QtGui.QMainWindow)
...
def normal_output_written(self, msg):
self.ui.text_edit.append(msg)
def new_select(self):
...
np = NewProject()
np.outputReady.connect(self.normal_output_written)
np.params = partial(np.create_new_project, section, project, otherargs)
np.start()
class NewProject(QtCore.QThread):
outputReady = QtCore.pyqtSignal(object)
errorReady = QtCore.pyqtSignal(object)
inputNeeded = QtCore.pyqtSignal(object)
def __init__(self):
super(NewProject, self).__init__()
self._inputQueue = Queue()
self.params = None
def run(self):
self.params()
def create_new_project(self, section, project, otherargs):
...
qprocess = QtCore.QProcess()
qprocess.start(command)
if not qprocess.waitForStarted():
# handle a failed command here
return
if not qprocess.waitForReadyRead():
# handle a timeout or error here
return
msg = str(self.qprocess.readAllStandardOutput())
self.outputReady.emit(msg)
Don't pass around the QProcess. Just emit the data. And create it from within the threads method so that it is automatically owned by that thread. Your outside classes should really not have any knowledge of that QProcess object. It doesn't even need to be a member attribute since its only needed during the operation.
Also make sure you are properly checking that your command both successfully started, and is running and outputting data.
Update
To clarify some problems you might be having (per the comments), I wanted to suggest that QProcess might not be the best option if you need to have interactive control with processes that expect periodic user input. It should work find for running scripts that just produce output from start to finish, though really using subprocess would be much easier. For scripts that need user input over time, your best bet may be to use pexpect. It allows you to spawn a process, and then watch for various patterns that you know will indicate the need for input:
foo.py
import time
i = raw_input("Please enter something: ")
print "Output:", i
time.sleep(.1)
print "Another line"
time.sleep(.1)
print "Done"
test.py
import pexpect
import time
child = pexpect.spawn("python foo.py")
child.setecho(False)
ret = -1
while ret < 0:
time.sleep(.05)
ret = child.expect("Please enter something: ")
child.sendline('FOO')
while True:
line = child.readline()
if not line:
break
print line.strip()
# Output: FOO
# Another line
# Done