Is there a way to use tqdm on a function call because i am not understanding how to display a progress bar because i don't have any loops:
this is where the function call happens :
if length == 2:
post().request(email,password)
everytime it makes a request, i want my progress bar to move, is there a way ?
So post is a class. As much I understood your question, you may try these:
For a progressbar for each request:
class post():
def __init__(self):
pass
def req(self):
for _ in trange(1):
print("req")
p = post()
p.req()
For single progressbar for n_req requests:
class post(tqdm.tqdm):
def __init__(self, n_req = 1):
tqdm.tqdm.__init__(self)
self.total = n_req
def req(self, i):
print("req", i)
self.update()
n_req = 5
with post(n_req) as p:
for i in range(n_req):
p.req(i)
#------------OR-----------------
n_req = 5
with post(n_req) as p:
p.req(1); p.req(2); p.req(3); p.req(4); p.req(5)
(I'm not sure why it's not working without with format. Probably because python objects or callbacks don't work same as c++ objects.)
Related
how is possible to pass a function to a thread like a standard variable,
when i pass function on args , i have a non iterable error
there is a summary of my code
from CallbackThread import CallbackThread
import time
import threading
class foo:
def runSomthing(self,callback):
#do things
callthread = threading.Thread(target=self.newthread, args=(callback))
callthread.start()
def newthread(self,calback):
print("runiing")
while True:
#if receve data (udp)
data = 0
#run mycallbacktest
calback(data)
def mycallbacktest(i):
print("hello world", i)
myobj = foo()
myobj.runSomthing(mycallbacktest)
i have see on similar topics things like that
https://gist.github.com/amirasaran/e91c7253c03518b8f7b7955df0e954bb
and i have try this based on this bu i not sure what this is doing, for me is just call a callback when thread if finished
class BaseThread(threading.Thread):
def __init__(self, callback=None, callback_args=None, *args, **kwargs):
target = kwargs.pop('target')
super(BaseThread, self).__init__(target=self.target_with_callback, *args, **kwargs)
self.callback = callback
self.method = target
self.callback_args = callback_args
def target_with_callback(self):
self.method()
if self.callback is not None:
self.callback(*self.callback_args)
but this d'ont solve what i tring to do
as suggested MisterMiyagi, the awnser a missig comma,
It should be args=(callback,) instead of args=(callback)
(this post is to mark this post close)
There's a situation where I want to check how many times an internal class method has been called. I have a sensitive cloud task that must be done in a count that depends upon some circunstances. I would like to strengthen my application with an unittest to assert the number of times a specific function has been called.
To do so in a much simpler scenario, I would like to make a test in the following script:
class HMT:
def __init__(self):
self.buildedList = []
def handle(self, string_to_explode: str):
for exploded_part in string_to_explode.split(","):
self.doCoolThings(exploded_part)
def doCoolThings(self, fetched_raw_string: str):
self.buildedList.append("Cool thing done: " + fetched_raw_string)
Depending on the string that I deliver to the handle function, the doCoolThings will be called N times (in this simple case, depends solely in the number of comas inside the string).
I can make a test works by just counting the number of resulting elements inside builderList:
import unittest
from HMT import HMT
class test_HMT(unittest.TestCase):
def setUp(self):
self.hmt = HMT()
def test_hmt_3(self):
string_to_test = "alpha,beta,gamma"
self.hmt.handle(string_to_test)
self.assertEqual(3, len(self.hmt.buildedList))
def test_hmt_2(self):
string_to_test = "delta,epsilon"
self.hmt.handle(string_to_test)
self.assertEqual(2, len(self.hmt.buildedList))
But in the real scenario, there's will not have an available public class list that always will match its number of elements to the times the function doCoolThings was called.
So, how do I check how many times the doCoolThings was called without needing to check the list elements count?
I know that I can just put a counter in the class that is increased each time doCoolThings is called and expose it externally to be checked afterwards. But I don't would like to mess up the code putting lines that is not directly related to my business rule.
After #jarmod comment, I came into this code's version:
def mydecorator(func):
def wrapped(*args, **kwargs):
wrapped.calls += 1
return func(*args, **kwargs)
wrapped.calls = 0
return wrapped
class HMT:
def __init__(self):
self.buildedList = []
def handle(self, string_to_explode: str):
for exploded_part in string_to_explode.split(","):
self.doCoolThings(exploded_part)
#mydecorator
def doCoolThings(self, fetched_raw_string: str, *args, **kwargs):
self.buildedList.append("Cool thing done: " + fetched_raw_string)
And the test:
import unittest
from HMT import HMT
class test_HMT(unittest.TestCase):
def test_hmt_3_dec(self):
hmt = HMT()
string_to_test = "epsilon,ota,eta"
hmt.handle(string_to_test)
self.assertEqual(3, hmt.doCoolThings.calls)
def test_hmt_3(self):
hmt = HMT()
string_to_test = "alpha,beta,gamma"
hmt.handle(string_to_test)
self.assertEqual(3, len(hmt.buildedList))
But still are not working properly. When I run tests, I receive:
.F
======================================================================
FAIL: test_hmt_3_dec (myTest.test_HMT)
----------------------------------------------------------------------
Traceback (most recent call last):
File "D:\Users\danil\tmp\myDec\myTest.py", line 10, in test_hmt_3_dec
self.assertEqual(3, hmt.doCoolThings.calls)
AssertionError: 3 != 6
----------------------------------------------------------------------
Ran 2 tests in 0.001s
FAILED (failures=1)
More tests shows that the tests runs twice but the counter do not reset either in new instantiation.
Anyway, the initial idea was to dynamically put an observer in a internal class method and externaly fetches each time when those method is triggered (still, not something that seems to be solved using decorator).
Thanks alot for answers (and as a plus to leave powered up, if someone knows how to reset the counter in the decorator I will appreciate as well).
from HMT import HMT
count = 0
def counter(f):
def wrap(*args, **kwargs):
global count
count += 1
return f(*args, **kwargs)
return wrap
class test_HMT(unittest.TestCase):
def setUp(self):
self.hmt = HMT()
# Add decorator.
self.hmt_no_decorator = self.hmt.doCoolThings
self.hmt.doCoolThings = counter(self.hmt.doCoolThings)
def test_doCoolThings_count(self):
repeat = 3
[self.hmt.doCoolThings() for _ in range(repeat)]
self.assertEqual(counter, repeat)
def tearDown(self):
# Remove decorator.
self.hmt.doCoolThings = self.hmt_no_decorator
...
doCoolThings is not modified in the business code. You simply get the counting behaviour for testing.
You can get rid of the global var by replacing the count var with an object.. or by doing anything else really. But does it matter in testing?
I'm studying the MVP pattern but having a hard time following the principles in order to update in real time a progress bar. As I understand the Presenter checks if there's any update in the Model and then outputs the result, so there's no instantiation of the Presenter in the Model, only the Presenter should instantiate the Model and the View.
My question is: how should I update the progress bar by following the MVP principle?
I could of course call presenter.update_progress_bar(i, total) from Model, but then it would infringe the MVP principle.
Here's a minimal working example:
PS: for now, I'm using CLI.
/main.py
import modules
def main():
modules.View(modules.Presenter).run()
if __name__ == "__main__":
main()
/modules/__init__.py
from modules.Model.Model import Model
from modules.Model.progressbar import ProgressBar
from modules.View.View import View
from modules.Presenter.Presenter import Presenter
/modules/Model/Model.py
class Model:
def __init__(self):
pass
def long_process(self):
import time
for i in range(10):
time.sleep(0.1)
print("Update the progress bar.")
return True
/modules/Model/progressbar.py
# MIT license: https://gist.github.com/vladignatyev/06860ec2040cb497f0f3
import sys
class ProgressBar:
def progress(count, total, status=''):
bar_len = 60
filled_len = int(round(bar_len * count / float(total)))
percents = round(100.0 * count / float(total), 1)
bar = '=' * filled_len + '-' * (bar_len - filled_len)
sys.stdout.write('[%s] %s%s ...%s\r' % (bar, percents, '%', status))
sys.stdout.flush()
/modules/View/View.py
import sys
class View:
def __init__(self, presenter):
self.presenter = presenter(self)
def run(self):
self.presenter.long_process()
def update_progress_bar(self, msg):
sys.stdout.write(msg)
def hide_progress_bar(self, msg):
sys.stdout.write(msg)
def update_status(self, msg):
print(msg)
/modules/Presenter/Presenter.py
class Presenter:
def __init__(self, view):
import modules
self.model = modules.Model()
self.view = view
def long_process(self):
if self.model.long_process():
self.view.update_status('Long process finished correctly')
else:
self.view.update_status('error')
def update_progress_bar(self, i, total):
from modules import ProgressBar
ProgressBar.progress(i, total)
self.view.update_progress_bar(ProgressBar.progress(i, total))
def end_progress_bar(self):
self.view.end_progress_bar('\n')
I could do:
class Model:
def __init__(self, presenter):
self.presenter = presenter # Violation of MVP
def long_process(self):
import time
for i in range(10):
time.sleep(0.1)
self.presenter.update_progress_bar(i, 10) # Violation of MVP
print("Update the progress bar.")
return True
But this is wrong since the Model now instantiates the Presenter. Any suggestions?
Use a callback:
import time
class Model:
def long_process(self, notify=lambda current, total: None):
for i in range(10):
time.sleep(0.1)
notify(i, 10)
return True
class Presenter:
def long_process(self):
result = self.model.long_process(lambda c, t: self.update_progress_bar(c, t)):
if result:
self.view.update_status('Long process finished correctly')
else:
self.view.update_status('error')
This keeps your model independant from the client code, while still allowing it (the model I mean) to notify it's caller.
As a side note, there are quite a few things in your code that are totally unpythonic:
1/ you don't have to put each class in a distinct module (it's actually considered an antipattern in Python), and even less in nested submodules (Python Zen: "flat is better than nested").
2/ you don't have to use classes when a plain function is enough (hint: Python functions are objects... actually, everything in Python is an object) - your ProgressBar class has no state and only one method, so it could just be a plain function (Python Zen: "simple is better than complex").
3/ imports should be at the top of the module, not in functions (if you have to put them in a function to solve cyclic dependancies issues then the proper solution is to rethink your design to avoid cyclic dependancies).
4/ module names should be all_lower
I am having trouble implementing the following scheme :
class A:
def __init__(self):
self.content = []
self.current_len = 0
def __len__(self):
return self.current_len
def update(self, new_content):
self.content.append(new_content)
self.current_len += 1
class B:
def __init__(self, id):
self.id = id
And I also have these 2 functions that will be called later in the main :
async def do_stuff(first_var, second_var):
""" this function is ideally called from the main in another
process. Also, first_var and second_var are not modified so it
would be nice if they could be given by reference without
having to copy them """
### used for first call
yield None
while len(first_var) < CERTAIN_NUMBER:
time.sleep(10)
while True:
## do stuff
if condition_met:
yield new_second_var ## which is a new instance of B
## continue doing stuff
def do_other_stuff(first_var, second_var):
while True:
queue = multiprocessing.JoinableQueue()
results = multiprocessing.Queue()
### do stuff
first_var.update(results)
The main looks like this at the moment :
first_var = A()
second_var = B()
while True:
async for new_second_var in do_stuff(first_var, second_var):
if new_second_var:
## stop the do_other_stuff that is currently running
## to re-launch it with the updated new_var
do_other_stuff(first_var, new_second_var)
else: ## used for the first call
do_other_stuff(first_var, second_var)
Here are my questions :
Is there a better solution to make this scheme work?
How can I implement the "stopping" part since there is a while True loop that fills first_var by reference?
Will the instance of A (first_var) be passed by reference to do_stuff if first_var doesn't get modified inside it?
Is it even possible to have an asynchronous generator in another process?
Is it even possible at all?
This is using Python 3.6 for the async generators.
I hope this is somewhat clear! Thanks a lot!
I was shocked to learn how little tutorials and guides there is to be found on the internet regarding parallel python (PP) and handling classes. I've ran into a problem where I want to initiate a couple of instances of the same class and after that retreive some variables (for instances reading 5 datafiles in parallel, and then retreive their data). Here's a simple piece of code to illustrate my problem:
import pp
class TestClass:
def __init__(self, i):
self.i = i
def doSomething(self):
print "\nI'm being executed!, i = "+str(self.i)
self.j = 2*self.i
print "self.j is supposed to be "+str(self.j)
return self.i
class parallelClass:
def __init__(self):
job_server = pp.Server()
job_list = []
self.instances = [] # for storage of the class objects
for i in xrange(3):
TC = TestClass(i) # initiate a new instance of the TestClass
self.instances.append(TC) # store the instance
job_list.append(job_server.submit(TC.doSomething, (), ())) # add some jobs to the job_list
results = [job() for job in job_list] # execute order 66...
print "\nIf all went well there's a nice bunch of objects in here:"
print self.instances
print "\nAccessing an object's i works ok, but accessing j does not"
print "i = "+str(self.instances[2].i)
print "j = "+str(self.instances[2].j)
if __name__ == '__main__' :
parallelClass() # initiate the program
I've added comments for your convenience. What am I doing wrong here?
You should use callbacks
A callbacks is a function that you pass to the submit call. That function will be called with the result of the job as argument (have a look at the API for more arcane usage).
In your case
Set up a callback:
class TestClass:
def doSomething(self):
j = 2 * self.i
return j # It's REQUIRED that you return j here.
def set_j(self, j):
self.j = j
Add the callback to the job submit call
class parallellClass:
def __init__(self):
#your code...
job_list.append(job_server.submit(TC.doSomething, callback=TC.set_j))
And you're done.
I made some improvements to the code to avoid using self.j in the doSomething call, and only use a local jvariable.
As mentioned in the comments, in pp, you only communicate the result of your job. That's why you have to return this variable, it will be passed to the callback.