I Have a counting function that I would like to start and restart while getting the live variables to use in another function my problem is while using threading it seams like even global variables don't seem to work to pass variables around. What I want the code to do is have a counter be triggered as needed or maybe free running I'm not sure yet. To be able to reset the counter and get the value of the counter.
Right now the counter will start and run fine but the print_stuff function keeps telling me that there is no attribute countval.
The count thread gets started at startup but I don't necessarily want it to start up immediately, I would like to trigger it as needed but I cant put count_thread.start() twice or it will through a error so I'm calling the thread at startup and then calling the function again to restart it as needed. Maybe there is a more elegant way of doing that.?
import threading
import time
def count():
global countval
for countval in range(3):
print('looping')
time.sleep(1)
def print_stuff():
global countval
e = input("press enter to start")
count()
while True:
if countval == 3:
print("time out")
count_thread = threading.Thread(target=count)
print_thread = threading.Thread(target=print_stuff)
print_thread.start()
count_thread.start()
print_stuff is getting to the if statement before the count function is able to create the variable. Just do them in the opposite order. Either that, or create a global countval = 0 to start things off.
To solve the no attribute problem you can use Queue,
and if you want to stop your counting thread you can set a global variable or you can pass a function (using lambda or inner function or ...) to do that.
Here is one way to do that:
import threading
import time
from queue import Queue
from typing import Callable
def count(q, stop_counting):
# type: (Queue, Callable[[], bool]) -> None
for countval in range(3):
if stop_counting():
print('stopped')
break
print(f'looping {countval}')
q.put(countval)
time.sleep(1)
def print_stuff(q):
# type: (Queue) -> None
while True:
countval = q.get()
print(f'countval gotten: {countval}')
if countval == 3:
print("time out")
def main():
flag_stop_counting = False
q = Queue()
def stop_counting():
return flag_stop_counting
count_thread = threading.Thread(target=count, args=(q, stop_counting,))
print_thread = threading.Thread(target=print_stuff, args=(q,))
print_thread.start()
count_thread.start()
time.sleep(1.25)
flag_stop_counting = True
if __name__ == '__main__':
main()
In this code:
counter checks if it should stop counting or not
counter puts the value that it made to q
print_stuff get the value from q (Note: he waits until counter puts his value in q)
To check that program works:
after 1.25 seconds we change the value of flag_stop_counting
But if you want your counter to only have a for, i guess it's better to don't make it as a thread and run it whenever you want.
Hope it was helpful.
I am trying to access the same global dictionary from different threads in python simultaneously. Thread safety at the accessing point is not a consern for me since all accesses are reads and dont modify the dictionary.
I changed my code to do the accesses from multiple threads but i have noticed no increase in the speed of the execution, after checking arround it seems that the interpreter serializes the accesses in effect making the change in my code null.
Is there an easy way to have a structure like concurrentHashMap of Java in python?
The part of the code in question follows:
class csvThread (threading.Thread):
def __init__(self, threadID, bizName):
threading.Thread.__init__(self)
self.threadID = threadID
self.bizName = bizName
def run(self):
thread_function(self.bizName)
def thread_function(biz):
first = True
bizTempImgMap = {}
for imag in bizMap[biz]:
if not similar(bizTempImgMap, imgMap[imag]):
bizTempImgMap[imag] = imgMap[imag]
if first:
a = imgMap[imag]
sum = a
else:
c = np.column_stack((a, imgMap[imag]))
sum += imgMap[imag]
a = c.max(1) #max
first = False
else:
print ("-")
csvLock.acquire()
writer.writerow([biz]+a.astype(np.str_).tolist()+(np.true_divide(sum, len(bizTempImgMap.keys()))).tolist())
csvLock.release()
csvLock = threading.Lock()
...
imgMap = img_vector_load('test_photos.csv')
bizMap = img_busyness_load('csv/test_photo_to_biz_ids.csv')
...
for biz in bizMap.keys():
if len(threads)<100:
thread = csvThread(len(threads), biz)
threads.append(thread)
thread.start()
else:
print("\nWaiting for threads to finish\n")
for t in threads:
t.join()
print("\nThreads Finished\n")
threads = []
"i have noticed no increase in the speed of the execution"
No speed increase will be done by using threads in python, since they all work on the same core.
Take a look to: GIL
Notice this, python threading should be used for concurrent arquitectures not for speed performance.
In case you want to keep this implementation use multiprocessing.
I have a question regarding the example posted below...
On my machine calcIt() function takes about 5 seconds to complete.
The same calcIt() function is called from inside of MyClass's callCalcIt() method.
Using while loop MyClass is "watching" for calcIt() function to finish.
Question: A while loop inside of calcIt() method prints out '...running' only once. Honestly I was expecting to see at least 'an infinite loop' type of behavior where '...running' would be printed thousand times per second. Observing a fact the while loop executes a print '...running' line only once makes me believe while loop watches very 'loosely' for calcIt()'s progress. If so, what other (other than while loop) approach should be used to make sure you get what you want: an instant feedback from calcIt() function?
def calcIt():
a=True
while a:
for i in range(25000000):
pass
a=False
return True
class MyClass(object):
def __init__(self):
super(MyClass, self).__init__()
def callCalcIt(self):
b=True
while b:
result=calcIt()
print '...running'
if result: b=False
print 0
calcIt()
print 1
c=MyClass()
c.callCalcIt()
print 2
EDITED LATER:
Posting a revised code with an implementation of solution suggested by Ebarr:
import threading
updateMe=[]
def calcIt():
a=True
while a:
for y in range(3):
for i in range(15000000):
pass
updateMe.append(1)
a=False
return True
class MyClass(object):
def __init__(self):
super(MyClass, self).__init__()
def callCalcIt(self):
thread = threading.Thread(target=calcIt)
thread.start()
print '\n...thread started'
while thread.is_alive():
if len(updateMe)==1: print '...stage 1'
if len(updateMe)==2: print '...stage 2'
if len(updateMe)==3: print '...stage 3'
def printUpdate(self):
print 'updateMe=', len(updateMe)
c=MyClass()
c.callCalcIt()
I'm not sure what you were expecting to happen, but the explanation is very simple. You are running a single threaded code. This means that all of the above will be executed in serial, so there will be no concurrency between the two while loops in your code.
What you appear to be asking is how to thread your code such that you can check the progress of a running function. If that is the case, you can turn calcIt into a thread.
import threading
class CalcIt(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
def run(self):
for i in range(25000000):
pass
You would then change callCalcIt to:
def callCalcIt(self):
thread = CalcIt()
thread.start()
while thread.is_alive():
print '...running'
Alternatively, you can make it simpler:
import threading
def calcIt():
for i in range(25000000):
pass
def callCalcIt():
thread = threading.Thread(target=calcIt)
thread.start()
while thread.is_alive():
print '...running'
callCalcIt()
I can come up with two ways of doing that, but both require some modification to the calcIt function.
Method 1, callbacks:
def calc_it(callback):
r = 25000000
for x in xrange(r+1):
if not (x % 1000):
callback(x, r) # report every 1000 ticks
class Monitor(object):
def print_status(self, x, r):
print "Done {0} out of {1}".format(x, r)
def call(self):
calc_it(self.print_status)
Method 2, generator:
def calc_it():
r = 25000000
for x in xrange(r+1):
if not (x % 1000): # report every 1000 ticks
yield x, r
class Monitor(object):
def call(self):
for x, r in calc_it():
print "Done {0} out of {1}".format(x, r)
(A side note: in neither case Monitor has to be a class, that's just for consistency with the original code.)
Not sure exactly what you are trying to accomplish, but you could possibly use my newly written generator state machine thingie. Like so:
from generatorstate import State
def calcIt():
while True:
for i in range(25000000):
pass
yield
tick = State(calcIt)
print 0
tick()
print 1
tick()
print 2
I've added a couple of examples, sneak a peek at those if you think it might be a fit.
I wrote a simple python pexpect script to ssh into a machine and perform a action. Now I need to do this action to multiple servers. I am using a list to hit all of the servers concurrently using multi-threading. My issue is due to everything being ran concurrently, each thread is running on the same server name. Is there a way to concurrently have each thread only run one of the listed servers?
#! /usr/bin/python
#Test script
import pexpect
import pxssh
import threading
import datetime
currentdate = datetime.datetime.now()
easterndate = (datetime.datetime.now() + datetime.timedelta(0, 3600))
#list of servers
serverlist = ["025", "089"]
#server number
sn = 0
ssh_new_conn = 'Are you sure you want to continue connecting'
class ThreadClass(threading.Thread):
def run(self):
index = 0
sn = serverlist[index]
print sn
username = '[a username]'
password = '[a password]'
hostname = '%(sn)s.[the rest of the host url]' % locals()
command = "/usr/bin/ssh %(username)s#%(hostname)s " % locals()
index = index + 1
now = datetime.datetime.now()
print command
p = pexpect.spawn(command, timeout=360)
***do some other stuff****
for i in range(len(severlist)):
t = ThreadClass()
t.start()
[update]
I may just trying doing this with a parent thread that calls the child thread and so forth....although it would be nice if multi-threading could work from a list or some sort of work queue.
The problem has nothing to do with "everything being ran concurrently". You're explicitly setting index = 0 at the start of the run function, so of course every thread works on index 0.
If you want each thread to deal with one server, just pass the index to each thread object:
class ThreadClass(threading.Thread):
def __init__(self, index):
super(ThreadClass, self).__init__()
self.index = index
def run(self):
sn = serverlist[self.index]
print sn
# same code as before, minus the index = index + 1 bit
for i in range(len(severlist)):
t = ThreadClass(i)
t.start()
(Of course you'll probably want to use serverlist instead of severlist and fix the other errors that make it impossible for your code to work.)
Or, more simply, pass the sn itself:
class ThreadClass(threading.Thread):
def __init__(self, sn):
super(ThreadClass, self).__init__()
self.sn = sn
def run(self):
print self.sn
# same code as last version, but use self.sn instead of sn
for sn in severlist:
t = ThreadClass(sn)
t.start()
Alternatively, if you really want to use a global variable, just make it global, and put a lock around it:
index = 0
index_lock = threading.Lock()
class ThreadClass(threading.Thread):
def run(self):
global index, index_lock
with index_lock:
sn = serverlist[index]
index += 1
print sn
# same code as first version
However, you might want to consider a much simpler design, with a pool or executor instead of an explicit worker thread and list of things to work on. For example:
def job(sn):
print sn
# same code as first version again
with concurrent.futures.ThreadPoolExecutor() as executor:
executor.map(job, serverlist)
This will only run, say, 4 or 8 or some other good "magic number" of jobs concurrently. Which is usually what you want. But if you want exactly one thread per server, just pass max_workers=len(serverlist) to the ThreadPoolExecutor constructor.
Besides being a whole lot less code to read, write, get wrong, debug, etc., it also has more functionality—e.g., you can get results and/or exceptions from the servers back to the main thread.
I need to wait in a script until a certain number of conditions become true?
I know I can roll my own eventing using condition variables and friends, but I don't want to go through all the trouble of implementing it, since some object property changes come from external thread in a wrapped C++ library (Boost.Python), so I can't just hijack __setattr__ in a class and put a condition variable there, which leaves me with either trying to create and signal a Python condition variable from C++, or wrap a native one and wait on it in Python, both of which sound fiddly, needlessly complicated and boring.
Is there an easier way to do it, barring continuous polling of the condition?
Ideally it would be along the lines of
res = wait_until(lambda: some_predicate, timeout)
if (not res):
print 'timed out'
Unfortunately the only possibility to meet your constraints is to periodically poll, e.g....:
import time
def wait_until(somepredicate, timeout, period=0.25, *args, **kwargs):
mustend = time.time() + timeout
while time.time() < mustend:
if somepredicate(*args, **kwargs): return True
time.sleep(period)
return False
or the like. This can be optimized in several ways if somepredicate can be decomposed (e.g. if it's known to be an and of several clauses, especially if some of the clauses are in turn subject to optimization by being detectable via threading.Events or whatever, etc, etc), but in the general terms you ask for, this inefficient approach is the only way out.
Another nice package is waiting - https://pypi.org/project/waiting/
install:
pip install waiting
Usage:
You pass a function that will be called every time as a condition, a timeout, and (this is useful) you can pass a description for the waiting, which will be displayed if you get TimeoutError.
using function:
from waiting import wait
def is_something_ready(something):
if something.ready():
return True
return False
# wait for something to be ready
something = # whatever
wait(lambda: is_something_ready(something), timeout_seconds=120, waiting_for="something to be ready")
# this code will only execute after "something" is ready
print("Done")
Note: the function must return a boolean - True when the wait is over, False otherwise
Here is another solution. The goal was to make threads to wait on each other before doing some work in a very precise order. The work can take unknown amount of time. Constant polling is not good for two reasons: it eats CPU time and action does not start immediately after condition is met.
class Waiter():
def __init__(self, init_value):
self.var = init_value
self.var_mutex = threading.Lock()
self.var_event = threading.Event()
def WaitUntil(self, v):
while True:
self.var_mutex.acquire()
if self.var == v:
self.var_mutex.release()
return # Done waiting
self.var_mutex.release()
self.var_event.wait(1) # Wait 1 sec
def Set(self, v):
self.var_mutex.acquire()
self.var = v
self.var_mutex.release()
self.var_event.set() # In case someone is waiting
self.var_event.clear()
And the way to test it
class TestWaiter():
def __init__(self):
self.waiter = Waiter(0)
threading.Thread(name='Thread0', target=self.Thread0).start()
threading.Thread(name='Thread1', target=self.Thread1).start()
threading.Thread(name='Thread2', target=self.Thread2).start()
def Thread0(self):
while True:
self.waiter.WaitUntil(0)
# Do some work
time.sleep(np.random.rand()*2)
self.waiter.Set(1)
def Thread1(self):
while True:
self.waiter.WaitUntil(1)
# Do some work
time.sleep(np.random.rand())
self.waiter.Set(2)
def Thread2(self):
while True:
self.waiter.WaitUntil(2)
# Do some work
time.sleep(np.random.rand()/10)
self.waiter.Set(0)
Waiter for multiprocessing:
import multiprocessing as mp
import ctypes
class WaiterMP():
def __init__(self, init_value, stop_value=-1):
self.var = mp.Value(ctypes.c_int, init_value)
self.stop_value = stop_value
self.event = mp.Event()
def Terminate(self):
self.Set(self.stop_value)
def Restart(self):
self.var.value = self.init_value
def WaitUntil(self, v):
while True:
if self.var.value == v or self.var.value == self.stop_value:
return
# Wait 1 sec and check aiagn (in case event was missed)
self.event.wait(1)
def Set(self, v):
exit = self.var.value == self.stop_value
if not exit: # Do not set var if threads are exiting
self.var.value = v
self.event.set() # In case someone is waiting
self.event.clear()
Please comment if this is still not the best solution.
You've basically answered your own question: no.
Since you're dealing with external libraries in boost.python, which may change objects at their leisure, you need to either have those routines call an event handler refresh, or work with a condition.
Here is the threading extention to Alex's solution:
import time
import threading
# based on https://stackoverflow.com/a/2785908/1056345
def wait_until(somepredicate, timeout, period=0.25, *args, **kwargs):
must_end = time.time() + timeout
while time.time() < must_end:
if somepredicate(*args, **kwargs):
return True
time.sleep(period)
return False
def wait_until_par(*args, **kwargs):
t = threading.Thread(target=wait_until, args=args, kwargs=kwargs)
t.start()
print ('wait_until_par exits, thread runs in background')
def test():
print('test')
wait_until_par(test, 5)
From the computational perspective there must be a check for all conditions somewhere, sometime. If you have two parts of code, one that generates conditions changes and the other one that should be executed when some are true, you can do the following:
Have the code that changes conditions in, say, main thread, and the code that should be launched when some conditions are true, in a worker thread.
from threading import Thread,Event
locker = Event()
def WhenSomeTrue(locker):
locker.clear() # To prevent looping, see manual, link below
locker.wait(2.0) # Suspend the thread until woken up, or 2s timeout is reached
if not locker.is_set(): # when is_set() false, means timeout was reached
print('TIMEOUT')
else:
#
# Code when some conditions are true
#
worker_thread = Thread(target=WhenSomeTrue, args=(locker,))
worker_thread.start()
cond1 = False
cond2 = False
cond3 = False
def evaluate():
true_conditions = 0
for i in range(1,4):
if globals()["cond"+str(i)]: #access a global condition variable one by one
true_conditions += 1 #increment at each true value
if true_conditions > 1:
locker.set() # Resume the worker thread executing the else branch
#Or just if true_conditions > 1: locker.set();
#true_conditions would need be incremented when 'True' is written to any of those variables
#
# some condition change code
#
evaluate()
For more information concerning this method, visit: https://docs.python.org/3/library/threading.html#event-objects
Proposed solution:
def wait_until(delegate, timeout: int):
end = time.time() + timeout
while time.time() < end:
if delegate():
return True
else:
time.sleep(0.1)
return False
Usage:
wait_until(lambda: True, 2)
I once used this in my code:
while not condition:
pass
Hope this helps
In 2022 now you could use https://trio-util.readthedocs.io/en/latest/#trio_util.AsyncValue
I think this comes closest to what you want in its "smoothest" form
This worked for me
direction = ''
t = 0
while direction == '' and t <= 1:
sleep(0.1)
t += 0.1
This is for waiting for a signal while making sure time limit of 1 second
here's how:
import time
i = false
while i == false:
if (condition):
i = true
break
Here's my Code I used during one of my Projects :
import time
def no() :
if (Condition !!!) :
it got true
oh()
else:
time.sleep(1) /Don't remove or don't blame me if ur system gets ""DEAD""
no()
def oh() : /Ur main program
while True:
if(bla) :
.......
no()
else :
time.sleep(1)
oh()
oh()
Hope it Helps