Would like to know if you can call an existing class function using Threading or Concurrent Futures without any issues? It works, but I'm curious about the implication in doing this. The reason I want to do this is because I want to keep class state information.
This is a sample with the general idea.
import concurrent.futures
import time
class Test:
def __init__(self) -> None:
super().__init__()
self.counter = 0
def run(self):
print('Test')
self.counter += 1
test1 = Test()
test2 = Test()
with concurrent.futures.ThreadPoolExecutor() as executer:
while True:
t1 = executer.submit(test1.run)
t2 = executer.submit(test2.run)
time.sleep(1.0)
Edit: the threads will use shared information downstream
Related
Let's consider the following example:
from pathos.pools import ProcessPool
class A:
def run(self, arg: int):
shared_variable = 100
def __run_parallel(arg: int):
local_variable = 0
# ...
if local_variable > shared_variable:
shared_variable = local_variable
ProcessPool(4).map(__run_parallel, range(1000))
It's quite obvious to see that there's a data race in if local_variable > shared_variable: with shared_variable = local_variable when using four processes.
Consequently, I'd like to introduce a locking mechanism around the if block, so I tried the following:
from pathos.pools import ProcessPool
from multiprocessing import Lock
class A:
def run(self, arg: int):
lock = Lock()
shared_variable = 100
def __run_parallel(arg: int):
local_variable = 0
# ...
lock.acquire()
if local_variable > shared_variable:
shared_variable = local_variable
lock.release()
ProcessPool(4).map(__run_parallel, range(1000))
However, I get the error RuntimeError: Lock objects should only be shared between processes through inheritance.
In the multiprocessing library, it seems as if the canonical way to achieve the desired mutual exclusion would be to use a Manager object.
However, how to do this idiomatically in pathos?
pathos leverages multiprocess, which has the same interface as multiprocessing, but uses dill. You can access it either of these ways.
>>> import pathos as pa
>>> import multiprocess as mp
>>> mp.Manager is pa.helpers.mp.Manager
True
in my project i have a class of threading.Thread like this:
class MakeHtml(threading.Thread):
def __init__(self, *rstext):
self.outhtml = [x for x in rstext]
self.retval = ''
threading.Thread.__init__(self)
def run(self):
...do something
in another file i call, every 10 seconds MakeHtml class
t = MakeHtml(mrr1, mrr2, mrr3, mrr4)
for create a thread but in this way i see that the thread is the same every time.
I need a new thread every time i call the MakeHtml Threading class, how can i do this?
Thanks in advance
MakeHtml extends Thread, but if you have only 1 instance of MakeHtml, you will have only one thread
For instance if you want 2 different thread you will have to do
t = MakeHtml(mrr1, mrr2, mrr3, mrr4) # one thread
t1 = MakeHtml(mrr1, mrr2, mrr3, mrr4) # another one
You can use :
import threading
def afunction(mm):
# do job
pass
threads = []
for mm in [mmr1, mmr2, mmr3n mmr4]:
t = threading.Thread(target=afunction, args=[mm,])
threads.append(t)
t.start()
[t.join() for t in threads]
I have a counter (training_queue) shared among many instances of a class. The class inherits threading.Thread, so it implements a run() method. When I call start(), I expect each thread to increment this counter, so when it reaches a limit no more threads are started. However, none of the threads modifies the variable. Here's the code:
class Engine(threading.Thread):
training_mutex = threading.Semaphore(MAX_TRAIN)
training_queue = 0
analysis_mutex = threading.Semaphore(MAX_ANALYSIS)
analysis_queue = 0
variable_mutex = threading.Lock()
def __init__(self, config):
threading.Thread.__init__(self)
self.config = config
self.deepnet = None
# prevents engine from doing analysis while training
self.analyze_lock = threading.Lock()
def run(self):
with self.variable_mutex:
self.training_queue += 1
print self.training_queue
with self.training_mutex:
with self.analyze_lock:
self.deepnet = self.loadLSTM3Model()
I protect the training_queue with a Lock, so it should be thread-safe. How ever, if I print its value its always 1. How does threading affect variable scope in this case?
Your understanding of how state is shared between threads is correct. However, you are using instance attribute "training_queue" instead of class attribute "training_queue".
That is, you always set training_queue to 1 for each new object.
For example:
import threading
class Engine(threading.Thread):
training_queue = 0
print_lock = threading.Lock()
def __init__(self, config):
threading.Thread.__init__(self)
def run(self):
with Engine.print_lock:
self.training_queue += 1
print self.training_queue
Engine('a').start()
Engine('b').start()
Engine('c').start()
Engine('d').start()
Engine('e').start()
Will return:
1
1
1
1
1
But:
import threading
class Engine(threading.Thread):
training_queue = 0
print_lock = threading.Lock()
def __init__(self, config):
threading.Thread.__init__(self)
def run(self):
with Engine.print_lock:
Engine.training_queue += 1 # <-here
print self.training_queue
Engine('a').start()
Engine('b').start()
Engine('c').start()
Engine('d').start()
Engine('e').start()
Returns:
1
2
3
4
5
Note self.training_queue vs Engine.training_queue
btw. I think += in python should be atomic so I wouldn't bother with the lock. However, not the usage of lock for printing to stdout in the example above.
I have a python class that communicates with a server. That python class has many functions, i was using few functions. But i want to access multiple servers data at same time using the python class i have.
Am trying something like this, but it'll run one after the other. And i wanted to get the data at same time.
import threading
from server_class import server
class runMonitor(threading.Thread):
def __init__(self,func):
self.func = func
threading.Thread.__init__(self)
def run(self):
self.func()
def monitorSB(ipAddr):
sb = server(ipAddr)
sb.readInfo()
print ('\nReading Registers...\n')
sb.read_rx()
sb.read_tx()
i = 0
while(1):
if i == 0:
print 'Monitoring Registers...'
i = 1
sb.monitor_tx()
sb.monitor_rx()
t = runMonitor(monitorSB('192.168.10.78'))
q = runMonitor(monitorSB('192.168.10.101'))
t.start()
q.start()
print ('\nTest Done...\n')
In the above code, i wanted to access both servers at same time. Help me how to run in parallel
The thing is by saying monitorSB('192.168.10.78') it will execute it before passing it to your thread. Try this:
import threading
from server_class import server
class runMonitor(threading.Thread):
def __init__(self,func, param):
self.func = func
self.param = param
threading.Thread.__init__(self)
def run(self):
self.func(self.param)
def monitorSB(ipAddr):
sb = server(ipAddr)
sb.readInfo()
print ('\nReading Registers...\n')
sb.read_rx()
sb.read_tx()
i = 0
while(1):
if i == 0:
print 'Monitoring Registers...'
i = 1
sb.monitor_tx()
sb.monitor_rx()
t = runMonitor(monitorSB, '192.168.10.78')
q = runMonitor(monitorSB, '192.168.10.101')
t.start()
q.start()
print ('\nTest Done...\n')
How can I implement conditional lock in threaded application, for instance I haw
30 threads that are calling function and for most off the time all threads can access is simultaneous, but depending on function input there can be condition when only one thread can do that one thing. (If value for input is repeated and some thread is still working then I need lock.)
I now that there is module threading with Rlock() but I don't now how to use it in a way that i described it in first part.
Edit: The question is actually about how to prevent any two threads from running the same function with the same argument at the same time. (Thanks to David for helping me formulate my question :) )
Try this: have a lock in the module where your function is, and if the input to the function is such that locking is required, acquire the lock inside the function. Otherwise don't.
l = threading.RLock()
def fn(arg):
if arg == arg_that_needs_lock:
l.acquire()
try:
# do stuff
finally:
l.release()
else:
# do other stuff
EDIT:
As far as I can tell now, the question is actually about how to prevent any two threads from running the same function with the same argument at the same time. There's no problem with two threads running the same function with different arguments at the same time, though. The simple method to do this, if all valid arguments to the function can be dictionary keys, is to create a dictionary of arguments to locks:
import threading
dict_lock = threading.RLock()
locks = {}
def fn_dict(arg):
dict_lock.acquire()
try:
if arg not in dict:
locks[arg] = threading.RLock()
l = locks[arg]
finally:
dict_lock.release()
l.acquire()
try:
# do stuff
finally:
l.release()
If your function can be called with many different arguments, though, that amounts to a lot of locks. Probably a better way is to have a set of all arguments with which the function is currently executing, and have the contents of that set protected by a lock. I think this should work:
set_condition = threading.Condition()
current_args = set()
def fn_set(arg):
set_condition.acquire()
try:
while arg in current_args:
set_condition.wait()
current_args.add(arg)
finally:
set_condition.release()
# do stuff
set_condition.acquire()
try:
current_args.remove(arg)
set_condition.notifyAll()
finally:
set_condition.release()
It sounds like you want something similar to a Readers-Writer lock.
This is probably not what you want, but might be a clue:
from __future__ import with_statement
import threading
def RWLock(readers = 1, writers = 1):
m = _Monitor(readers, writers)
return (_RWLock(m.r_increment, m.r_decrement), _RWLock(m.w_increment, m.w_decrement))
class _RWLock(object):
def __init__(self, inc, dec):
self.inc = inc
self.dec = dec
def acquire(self):
self.inc()
def release(self):
self.dec()
def __enter__(self):
self.inc()
def __exit__(self):
self.dec()
class _Monitor(object):
def __init__(self, max_readers, max_writers):
self.max_readers = max_readers
self.max_writers = max_writers
self.readers = 0
self.writers = 0
self.monitor = threading.Condition()
def r_increment(self):
with self.monitor:
while self.writers > 0 and self.readers < self.max_readers:
self.monitor.wait()
self.readers += 1
self.monitor.notify()
def r_decrement(self):
with self.monitor:
while self.writers > 0:
self.monitor.wait()
assert(self.readers > 0)
self.readers -= 1
self.monitor.notify()
def w_increment(self):
with self.monitor:
while self.readers > 0 and self.writers < self.max_writers:
self.monitor.wait()
self.writers += 1
self.monitor.notify()
def w_decrement(self):
with self.monitor:
assert(self.writers > 0)
self.writers -= 1
self.monitor.notify()
if __name__ == '__main__':
rl, wl = RWLock()
wl.acquire()
wl.release()
rl.acquire()
rl.release()
(Unfortunately not tested)