Develop Reference

How to synchronize threads with shared global variable in python?

i'm currently trying to unterstand threading in python and i wrote a program that ideally would have 2 threads alternating between incrementing and decrementing a global variable but no matter how i spread out the lock it inevitably becomes out of sync.
number = 0
lock = threading.Lock()
def func1():
global number
global lock
while True:
try:
lock.acquire()
number += 1
finally:
lock.release()
print(f"number 1 is: {number}")
time.sleep(0.1)
def func2():
global number
global lock
while True:
try:
lock.acquire()
number -= 1
finally:
lock.release()
print(f"number 2 is: {number}")
time.sleep(0.1)
t1 = threading.Thread(target=func1)
t1.start()
t2 = threading.Thread(target=func2)
t2.start()
t1.join()
t2.join()
the output should look something like this:
number 1 is: 1
number 2 is: 0
number 1 is: 1
number 2 is: 0
number 1 is: 1
number 2 is: 0
number 1 is: 1
number 2 is: 0
but right now it looks like this:
number 1 is: 1
number 2 is: 0
number 1 is: 1
number 2 is: 0
number 2 is: -1number 1 is: 0
number 2 is: -1number 1 is: 0
number 1 is: 1number 2 is: 0
any idea how to do this without falling out of sync?

First, avoid using global variables with threads in python. Use a queue to share the variables instead.
Second, the lock acquisition in non-deterministic. At the moment a lock is released, you have no guarantee that the other thread will grab it. There is always a certain probability that the thread that just released the lock can grab it again before the other thread.
But in your case, you can avoid problems because you know the state that the variable needs to be to accept modifications by one thread or the other. So, you can enforce the protection for modification by verifying if the variable is in the right state to accept a modification.
Something like:
from threading import Thread
import time
from queue import Queue
def func1(threadname, q):
while True:
number = q.get()
if number == 0:
number += 1
print(f"number 1 is: {number}")
q.put(number)
time.sleep(0.1)
def func2(threadname, q):
while True:
number = q.get()
if number == 1:
number -= 1
print(f"number 2 is: {number}")
q.put(number)
time.sleep(0.1)
queue = Queue()
queue.put(0)
t1 = Thread(target=func1, args=("Thread-1", queue))
t2 = Thread(target=func2, args=("Thread-2", queue))
t1.start()
t2.start()
t1.join()
t2.join()

thanks for all your answers, i remember seing someone in the comments mentioned using events or something like that and that solved the issue. here's the code:
number = 0
event_number = threading.Event()
event_number.clear()
def func1():
global number
global event_number
while True:
if not event_number.is_set():
number += 1
print(f"func 1 is {number}")
event_number.set()
else:
pass
time.sleep(2)
def func2():
global number
global event_number
while True:
if event_number.is_set():
number -= 1
print(f"func 2 is {number}")
event_number.clear()
else:
pass
time.sleep(2)
t1 = threading.Thread(target=func1)
t2 = threading.Thread(target=func2)
t1.start()
t2.start()
t1.join()
t2.join()
now i notice that sometimes one of the loops will either not wait it's alloted time and print right away or wait double the time but at least the number only stays within those 2 values.

For starters, time.sleep is not exactly accurate. And depending on the python-implementation you're using (most likely cpython) multithreading might not quite work the way you're expecting it to. These two factors allow the initially correct timing of your threads to get out of sync within fairly short time.
There solution for this problem is to enforce alternate operation on the variable by the two threads via two locks:
import time
import threading
var = 0
def runner(op, waitfor, release):
global var
while True:
try:
# wait for resource to free up
waitfor.acquire()
# operation
var = op(var)
print(f"var={var}")
finally:
# notify other thread
release.release()
time.sleep(0.1)
# init locks for thread-synchronization
lock_a = threading.Lock()
lock_b = threading.Lock()
lock_a.acquire()
lock_b.acquire()
# create and start threads (they'll wait for their lock to be freed)
thread_a = threading.Thread(target=runner, args=(lambda v: v - 1, lock_a, lock_b))
thread_b = threading.Thread(target=runner, args=(lambda v: v + 1, lock_b, lock_a))
thread_a.start()
thread_b.start()
# let thread_b start the first operation by releasing the lock
lock_b.release()
In the above code, each thread has a lock that can be used to notify it, that the resource may be used by it. Thus threads can hand control over the global variable to each other.

Python: Sharing a time Lock in between spawned Processes so that there is a delay between them

I am trying to print the id's in this list with them having a delay in between the start and end of the process, and having a delay between the queue.get(which I implement using a threading.Timer with a shared lock). The problem I am having is that while my current setup of having a Timer allows me to lock the processes so that there is a 2 second period after one process acquires the record from the queue that all other processes cannot start, my program only closes 2 of the 4 processes at the end of the program run. How can I fix this so that all the processes close and the program can exit.
My output below shows this as I want there to be 2 more "worker closed" notifications:
Process started
Process started
Process started
Process started
begin 1 : 1560891818.0307562
begin 2 : 1560891820.0343137
begin 3 : 1560891822.0381632
end 2 : 3.0021514892578125
end 1 : 6.004615068435669
begin 4 : 1560891824.0439706
begin 5 : 1560891826.0481522
end 4 : 3.004107713699341
end 3 : 6.005637168884277
begin 6 : 1560891828.0511773
begin 7 : 1560891830.0557532
end 6 : 3.0032966136932373
end 5 : 6.006829261779785
begin 8 : 1560891832.056265
begin 9 : 1560891834.0593572
end 8 : 3.011284112930298
end 7 : 6.005618333816528
begin 10 : 1560891836.0627353
end 10 : 3.0014095306396484
worker closed
end 9 : 6.000675916671753
worker closed
import multiprocessing
from time import sleep, time
import threading
class TEMP:
lock = multiprocessing.Lock()
id_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
queue = multiprocessing.Queue(10)
DELAY = 2
def mp_worker(self, queue, lock):
while queue.qsize() > 0:
lock.acquire()
# Release the lock after a delay
threading.Timer(self.DELAY,lock.release).start()
record = queue.get()
start_time = time()
print("begin {0} : {1}".format(record, start_time))
if (record % 2 == 0):
sleep(3)
else:
sleep(6)
print("end {0} : {1}".format(record, time() - start_time))
threading.Timer.join()
print("worker closed")
def mp_handler(self):
# Spawn two processes, assigning the method to be executed
# and the input arguments (the queue)
processes = [multiprocessing.Process(target=self.mp_worker, args=([self.queue, self.lock])) \
for _ in range(4)]
for process in processes:
process.start()
print('Process started')
for process in processes:
process.join()
def start_mp(self):
for id in self.id_list:
self.queue.put(id)
self.mp_handler()
if __name__ == '__main__':
temp = TEMP()
temp.start_mp()

I actually fixed this problem. The main reason my code was not joining was because my code was checking if the queue was empty, waiting for a delay, then attempting to get something from the queue. This meant that towards the end of the program that while the queue had become empty and 2 of the 4 processes successfully finished at the same time, the remaining 2 processes were in a delay. When this delay ended they attempted to get something from the queue but since the queue was empty they just blocked the remainder of the process's code from running which meant that they could never join back up.
I fixed this by also checking if the queue is empty right before the process attempts to get something from the queue. My fixed workerfunction is below:
def mp_worker(self, queue, lock):
while not queue.empty():
print(mp.current_process().name)
lock.acquire()
# Release the lock after a delay
timer = Timer(self.DELAY, lock.release)
timer.start()
if not queue.empty():
record = queue.get(False)
start_time = time()
print("begin {0} : {1}".format(record, start_time))
if (record % 2 == 0):
sleep(3)
else:
sleep(6)
print("end {0} : {1}".format(record, time() - start_time))
print("{0} closed".format(mp.current_process().name))

python multithreading - confused with output

I am trying to create a multithreading program that will run one thread for three seconds, then 'pause' the thread using event objects, wait for a few seconds, then repeat the process over again. here is my code:
import threading
import time
counter = 1
def control():
global counter
test_event.wait()
while(test_event.is_set()):
print 'waiting 3 seconds'
time.sleep(3)
event.set()
time.sleep(3)
event.clear()
if(counter == 1):
counter = counter - 1
def function():
global counter
event.wait()
while (event.is_set()):
test_event.clear()
print 'event is set to high'
time.sleep(1)
print 'event is set to low'
test_event.set()
event = threading.Event()
test_event = threading.Event()
t1 = threading.Thread(target = control)
t2 = threading.Thread(target = function)
t1.start()
t2.start()
while(counter == 1):
test_event.set()
the output I am desiring should look like this:
waiting 3 seconds
event is set to high
event is set to high
event is set to high
event is set to low
waiting 3 seconds
event is set to high
....
....
and that keeps repeating.
the output I am currently receiving however is this:
waiting three seconds
event is set to high
event is set to high
event is set to high
waiting 3 seconds
event is set to low
waiting 3 seconds
waiting 3 seconds
waiting 3 seconds
...
...
and that keeps repeating.
Can anybody detect where I am messing up, and possibly offer any advice for me to fix? I cannot seem to figure out what I am doing wrong.
P.S. I am aware that this code probably is constructed very poorly, but I am completely new to multithreading...sorry in advance!

Causing a thread to stop while stuck within a while loop?

Is is possible to stop a thread prematurely when it is stuck inside a while loop? Below is my sample code, which runs correctly, since each time it calls loop_thread it will check to see if the threading.Event() flag is set. When attempting to run the code for a file that processes information much longer than each second, there is no way to stop the entire function from continuing its execution until the next iteration. For example, if I run dld_img_thread, it takes about 5 minutes to complete its execution and recheck the while loop to see if should proceed. What I want to have happen is kill the dld_img_thread at a time shorter than 5 minutes (e.g. 1 minute). I don't care if the data is lost, just that the thread stops before the function finishes execution. Thank you
import threading, time, pythoncom, read_mt0
import powerfail_debugport_reader as pf_dbg_rdr
import powerfail_firmware_downloader as pf_fwdld
def loop_thread(thread_name, thread_event):
loopCnt = 0
print "\nstarting {}".format(thread_name)
print "is {0} alive? {1}\n".format(thread_name, L00P_thread.is_alive())
while not thread_event.is_set():
print("value of loopCnt = {}".format(loopCnt))
loopCnt += 1
time.sleep(1)
print('stopping {}\n'.format(thread_name))
def image_dld(thread_name, thread_event):
pythoncom.CoInitializeEx(pythoncom.COINIT_MULTITHREADED)
print "\nstarting {}".format(thread_name)
print "is {0} alive? {1}\n".format(thread_name, dld_img_thread.is_alive())
while not thread_event.is_set():
pf_fwdld.power_fail_test()
print('stopping {}'.format(thread_name))
def debug_port_thread(thread_name, thread_event):
pythoncom.CoInitializeEx(pythoncom.COINIT_MULTITHREADED)
print "\nstarting {}".format(thread_name)
print "is {0} alive? {1}\n".format(thread_name, debug_thread.is_alive())
pf_dbg_rdr.debug_port_reader()
print('\nstopping {}'.format(thread_name))
def main():
global L00P_thread, debug_thread
pf_dbg_rdr.samurai_event = threading.Event()
L00P_thread = threading.Thread(target=loop_thread, \
args=('L00P_thread', pf_dbg_rdr.samurai_event))
dld_img_thread = threading.Thread(target=image_dld, \
args=('image_download', pf_dbg_rdr.samurai_event))
debug_thread = threading.Thread(target=debug_port_thread, \
args=('debug_port_reader', pf_dbg_rdr.samurai_event))
L00P_thread.start()
dld_img_thread.start()
debug_thread.start()
debug_thread.join()
if __name__ == '__main__':
main()
print('processes stopped')
print "Exiting Main Thread"

Use a second variable in your while condition that you can change once your timeout is reached.
For example:
shouldRun = True
while not thread_event.is_set() and shouldRun:
print("value of loopCnt = {}".format(loopCnt))
loopCnt += 1
time.sleep(1)
if loopCnt > 60: shouldRun = False
would stop after 60 iterations (about 60 seconds given you sleep for 1 second on each iteration).

Print value from inside thread during it's execution

I've just been trying to get threading working properly and I've hit a problem. The default thread module doesn't seem to be able to return values, so I looked up a solution and found this answer - how to get the return value from a thread in python?
I've got this working for getting multiple threads running, but I can't seem to print any values from inside the thread until they've all finished. Here is the code I currently have:
import random
from multiprocessing.pool import ThreadPool
#only 2 threads for now to make sure they don't all finish at once
pool = ThreadPool(processes=2)
#should take a few seconds to process
def printNumber(number):
num = random.randint( 50000, 500000 )
for i in range( num ):
if i % 10000 == 0:
print "Thread " + str( number ) + " progress: " + str( i )
test = random.uniform( 0, 10 ) ** random.uniform( 0, 1 )
return number
thread_list = []
#Execute threads
for i in range(1,10):
m = pool.apply_async(printNumber, (i,))
thread_list.append(m)
#Wait for values and get output
totalNum = 0
for i in range( len( thread_list ) ):
totalNum += thread_list[i].get()
print "Thread finished"
# Demonstrates that the main process waited for threads to complete
print "Done"
What happens, is you get 9x "Thread finished", then "Done", then everything that was printed by the threads.
However, remove the #wait for values part, and it prints them correctly. Is there any way I can keep it waiting for completion, but print things from inside the function?
Edit: Here is the output (a bit long to add to the post), it weirdly reverses the print order - http://pastebin.com/9ZRhg52Q