Develop Reference

How do you share data between functions in threads with python

I Have a counting function that I would like to start and restart while getting the live variables to use in another function my problem is while using threading it seams like even global variables don't seem to work to pass variables around. What I want the code to do is have a counter be triggered as needed or maybe free running I'm not sure yet. To be able to reset the counter and get the value of the counter.
Right now the counter will start and run fine but the print_stuff function keeps telling me that there is no attribute countval.
The count thread gets started at startup but I don't necessarily want it to start up immediately, I would like to trigger it as needed but I cant put count_thread.start() twice or it will through a error so I'm calling the thread at startup and then calling the function again to restart it as needed. Maybe there is a more elegant way of doing that.?
import threading
import time
def count():
global countval
for countval in range(3):
print('looping')
time.sleep(1)
def print_stuff():
global countval
e = input("press enter to start")
count()
while True:
if countval == 3:
print("time out")
count_thread = threading.Thread(target=count)
print_thread = threading.Thread(target=print_stuff)
print_thread.start()
count_thread.start()

print_stuff is getting to the if statement before the count function is able to create the variable. Just do them in the opposite order. Either that, or create a global countval = 0 to start things off.

To solve the no attribute problem you can use Queue,
and if you want to stop your counting thread you can set a global variable or you can pass a function (using lambda or inner function or ...) to do that.
Here is one way to do that:
import threading
import time
from queue import Queue
from typing import Callable
def count(q, stop_counting):
# type: (Queue, Callable[[], bool]) -> None
for countval in range(3):
if stop_counting():
print('stopped')
break
print(f'looping {countval}')
q.put(countval)
time.sleep(1)
def print_stuff(q):
# type: (Queue) -> None
while True:
countval = q.get()
print(f'countval gotten: {countval}')
if countval == 3:
print("time out")
def main():
flag_stop_counting = False
q = Queue()
def stop_counting():
return flag_stop_counting
count_thread = threading.Thread(target=count, args=(q, stop_counting,))
print_thread = threading.Thread(target=print_stuff, args=(q,))
print_thread.start()
count_thread.start()
time.sleep(1.25)
flag_stop_counting = True
if __name__ == '__main__':
main()
In this code:
counter checks if it should stop counting or not
counter puts the value that it made to q
print_stuff get the value from q (Note: he waits until counter puts his value in q)
To check that program works:
after 1.25 seconds we change the value of flag_stop_counting
But if you want your counter to only have a for, i guess it's better to don't make it as a thread and run it whenever you want.
Hope it was helpful.

python threading.lock() not working

I created a function to modify a global variable, which adds 1 each time. Then I create 10 threads to call this function; each thread will call the function 10 times. However, the variable's final value is not the same as if I call function 100 times in one thread. Is anything wrong with how I use threading.lock? Below is my code:
num=0
lockOne=threading.Lock()
def subPro():
global num
lockOne.acquire()
num+=1
lockOne.release()

You are probably not waiting for threads to complete. You have to use Thread.join:
import threading
num=0
lockOne=threading.Lock()
def subPro():
global num
lockOne.acquire()
num+=1
lockOne.release()
def run():
for i in range(10):
subPro()
# Start all threads
threads = [threading.Thread(target=run) for x in range(10)]
for thread in threads:
thread.start()
# Wait for completion
for thread in threads:
thread.join()
print(num)

How to use a thread pool to do infinite loop function?

I want to do a infinite loop function.
Here is my code
def do_request():
# my code here
print(result)
while True:
do_request()
When use while True to do this, it's a little slow, so I want to use a thread pool to concurrently execute the function do_request(). How to do this ?
Just like use ab (Apache Bench) to test HTTP server.

Finally, I've solved this problem. I use a variable to limit the thread number.
Here is my final code, solved my problem.
import threading
import time
thread_num = 0
lock = threading.Lock()
def do_request():
global thread_num
# -------------
# my code here
# -------------
with lock:
thread_num -= 1
while True:
if thread_num <= 50:
with lock:
thread_num += 1
t = threading.Thread(target=do_request)
t.start()
else:
time.sleep(0.01)
Thanks for all replies.

You can use threading in Python to implement this.
Can be something similar to this (when using two extra threads only):
import threading
# define threads
task1 = threading.Thread(target = do_request)
task2 = threading.Thread(target = do_request)
# start both threads
task1.start()
task2.start()
# wait for threads to complete
task1.join()
task2.join()
Basically, you start as many threads as you need (make sure you don't get too many, so your system can handle it), then you .join() them to wait for tasks to complete.
Or you can get fancier with multiprocessing Python module.

Try the following code:
import multiprocessing as mp
import time
def do_request():
while(True):
print('I\'m making requests')
time.sleep(0.5)
p = mp.Process(target=do_request)
p.start()
for ii in range(10):
print 'I\'m also doing other things though'
time.sleep(0.7)
print 'Now it is time to kill the service thread'
p.terminate()
The main thread stars a service thread that does the request and goes on until it has to, and then it finishes up the service thread.

Maybe you can use the concurrent.futures.ThreadPoolExecutor
from concurrent.futures import ThreadPoolExecutor
import time
def wait_on_b(hello):
time.sleep(1)
print(hello) # b will never complete because it is waiting on a.
return 5
def wait_on_a():
time.sleep(1)
print(a.result()) # a will never complete because it is waiting on b.
return 6
executor = ThreadPoolExecutor(max_workers=2)
a = executor.submit(wait_on_b, 3)
b = executor.submit(wait_on_a)

How about this?
from threading import Thread, Event
class WorkerThread(Thread):
def __init__(self, logger, func):
Thread.__init__(self)
self.stop_event = Event()
self.logger = logger
self.func = func
def run(self):
self.logger("Going to start the infinite loop...")
#Your code
self.func()
concur_task = WorkerThread(logger, func = do_request)
concur_task.start()
To end this thread...
concur_task.stop_event.set()
concur_task.join(10) #or any value you like

How properly close thread in Python

I'm having trouble understanding threads in Python. I have this program:
import _thread, time
def print_loop():
num = 0
while 1:
num = num + 1
print(num)
time.sleep(1)
_thread.start_new_thread(print_loop, ())
time.sleep(10)
And my question is if I need to close the thread print_loop, because it looks to me that both threads end when the main thread ends. Is this proper way to handle threads?

First, avoid using the low-level API unless you absolutely have to. The threading module is preferred over _thread. In general in Python, avoid anything starting with an underscore.
Now, the method you are looking for is called join. I.e.
import time
from threading import Thread
stop = False
def print_loop():
num = 0
while not stop:
num = num + 1
print(num)
time.sleep(1)
thread = Thread(target=print_loop)
thread.start()
time.sleep(10)
stop = True
thread.join()

Getting multi-threading to work with a list using python pexpect

I wrote a simple python pexpect script to ssh into a machine and perform a action. Now I need to do this action to multiple servers. I am using a list to hit all of the servers concurrently using multi-threading. My issue is due to everything being ran concurrently, each thread is running on the same server name. Is there a way to concurrently have each thread only run one of the listed servers?
#! /usr/bin/python
#Test script
import pexpect
import pxssh
import threading
import datetime
currentdate = datetime.datetime.now()
easterndate = (datetime.datetime.now() + datetime.timedelta(0, 3600))
#list of servers
serverlist = ["025", "089"]
#server number
sn = 0
ssh_new_conn = 'Are you sure you want to continue connecting'
class ThreadClass(threading.Thread):
def run(self):
index = 0
sn = serverlist[index]
print sn
username = '[a username]'
password = '[a password]'
hostname = '%(sn)s.[the rest of the host url]' % locals()
command = "/usr/bin/ssh %(username)s#%(hostname)s " % locals()
index = index + 1
now = datetime.datetime.now()
print command
p = pexpect.spawn(command, timeout=360)
***do some other stuff****
for i in range(len(severlist)):
t = ThreadClass()
t.start()
[update]
I may just trying doing this with a parent thread that calls the child thread and so forth....although it would be nice if multi-threading could work from a list or some sort of work queue.

The problem has nothing to do with "everything being ran concurrently". You're explicitly setting index = 0 at the start of the run function, so of course every thread works on index 0.
If you want each thread to deal with one server, just pass the index to each thread object:
class ThreadClass(threading.Thread):
def __init__(self, index):
super(ThreadClass, self).__init__()
self.index = index
def run(self):
sn = serverlist[self.index]
print sn
# same code as before, minus the index = index + 1 bit
for i in range(len(severlist)):
t = ThreadClass(i)
t.start()
(Of course you'll probably want to use serverlist instead of severlist and fix the other errors that make it impossible for your code to work.)
Or, more simply, pass the sn itself:
class ThreadClass(threading.Thread):
def __init__(self, sn):
super(ThreadClass, self).__init__()
self.sn = sn
def run(self):
print self.sn
# same code as last version, but use self.sn instead of sn
for sn in severlist:
t = ThreadClass(sn)
t.start()
Alternatively, if you really want to use a global variable, just make it global, and put a lock around it:
index = 0
index_lock = threading.Lock()
class ThreadClass(threading.Thread):
def run(self):
global index, index_lock
with index_lock:
sn = serverlist[index]
index += 1
print sn
# same code as first version
However, you might want to consider a much simpler design, with a pool or executor instead of an explicit worker thread and list of things to work on. For example:
def job(sn):
print sn
# same code as first version again
with concurrent.futures.ThreadPoolExecutor() as executor:
executor.map(job, serverlist)
This will only run, say, 4 or 8 or some other good "magic number" of jobs concurrently. Which is usually what you want. But if you want exactly one thread per server, just pass max_workers=len(serverlist) to the ThreadPoolExecutor constructor.
Besides being a whole lot less code to read, write, get wrong, debug, etc., it also has more functionality—e.g., you can get results and/or exceptions from the servers back to the main thread.