I am compiling my Python script into a Windows Executable. The script simply downloads a a files and saves them locally - each download uses a different thread. I am finding that my simple application exits before any of the threads finish. But I am not entirely sure?
Does my script below exit before the threads finish or does the script wait till they are done? AND If the script does exit before the threads finish - How can I stop this?
Whats they standard practice to avoid this? Should I use a while loop that checks if any threads are still alive or is there a standard way of doing this?
import thread
import threading
import urllib2
def download_file():
response = urllib2.urlopen("http://website.com/file.f")
print "Res: " + str(response.read())
raw_input("Press any key to exit...")
def main():
# create thread and run
#thread.start_new_thread (run_thread, tuple())
t = threading.Thread(target=download_file)
t.start()
if __name__ == "__main__":
main()
# The below prints before "Res: ..." which makes me think the script exits before the thread has completed
print("script exit")
What you are looking for is the join() function on your newly created thread, which will block the execution of code until the thread is done. I took the liberty of removing your def main() as it is completely not needed here and only creates confusion.
If you want to wrap the launch of all downloads into a neat function, then pick a descriptive name for it.
import thread
import threading
import urllib2
def download_file():
response = urllib2.urlopen("http://website.com/file.f")
print "Res: " + str(response.read())
raw_input("Press any key to exit...")
if __name__ == "__main__":
t = threading.Thread(target=download_file)
t.start()
t.join()
# The below prints before "Res: ..." which makes me think the script exits before the thread has completed
print("script exit")
Related
I have a loop which makes a get request to a webservice to fetch data and do some stuff, but I want to 'manually' terminate the thread/event, which I achieved with the following example:
from threading import Event
exit = Event()
if external_condition():
exit.set()
for _ in range(mins):
fetch_data_and_do_stuff()
exit.wait(10) #wait 10 seconds
With that, the only thing that terminates it's the sleep time between loops. How can I also kill the loop so it doesn't keep running until it gets to the last iteration?
nvm i've solved it like this
from threading import Event
exit = Event()
if external_condition():
exit.set()
for _ in range(mins):
fetch_data_and_do_stuff()
if exit.wait(10):
break
the condition returns true when killed and also sleeps the 10 seconds, so it works
you have 2 options ,
kill the thread or process entirely
or making the loop's boolean false. going that way
you could use a global variable in this way: [Python 3.7] , run it to see
from threading import Thread
from time import sleep
global glob
glob=True
def threaded_function():
while glob:
print("\n [Thread] this thread is running until main function halts this")
sleep(0.8)
if __name__ == "__main__":
thread = Thread(target = threaded_function, args = ())
thread.start()
for i in range(4,0,-1):
print("\n [Main] thread will be terminated in "+str(i)+" seconds")
sleep(1)
glob=False
while True:
print("[Main] program is over")
sleep(1)
I just start a new thread:
self.thread = ThreadedFunc()
self.thread.start()
after something happens I want to exit my program so I'm calling os._exit():
os._exit(1)
The program still works. Everything is functional and it just looks like the os._exit() didn't execute.
Is there a different way to exit a whole program from different thread? How to fix this?
EDIT: Added more complete code sample.
self.thread = DownloadThread()
self.thread.data_downloaded.connect(self.on_data_ready)
self.thread.data_progress.connect(self.on_progress_ready)
self.progress_initialized = False
self.thread.start()
class DownloadThread(QtCore.QThread):
# downloading stuff etc.
sleep(1)
subprocess.call(os.getcwd() + "\\another_process.exe")
sleep(2)
os._exit(1)
EDIT 2: SOLVED! There is a quit(), terminate() or exit() function which just stops the thread. It was that easy. Just look at the docs.
Calling os._exit(1) works for me.
You should use the standard lib threading.
I guess you are using multiprocessing, which is a process-based “threading” interface, which uses similar API to threading, but creates child process instead of child thread. so os._exit(1) only exits child process, not affecting the main process
Also you should ensure you have called join() function in the main thread. Otherwise, it is possible that the operating system schedules to run the main thread to the end before starting to do anything in child thread.
sys.exit() does not work because it is the same as raising a SystemExit exception. Raising an exception in thread only exits that thread, rather than the entire process.
Sample code. Tested under ubuntu by python3 thread.py; echo $?.
Return code is 1 as expected
import os
import sys
import time
import threading
# Python Threading Example for Beginners
# First Method
def greet_them(people):
for person in people:
print("Hello Dear " + person + ". How are you?")
os._exit(1)
time.sleep(0.5)
# Second Method
def assign_id(people):
i = 1
for person in people:
print("Hey! {}, your id is {}.".format(person, i))
i += 1
time.sleep(0.5)
people = ['Richard', 'Dinesh', 'Elrich', 'Gilfoyle', 'Gevin']
t = time.time()
#Created the Threads
t1 = threading.Thread(target=greet_them, args=(people,))
t2 = threading.Thread(target=assign_id, args=(people,))
#Started the threads
t1.start()
t2.start()
#Joined the threads
t1.join() # Cannot remove this join() for this example
t2.join()
# Possible to reach here if join() removed
print("I took " + str(time.time() - t))
Credit: Sample code is copied and modified from https://www.simplifiedpython.net/python-threading-example/
I'm working on a Python launcher which should execute a few programs in my list by calling subprocess. The code is correct, but it works very strangely.
In short, it doesn't work without some sleep or input command in main.
Here is the example:
import threading
import subprocess
import time
def executeFile(file_path):
subprocess.call(file_path, shell=True)
def main():
file = None
try:
file = open('./config.ini', 'r');
except:
# TODO: add alert widget
print("cant find a file")
pathes = [ path.strip() for path in file.readlines() ]
try:
for idx in range(len(pathes)):
print(pathes[idx])
file_path = pathes[idx];
newThread = threading.Thread(target=executeFile, args=(file_path,))
newThread.daemon = True
newThread.start()
except:
print("cant start thread")
if __name__ == '__main__':
main()
# IT WORKS WHEN SLEEP EXISTS
time.sleep(10)
# OR
# input("Press enter to exit ;)")
but without input or sleep it doesn't work:
if __name__ == '__main__':
# Doesn't work
main()
Could someone explain me, please, why it happens?
I have some idea but I'm not sure. Maybe it's because subprocess is asynchronyous and the program executes and closes itself BEFORE the subprocess execution.
In case of sleep and input, the program suspends and subprocess has enough time to execute.
Thanks for any help!
As soon as the last thread is started, your main() returns. That in turn will exit your Python program. That stops all your threads.
From the documentation on daemon threads:
Note: Daemon threads are abruptly stopped at shutdown. Their resources (such as open files, database transactions, etc.) may not be released properly. If you want your threads to stop gracefully, make them non-daemonic and use a suitable signalling mechanism such as an Event.
The simple fix would be to not use daemon threads.
As an aside, I would suggest some changes to your loop. First, iterate over pathes directly instead of using indices. Second; catch errors for each thread seperately, so one error doesn't leave remaining files unprocessed.
for path in pathes:
try:
print(path)
newThread = threading.Thread(target=executeFile, args=(path,))
newThread.start()
except:
print("cant start thread for", path)
Another option would be to skip threads entirely, and just maintain a list of running subprocesses:
import os
import subprocess
import time
def manageprocs(proclist):
"""Check a list of subprocesses for processes that have
ended and remove them from the list.
:param proclist: list of Popen objects
"""
for pr in proclist:
if pr.poll() is not None:
proclist.remove(pr)
# since manageprocs is called from a loop,
# keep CPU usage down.
time.sleep(0.5)
def main():
# Read config file
try:
with open('./config.ini', 'r') as f:
pathes = [path.strip() for path in f.readlines()]
except FileNotFoundError:
print("cant find config file")
exit(1)
# List of subprocesses
procs = []
# Do not launch more processes concurrently than your
# CPU has cores. That will only lead to the processes
# fighting over CPU resources.
maxprocs = os.cpu_count()
# Launch all subprocesses.
for path in pathes:
while len(procs) == maxprocs:
manageprocs(procs)
procs.append(subprocess.Popen(path, shell=True))
# Wait for all subprocesses to finish.
while len(procs) > 0:
manageprocs(procs)
if __name__ == '__main__':
main()
I'm not too familiar with threading, and probably not using it correctly, but I have a script that runs a speedtest a few times and prints the average. I'm trying to use threading to call a function which displays something while the tests are running.
Everything works fine unless I try to put input() at the end of the script to keep the console window open. It causes the thread to run continuously.
I'm looking for some direction in terminating a thread correctly. Also open to any better ways to do this.
import speedtest, time, sys, datetime
from threading import Thread
s = speedtest.Speedtest()
best = s.get_best_server()
def downloadTest(tries):
x=0
downloadList = []
for x in range(tries):
downSpeed = (s.download()/1000000)
downloadList.append(downSpeed)
x+=1
results_dict = s.results.dict()
global download_avg, isp
download_avg = (sum(downloadList)/len(downloadList))
download_avg = round(download_avg,1)
isp = (results_dict['client']['isp'])
print("")
print(isp)
print(download_avg)
def progress():
while True:
print('~ ',end='', flush=True)
time.sleep(1)
def start():
now=(datetime.datetime.today().replace(microsecond=0))
print(now)
d = Thread(target= downloadTest, args=(3,))
d.start()
d1 = Thread(target = progress)
d1.daemon = True
d1.start()
d.join()
start()
input("Complete...") # this causes progress thread to keep running
There is no reason for your thread to exit, which is why it does not terminate. A daemon thread normally terminates when your programm (all other threads) terminate, which does not happen in this as the last input does not quit.
In general it is a good idea to make a thread stop by itself, rather than forcefully killing it, so you would generally kill this kind of thread with a flag. Try changing the segment at the end to:
killflag = False
start()
killflag = True
input("Complete...")
and update the progress method to:
def progress():
while not killflag:
print('~ ',end='', flush=True)
time.sleep(1)
I'm struggling with a issue for some time now.
I'm building a little script which uses a main loop. This is a process that needs some attention from the users. The user responds on the steps and than some magic happens with use of some functions
Beside this I want to spawn another process which monitors the computer system for some specific events like pressing specif keys. If these events occur then it will launch the same functions as when the user gives in the right values.
So I need to make two processes:
-The main loop (which allows user interaction)
-The background "event scanner", which searches for specific events and then reacts on it.
I try this by launching a main loop and a daemon multiprocessing process. The problem is that when I launch the background process it starts, but after that I does not launch the main loop.
I simplified everything a little to make it more clear:
import multiprocessing, sys, time
def main_loop():
while 1:
input = input('What kind of food do you like?')
print(input)
def test():
while 1:
time.sleep(1)
print('this should run in the background')
if __name__ == '__main__':
try:
print('hello!')
mProcess = multiprocessing.Process(target=test())
mProcess.daemon = True
mProcess.start()
#after starting main loop does not start while it prints out the test loop fine.
main_loop()
except:
sys.exit(0)
You should do
mProcess = multiprocessing.Process(target=test)
instead of
mProcess = multiprocessing.Process(target=test())
Your code actually calls test in the parent process, and that call never returns.
You can use the locking synchronization to have a better control over your program's flow. Curiously, the input function raise an EOF error, but I'm sure you can find a workaround.
import multiprocessing, sys, time
def main_loop(l):
time.sleep(4)
l.acquire()
# raise an EOFError, I don't know why .
#_input = input('What kind of food do you like?')
print(" raw input at 4 sec ")
l.release()
return
def test(l):
i=0
while i<8:
time.sleep(1)
l.acquire()
print('this should run in the background : ', i+1, 'sec')
l.release()
i+=1
return
if __name__ == '__main__':
lock = multiprocessing.Lock()
#try:
print('hello!')
mProcess = multiprocessing.Process(target=test, args = (lock, ) ).start()
inputProcess = multiprocessing.Process(target=main_loop, args = (lock,)).start()
#except:
#sys.exit(0)