Develop Reference

Clean up a thread without .join() and without blocking the main thread

I am in a situation where I have two endpoints I can ask for a value, and one may be faster than the other. The calls to the endpoints are blocking. I want to wait for one to complete and take that result without waiting for the other to complete.
My solution was to issue the requests in separate threads and have those threads set a flag to true when they complete. In the main thread, I continuously check the flags (I know it is a busy wait, but that is not my primary concern right now) and when one completes it takes that value and returns it as the result.
The issue I have is that I never clean up the other thread. I can't find any way to do it without using .join(), which would just block and defeat the purpose of this whole thing. So, how can I clean up that other, slower thread that is blocking without joining it from the main thread?

What you want is to make your threads daemons, so when you get the result and finish your main, the other running thread will be forced to finish. You do that by changing the daemon keyword to True:
tr = threading.Thread(daemon=True)
From the threading docs:
The significance of this flag is that the entire Python program exits
when only daemon threads are left.
Although:
Daemon threads are abruptly stopped at shutdown. Their resources (such
as open files, database transactions, etc.) may not be released
properly. If you want your threads to stop gracefully, make them
non-daemonic and use a suitable signalling mechanism such as an Event.
I don't have any particular experience with Events so can't elaborate on that. Feel free to click the link and read on.

One bad and dirty solution is to implement a methode for the threads which close the socket which is blocking. Now you have to catch the exception in the main thread.

Run away multi-threading script that continues to run after canceled python

This is a two part question,
After I cancel my script it still continues run, what I'm doing is queering an exchange api and saving the data for various assets.
My parent script can be seen here you can see i'm testing it out with just 3 assets, a sample of one of the child scripts can be seen here.
After I cancel the script the script for BTC seems to still be running and new .json files are still being generated in it's respective folder. The only way to stop it is to delete the folder and create it again.
This is really a bonus, my code was working with two assets but now with the addition of another it seems to only take in data for BTC and not the other 2.

Your first problem is that you are not really creating worker threads.
t1 = Thread(target=BTC.main()) executes BTC.main() and uses its return code to try to start a thread. Since main loops forever, you don't start any other threads.
Once you fix that, you'll still have a problem.
In python, only the root thread sees signals such as ctrl-c. Other threads will continue executing no matter how hard you press the key. When python exits, it tries to join non-daemon threads and that can cause the program to hang. The main thread is waiting for a thread to terminate, but the thread is happily continuing with its execution.
You seem to be depending on this in your code. Your parent starts a bunch of threads (or will, when you fix the first bug) and then exits. Really, its waiting for the threads to exit. If you solve the problem with daemon threads (below), you'll also need to add code for your thread to wait and not exit.
Back to the thread problem...
One solution is to mark threads as "daemon" (do mythread.daemon = True before starting the thread). Python won't wait for those threads and the threads will be killed when the main thread exits. This is great if you don't care about what state the thread is in while terminating. But it can do bad things like leave partially written files laying around.
Another solution is to figure out some way for the main thread to interrupt the thread. Suppose the threads waits of socket traffic. You could close the socket and the thread would be woken by that event.
Another solution is to only run threads for short-lived tasks that you want to complete. Your ctrl-c gets delayed a bit but you eventually exit. You could even set them up to run off of a queue and send a special "kill" message to them when done. In fact, python thread pools are a good way to go.
Another solution is to have the thread check a Event to see if its time to exit.

python: How to unblock a script that uses recvfrom

I'm using a client to connect a socket via UDP in python. I have two threads. After a KeyboardInterrupt, the first thread still is waiting for a connection via recvfrom.
(...)
udp = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
(...)
udp.shutdown(socket.SHUT_RDWR) # Does not work
udp.close() # Does not work
I have a global variable that all threads share and I update this value after a KeyboardInterrupt. But the thread don't close.
How can I exit from this thread?
Thanks in advance.

Some clarification might be needed on your part, but this is what I understand from your question. If your KeyboardInterrupt kills the first thread and you need paired threads to close when this occurs, then you should consider using a daemon thread. This will guarantee the end of these threads once the parent thread finishes. However, as mentioned in the docs:
Daemon threads are abruptly stopped at shutdown. Their resources (such as open files, database transactions, etc.) may not be released properly. If you want your threads to stop gracefully, make them non-daemonic and use a suitable signalling mechanism such as an Event.
Events are nothing other than signals meant to be utilized between threads. Just like a status flag, you can check whether an event has been 'set' and take proper precautions to handle resources before finishing execution within the thread.

Why non-daemon threads with blocking operations still running after interrupting control thread in Python

I can't figure out the fundamental reason about why non-daemon threads (daemon thread description) still running after control thread interrupted. Maybe it's not python specific...
I'm expecting such behavior: when I'm interrupting control thread, then every other threads, generated my program, should shutdown too.
But if a thread performs blocking operation (e.g. I/O action), then this thread still running after control thread interrupted.
Python docs says that:
If you want your threads to stop gracefully, make them non-daemonic and use a suitable signalling mechanism such as an Event.
It's ok - we can communicate with threads through events, signals, etc., using select() for non-block such operations and make reacting to events conditions in loops.
But I'd like to know why things go in such way.
UPDATE:
I thought it's common problem and code snippet is not really necessary here, but here it is:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import threading
import time
def func_with_blocking_operation():
time.sleep(5)
threading.Thread(target=func_with_blocking_operation).start()
Here after interrupting control thread by hitting Ctrl+C in control terminal (SIGINT signal was sent to control thread), program still running while sleep function not exited. Maybe it is because non-daemon threads by default join()ed in Python?
Thanks

Stopping a thread in python

I am creating a thread in my Python app with thread.start_new_thread.
How do I stop it if it hasn't finished in three seconds time?

You can't do that directly. Anyway aborting a thread is not good practice - rather think about using synchronization mechanisms that let you abort the thread in a "soft" way.
But daemonic threads will automatically be aborted if no non-daemonic threads remain (e.g. if the only main thread ends). Maybe that's what you want.

If you really need to do this (e.g. the thread calls code that may hang forever) then consider rewriting your code to spawn a process with the multiprocessing module. You can then kill the process with the Process.terminate() method. You will need 2.6 or later for this, of course.

You cannot. Threads can't be killed from outside. The only thing you can do is add a way to ask the thread to exit. Obviously you won't be able to do this if the thread is blocked in some systemcall.

As noted in a related question, you might be able to raise an exception through ctypes.pythonapi, but not while it's waiting on a system call.