I have a thread code in python like this. But I am not sure whether I am doing in correct way or not.
Class MyThread(threading.thread):
def __init__(self, thread_id, thread_name):
self.thread_name = thread_name
self.thread_id = thread_id
def run(self):
do_something()
def do_something():
while True:
do_something_else()
time.sleep(5)
Class SomeClass:
def __init__():
pass
def run():
thread1 = MyThread(1, "thread1")
thread2 = MyThread(2, "thread2")
thread3 = MyThread(3, "thread3")
def main():
agent = Someclass()
agent.run()
Whether this is the safe way to deal with multiple thread? How does it impact other applications? Is there a chance, that execution of one thread can hinder the execution of others? What happens , if the threads got blocked in any cycle?
Also how to make sure that, thread doesn't gets blocked for forever b'coz of any reason. If it gets blocked , then after fixed timeinterval it should come out gracefully and continue in next loop.
That is why Python and some other languages introduce the lock
This page will help you, you need to read something about Lock, RLock and Condition
Your code's thread safety is really dependent on what's in do_something() and do_something_else(). It's thread safe if you're only modifying local variables. But the moment you start reading/modifying shared variables/storage, like a file or a global variable, then you need to use something like locks or semaphores to ensure thread safety.
You can read about Python's threading module here.
This Wikipedia articles on synchronization and locks may be helpful to you too.
If you need examples for writing multi-threading code, here's a good example using different synchronization mechanisms.
Related
I'm starting to write a program that uses threads but after searching how to start threads in Python I have found two methods that accomplish the same thing. There must be a difference or advantage one over the other. Confused which road I should go down.
My thread is going to be ran in the background continuously and never stop until the program is told to by the user. Also one or more arguments will be passed to the thread when started.
one way using classes:
from threading import Thread
class myClassA(Thread):
def __init__(self):
Thread.__init__(self)
self.daemon = True
self.start()
def run(self):
while True:
print 'A'
myClassA()
while True:
pass
Second way using methods:
from threading import Thread
def runA():
while True:
print 'A\n'
if __name__ == "__main__":
t1 = Thread(target = runA)
t1.setDaemon(True)
t1.start()
while True:
pass
My rule of thumb for using classes is that you shouldn't use them until you find a good use case for them. One use case would be if you wanted to define multiple methods to interact with the thread. But usually developers can't see the future when designing classes so it's better to just code using functions, and when you see a use case for classes refactor your code. What I mean is, you might spend a lot of time designing a class and not even end up using or needing a lot of the functionality you implemented; so you wasted your time and made your code complex for no reason.
I'm trying to write a mini-game that allows me to practice my python threading skill. The game itself involves with timed bombs and citys that have them.
Here is my code:
class City(threading.Thread):
def __init__(self, name):
super().__init__()
self.name = name
self.bombs = None
self.activeBomb = None
self.bombID = 0
self.exploded = False
def addBomb(self, name, time, puzzle, answer, hidden=False):
self.bombs.append(Bomb(name, self.bombID, time, puzzle, answer, hidden))
self.activeBomb.append(self.bombID)
self.bombID += 1
def run(self):
for b in self.bombs:
b.start()
while True:
# listen to the bombs in the self.bombs # The part that I dont know how
# if one explodes
# print(self.name + ' has been destroyed')
# break
# if one is disarmed
# remove the bombID from the activeBomb
# if all bombs are disarmed (no activeBomb left)
# print('The city of ' + self.name + ' has been cleansed')
# break
class Bomb(threading.Thread):
def __init__(self, name, bombID, time, puzzle, answer, hidden=False):
super(Bomb, self).__init__()
self.name = name
self.bombID = bombID
self._timer = time
self._MAXTIME = time
self._disarmed = False
self._puzzle = puzzle
self._answer = answer
self._denoted = False
self._hidden = hidden
def run(self):
# A bomb goes off!!
if not self._hidden:
print('You have ' + str(self._MAXTIME)
+ ' seconds to solve the puzzle!')
print(self._puzzle)
while True:
if self._denoted:
print('BOOM')
// Communicate to city that bomb is denoted
break
elif not self._disarmed:
if self._timer == 0:
self._denoted = True
else:
self._timer -= 1
sleep(1)
else:
print('You have successfully disarmed bomb ' + str(self.name))
// Communicate to city that this bomb is disarmed
break
def answerPuzzle(self, ans):
print('Is answer ' + str(ans) + ' ?')
if ans == self._answer:
self._disarmed = True
else:
self._denotaed = True
def __eq__(self, bomb):
return self.bombID == bomb.bombID
def __hash__(self):
return id(self)
I currently don't know what is a good way for the City class to effectively keep track of the
bomb status.
The first thought I had was to use a for loop to have the City to check all the bombs in the
City, but I found it being too stupid and inefficient
So here is the question:
What is the most efficient way of implementing the bomb and City so that the city immediately know the state change of a bomb without having to check it every second?
PS: I do NOT mean to use this program to set off real bomb, so relax :D
A good case to use queue. Here is an example of the so-called producer - consumer pattern.
The work threads will run forever till your main program is done (that is what the daemon part and the "while True" is for). They will diligently monitor the in_queue for work packages. They will process the package until none is left. So when the in_queue is joined, your work threads' jobs are done. The out_queue here is an optional downstream processing step. So you can assemble the pieces from the work threads to a summary form. Useful when they are in a function.
If you need some outputs, like each work thread will print the results out to the screen or write to one single file, don't forget to use semaphore! Otherwise, your output will stumble onto each other.
Good luck!
from threading import Thread
import Queue
in_queue = Queue.Queue()
out_queue = Queue.Queue()
def work():
while True:
try:
sonId = in_queue.get()
###do your things here
result = sonID + 1
###you can even put your thread results again in another queue here
out_queue.put(result) ###optional
except:
pass
finally:
in_queue.task_done()
for i in range(20):
t = Thread(target=work)
t.daemon = True
t.start()
for son in range(10):
in_queue.put(son)
in_queue.join()
while not out_queue.empty():
result = out_queue.get()
###do something with your result here
out_queue.task_done()
out_queue.join()
The standard way of doing something like this is to use a queue - one thread watches the queue and waits for an object to handle (allowing it to idle happily), and the other thread pushes items onto the queue.
Python has the queue module (Queue in 2.x). Construct a queue in your listener thread and get() on it - this will block until something gets put on.
In your other thread, when a relevant event occurs, push it onto the queue and the listener thread will wake up and handle it. If you do this in a loop, you have the behaviour you want.
The easiest way would be to use a scheduler library. E.g. https://docs.python.org/2/library/sched.html. Using this you can simply schedule bombs to call a function or method at the time they go off. This is what I would recommend if you did not wanted to learn about threads.
E.g.
import sched
s = sched.scheduler(time.time, time.sleep)
class Bomb():
def explode(self):
if not self._disarmed:
print "BOOM"
def __init__(self, time):
s.enter(self._MAXTIME, 1, self.explode)
However, that way you will not learn about threads.
If you really want to use threads directly, then you can simply let the bombs call sleep until it is their time to go off. E.g.
class Bomb(threading.Thread)
def run(self):
time.sleep.(self._MAXTIME)
if not self._disarmed:
print "BOOM"
However, this is not a nice way to handle threads, since the threads will block your application. You will not be able to exit the application until you stop the threads. You can avoid this by making the thread a daemon thread. bomb.daemon = True.
In some cases, the best way to handle this is to actually "wake up" each second and check the status of the world. This may be the case when you need to perform some cleanup actions when the thread is stopped. E.g. You may need to close a file. Checking each second may seem wasteful, but it is actually the proper way to handle such problems. Modern desktop computers are mostly idle. To be interrupted for a few milliseconds each second will not cause them much sweat.
class Bomb(threading.Thread)
def run(self):
while not self._disarmed:
if time.now() > self.time_to_explode:
print "BOOM"
break
else:
time.sleep.(1)
Before you start "practising threading with python", I think it is important to understand Python threading model - it is Java threading model, but comes with a more restrictive option:
https://docs.python.org/2/library/threading.html
The design of this module is loosely based on Java’s threading model.
However, where Java makes locks and condition variables basic behavior
of every object, they are separate objects in Python. Python’s Thread
class supports a subset of the behavior of Java’s Thread class;
currently, there are no priorities, no thread groups, and threads
cannot be destroyed, stopped, suspended, resumed, or interrupted. The
static methods of Java’s Thread class, when implemented, are mapped to
module-level functions.
Locks being in separate objects, and not per-object, following the diagram below, means less independent scheduling even when different objects are accessed - because possibly even same locks are necessary.
For some python implementation - threading is not really fully concurrent:
http://uwpce-pythoncert.github.io/EMC-Python300-Spring2015/html_slides/07-threading-and-multiprocessing.html#slide-5
A thread is the entity within a process that can be scheduled for
execution
Threads are lightweight processes, run in the address space of an OS
process.
These threads share the memory and the state of the process. This
allows multiple threads access to data in the same scope.
Python threads are true OS level threads
Threads can not gain the performance advantage of multiple processors
due to the Global Interpreter Lock (GIL)
http://uwpce-pythoncert.github.io/EMC-Python300-Spring2015/html_slides/07-threading-and-multiprocessing.html#slide-6
And this (from above slide):
Let us say we have a python function magical_attack(energy) which may or may not last more than a second. It could even be an infinite loop? How would I run, but if it goes over a second, terminate it, and tell the rest of the program. I am looking for a sleek module to do this. Example:
import timeout
try: timout.run(magical_attack(5), 1)
except timeout.timeouterror:
blow_up_in_face(wizard)
Note: It is impossible to modify the function. It comes from the outside during runtime.
The simplest way to do this is to run the background code in a thread:
t = threading.Thread(target=magical_attack, args=(5,))
t.start()
t.join(1)
if not t.isAlive():
blow_up_in_face(wizard)
However, note that this will not cancel the magical_attack function; it could still keep spinning along in the background for as long as it wants even though you no longer care about the results.
Canceling threads safely is inherently hard to do, and different on each platform, so Python doesn't attempt to provide a way to do it. If you need that, there are three alternatives:
If you can edit the code of magical_attack to check a flag every so often, you can cancel it cooperatively by just setting that flag.
You can use a child process instead of a thread, which you can then kill safely.
You can use ctypes, pywin32, PyObjC, etc. to access platform-specific routines to kill the thread. But you have to really know what you're doing to make sure you do it safely, and don't confuse Python in doing it.
As Chris Pak pointed out, the futures module in Python 3.2+ makes this even easier. For example, you can throw off thousands of jobs without having thousands of threads; you can apply timeouts to a whole group of jobs as if they were a single job; etc. Plus, you can switch from threads to processes with a trivial one-liner change. Unfortunately, Python 2.7 does not have this module—but there is a quasi-official backport that you can install and use just as easily.
Abamert beat me there on the answer I was preparing, except for this detail:
If, and only if, the outside function is executed through the Python interpreter, even though you can't change it (for example, from a compiled module), you might be able to use the technique described in this other question to kill the thread that calls that function using an exception.
Is there any way to kill a Thread in Python?
Of course, if you did have control over the function you were calling, the StoppableThread class from that answer works well for this:
import threading
class StoppableThread(threading.Thread):
"""Thread class with a stop() method. The thread itself has to check
regularly for the stopped() condition."""
def __init__(self):
super(StoppableThread, self).__init__()
self._stop = threading.Event()
def stop(self):
self._stop.set()
def stopped(self):
return self._stop.isSet()
class Magical_Attack(StoppableThread):
def __init__(self, enval):
self._energy = enval
super(Magical_Attack, self).__init__()
def run(self):
while True and not self.stopped():
print self._energy
if __name__ == "__main__":
a = Magical_Attack(5)
a.start()
a.join(5.0)
a.stop()
I tried to put together a very simple multithreading model, and so far it seems to work. My question is how am I sure that two threads will not grab the same value from the queue simultaneously and give me repeats? Is there just some built in method that prevents this? I added a delay to try to put time between when each thread will get a value from the queue, is this necessary?
from Queue import Queue
from threading import Thread
import time
class myThread(Thread):
def __init__(self,queue):
Thread.__init__(self)
self.queue = queue
def run(self):
while True:
time.sleep(0.0001) #not sure if this is good enough to compensate for overlap in pulling the same value out of a queue
task = self.queue.get() #pull a number from the queue,
if task != None:
out.append(task) #This will be where you
else:
break
queue = Queue()
out = []
for i in range(10):
t = myThread(queue)
t.start()
for i in range(100):
queue.put(i)
print out
The Queue class implements locking to prevent this happening, see http://docs.python.org/library/queue.html, in particular:
It is especially useful in threaded programming when information must
be exchanged safely between multiple threads. The Queue class in this
module implements all the required locking semantics.
So you don't need any delay, the queue should work exactly as you want. That's what it was written for (basically) :-)
You code are correct. The sleep do not is necessary. The Queue is synchronization. Exist a lock that avoid threads to pull the same task.
The whole point of queue.Queue is that it guarantees this property for you. No sleep is needed (and sleeping is never a good way around concurrency glitches).
I only know basic concepts for multi-threading, and I currently encounter a situation that needs some help.
I have two tasks to finish, and both should be executed continuously. The thing is that the second task should start only after the first thread did some jobs first. Right now the two thread classes look roughly like the following:
finished = False # shared flag
class first(threading.Thread):
def __init__(self, cond, finished):
threading.Thread.__init__(self)
self.cond = cond
self.finished = finished
def run(self):
self.cond.aquire()
do_something()
self.finished = True #change the flag
self.cond.notify()
self.cond.release()
do_something_else()
class second(threading.Thread):
def __init__(self, cond, finished):
threading.Thread.__init__(self)
self.cond = cond
self.finished = finished
def run(self):
self.cond.aquire()
while self.finished == False:
self.cond.wait()
self.cond.release()
do_something()
However, the fact is that the program still executes randomly regardless of the wait() and notify(). Can anybody help me with this issue? Thanks.
self.finished in class first is a copy of the value of the global finished, not a reference to it, so it has no live relationship to the self.finished of class second.
You should probably create a global Queue object (which is designed to be used with the threading module). Have both classes refer to the queue, and have the first thread write a go-ahead signal to the queue, and the second thread block until it reads the go-ahead.
You can avoid synchronization altogether. Use 3 threads instead of 2.
Thread 1a 'does some job' and terminates.
Thread 1b starts where 1a ended, and
Thread 2 starts independently.
(Also I suppose you know that you cannot effectively share CPU with Python threads; these are only good for I/O waiting in parallel. When you need CPU-bound parallelization, you use multiprocessing.)