I've been mucking around with python for a little while and I have recently come up with something involving multithreading... without further ado... heres what I have...
import pythoncom
import wmi
import threading
class Info(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
def run(self):
pythoncom.CoInitialize()
c = wmi.WMI()
detect = c.Win32_ComputerShutdownEvent.watch_for()
detect()
return
if __name__ == '__main__':
Info().start()
for process in c.Win32_Process(Name="something.exe"):
result = process.Terminate()
So my question is... Why does this work? It may be an overall question regarding the process of the inheritence of threading.Thread... but there is no start() def in the class Info() so why does the run def begin?
This is actually a pretty handy application I needed to use to stop an application that always seems to hang when windows shuts down... finding when the windows shutdown event happens was a bit of a headache but luckily tim golden's masterpiece saves the day!
Because it's defined in the parent. Parent classes are checked for attributes if they're not found (or handled) in the child class.
Subclasses of Thread automatically call their run(), when you call start() on them. Start is defined in Thread.
From the docs
There are two ways to specify the activity: by passing a callable object to the constructor, or by overriding the run() method in a subclass.
and from docs on start()
It arranges for the object’s run() method to be invoked in a separate thread of control.
Don't you intent to wait the thread has ended before killing process ?
If so:
if __name__ == '__main__':
info = Info()
info.start()
info.join()
for process in c.Win32_Process(Name="something.exe"):
result = process.Terminate()
Related
I am new to parallelization in general and concurrent.futures in particular. I want to benchmark my script and compare the differences between using threads and processes, but I found that I couldn't even get that running because when using ProcessPoolExecutor I cannot use my global variables.
The following code will output Helloas I expect, but when you change ThreadPoolExecutor for ProcessPoolExecutor, it will output None.
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
greeting = None
def process():
print(greeting)
return None
def main():
with ThreadPoolExecutor(max_workers=1) as executor:
executor.submit(process)
return None
def init():
global greeting
greeting = 'Hello'
return None
if __name__ == '__main__':
init()
main()
I don't understand why this is the case. In my real program, init is used to set the global variables to CLI arguments, and there are a lot of them. Hence, passing them as arguments does not seem recommended. So how do I pass those global variables to each process/thread correctly?
I know that I can change things around, which will work, but I don't understand why. E.g. the following works for both Executors, but it also means that the globals initialisation has to happen for every instance.
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
greeting = None
def init():
global greeting
greeting = 'Hello'
return None
def main():
with ThreadPoolExecutor(max_workers=1) as executor:
executor.submit(process)
return None
def process():
init()
print(greeting)
return None
if __name__ == '__main__':
main()
So my main question is, what is actually happening. Why does this code work with threads and not with processes? And, how do I correctly pass set globals to each process/thread without having to re-initialise them for every instance?
(Side note: because I have read that concurrent.futures might behave differently on Windows, I have to note that I am running Python 3.6 on Windows 10 64 bit.)
I'm not sure of the limitations of this approach, but you can pass (serializable?) objects between your main process/thread. This would also help you get rid of the reliance on global vars:
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor
def process(opts):
opts["process"] = "got here"
print("In process():", opts)
return None
def main(opts):
opts["main"] = "got here"
executor = [ProcessPoolExecutor, ThreadPoolExecutor][1]
with executor(max_workers=1) as executor:
executor.submit(process, opts)
return None
def init(opts): # Gather CLI opts and populate dict
opts["init"] = "got here"
return None
if __name__ == '__main__':
cli_opts = {"__main__": "got here"} # Initialize dict
init(cli_opts) # Populate dict
main(cli_opts) # Use dict
Works with both executor types.
Edit: Even though it sounds like it won't be a problem for your use case, I'll point out that with ProcessPoolExecutor, the opts dict you get inside process will be a frozen copy, so mutations to it will not be visible across processes nor will they be visible once you return to the __main__ block. ThreadPoolExecutor, on the other hand, will share the dict object between threads.
Actually, the first code of the OP will work as intended on Linux (tested in Python 3.6-3.8) because
On Unix a child process can make use of a shared resource created in a
parent process using a global resource.
as explained in multiprocessing doc. However, for a mysterious reasons, it won't work on my Mac running Mojave (which is supposed to be a UNIX-compliant OS; tested only with Python 3.8). And for sure, it won't work on Windows, and it's in general not a recommended practice with multiple processes.
Let's image a process is a box while a thread is a worker inside a box. A worker can only access the resources in the box and cannot touch the other resources in other boxes.
So when you use threads, you are creating multiple workers for your current box(main process). But when you use process, you are creating another box. In this case, the global variables initialised in this box is completely different from ones in another box. That's why it doesn't work as you expect.
The solution given by jedwards is good enough for most situations. You can expilictly package the resources in current box(serialize variables) and deliver it to another box(transport to another process) so that the workers in that box have access to the resources.
A process represents activity that is run in a separate process in the OS meaning of the term while threads all run in your main process. Every process has its own unique namespace.
Your main process sets the value to greeting by calling init() inside your __name__ == '__main__'condition for its own namespace. In your new process, this does not happen (__name__ is '__mp_name__' here) hence greeting remains None and init() is never actually called unless you do so explicitly in the function your process executes.
While sharing state between processes is generally not recommended, there are ways to do so, like outlined in #jedwards answer.
You might also want to check Sharing State Between Processes from the docs.
Let us say we have a python function magical_attack(energy) which may or may not last more than a second. It could even be an infinite loop? How would I run, but if it goes over a second, terminate it, and tell the rest of the program. I am looking for a sleek module to do this. Example:
import timeout
try: timout.run(magical_attack(5), 1)
except timeout.timeouterror:
blow_up_in_face(wizard)
Note: It is impossible to modify the function. It comes from the outside during runtime.
The simplest way to do this is to run the background code in a thread:
t = threading.Thread(target=magical_attack, args=(5,))
t.start()
t.join(1)
if not t.isAlive():
blow_up_in_face(wizard)
However, note that this will not cancel the magical_attack function; it could still keep spinning along in the background for as long as it wants even though you no longer care about the results.
Canceling threads safely is inherently hard to do, and different on each platform, so Python doesn't attempt to provide a way to do it. If you need that, there are three alternatives:
If you can edit the code of magical_attack to check a flag every so often, you can cancel it cooperatively by just setting that flag.
You can use a child process instead of a thread, which you can then kill safely.
You can use ctypes, pywin32, PyObjC, etc. to access platform-specific routines to kill the thread. But you have to really know what you're doing to make sure you do it safely, and don't confuse Python in doing it.
As Chris Pak pointed out, the futures module in Python 3.2+ makes this even easier. For example, you can throw off thousands of jobs without having thousands of threads; you can apply timeouts to a whole group of jobs as if they were a single job; etc. Plus, you can switch from threads to processes with a trivial one-liner change. Unfortunately, Python 2.7 does not have this module—but there is a quasi-official backport that you can install and use just as easily.
Abamert beat me there on the answer I was preparing, except for this detail:
If, and only if, the outside function is executed through the Python interpreter, even though you can't change it (for example, from a compiled module), you might be able to use the technique described in this other question to kill the thread that calls that function using an exception.
Is there any way to kill a Thread in Python?
Of course, if you did have control over the function you were calling, the StoppableThread class from that answer works well for this:
import threading
class StoppableThread(threading.Thread):
"""Thread class with a stop() method. The thread itself has to check
regularly for the stopped() condition."""
def __init__(self):
super(StoppableThread, self).__init__()
self._stop = threading.Event()
def stop(self):
self._stop.set()
def stopped(self):
return self._stop.isSet()
class Magical_Attack(StoppableThread):
def __init__(self, enval):
self._energy = enval
super(Magical_Attack, self).__init__()
def run(self):
while True and not self.stopped():
print self._energy
if __name__ == "__main__":
a = Magical_Attack(5)
a.start()
a.join(5.0)
a.stop()
i need a help with a python project:
Example:
class MyFrame(wx.Frame):
def __init__(self, parent, title):
super(MyFrame, self).__init__(parent, title=title, size=(330, 300))
self.InitUI()
self.Centre()
self.Show()
def InitUI(self):
"""
Subprocess
"""
subprocess.execMethodFromClass( self , 'Connection' , args1 , args2 , ... )
def Connection( self ):
self.connection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.connection.connect(( '192.0.1.135' , 3345 ))
while True:
data = self.connection.recv(1024)
if not data:
break
else:
print data
Show:
subprocess.execMethodFromClass( self , 'Connection' , args1 , args2 , ... )
Thanks!
As the friendly dogcow says, to run a function in a child process, all you have to do is use a multiprocessing.Process:
p = multiprocessing.Process(target=f, args=('bob',))
p.start()
p.join()
Of course you probably want to hang onto p and join it later in most* real-life use cases. You're obviously not getting any parallelism by spawning a new process just to make your main process sit around and wait.
So, in your case, that's just:
p = multiprocessing.Process(target=self.Connection, args=(args1, args2))
But this probably won't work in your case, because you're trying to call a method on the current self object.
First, depending on your platform and Python version, multiprocessing may have to pass the bound method self.Connection to the child by pickling it and sending it over a pipe. This involves pickling the self instance as well as the method. So it will only work if MyFrame objects are pickleable. And I'm pretty sure that a wx.Frame can't be pickled.
And even if you do get the self object to the child, it will obviously be a copy, not a shared instance. So, when the child process's Connection method sets self.connection = …, that won't affect the original parent process's self.
Even worse if you try to call any wx.Frame methods. Even if all the Python stuff worked, on most platforms, trying to modify GUI resources like windows from the wrong process will not work.
The only kinds of objects you can actually share are the kinds you can put in multiprocessing.Value or multiprocessing.sharedctypes.
The way around this is to factor out the code you want to childify into a separate, isolated function, that shares as little as possible (ideally nothing, or nothing but a Queue or Pipe) with the parent.
For your example, this is easy:
class Client(object):
def connect_and_fetch(self):
self.connection = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.connection.connect(( '192.0.1.135' , 3345 ))
while True:
data = self.connection.recv(1024)
if not data:
break
else:
print data
def do_client():
client = Client()
connect_and_fetch()
class MyFrame(wx.Frame):
# ...
def Connection(self):
self.child = multiprocessing.Process(target=do_client)
self.child.start()
# and now put a self.child.join() somewhere
In fact, you don't even need a class at all here, because the only use you have for self is to store a variable that could just as easily be a local. But I'm guessing in your real-life program, there's a bit more state than that.
There's an interesting (if a bit outdated) example on the wxpython wiki, called MultiProcessing, which looks like it does most of what you want and more. (It's using a classmethod for the child process instead of a standalone function for some reason, and using old-style syntax for it because it's old, but hopefully it's still helpful.)
If you're using wx for your GUI, you may want to consider using its inter-process mechanisms instead of the native Python ones. While it's more complicated and less pythonic in the general case, when you're trying to integrate a child process and its communications pipe into your main event loop, why not let wx take care of it?
The alternative is to create a thread to wait on the child process and/or whatever Pipe or Queue you give it, and then create and post wx.Events to the main thread.
* Most, not all. For example, if f temporarily uses up a whole lot of memory, running it in a child process means you release that memory to the OS as quickly as possible. Or, if it calls badly-written third-party/legacy/whatever code that has nasty and poorly-documented global side-effects, you're isolated from those side-effects. And so on.
From http://docs.python.org/dev/library/multiprocessing.html:
from multiprocessing import Process
def f(name):
print('hello', name)
if __name__ == '__main__':
p = Process(target=f, args=('bob',))
p.start()
p.join()
You can't. You use subprocess to call another application or script to run in a separate process.
subprocess.Popen(cmds)
If you need to run some long running process, look into threads or the multiprocessing module. Here are some links:
http://docs.python.org/2/library/multiprocessing.html
http://wiki.wxpython.org/LongRunningTasks
http://www.blog.pythonlibrary.org/2012/08/03/python-concurrency-porting-from-a-queue-to-multiprocessing/
http://www.blog.pythonlibrary.org/2010/05/22/wxpython-and-threads/
well,I wrote a little snappet trying to know how to use python threading .
But strangely the following code just quit quickly without the expected output.
Is it because I shouldn't spawn threads by overiding the run() method?
import threading
from time import sleep
class mythread(threading.Thread):
def __init__(self,target=None,thread_num=5):
threading.Thread.__init__(self,target=None)
self.thn = thread_num
def run(self):
for i in range(self.thn):
t = threading.Thread(target=self.myfunc)
t.start()
t.join()
myfunc(self.thn)
def myfunc(num):
print num,'\tI am doing sth.'
sleep(0.5)
print num,'\tI have done it.'
mythread()
You need to start the thread to make it actually do something:
t = mythread()
t.start()
If you bother to accept a target parameter in your constructor (why?), you shouldn't ignore this parameter. Maybe you want to pass it on to the Thread constructor. (Why?)
When you write mythread(), you instantiate the object. THe default constructor will be called, so __init__() will be executed.
You constructor doesn't have the any instruction of starting the thread.
obj = functioning()
from threading import Thread
Thread(target=obj.runCron(cronDetails)).start()
print "new thread started..."
I am runnning this, this should run as new thread for runCron function and should print new thread started. but this is not printing new thread started and not creating new thread
You question is missing some details, e.g. what error message you are getting, etc. – below is a working example mimicked after your code.
#!/usr/bin/env python
import time
class Obj(object):
def runCron(self, cronDetails):
time.sleep(1)
print cronDetails
obj = Obj()
cronDetails = "I'm here."
from threading import Thread
# Note, that the `target` is a function object
# (or a callable in general), we don't actually call it yet!
t = Thread(target=obj.runCron, args=(cronDetails, ))
t.start()
print "New thread started (should be here in a second) ..."
It prints:
New thread started (should be here in a second) ...
I'm here.
Looks like you want to call obj.runCron(cronDetails) inside the thread. But what that code does is to call obj.runCron(cronDetails) first, and then pass the result of that to the Thread class.
If that's the case, the code below should fix it:
obj = functioning()
from threading import Thread
Thread(target=obj.runCron, args=(cronDetails,)).start()
print "new thread started..."
Note that I'm not calling obj.runCron myself anymore, but passing that method with the arguments separately to threading.Thread so it can be called inside the thread with the correct arguments.
If that doesn't do what you want, please provide more info as I asked in the comment.