I'm trying to run some simple threading in Python using:
t1 = threading.Thread(analysis("samplequery"))
t1.start()
other code runs in here
t1.join()
Unforunately I'm getting the error:
"AssertionError: group argument must be none for now"
I've never implemented threading in Python before, so I'm a bit unsure as to what's going wrong. Does anyone have any idea what the problem is?
I'm not sure if it's relevant at all, but analysis is a method imported from another file.
I had one follow up query as well. Analysis returns a dictionary, how would I go about assigning that for use in the original method?
Thanks
You want to specify the target keyword parameter instead:
t1 = threading.Thread(target=analysis("samplequery"))
You probably meant to make analysis the run target, but 'samplequery the argument when started:
t1 = threading.Thread(target=analysis, args=("samplequery",))
The first parameter to Thread() is the group argument, and it currently only accepts None as the argument.
From the threading.Thread() documentation:
This constructor should always be called with keyword arguments. Arguments are:
group should be None; reserved for future extension when a ThreadGroup class is implemented.
target is the callable object to be invoked by the run() method. Defaults to None, meaning nothing is called.
You need to provide the target attribute:
t1 = threading.Thread(target = analysis, args = ('samplequery',))
Related
I am using a library that takes some keyword arguments during the initialization of the object. These keyword arguments define the callback objects/functions that are called when a specific event occurs. The callback objects themselves need to reference the object that called them and have some additional data that comes from the class/object instance itself.
What I would like to do is pass a reference to another object into one of these call backs when initializing in order to avoid using a global variable.
How/where would I pass the reference to the object? Do I need to subclass? If I do need to subclass can I accomplish this without understanding the working behind the class/library? I tried looking at the source to try and figure something out but unfortunately it's a bit past my understanding.
Example which will hopefully get my point across:
def One(eI):
*...do some stuff...*
def Two(eI, data, ref_I_want_to_add):
*do some other stuff*
ref_I_want_to_pass = anotherClass(diff_arg,diff_arg2):
*...you know the deal...*
eI = exampleClass(argOne=One, ..., argN=N)
The first thing that comes to mind is to use something like
eI = exampleClass(argOne=One, argTwo=Two(ref_I_want_to_pass), ..., argN)
But that obviously doesnt work because I get a TypeError: argTwo takes 0 positional arguments but 1 was given
More precisely I am using the websocket client library and in order to create a WebSocketApp object which handles the connection you instaniate the object with something like
ws = websocket.WebSocketApp(uri, on_open=on_open, on_message=on_message, on_ping=on_ping)
now I have a function named on_message that handles the incoming messages and I told the ws object during instantiation of WebSocketApp that I would like to use that function to handle the incoming messages or rather what to do with them.
I would like to update another object when a new message comes in, now I can of course use a global variable for that but it doesn't seem like the right way to do it and it would be nice if I could pass a reference to that object when initializing a WebSocketApp object, with something like:
ws = websock.WebSocketApp(uri, on_message=on_message(ref_to_Obj), on_open=on_open, on_ping=on_ping)
But obviously that doesnt work because again I get a TypeError: argTwo takes 0 positional arguments but 1 was given error.
I'm not entirely sure how to do this. Subclassing comes to mind but even then I'm kind of lost, I will admit that subclassing is a topic I need more work on, I get the idea behind it and can do all the basic examples they use in tutorials but more complex classes can stump me.
If you want a local variable to be passed onto a function, you need to create a closure around that variable. We can do that with lambda.
eI = exampleClass(argOne=One, ..., argN=lambda x, y: N(x, y, ref_I_want_to_pass))
I'm attempting to thread a function call in my Python catastr^H^H^H^H^H^Hreation, and I've read up on how to use the threading.Thread() call. My function takes a simple string argument, so theoretically it should be as easy as:
thread = threading.Thread(target = my_func, args = (string_var, ))
bearing in mind that the args() needs to be a tuple. Got it. However, it appears as though I'm still doing something wrong because I continually get the barffage from Python:
TypeError: my_func() takes 1 positional argument but 2 were given
I'm a bit stumped here. Any guidance?
Thanks!
please provide some code for us to help you.
But before you do your post could be a possible duplicate of this post.
Seems the issue is that because it's a method (thanks gribvirus74 for the idea) and I'm attempting to thread it, it won't inherit the self. And that appears to be the issue. I moved the function outside of the class and called it with the Thread(). Works fine now.
If it's a method, then you can write the following code (assuming the class name is SomeClass and it has a method called foo with one argument):
x = SomeClass()
thread = threading.Thread(target=SomeClass.foo, args=(x, 'your method argument'))
I'm currently learning curses in python, and I found this piece of code online that is confusing me.
import curses
def draw_menu(stdscr):
# do stuff
# if you want more code just let me know
def main():
curses.wrapper(draw_menu)
if __name__ == "__main__":
main()
When I run this I don't get the expected missing 1 required positional argument error, since there is no parameter being passed in the curses.wrapper(draw_menu) line. Is this a curses thing? Any help is greatly appreciated.
A function is a datatype, just as much as strings, integers, and so on.
def my_function(txt):
print(txt)
here type(my_function) # => <class 'function'>
You invoke the code inside the function when you call it with parenthesis : my_function('hello') # => prints hello
Until then you can perfectly pass a function as an argument to another function.
And that last one can call the one you passed giving it some parameters.
Like in your case, I'd guess that curses.wrapper() creates a screen interface that it passes as argument your draw_menu() function.
And you can probably use that screen object to build your curse app.
See this : Python function as a function argument?
There's a big difference between curses.wrapper(draw_menu) and curses.wrapper(draw_menu()). curses.wrapper(draw_menu) calls curses.wrapper and passes the function draw_menu into it as an argument. In contrast, curses.wrapper(draw_menu()) would call draw_menu and pass its return value into curses.wrapper.
curses.wrapper will call the function you pass it. From that link:
Initialize curses and call another callable object, func, which should be the rest of your curses-using application.
E.g., it will call draw_menu when curses is completely initialized.
Here is the signature for curses.wrapper from here.
curses.wrapper(func, /, *args, **kwargs)
It says that you need to give curses.wrapper a function reference argument followed by zero or more arguments and keyword arguments. Your code satisfies those requirements.
Python allows function signatures like this to enable developers a lot of flexibility regarding what can be passed in by the caller.
from threading import Thread
def test_first(*args):
'''
some code
:param args:
:return:
'''
td_num = Thread(target=test_first([1,2,3,4,5]))
td_char = Thread(target=test_first(['A','B','C','D','E']))
td_welcome = Thread(target=test_first("Welcome"))
td_num.start()
td_char.start()
td_welcome.start()
td_num.join()
td_char.join()
td_welcome.join()
I have one function which i'm calling from multiple threads, but instead of executing in parallel it is executing in sequence. Any suggestion what wrong in this?
You are calling the functions when you pass them to Thread.
This:
Thread(target=test_first([1,2,3,4,5]))
first calls the function test_first(), then passes the result of the function to Thread(). That means the function is called and complete before the thread is created.
The docs are pretty clear:
target is the callable object to be invoked by the run() method.
Defaults to None, meaning nothing is called.
You should instead pass the callable directly to the Thread and pass the arguments in separately:
td_num = Thread(target=test_first, args=([1,2,3,4,5],))
I have 8 CPU core and 200 tasks to done. Each tasks are isolate. There is no need to wait or share the result. I'm looking for a way to run 8 tasks/processes at a time (Maximum) and when one of them finished. The remaining task will automatic start process.
How to know when the child process was done and start a new child process. First I'm trying to use process(multiprocessing) and it's hard to figure out. Then I try to use pool and face with the pickle problem cause I need to use dynamic instantiate.
Edited : Adding my code of Pool
class Collectorparallel():
def fire(self,obj):
collectorController = Collectorcontroller()
collectorController.crawlTask(obj)
def start(self):
log_to_stderr(logging.DEBUG)
pluginObjectList = []
for pluginName in self.settingModel.getAllCollectorName():
name = pluginName.capitalize()
#Get plugin class and instanitiate object
module = __import__('plugins.'+pluginName,fromlist=[name])
pluginClass = getattr(module,name)
pluginObject = pluginClass()
pluginObjectList.append(pluginObject)
pool = Pool(8)
jobs = pool.map(self.fire,pluginObjectList)
pool.close()
print pluginObjectList
pluginObjectList got something like
[<plugins.name1.Name1 instance at 0x1f54290>, <plugins.name2.Name2 instance at 0x1f54f38>]
PicklingError: Can't pickle : attribute lookup builtin.instancemethod failed
but the Process version work fine
Warning this is kinda subjective to deployment and situation but my current setup is as follows
I have a worker program, I fire up 6 copies (I have 6 cores).
Each worker does the following;
Connect to a Redis instance
Try and pop some work of a specific list
Pushes back logging information
Either idles or terminates on a lack of work in the 'queue'
Then each program is essentially standalone while still doing the work you require with a separate queuing system. As you have no go-between on your processes, this might be a solution to your problem.
I'm not an expert in multiprocessing in Python but I tried some fiew things with this help http://www.tutorialspoint.com/python/python_multithreading.htm and this one too http://www.devshed.com/c/a/Python/Basic-Threading-in-Python/1/ .
You can for example use this method isAlive which answering your question.
The solution to your problem is trivial. First of all, note that methods cannot be pickled. In fact only the types listed in pickle's documentation can be pickled:
None, True, and False
integers, long integers, floating point numbers, complex numbers
normal and Unicode strings
tuples, lists, sets, and dictionaries containing only picklable objects
functions defined at the top level of a module
built-in functions defined at the top level of a module
classes that are defined at the top level of a module
instances of such classes whose __dict__ or the result of calling __getstate__() is picklable (see section The pickle protocol
for details).
[...]
Note that functions (built-in and user-defined) are pickled by
“fully qualified” name reference, not by value. This means that
only the function name is pickled, along with the name of the module the function is defined in. Neither the function’s code, nor any of
its function attributes are pickled. Thus the defining module must be
importable in the unpickling environment, and the module must contain
the named object, otherwise an exception will be raised. [4]
Similarly, classes are pickled by named reference, so the same
restrictions in the unpickling environment apply. Note that none of
the class’s code or data is pickled[...]
Clearly a method isn't a function defined at the top level of a module, hence it cannot be pickled.(read carefully that part of the documentation to avoid future problems with pickle!) But it is absolutely trivial to replace the method with a global function and passing self as additional parameter:
import itertools as it
def global_fire(argument):
self, obj = argument
self.fire(obj)
class Collectorparallel():
def fire(self,obj):
collectorController = Collectorcontroller()
collectorController.crawlTask(obj)
def start(self):
log_to_stderr(logging.DEBUG)
pluginObjectList = []
for pluginName in self.settingModel.getAllCollectorName():
name = pluginName.capitalize()
#Get plugin class and instanitiate object
module = __import__('plugins.'+pluginName,fromlist=[name])
pluginClass = getattr(module,name)
pluginObject = pluginClass()
pluginObjectList.append(pluginObject)
pool = Pool(8)
jobs = pool.map(global_fire, zip(it.repeat(self), pluginObjectList))
pool.close()
print pluginObjectList
Note that, since Pool.map calls the given function with only one argument, we have to "pack together" both self and the actual argument. To do this I have zipped it.repeat(self) and the original iterable.
If you do not care about the order in which the calls are done then using pool.imap_unordered might provide better performances. However it returns an iterable and not a list, so if you want the list of results you'll have to do jobs = list(pool.imap_unordered(...)).
I believe that this code will remove all pickling problems.
class Collectorparallel():
def __call__(self,cNames):
for pluginName in cNames:
name = pluginName.capitalize()
#Get plugin class and instanitiate object
module = __import__('plugins.'+pluginName,fromlist=[name])
pluginClass = getattr(module,name)
pluginObject = pluginClass()
pluginObjectList.append(pluginObject)
collectorController = Collectorcontroller()
collectorController.crawlTask(obj)
def start(self):
log_to_stderr(logging.DEBUG)
pool = Pool(8)
jobs = pool.map(self,self.settingModel.getAllCollectorName())
pool.close()
What has happened here is that Collectorparallel has been turned into a callable. The list of plugin names is used as the iterable for the pool, the actual determination of the plugins and their instantiation is done in each of the worker processes, and the class instance object is used as the callable for each worker process.