In Python, what do you do if you are using a multiprocessing and you need to give the function an extra agruement?
Example:
if value == "Y":
pool = multiprocessing.Pool(processes=8)
pool.map(verify_headers, url_list)<-need to give parameter for a password
pool.close()
pool.join()
print "Done..."
and the function would be something like:
def verify_headers(url, password):
pass
Pool.map takes a function of one argument and an iterable to produce that argument. We can turn your function of two arguments into a function of one argument by wrapping it in another function body:
def verify_headers_with_passowrd(url):
return verify_headers(url, 'secret_password')
And pass that to pool.map instead:
pool.map(verify_headers_with_password, url_list)
so long as verify_headers can take password as a keyword argument, we can shorten that a little: you can use functools.partial
pool.map(functools.partial(verify_headers, password='secret_password'), url_list)
Edit: as Bakuriu points out, multiprocessing passes data round by pickling, so the following doesn't work:
pool.map(lambda url: verify_headers(url, 'secret_password'), url_list)
Since lambda's are functions without a name, and pickle serialzes functions by name.
i believe
from functools import partial
and
pool.map(partial(verify_headers,password=password),url_list)
should work?
edit: fixed based on recommendations below
You define a function, right after the original, that accepts as argument a 2-element tuple:
def verify_headers_tuple(url_passwd):
return verify_headers(*url_passwd)
Then you can zip the original url_list with itertools.repeat(password):
pool.map(verify_headers_tuple, it.izip(url_list, it.repeat(password)))
Note that the function passed to Pool.map must be defined at the top level of a module(due to pickling restrictions), which means you cannot use partial or lambda to create a "curried function".
Related
I've built a function where I want to leverage some multi-threading in Python. This function takes two arguments: A List type of argument (where I leverage multi-threading) and a non iterable variable. The way that I've coded is not working and I don't know how to invoke it
("ssns" variable would be a List of ssn and cursor is simply the cursor used for writing in SQL Server DB). Thank you!
def main(ssns):
t1 = time.perf_counter()
create_folders()
conn = pymssql.connect(db.server, db.user, db.password, db.db)
cursor = conn.cursor(as_dict=True)
with concurrent.futures.ThreadPoolExecutor() as executor:
results = executor.map(f.get_pdf_multi_thread, ssns, cursor)
conn.commit()
cursor.close()
conn.close()
If I'm understanding the question correctly:
You have the function f.get_pdf_multi_thread which takes 2 arguments: a List of SSNs and a cursor. You want to call f.get_pdf_multi_thread(ssns, cursor), using the ThreadPoolExecutor class for asynchronicity.
Answer:
I think you want to either:
(1) use executor.submit instead of executor.map in order to accept multiple arguments, or
(2) change your function to accept two non-iterables, e.g. f.get_pdf_single(ssn, cursor), and use an anonymous (lambda) function within the call to executor.map:
results = executor.map(lambda s, c: f.get_pdf_single(s, c), ssns)
The method executor.map actually does the work of making multiple calls to the given function AND distributing them across threads for you. The function passed in should be a function that takes one item of your iterable as an argument.
I was unable to find a reasonable way to create a variable which calls a function requiring parameters.
Here is a simplified version of my code. I would like ‘print_hello’ to print ‘hello’ when it is called, and not when it is defined.
print_hello = print(‘hello’)
When I define ‘print_hello’, it calls print(‘hello’). When I call ‘print_hello’, it gives me an error. How do I fix this?
If you just want a function that does precisely what you describe, Sheldore's answer is the simplest way to go (and more Pythonic than using a named lambda).
An alternative approach is to make a partial application of the function with functools.partial, which allows you to pass additional arguments at call time:
from functools import partial
print_hello = partial(print, "hello")
print_hello() # Prints "hello" to stdout
print_hello(file=sys.stderr) # Prints "hello" to stderr
print_hello("world") # Prints "hello world" to stdout
Just define print_hello as a lambda function
>>> print_hello = lambda: print('hello')
>>> print_hello()
hello
To delay execution, you'll have to wrap the call to print in another function. A lambda is less code than defining another function.
Note: that pep08 recommends using a def function rather than a lambda when assigning to a variable. See here. So #Sheldores answer is probably the way to go.
You need to define a function. In python a function is defined using def as shown in a simple example for your purpose below. You then call the function using the function name and (), for instance print_hello().
def print_hello(): # <--- Does not accept an argument
print('hello')
print_hello() # <--- No argument is passed
# hello
Another example to give you more idea on how to pass an argument to the function. You can define a variable that contains the string you want to print, let's say to_print and then pass this as an argument to your function during calling it. While explaining more details is out of the scope of this answer, the two examples I gave should get you started. For more details, you can refer to the official docs here
def print_hello(to_print): # <--- Accepts an argument
print(to_print)
to_print = "hello"
print_hello(to_print) # <--- Argument is passed
# hello
You could use a lambda expression:
print_hello = lambda: print('hello')
Or an actual function definition:
def print_hello(): print('hello')
Or functools.partial (this is different in that you can still use other arguments for print whereas you lose that functionality with the others unless specified in the definitions)
from functools import partial
print_hello = partial(print, 'hello')
To use any of these:
print_hello()
#'hello'
def return_total():
dE_total = 0
for num in range(len(self.layer)):
dE_total += self.layer[num].backprop(delta[num])
return dE_total
I have the above method inside a class. I need to call the backprop() method using multithreading. Usually the length of self.layer is small. I was planning to try ThreadPoolExecutor's map() method. As far as i know it is used to call a function and iterable value. Here each thread should execute for a different method with a input paramater. Is there any way to go about doing this?
with ThreadPoolExecutor() as executor:
dE_total += executor.map(self.layer.backprop, delt)
I am aware the above code does not make any sense. I'm looking for something similar to the above idea.
Thanks for any help in advance
If I'm interpreting this correctly, you could write a method which takes the function as argument. This can then be passed to executor.map, e.g.:
def func_caller(funcs, params):
return func(*params)
dE_total += sum(executor.map(func_caller, funcs_params))
or similar, with funcs_params some appropriate list of tuples of functions and parameters. The argument unpacking might need to be adjusted.
I have an OrderedDict whose values I would like to be functions, but have encountered unexpected behaviour. Initializing:
from collections import OrderedDict
options_dict=OrderedDict(["A",call_func_A(arg1,arg2)],
["B",call_func_B(arg1,arg3)],
["C",call_func_C(arg1,arg4)]
# Select options
options=["A","C"]
# Execute
result={}
for opt in options:
result[opt]=options_dict[opt]
# Return result (or whatever)
print result
Functions call_func_A, call_func_B and call_func_C turn out to be executed when options_dict is declared, rather than in the subsequent for loop over options.
I'd like the function calls to wait until the for loop.
What's going on?
The functions are called before the dictionary is created. You made the call.
However, you can defer the function calls by nesting it within another function to be called later:
options_dict = OrderedDict([("A", lambda: call_func_A(arg1,arg2)),
("B", lambda: call_func_B(arg1,arg3)),
("C", lambda: call_func_C(arg1,arg4))])
# Select options
options = ["A", "C"]
# Execute
result = {}
for opt in options:
result[opt] = options_dict[opt]() # <- call
The same effect can be achieved with functools.partial, with an extra import statement to execute.
On another note, since your function arguments are presumably invariant, I don't think when the calls are made is important here. You might as well keep your initial approach of having the functions called at the dict creation time.
First of all, you are declaring the OrderedDict incorrectly. The constructor expects a list of tuples. Instead, you are giving it multiple lists. Do it like so:
options_dict=OrderedDict([("A",call_func_A(arg1, arg2)),
("B",call_func_B(arg1, arg3)),
("C",call_func_C(arg1, arg4))])
Second, when you declare options_dict, you don't pass the functions as the values of the dict, but rather their results:
options_dict=OrderedDict(["A",call_func_A(arg1,arg2)],
["B",call_func_B(arg1,arg3)],
["C",call_func_C(arg1,arg4)])
You are calling them by doing call_func_A(arg1, arg2). One relatively simple way of avoiding that is by omitting the args:
options_dict=OrderedDict([("A",call_func_A),
("B",call_func_B),
("C",call_func_C)])
You can store the args in a second OrderedDict:
args_dict=OrderedDict([("A",[arg1, arg2]),
("B",[arg3, arg4]),
("C",[arg5, arg6])])
And then to call them:
result={}
for opt in options:
result[opt]=options_dict[opt](*args_dict[opt])
I am Trying to emulate the following code with something more elegant:
def makeHomeFunc(axisNumber):
def fClosure(null_pass):
stage.home(axisNumber)
return fClosure
In short this is a function factory, that acts like functools.partial with the additional property of discarding the values passed to the produced function, and running the inner wrapped function with the factory's call.
Thanks