Related
I was profiling Erlang's lists:reverse Built in Function (BIF) to see how well it scales with the size of the input. More specifically, I tried:
1> X = lists:seq(1, 1000000).
[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,
23,24,25,26,27,28,29|...]
2> timer:tc(lists, reverse, [X]).
{57737,
[1000000,999999,999998,999997,999996,999995,999994,999993,
999992,999991,999990,999989,999988,999987,999986,999985,
999984,999983,999982,999981,999980,999979,999978,999977,
999976,999975,999974|...]}
3> timer:tc(lists, reverse, [X]).
{46896,
[1000000,999999,999998,999997,999996,999995,999994,999993,
999992,999991,999990,999989,999988,999987,999986,999985,
999984,999983,999982,999981,999980,999979,999978,999977,
999976,999975,999974|...]}
4> Y = lists:seq(1, 10000000).
[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,
23,24,25,26,27,28,29|...]
5> timer:tc(lists, reverse, [Y]).
{434079,
[10000000,9999999,9999998,9999997,9999996,9999995,9999994,
9999993,9999992,9999991,9999990,9999989,9999988,9999987,
9999986,9999985,9999984,9999983,9999982,9999981,9999980,
9999979,9999978,9999977,9999976,9999975,9999974|...]}
6> timer:tc(lists, reverse, [Y]).
{214173,
[10000000,9999999,9999998,9999997,9999996,9999995,9999994,
9999993,9999992,9999991,9999990,9999989,9999988,9999987,
9999986,9999985,9999984,9999983,9999982,9999981,9999980,
9999979,9999978,9999977,9999976,9999975,9999974|...]}
Ok, so far it seems like the reverse BIF scales in approximately linear time with respect to the input (e.g. multiply the size of the input by 10 and the size of time taken also increases by a factor of 10). In pure Erlang that would make sense since we would use something like tail recursion to reverse the list. I guess that even as a BIF implemented in C, the algorithm for reversing seems a list to be the same (maybe because of the way lists are just represented in Erlang?).
Now I wanted to compare this with something another language - perhaps another dynamically typed language that I already use. So I tried a similar thing in Python - taking care to, very explicitly, use actual lists instead of generators which I anticipate would affect the performance of Python positively in this test, giving it an unfair advantage.
import time
ms_conv_factor = 10**6
def profile(func, *args):
start = time.time()
func(args)
end = time.time()
elapsed_seconds = end - start
print(elapsed_seconds * ms_conv_factor, flush=True)
x = list([i for i in range(0, 1000000)])
y = list([i for i in range(0, 10000000)])
z = list([i for i in range(0, 100000000)])
def f(m):
return m[::-1]
def g(m):
return reversed(m)
if __name__ == "__main__":
print("All done loading the lists, starting now.", flush=True)
print("f:")
profile(f, x)
profile(f, y)
print("")
profile(f, x)
profile(f, y)
print("")
profile(f, z)
print("")
print("g:")
profile(g, x)
profile(g, y)
print("")
profile(g, x)
profile(g, y)
print("")
profile(g, z)
This seems to suggest that after the function has been loaded and run once, the length of the input makes no difference and the reversal times are incredibly fast - in the range of ~0.7µs.
Exact result:
All done loading the lists, starting now.
f:
1.430511474609375
0.7152557373046875
0.7152557373046875
0.2384185791015625
0.476837158203125
g:
1.9073486328125
0.7152557373046875
0.2384185791015625
0.2384185791015625
0.476837158203125
My first, naive, guess was that python might be able to recognize the reverse construct and create something like a reverse iterator and return that (Python can work with references right? Maybe it was using some kind of optimization here). But I don't think that theory makes sense since the original list and the returned list are not the same (changing one shouldn't change the other).
So my question(s) here is(are):
Is my profiling technique here flawed? Have I written the tests in a way that favor one language over the other?
What is the difference in implementation of lists and their reversal in Erlang vs Python that make this situation (of Python being WAY faster) possible?
Thanks for your time (in advance).
This seems to suggest that after the function has been loaded and run
once, the length of the input makes no difference and the reversal
times are incredibly fast - in the range of ~0.7µs.
Because your profiling function is incorrect. It accepts variable positional arguments, but when it passes them to the function, it doesn't unpack them so you are only ever working with a tuple of length one. You need to do the following:
def profile(func, *args):
start = time.time()
func(*args) # Make sure to unpack the args!
end = time.time()
elapsed_seconds = end - start
print(elapsed_seconds * ms_conv_factor, flush=True)
So notice the difference:
>>> def foo(*args):
... print(args)
... print(*args)
...
>>> foo(1,2,3)
(1, 2, 3)
1 2 3
Also note, reversed(m) creates a reversed iterator, so it doesn't actually do anything until you iterate over it. So g will still be constant time.
But rest assured, reversing a list in Python takes linear time.
I am looking for a way to partially apply functions in python that is simple to understand, readable, resusable and as little error prone to coder mistakes as possible. Most of all I want the style to be as performant as possible - less frames on the stack is nice, and less memory footprint for the partially applied functions is also desirable. I have considered 4 styles and written examples below:
import functools
def multiplier(m):
def inner(x):
return m * x
return inner
def divide(n,d):
return n/d
def divider(d):
return functools.partial(divide,d=d)
times2 = multiplier(2)
print(times2(3)) # 6
by2 = divider(2)
print(by2(6)) # 3.0
by3 = functools.partial(divide,d=3)
print(by3(9)) # 3.0
by4 = lambda n: divide(n,4)
print(by4(12)) # 3.0
My analysis of them are:
times2 is a nested thing. I guess python makes a closure with the m bound, and everything is nice. The code is readable (I think) and simple to understand. No external libraries. This is the style I use today.
by2 has an explicit named function which makes it simple for the user. It uses functools, so it gives you extra imports. I like this style to some extent since it is transparent, and give me the option to use divide in other ways if I want to. Contrast this with inner which is not reachable.
by3 is like by2, but forces the reader of the code to be comfortable with functools.partial since they have it right in the face. what I like less is that PyCharm cant give my tooltips with what the arguments to functools.partial should be, since they are effectively arguments to by3. I have to know the signature of divide myself everytime I define some new partial application.
by4 is simple to type, since I can get autocompletion. It needs no import of functools. I think it looks non-pythonic though. Also, I always feel uncomfortable about scoping of variables / closures with lambdas work in python. Never sure how that behaves....
What is the logical difference between the styles and how does that affect memory and CPU?
The first way seems to be the most efficient. I tweaked your code so that all 4 functions compute exactly the same mathematical function:
import functools,timeit
def multiplier(m):
def inner(x):
return m * x
return inner
def mult(x,m):
return m*x
def multer(m):
return functools.partial(mult,m=m)
f1 = multiplier(2)
f2 = multer(2)
f3 = functools.partial(mult,m=2)
f4 = lambda x: mult(x,2)
print(timeit.timeit('f1(10)',setup = 'from __main__ import f1'))
print(timeit.timeit('f2(10)',setup = 'from __main__ import f2'))
print(timeit.timeit('f3(10)',setup = 'from __main__ import f3'))
print(timeit.timeit('f4(10)',setup = 'from __main__ import f4'))
Typical output (on my machine):
0.08207898699999999
0.19439769299999998
0.20093803199999993
0.1442435820000001
The two functools.partial approaches are identical (since one of them is just a wrapper for the other), the first is twice as fast, and the last is somewhere in between (but closer to the first). There is a clear overhead in using functools over a straightforward closure. Since the closure approach is arguably more readable as well (and more flexible than the lambda which doesn't extend well to more complicated functions) I would just go with it.
Technically you're missing one other option as operator.mul does the same thing you're looking to do and you can just use functools.partial on it to get a default first argument without having to reinvent the wheel.
Not only is it the fastest option it also uses the least space compared to a custom function or a lambda statement. The fact that it's a partial is why it uses the same space as the others and I think that's the best route here.
from timeit import timeit
from functools import partial
from sys import getsizeof
from operator import mul
def multiplier(m):
def inner(x):
return m * x
return inner
def mult(x,m):
return m*x
def multer(m):
return partial(mult,m=m)
f1 = multiplier(2)
f2 = multer(2)
f3 = partial(mult,m=2)
f4 = lambda n: mult(n,2)
f5 = partial(mul, 2)
from_main = 'from __main__ import {}'.format
print(timeit('f1(10)', from_main('f1')), getsizeof(f1))
print(timeit('f2(10)', from_main('f2')), getsizeof(f2))
print(timeit('f3(10)', from_main('f3')), getsizeof(f3))
print(timeit('f4(10)', from_main('f4')), getsizeof(f4))
print(timeit('f5(10)', from_main('f5')), getsizeof(f5))
Output
0.5278953390006791 144
1.0804575479996856 96
1.0762036349988193 96
0.9348237040030654 144
0.3904160970050725 96
This should answer your question as far as memory usage and speed.
This may be a very easy question but definitely worn me out.
To use multiprocessing, I wrote the following code. the main function creates two processes which both use the same function , called prepare_input_data() but process different input datasets. this function must return multiple objects and values for each input to be used in the next steps of the code (not include here).
What I want is to get more than one value or object as a return from the function I am using in multiprocessing.
def prepare_input_data(inputdata_address,temporary_address, output):
p=current_process()
name = p.name
data_address = inputdata_address
layer = loading_layer(data_address)
preprocessing_object = Preprocessing(layer)
nodes= preprocessing_object.node_extraction(layer)
tree = preprocessing_object.index_nodes()
roundabouts_dict , roundabouts_tree= find_roundabouts(layer.address, layer, temporary_address)
#return layer, nodes, tree, roundabouts_dict, roundabouts_tree
#return [layer, nodes, tree, roundabouts_dict, roundabouts_tree]
output.put( [layer, nodes, tree, roundabouts_dict, roundabouts_tree])
if __name__ == '__main__':
print "the data preparation in multi processes starts here"
output=Queue()
start_time=time.time()
processes =[]
#outputs=[]
ref_process = Process(name ="reference", target=prepare_input_data, args=("D:/Ehsan/Skane/Input/Skane_data/Under_processing/identicals/clipped/test/NVDB_test3.shp", "D:/Ehsan/Skane/Input/Skane_data/Under_processing/temporary/",output))
cor_process = Process(name ="corresponding", target=prepare_input_data, args=("D:/Ehsan/Skane/Input/Skane_data/Under_processing/identicals/clipped/test/OSM_test3.shp", "D:/Ehsan/Skane/Input/Skane_data/Under_processing/temporary/",output))
#outputs.append(ref_process.start)
#outputs.append(cor_process.start)
ref_process.start
cor_process.start
processes.append(ref_process)
processes.append(cor_process)
for p in processes:
p.join()
print "the whole data preparation took ",time.time()-start_time
results={}
for p in processes:
results[p.name]=output.get()
########################
#ref_info = outputs[0]
# ref_nodes=ref_info[0]
Previous ERROR
when I use return,ref_info[0] has Nonetype.
ERROR:
based on the answer here I changed it to a Queueu object passed to the function then I used put() to add the results and get() to retrieve them for the further processing.
Traceback (most recent call last):
File "C:\Python27\ArcGISx6410.2\Lib\multiprocessing\queues.py", line 262, in _feed
send(obj)
UnpickleableError: Cannot pickle <type 'geoprocessing spatial reference object'> objects
Could you please help me solve how to return more than one value from a function in multiprocessing?
Parallel programming with shared state is a rocky road that even experienced programmers get wrong. A much more beginner-friendly method is to copy data around. This is the only way to move data between subprocesses (not quite true, but that's an advanced topic).
Citing https://docs.python.org/2/library/multiprocessing.html#exchanging-objects-between-processes, you'll want to setup a multiprocessing.Queue to fill with your returned data for each of your subprocesses. Afterward you can pass the queue to be read from to the next stage.
For multiple different datasets, such as your layer, nodes, tree, etc, you can use multiple queues to differentiate each return value. It may seem a bit cluttered to use a queue for each, but it's simple and understandable and safe.
Hope that helps.
if you use jpe_types.paralel's Process it will return the return value of the Processes target function like so
import jpe_types.paralel
def fun():
return 4, 23.4, "hi", None
if __name__ == "__main__":
p = jpe_types.paralel.Process(target = fun)
p.start()
print(p.join())
otherwise you could
import multiprocessing as mp
def fun(returner):
returner.send((1, 23,"hi", None))
if __name__ == "__main__":
processes = []
for i in range(2):
sender, recever = mp.Pipe()
p = mp.Process(target = fun, args=(sender,))
p.start()
processes.append((p, recever))
resses = []
for p, rcver in processes:
p.join()
resses.append(rcver.recv())
print(resses)
using the conection will garantee that the retun's don't get scrambeld
If you are looking to get multiple return values from multiprocessing, then you can do that. Here's a simple example, first in serial python, then with multiprocessing:
>>> a,b = range(10), range(10,0,-1)
>>> import math
>>> map(math.modf, (1.*i/j for i,j in zip(a,b)))
[(0.0, 0.0), (0.1111111111111111, 0.0), (0.25, 0.0), (0.42857142857142855, 0.0), (0.6666666666666666, 0.0), (0.0, 1.0), (0.5, 1.0), (0.3333333333333335, 2.0), (0.0, 4.0), (0.0, 9.0)]
>>>
>>> from multiprocessing import Pool
>>> res = Pool().imap(math.modf, (1.*i/j for i,j in zip(a,b)))
>>> for i,ai in enumerate(a):
... x,y = res.next()
... print("{x},{y} = modf({u}/{d})").format(x=x,y=y,u=ai,d=b[i])
...
0.0,0.0 = modf(0/10)
0.111111111111,0.0 = modf(1/9)
0.25,0.0 = modf(2/8)
0.428571428571,0.0 = modf(3/7)
0.666666666667,0.0 = modf(4/6)
0.0,1.0 = modf(5/5)
0.5,1.0 = modf(6/4)
0.333333333333,2.0 = modf(7/3)
0.0,4.0 = modf(8/2)
0.0,9.0 = modf(9/1)
So to get multiple values in the return from a function with multiprocessing, you only need to have a function that returns multiple values… you will just get the values back as a list of tuples.
The major issue with multiprocessing, as you can see from your error… is that most functions don't serialize. So, if you really want to do what it seems like you want to do… I'd strongly suggest you use pathos (as discussed below). The largest barrier you will have with multiprocessing is that the functions you are passing as the target must be serializable. There are several modifications you can make to your prepare_input_data function… the first of which is to make sure it is encapsulated. If your function is not fully encapsulated (e.g. it has name-reference lookups outside of it's own scope), then it probably won't pickle with pickle. That means, you need to include all imports inside the target function and pass any other variables in through the function input. The error you are seeing (UnPicklableError) is due to your target function and it's dependencies not being able be serialized -- and not that you can't return multiple values from multiprocessing.
While I'd encapsulate the target function anyway as a matter of good practice, it can be a bit tedious and could slow your code down a hair. I also suggest that you convert your code to use dill and pathos.multiprocessing -- dill is an advanced serializer that can pickle almost all python objects, and pathos provides a multiprocessing fork that uses dill. That way, you can pass most python objects in the pipe (i.e. apply) or the map that is available form the Pool object and not worry too much about sweating too hard refactoring your code to make sure plain old pickle and multiprocessing can handle it.
Also, I'd use an asynchronous map instead of doing what you are doing above. pathos.multiprocessing has the ability to take multiple arguments in the map function, so you don't need to wrap them in the tuple args as you've done above. The interface should be much cleaner with an asynchronous map, and you can return multiple arguments if you need to… just pack them in a tuple.
Here's some examples that should demonstrate what I'm referring to above.
Return multiple values:
>>> from pathos.multiprocessing import ProcessingPool as Pool
>>> def addsub(x,y):
... return x+y, x-y
...
>>> a,b = range(10),range(-10,10,2)
>>> res = Pool().imap(addsub, a, b)
>>>
>>> for i,ai in enumerate(a):
... add,sub = res.next()
... print("{a} + {b} = {p}; {a} - {b} = {m}".format(a=ai,b=b[i],p=add,m=sub))
...
0 + -10 = -10; 0 - -10 = 10
1 + -8 = -7; 1 - -8 = 9
2 + -6 = -4; 2 - -6 = 8
3 + -4 = -1; 3 - -4 = 7
4 + -2 = 2; 4 - -2 = 6
5 + 0 = 5; 5 - 0 = 5
6 + 2 = 8; 6 - 2 = 4
7 + 4 = 11; 7 - 4 = 3
8 + 6 = 14; 8 - 6 = 2
9 + 8 = 17; 9 - 8 = 1
>>>
Asynchronous map:
Python multiprocessing - tracking the process of pool.map operation
pathos:
Can't pickle <type 'instancemethod'> when using python's multiprocessing Pool.map()
pathos:
What can multiprocessing and dill do together?
We still can't run your code… but if you post code that can be run, it might be more possible help edit your code (using the pathos fork and the asynchronous map or otherwise).
FYI: A release for pathos is a little bit overdue (i.e. late), so if you want to try it, it's best to get the code here: https://github.com/uqfoundation
My question is fairly basic yet might need a challenging solution.
Essentially, I have an arbitrary function which we will call some_function.
def some_function(n):
for i in range(n):
i+i
r = 1
r = r+1
And I want to count the number of operations took place in an arbitrary call to this function is executed (e.g. some_function(5). there are 7 operations that took place).
How would one count the number of operations that took place within a function call? I cannot modify some_function.
I think you're really after what others already told you - the big O notation.
But if you really want to know the actual number of instructions executed you can use this on linux:
perf stat -e instructions:u python yourscript.py
Which will output:
Performance counter stats for 'python yourscript.py':
22,260,577 instructions:u
0.014450363 seconds time elapsed
Note though that it includes all the instructions for executing python itself. So you'd have to find your own reference.
Using byteplay:
Example:
#!/usr/bin/env python
from byteplay import Code
def some_function(n):
for i in range(n):
i + i
r = 1
r = r + 1
def no_of_bytecode_instructions(f):
code = Code.from_code(f.func_code)
return len(code.code)
print(no_of_bytecode_instructions(some_function))
Output:
$ python -i foo.py
28
>>>
NB:
This still gives you no idea how complex f is here.
"Number of Instructions" != "Algorithm Complexity" (not by itself)
See: Big O
Algorithm complexity is a measure of the no. of instructions executed
relative to the size of your input data set(s).
Some naive examples of "complexity" and Big O:
def func1(just_a_list):
"""O(n)"""
for i in just_a_list:
...
def func2(list_of_lists):
"""O(n^2)"""
for i in list_of_lsits:
for j in i:
...
def func3(a_dict, a_key):
"""O(1)"""
return a_dict[a_key]
With reference to the following link: http://docs.python.org/faq/library.html#what-kinds-of-global-value-mutation-are-thread-safe
I wanted to know if the following:
(x, y) = (y, x)
will be guaranteed atomic in cPython. (x and y are both python variables)
Let's see:
>>> x = 1
>>> y = 2
>>> def swap_xy():
... global x, y
... (x, y) = (y, x)
...
>>> dis.dis(swap_xy)
3 0 LOAD_GLOBAL 0 (y)
3 LOAD_GLOBAL 1 (x)
6 ROT_TWO
7 STORE_GLOBAL 1 (x)
10 STORE_GLOBAL 0 (y)
13 LOAD_CONST 0 (None)
16 RETURN_VALUE
It doesn't appear that they're atomic: the values of x and y could be changed by another thread between the LOAD_GLOBAL bytecodes, before or after the ROT_TWO, and between the STORE_GLOBAL bytecodes.
If you want to swap two variables atomically, you'll need a lock or a mutex.
For those desiring empirical proof:
>>> def swap_xy_repeatedly():
... while 1:
... swap_xy()
... if x == y:
... # If all swaps are atomic, there will never be a time when x == y.
... # (of course, this depends on "if x == y" being atomic, which it isn't;
... # but if "if x == y" isn't atomic, what hope have we for the more complex
... # "x, y = y, x"?)
... print 'non-atomic swap detected'
... break
...
>>> t1 = threading.Thread(target=swap_xy_repeatedly)
>>> t2 = threading.Thread(target=swap_xy_repeatedly)
>>> t1.start()
>>> t2.start()
>>> non-atomic swap detected
Yes, yes it will.
I stand corrected.
Kragen Sitaker
writes:
Someone recommended using the idiom
spam, eggs = eggs, spam
to get a thread-safe swap. Does this really work? (...)
So if this thread loses control anywhere between the first LOAD_FAST
and the last STORE_FAST, a value could get stored by another thread
into "b" which would then be lost. There isn't anything keeping this
from happening, is there?
Nope. In general not even a simple
assignment is necessarily thread safe
since performing the assignment may
invoke special methods on an object
which themselves may require a number
of operations. Hopefully the object
will have internally locked its
"state" values, but that's not always
the case.
But it's really dictated by what
"thread safety" means in a particular
application, because to my mind there
are many levels of granularity of such
safety so it's hard to talk about
"thread safety". About the only thing
the Python interpreter is going to
give you for free is that a built-in
data type should be safe from internal
corruption even with native threading.
In other words if two threads have
a=0xff and a=0xff00, a will end up
with one or the other, but not
accidentally 0xffff as might be
possible in some other languages if a
isn't protected.
With that said, Python also tends to
execute in such a fashion that you can
get away with an awful lot without
formal locking, if you're willing to
live on the edge a bit and have
implied dependencies on the actual
objects in use. There was a decent
discussion along those lines here in
c.l.p a while back - search
groups.google.com for the "Critical
sections and mutexes" thread among
others.
Personally, I explicitly lock shared
state (or use constructs designed for
exchanging shared information properly
amongst threads, such as Queue.Queue)
in any multi-threaded application. To
my mind it's the best protection
against maintenance and evolution down
the road.
--
-- David
Python atomic for shared data types.
https://sharedatomic.top
The module can be used for atomic operations under multiple processs and multiple threads conditions. High performance python! High concurrency, High performance!
atomic api Example with multiprocessing and multiple threads:
You need the following steps to utilize the module:
create function used by child processes, refer to UIntAPIs, IntAPIs, BytearrayAPIs, StringAPIs, SetAPIs, ListAPIs, in each process, you can create multiple threads.
def process_run(a):
def subthread_run(a):
a.array_sub_and_fetch(b'\x0F')
threadlist = []
for t in range(5000):
threadlist.append(Thread(target=subthread_run, args=(a,)))
for t in range(5000):
threadlist[t].start()
for t in range(5000):
threadlist[t].join()
create the shared bytearray
a = atomic_bytearray(b'ab', length=7, paddingdirection='r', paddingbytes=b'012', mode='m')
start processes / threads to utilize the shared bytearray
processlist = []
for p in range(2):
processlist.append(Process(target=process_run, args=(a,)))
for p in range(2):
processlist[p].start()
for p in range(2):
processlist[p].join()
assert a.value == int.to_bytes(27411031864108609, length=8, byteorder='big')