I am trying to scan over iterable properties of n objects. I am looking for a pythonic way to perform functions in nested loops of arbitrary depth by passing functions to method calls of the loop one level up. I haven't been able to get more than the most inner loop to run when the depth is 3. Here is a non-working python pseudo code where I am querying a different value at each point in the loops. The other difficulty is I am trying to capture the output and pass it to the next outer loop
class Parent(object):
def __init__(self):
self.iterable = [None] * 2
self.result = self.iterable[:]
def loop(self, query_func):
def innerloop():
for i, x in enumerate(self.iterable):
self.result[i] = query_func(x)
return self.result[:]
return innerloop
class ChildA(Parent):
def __init___(self, A, object_to_queryA):
self.iterableA = [valueA for valueA in range(A)]
self.resultA = self.iterableA[:]
self.object_to_query = object_to_queryA
def query_valueA(self, x):
return self.object_to_query.some_query_function(x)
class ChildB(Parent):
def __init___(self, B, object_to_queryB):
self.iterableB = [valueB for valueB in range(B))]
self.resultB = self.iterableB[:]
self.object_to_query = object_to_queryB
def query_valueB(self, x):
return self.object_to_query.some_other_query_function(x)
class ChildC(Parent):
def __init___(self, C, , object_to_queryC):
self.iterableC = [valueC for valueC in range(C))]
self.resultC = self.iterableC[:]
self.object_to_query = object_to_queryC
def query_valueC(self, x):
return self.object_to_query.yet_another_query_function(x)
I want to be able to call these loops as follows:
import numpy
query_objA, query_objB, query_objC = (SomeObjA(), SomeObjB(), SomeObjC())
A, B, C = (len(query_objA.data), len(query_objB.data), len(query_objC.data))
instA = ChildA(A, query_objA)
instB = ChildB(B, query_objB)
instC = ChildC(C, query_objC)
my_scanning_func = ChildA.loop(ChildB.loop(ChildC.loop))
my_queries = numpy.array(my_scanning_func()).reshape(A,B,C)
# Equally valid call example below:
my_scanning_func2 = ChildB.loop(ChildC.loop(ChildA.loop))
my_queries2 = numpy.array(my_scanning_func2()).reshape(B,C,A)
The ultimate functionality im looking for would be similar to below, but for arbitrary depth and order:
for i, x in enumerate(query_objA.data):
response[i] = instA.some_query_function(x)
for j, y in enumerate(query_objB.data):
response[i][j] = instB.some_other_query_function(y)
for k, z in enumerate(query_objC.data):
response[i][j][k] = instC.yet_another_query_function(z)
Bonus points if this can be done via an inherited recursive function, rather than defining separate looping methods for each child, as I tried to do above. Last Note: I am trying to write Python 2.7 compatible code. Thanks in advance!
After much discussion with the OP I have a better idea of how you could generalize the construction of these arrays, first it seems that your objects would be designed to both iterate over predefined states or query the present state (possibly with only one of these being valid) so the iterface for object would be abstracted to something like this:
class Apparatus_interface:
def __init__(self,*needed_stuff):
#I have no idea how you are actually interacting with the device
self._device = SET_UP_OBJECT(needed_stuff)
#when iterating over this object we need to know how many states there are
#so we can predefine the shape (dimensions) of our arrays
self.num_of_states = 5
#it would make sense for each object to define
#the type of value that .query() returns (following spec of numpy's dtype)
self.query_type = [('f1', float), ('f2', float)]
def __iter__(self):
"""iterates over the physical positions/states of the apperatus
the state of the device is only active in between iterations
* calling list(device) doesn't give you any useful information, just a lot of mechanical work
"""
for position in range(self.num_of_states):
# ^ not sure what this should be either, you will have a better idea
self._device.move_to(position) #represents a physical change in the device
yield position #should it generate different information?
def query(self):
return self._device.query()
with this interface you would generate your array by iterating (nested loop) over a number of devices and at each combination of states between them you query the state of another device (and record that value into an array)
Normally you'd be able to use itertools.product to generate the combinations of states of the devices however due to optimizations itertools.product would run the iteration code that affects the physical device before it is used in iteration, so you will need an implementation that does not apply this kind of optimization:
#values is a list that contains the current elements generated
#the loop: for values[depth] in iterables[depth] basically sets the depth-th element to each value in that level of iterable
def _product(iterables, depth, values):
if len(iterables)-depth == 1:
for values[depth] in iterables[depth]:
yield tuple(values)
else:
for values[depth] in iterables[depth]:
#yield from _product(iterables, depth+1, values)
for tup in _product(iterables, depth+1, values):
yield tup
def product(*iterables):
"""
version of itertools.product to activate side-effects of iteration
only works with iterables, not iterators.
"""
values = [None]*len(iterables)
return _product(iterables, 0, values)
Now for actually generating the array - first a process that iterates through the product of all states and makes a query at each one, note that states variable is unused as I'm going to assume the placement in the numpy array will be determined by the order the states get iterated not the values produced
def traverse_states(variable_devices, queried_device):
"""queries a device at every combination of variable devices states"""
for states in product(*variable_devices):
yield queried_device.query()
then the function to put the array together is quite strait forward:
def array_from_apparatus(variable_devices, queried_object, dtype=None):
# the # of states in each device <==> # of elements in each dimension
arr_shape = [device.num_of_states for device in variable_devices]
iterator = traverse_states(variable_devices, queried_object)
if dtype is None:
dtype = queried_object.query_type
array = numpy.fromiter(iterator, dtype=dtype)
array.shape = arr_shape #this will fail if .num_of_states doesn't match the actual number of iterations
return array
I'm not sure how I could make a decent test of this but I believe it would work or at least be close.
I'm not sure if this answers your question but I think it is at least relevant, if you want to generate a numpy array such that array[tup] = func(tup) where tup is a tuple of integer indices you could use itertools.product in combination with numpy.fromiter like this:
import itertools
#from itertools import imap as map #for python 2
import numpy
def array_from_func(dimensions, func, dtype=float):
ranges = (range(i) for i in dimensions) #ranges of indices for all dimensions
all_indices = itertools.product(*ranges) #will iterate over all locations regardless of # of dimensions
value_gen = map(func, all_indices) #produces each value for each location
array = numpy.fromiter(value_gen, dtype=dtype)
array.shape = dimensions #modify the shape in place, .reshape would work but makes a copy.
return array
This is useful to me to see how indices relate to the actual array output, here are three demos to show basic functionality (second one I figured out recently)
from operator import itemgetter
>>> array_from_func((2,3,4), itemgetter(1),int) #second index
array([[[0, 0, 0, 0],
[1, 1, 1, 1],
[2, 2, 2, 2]],
[[0, 0, 0, 0],
[1, 1, 1, 1],
[2, 2, 2, 2]]])
>>> def str_join(it):
return ",".join(map(str,it))
#the '<U5' in next line specifies strings of length 5, this only works when the string will actually be length 5
#changing to '<U%d'%len(str_join(dims)) would be more generalized but harder to understand
>>> print(array_from_func((3,2,7), str_join, '<U5'))
[[['0,0,0' '0,0,1' '0,0,2' '0,0,3' '0,0,4' '0,0,5' '0,0,6']
['0,1,0' '0,1,1' '0,1,2' '0,1,3' '0,1,4' '0,1,5' '0,1,6']]
[['1,0,0' '1,0,1' '1,0,2' '1,0,3' '1,0,4' '1,0,5' '1,0,6']
['1,1,0' '1,1,1' '1,1,2' '1,1,3' '1,1,4' '1,1,5' '1,1,6']]
[['2,0,0' '2,0,1' '2,0,2' '2,0,3' '2,0,4' '2,0,5' '2,0,6']
['2,1,0' '2,1,1' '2,1,2' '2,1,3' '2,1,4' '2,1,5' '2,1,6']]]
>>> array_from_func((3,4), sum) #the sum of the indices, not as useful but another good demo
array([[ 0., 1., 2., 3.],
[ 1., 2., 3., 4.],
[ 2., 3., 4., 5.]])
I think this is along the lines of what you are trying to accomplish but I'm not quite sure... please give me feedback if I can be more specific about what you need.
Related
I am trying to optimize the inner loop of my python code. I've been reading about map, and reduce, but struggle to apply these concepts to the following code, since it also contains a multiplication. My data structure looks like this
f.m: [NDArray[float]]
f.l: [NDArray[float]]
f.h: [NDArray[float]]
I have several of these in a list and I would like to calculate the sum for each array element (i.e., m, l, h) in the list. Right now, I use a loop to iterate through the list of arrays. This scenario can be done with map etc. However, each array also carries a sign (+1. vs -1). Is there a way to optimize this and keeping the sign separate?
f1 = type('test', (object, ), {})()
f2 = type('test', (object, ), {})()
f1.n = "f1"
f1.m = f1.l = [1, 2]
f2.n = "f2"
f2.m = f2.l = [2, 4]
flux_list = [f1, f2]
dirs = {"f1": -1, "f2": 1}
new = [0, 0]
i = 0 # set in outer loop
for f in flux_list:
direction = dirs[f.n]
new[0] += f.m[i] * direction
new[1] += f.l[i] * direction
print(new)
One way to avoid the above loop is to create the data structure outside of the object instances, and then use numpy.view() to create individual views on the data.
unlike np.r_, np.view() does not create a new copy of the data, so anything we do to the data is visible in all other views (similar to shared memory).
I.e., we can create a view where f1.m and f2.m appear to be adjacent to each other, and we can use np.sum on the view rather than a loop over each f1 to fn. While this involves a small overhead in creating the data structure, all of this can be done before we start iterating over the data.
Last but not least, the multiplication step can be moved outside the loop by a suitable grouping of f.m (I,.e.collect all f.m with positive sign into one group, and do the same for those with q negative sign).
The code below illustrates the principle:
import numpy as np
# create the data structure first
rand = np.random.RandomState(42)
A = rand.randint(10,size=25).reshape(5,5)
row = np.arange(5)
# create some empty object instances
f1 = type('test', (object, ), {})()
f2 = type('test', (object, ), {})()
# Assign views into A to each object
f1.m = A.view()[row[:, np.newaxis], 0]
f1.l = A.view()[row[:, np.newaxis], 3]
f2.m = A.view()[row[:, np.newaxis], 1]
f2.l = A.view()[row[:, np.newaxis], 2]
# Create view in such a way that we can directly sum the data
# in this case we want to add f1.m and f2.m
col = np.array([0,1])
C=A.view()[row[:, np.newaxis], col]
# do the sum over the view which will then also be
# reflected in A
np.sum(C,axis=1)
I have two numpy arrays "Elements" and "nodes". My aim is to gather some data of these arrays.
I need to replace "Elements" data of the two last columns by the two coordinates contains
in "nodes" array. The two arrays are very huge, I have to automate it.
This posts refers to an old one: Replace data of an array by 2 values of a second array
with a difference that arrays are very huge (Elements: (3342558,5) and nodes: (581589,4)) and the previous way out does not work.
An example :
import numpy as np
Elements = np.array([[1.,11.,14.],[2.,12.,13.]])
nodes = np.array([[11.,0.,0.],[12.,1.,1.],[13.,2.,2.],[14.,3.,3.]])
results = np.array([[1., 0., 0., 3., 3.],
[2., 1., 1., 2., 2.]])
The previous way out proposed by hpaulj
e = Elements[:,1:].ravel().astype(int)
n=nodes[:,0].astype(int)
I, J = np.where(e==n[:,None])
results = np.zeros((e.shape[0],2),nodes.dtype)
results[J] = nodes[I,:1]
results = results.reshape(2,4)
But with huge arrays, this script does not work:
DepreciationWarning: elementwise comparison failed; this will raise an error in the future...
Most of the game would be to figure out the corresponding matching indices from Elements in nodes.
Approach #1
Since it seems you are open to conversion to integer, let's assume we could take them as integers. With that, we could use an array-assignment + mapping based method, as shown below :
ar = Elements.astype(int)
a = ar[:,1:].ravel()
nd = nodes[:,0].astype(int)
n = a.max()+1
# for generalized case of neagtive ints in a or nodes having non-matching values:
# n = max(a.max()-min(0,a.min()), nd.max()-min(0,nd.min()))+1
lookup = np.empty(n, dtype=int)
lookup[nd] = np.arange(len(nd))
indices = lookup[a]
nc = (Elements.shape[1]-1)*(nodes.shape[1]-1) # 4 for given setup
out = np.concatenate((ar[:,0,None], nodes[indices,1:].reshape(-1,nc)),axis=1)
Approach #2
We could also use np.searchsorted to get those indices.
For nodes having rows sorted based on first col and matching case, we can simply use :
indices = np.searchsorted(nd, a)
For not-necessarily sorted case and matching case :
sidx = nd.argsort()
idx = np.searchsorted(nd, a, sorter=sidx)
indices = sidx[idx]
For non-matching case, use an invalid bool array :
invalid = idx==len(nd)
idx[invalid] = 0
indices = sidx[idx]
Approach #3
Another with concatenation + sorting -
b = np.concatenate((nd,a))
sidx = b.argsort(kind='stable')
n = len(nd)
v = sidx<n
counts = np.diff(np.flatnonzero(np.r_[v,True]))
r = np.repeat(sidx[v], counts)
indices = np.empty(len(a), dtype=int)
indices[sidx[~v]-n] = r[sidx>=n]
To detect non-matching ones, use :
nd[indices] != a
Port the idea here to numba :
from numba import njit
def numba1(Elements, nodes):
a = Elements[:,1:].ravel()
nd = nodes[:,0]
b = np.concatenate((nd,a))
sidx = b.argsort(kind='stable')
n = len(nodes)
ncols = Elements.shape[1]-1
size = nodes.shape[1]-1
dt = np.result_type(Elements.dtype, nodes.dtype)
nc = ncols*size
out = np.empty((len(Elements),1+nc), dtype=dt)
out[:,0] = Elements[:,0]
return numba1_func(out, sidx, nodes, n, ncols, size)
#njit
def numba1_func(out, sidx, nodes, n, ncols, size):
N = len(sidx)
for i in range(N):
if sidx[i]<n:
cur_id = sidx[i]
continue
else:
idx = sidx[i]-n
row = idx//ncols
col = idx-row*ncols
cc = col*size+1
for ii in range(size):
out[row, cc+ii] = nodes[cur_id,ii+1]
return out
Would you consider using pandas?
import pandas as pd
Elements = np.array([[1.,11.,14.],[2.,12.,13.]])
nodes = np.array([[11.,0.,0.],[12.,1.,1.],[13.,2.,2.],[14.,3.,3.]])
df_elements = pd.DataFrame(Elements,columns = ['idx','node1','node2'])
df_nodes = pd.DataFrame(nodes, columns = ['node_id','x','y'])
#Double merge to get the coordinates from df_nodes
results = df_elements.merge(df_nodes, left_on = 'node1', right_on="node_id", how='left').merge(df_nodes, left_on="node2",right_on = "node_id", how='left')[['idx',"x_x",'y_x','x_y','y_y']].values
Output
array([[1., 0., 0., 3., 3.],
[2., 1., 1., 2., 2.]])
First, let's estimate the sizes of the arrays to see if we will encounter a memory error
from sys import getsizeof
Element_size = getsizeof(np.random.randint(0,100,(3342558,5))) / (1024**3)
nodes_size = getsizeof(np.random.randint(0,100,(581589,4))) / (1024**3)
result_size = getsizeof(np.random.randint(0,100,(3342558,13))) / (1024**3)
total_size = Element_size + nodes_size + result_size
Running this script (13=(5-1)*(4-1)+1), the total_size is about 0.46 GB, this means we don't need to worry too much about memory error, but we should still do our best to avoid making copies of an array.
We first create arrays to work with
elements = np.random.randint(0,100,(100,5))
elements[:,0] = np.arange(100)
nodes = np.random.randint(0,100,(300,4))
# create an empty result array
results = np.empty((100,13)).astype(elements.dtype)
results[:,:5] = elements
As you can see, we create the array results in the first place, there are two benefits to create this array at the beginning:
Most operations can be in-place operations performed on results.
If the memory space is not sufficient, you will know this when you create results.
With these arrays, you can solve your problem with
aux_inds = np.arange(4)
def argmax_with_exception(row):
mask = row[1:5][:,None] == nodes[:,0]
indices = np.argmax(mask,axis=1)
node_slices = nodes[indices][:,1:]
# if a node in Element is not found in the array nodes
not_found = aux_inds[~np.any(mask,axis=1)]
node_slices[not_found] = np.ones(3) * -999
row[1:] = node_slices.flatten()
np.apply_along_axis(argmax_with_exception,1,results)
in which, if a node in Element is not found in nodes, its value will be assigned to (-999,-999,-999).
In this approach, np.apply_along_axis(argmax_with_exception,1, results) will perform an in-place operation on the array results, therefore, it is unlikely you will run into memory error as long as the arrays can be created in the first place. If, however, the machine you are working with has a very small RAM, you can save the array Elements to disk in the first place, then load it into results with results[:,:5] = np.load('Elements.npy')
In order to understand the pythonic solution first look at the solution provided by sgnfis on the old post:
Old solution
import numpy as np
# I used numpy 1.10.1 here
Elements = np.array([[1.,11.,14.],[2.,12.,13.]])
nodes = np.array([[11.,0.,0.],[12.,1.,1.],[13.,2.,2.],[14.,3.,3.]])
# Create an array with enough rows and five columns
res = np.zeros((np.shape(Elements)[0],5))
for i in range(np.shape(Elements)[0]):
res[i,0] = Elements[i,0] # The first column stays the same
# Find the Value of the 2nd column of Elements in the first column of nodes.
nodesindex = np.where(nodes[:,0]==Elements[i,1])
# Replace second and third row of the results with the ventries from nodes.
res[i,1:3]=nodes[nodesindex,1:3]
#Do the same for the 3rd column of Elements
nodesindex = np.where(nodes[:,0]==Elements[i,2])
res[i,3:5]=nodes[nodesindex,1:3]
print(res)
The above solution is now turned into pythonic solution as given below:
New Solution:
import numpy as np
Elements = np.array([[1.,11.,14.],[2.,12.,13.]])
nodes = np.array([[11.,0.,0.],[12.,1.,1.],[13.,2.,2.],[14.,3.,3.]])
# Create an array with enough rows and five columns
res = np.zeros((np.shape(Elements)[0],5))
res[:,0] = Elements[:,0] # The first column stays the same
res[:,1:3]=[nodes[np.where(nodes[:,0]==Elements[i,1]),1:3] for i in range(np.shape(Elements)[0])]
res[:,3:5]=[nodes[np.where(nodes[:,0]==Elements[i,2]),1:3] for i in range(np.shape(Elements)[0])]
print(res)
this involves a problem that I encountered when try to solve a linked-list reverse problem.
First let me put some preliminary codes for the definition of the linked list and quick method to generate a linked list:
class ListNode:
def __init__(self, x):
self.val = x
self.next = None
def __repr__(self):
if self.next:
return "{}->{}".format(self.val, repr(self.next))
else:
return "{}".format(self.val)
def genNode(*nodes, end=None):
if len(nodes) == 1 and type(nodes[0]) == list:
nodes = nodes[0]
for i in nodes[::-1]:
n = ListNode(i)
n.next, end = end, n
return n if nodes else None
The problem I have is that I found the swapping mechanism is still depending on the sequence of the variable that I write.
Originally when we talk about swapping values in python we can do:
a, b = b, a
and it should work the same way if I have
b, a = a, b
This reverse linked list method that I am trying to write has 3 variables swapping, the idea is simple, to create a dummy head, and consistently adding nodes between dummy and dummy.next, so that it can be reversed.
def rev(head):
dummy = ListNode('X')
while head:
dummy.next, head.next, head = head, dummy.next, head.next
return dummy.next
a = genNode(1,2,3,4)
print(rev(a)) # >>> 4->3->2->1
But If I slightly switch the sequence of the 3 variables:
def rev2(head):
dummy = ListNode('X')
while head:
dummy.next, head, head.next, = head, head.next, dummy.next,
return dummy.next
a = genNode(1,2,3,4)
print(rev2(a)) # >>> AttributeError: 'NoneType' object has no attribute 'next'
So it does seem like that the sequence matters here, and can anyone let me know how python evaluate swapping values if there is more than 2 variables.
Thanks!
left to right
Look at https://docs.python.org/3/reference/simple_stmts.html#assignment-statements
CPython implementation detail: In the current implementation, the syntax for targets is taken to be the same as for expressions, and invalid syntax is rejected during the code generation phase, causing less detailed error messages.
Although the definition of assignment implies that overlaps between the left-hand side and the right-hand side are ‘simultaneous’ (for example a, b = b, a swaps two variables), overlaps within the collection of assigned-to variables occur left-to-right, sometimes resulting in confusion. For instance, the following program prints [0, 2]:
x = [0, 1]
i = 0
i, x[i] = 1, 2 # i is updated, then x[i] is updated
print(x)
A simple example below should show you the caveat of using swapping for a class like ListNode
Let's define a 3 element linked list.
a = ListNode(1)
b = ListNode(2)
c = ListNode(3)
a.next = b
b.next = c
print(a)
#1->2->3
Now if we swap say b and c, it won't have any effect
b,c = c,b
print(a)
#1->2->3
If we swap a and b, the linked list changes.
a,b=b,a
print(a)
#2->3
Similarly for a and c swap.
a,c=c,a
print(a)
#3
So you can see that using the simple swap logic is inconsistent in how it applies to a ListNode, hence should be avoided.
Interesting discussion, kind of extending the answer above, I created this new example below.
x = [1, 0]
i = 0
i, x[i] = x[i], i
print(i, x)
>> 1 [1, 0]
Let's go through step by step to see what is going one with i, x[i] = x[i], i.
Initially, all variables are at the previous stage, i.e., i=0, so x[i] is x[0]=1 for both sides. We have 0, x[0] = x[0], 0 or 0, 1 = 1, 0
The exchange/assignment starts from left to right. The left part of comma,i = x[i] takes place first, that is i = 1, i value changes from 0 to 1.
Importantly, when the right part of comma takes place, the value of i already changed. We are actually looking at, 1, x[1] = 1, 0, the confusing part is that the i on the right will not change and its value is still 0, not the new value, 1, x[1] = 1, i. Thus, the final state is, 1, x[1] = 1, 0.
If there are more than two variables it works identically as with two. You put them in the desired final order:
>>> a = 1
>>> b = 2
>>> c = 3
>>> c,b,a = a,b,c
>>> a,b,c
(3, 2, 1)
In my code I use the following structure to avoid conditions in a for loop:
if patch_type == "zeros":
patch_fct = np.zeros
elif patch_type == "ones":
patch_fct = np.ones
elif patch_type == "rand":
patch_fct = np.random.random_sample
else:
raise "Error"
for k in range(10**9):
m, n = comp_size()
bla = patch_fct((m,n))
where patch_fct can be easily used with tupels.
Now I want to use the same approach to create a patch_fct that takes a tupel and returns uniformly distributed random numbers between -1 and 1. How can I do that?
I would like to do something like:
patch_fct = 2. * (np.random.random_sample - 0.5)
The approach from above does not seem to be the right one.
If you need a function that does not already exist, you can just define it and use its name from now on.
For example:
if foo:
def patch_fct(tup):
return 2*(np.random.random_sample(tup) - 0.5)
elif bar:
def patch_fct(tup):
# do something else
else:
patch_fct = another_existing_function
The chain of ifs and elses can be written a bit more smoothly with the help of a dictionary.
For your original code, you could write
patch_functions = {'zeros': np.zeros,
'ones': np.ones,
'rand': np.random.random_sample}
and then use it like this:
>>> patch_functions['zeros'](5)
>>> array([0., 0., 0., 0., 0.])
This will automatically throw a KeyError if you are trying to access a key that does not exist in the dictionary.
You can also put self-defined functions inside a dictionary, either by defining them prior to insertion or using anonymous lambda functions. Demo:
>>> def fun1(tup):
...: return sum(tup) + 1
>>>
>>> my_functions = {'my_sum': fun1, 'my_random': lambda tup: 2*(np.random.random_sample(tup) - 0.5)}
>>> my_functions['my_sum']((2, 5))
>>> 8
>>> my_functions['my_random']((2, 5))
>>>
array([[-0.20203832, -0.23868021, 0.72052191, 0.72931098, -0.57160796],
[-0.45117601, -0.95461634, -0.52232593, -0.24011216, -0.83875935]])
numpy provides such a function explicitly:
numpy.random.uniform(low=-1, high=1, size=None)
size is the amount of times to draw - this can be a tuple stating the dimensions of the resulting array. size=(10,10) will yield a 10x10 matrix.
If I understand you correctly then:
def patch_fct(size):
return numpy.random.uniform(low=-1, high=1, size=size)
and size can be a tuple (or not).
In general searching numpy with some math/probability thing will yield the correct answer on the first hit.
I'm rather new to NumPy. Anyone have an idea for making this code, especially the nested loops, more compact/efficient? BTW, dist and data are three-dimensional numpy arrays.
def interpolate_to_distance(self,distance):
interpolated_data=np.ndarray(self.dist.shape[1:])
for j in range(interpolated_data.shape[1]):
for i in range(interpolated_data.shape[0]):
interpolated_data[i,j]=np.interp(
distance,self.dist[:,i,j],self.data[:,i,j])
return(interpolated_data)
Thanks!
Alright, I'll take a swag with this:
def interpolate_to_distance(self, distance):
dshape = self.dist.shape
dist = self.dist.T.reshape(-1, dshape[-1])
data = self.data.T.reshape(-1, dshape[-1])
intdata = np.array([np.interp(distance, di, da)
for di, da in zip(dist, data)])
return intdata.reshape(dshape[0:2]).T
It at least removes one loop (and those nested indices), but it's not much faster than the original, ~20% faster according to %timeit in IPython. On the other hand, there's a lot of (probably unnecessary, ultimately) transposing and reshaping going on.
For the record, I wrapped it up in a dummy class and filled some 3 x 3 x 3 arrays with random numbers to test:
import numpy as np
class TestClass(object):
def interpolate_to_distance(self, distance):
dshape = self.dist.shape
dist = self.dist.T.reshape(-1, dshape[-1])
data = self.data.T.reshape(-1, dshape[-1])
intdata = np.array([np.interp(distance, di, da)
for di, da in zip(dist, data)])
return intdata.reshape(dshape[0:2]).T
def interpolate_to_distance_old(self, distance):
interpolated_data=np.ndarray(self.dist.shape[1:])
for j in range(interpolated_data.shape[1]):
for i in range(interpolated_data.shape[0]):
interpolated_data[i,j]=np.interp(
distance,self.dist[:,i,j],self.data[:,i,j])
return(interpolated_data)
if __name__ == '__main__':
testobj = TestClass()
testobj.dist = np.random.randn(3, 3, 3)
testobj.data = np.random.randn(3, 3, 3)
distance = 0
print 'Old:\n', testobj.interpolate_to_distance_old(distance)
print 'New:\n', testobj.interpolate_to_distance(distance)
Which prints (for my particular set of randoms):
Old:
[[-0.59557042 -0.42706077 0.94629049]
[ 0.55509032 -0.67808257 -0.74214045]
[ 1.03779189 -1.17605275 0.00317679]]
New:
[[-0.59557042 -0.42706077 0.94629049]
[ 0.55509032 -0.67808257 -0.74214045]
[ 1.03779189 -1.17605275 0.00317679]]
I also tried np.vectorize(np.interp) but couldn't get that to work. I suspect that would be much faster if it did work.
I couldn't get np.fromfunction to work either, as it passed (2) 3 x 3 (in this case) arrays of indices to np.interp, the same arrays you get from np.mgrid.
One other note: according the the docs for np.interp,
np.interp does not check that the x-coordinate sequence xp is increasing. If
xp is not increasing, the results are nonsense. A simple check for
increasingness is::
np.all(np.diff(xp) > 0)
Obviously, my random numbers violate the 'always increasing' rule, but you'll have to be more careful.