Develop Reference

How can I create a numba callable that has adjustable parameters?

I would like to create a numba-compiled python callable (a function that I can use in another Numba-compiled function) that has an internal array that I can adjust to influence the result of the function call. In pure python, this would correspond to a class with a __call__ method:
class Test:
def __init__(self, arr):
self.arr = arr
def __call__(self, idx):
res = 0
for i in idx:
res += self.arr[i]
return res
t = Test([0, 1, 2])
print(t([1, 2]))
t.arr = [1, 2, 3]
print(t([1, 2]))
which prints 3 and 5, respectively, so the result was different after I modified the internal array arr.
A literal translation to Numba using jitclass and numpy arrays looks like this
import numpy as np
import numba as nb
#nb.jitclass([('arr', nb.double[:])])
class Test:
def __init__(self, arr):
self.arr = arr.astype(np.double)
def __call__(self, idx):
res = 0
for i in idx:
res += self.arr[i]
return res
t = Test(np.arange(3))
print(t(np.array([1, 2])))
t.arr = np.arange(3) + 1
print(t(np.array([1, 2])))
Unfortunately, this fails with TypeError: 'Test' object is not callable, since Numba does not seem to support __call__, yet.
I then tried to solve the problem using closures
import numpy as np
import numba as nb
arr = np.arange(5)
#nb.jit
def call(idx):
res = 0
for i in idx:
res += arr[i]
return res
print(call(np.array([1, 2])))
arr += 1
print(call(np.array([1, 2])))
but this prints 3 twice, since closures copy the data in arr into an internal representation, which I then cannot (easily?) change from the outside.
I even tried to trick Numba, by using ctypes pointers on Numpy arrays I combination with numba.carray, but Numba still seems to copy the data, so I cannot manipulate it.
I understand that Numba wants to control the memory and avoid access to memory regions that might not be used anymore. However, I have a specific use case where I would like to avoid passing around the extra array arr and rather adjust the internal copy somehow. Is there any way to achieve this?
EDIT:
I tried the suggestion by Daniel in the comments to use a method different than __call__, but this also does not work. Here is what I thought might work:
#nb.jitclass([('arr', nb.double[:])])
class Test:
def __init__(self, arr):
self.arr = arr
def call(self, idx):
return self.arr[idx]
a = Test(np.arange(5).astype(np.double))
print(a.call(3))
a.arr += 1
print(a.call(3))
#nb.njit
def rhs(idx):
return a.call(idx)
rhs(3)
This prints 3 and 4, so the array arr can indeed be manipulated. However, using the instance a in a compiled method fails with a NotImplementedError, so I suspect this use case is not (yet) supported by Numba.

Divide the problem in two parts, a numba function and a pure python class:
import numpy as np
import numba
#numba.jit
def calc(arr, idx):
res = 0
for i in idx:
res += arr[i]
return res
class Test:
def __init__(self, arr):
self.arr = arr.astype(np.double)
def __call__(self, idx):
return calc(self.arr, idx)
t = Test(np.arange(3))
print(t(np.array([1, 2])))
t.arr = np.arange(3) + 1
print(t(np.array([1, 2])))

I believe you need #property before the methods of the class but this may not be the only issue
#nb.jitclass([('arr', nb.double[:])])
class Test:
def __init__(self, arr):
self.arr = arr
#property
def call(self, idx):
return self.arr[idx]
a = Test(np.arange(5).astype(np.double))
print(a.call(3))
a.arr += 1
print(a.call(3))
#nb.njit
def rhs(idx):
return a.call(idx)
rhs(3)

This effect is the result of nopython compilation. If your goal is to create such callable at any costs, even possibly without taking benefits from jit-compilation - object compilation mode is a simple solution for your problem. This may be acheived in your closure example code simply by providing forceobj=True parameter to #nb.jit decorator.
This code prints 3 and 5 respectively:
import numpy as np
import numba as nb
arr = np.arange(5)
#nb.jit(forceobj=True)
def call(idx):
res = 0
for i in idx:
res += arr[i]
return res
print(call(np.array([1, 2])))
arr += 1
print(call(np.array([1, 2])))

Fastest way to sort in Python (no cython)

I have a problem where I've to sort a very big array(shape - 7900000X4X4) with a custom function. I used sorted but it took more than 1 hour to sort. My code was something like this.
def compare(x,y):
print('DD '+str(x[0]))
if(np.array_equal(x[1],y[1])==True):
return -1
a = x[1].flatten()
b = y[1].flatten()
idx = np.where( (a>b) != (a<b) )[0][0]
if a[idx]<0 and b[idx]>=0:
return 0
elif b[idx]<0 and a[idx]>=0:
return 1
elif a[idx]<0 and b[idx]<0:
if a[idx]>b[idx]:
return 0
elif a[idx]<b[idx]:
return 1
elif a[idx]<b[idx]:
return 1
else:
return 0
def cmp_to_key(mycmp):
class K:
def __init__(self, obj, *args):
self.obj = obj
def __lt__(self, other):
return mycmp(self.obj, other.obj)
return K
tblocks = sorted(tblocks.items(),key=cmp_to_key(compare))
This worked but I want it to complete in seconds. I don't think any direct implementation in python can give me the performance I need, so I tried cython. My Cython code is this, which is pretty simple.
cdef int[:,:] arrr
cdef int size
cdef bool compare(int a,int b):
global arrr,size
cdef int[:] x = arrr[a]
cdef int[:] y = arrr[b]
cdef int i,j
i = 0
j = 0
while(i<size):
if((j==size-1)or(y[j]<x[i])):
return 0
elif(x[i]<y[j]):
return 1
i+=1
j+=1
return (j!=size-1)
def sorted(np.ndarray boxes,int total_blocks,int s):
global arrr,size
cdef int i
cdef vector[int] index = xrange(total_blocks)
arrr = boxes
size = s
sort(index.begin(),index.end(),compare)
return index
This code in cython took 33 seconds! Cython is the solution, but I am looking for some alternate solutions which can run directly on python. For example numba. I tried Numba, but I didn't get satisfying results. Kindly help!

It is hard to give an answer without a working example. I assume, that arrr in your Cython code was a 2D-array and I assume that size was size=arrr.shape[0]
Numba Implementation
import numpy as np
import numba as nb
from numba.targets import quicksort
def custom_sorting(compare_fkt):
index_arange=np.arange(size)
quicksort_func=quicksort.make_jit_quicksort(lt=compare_fkt,is_argsort=False)
jit_sort_func=nb.njit(quicksort_func.run_quicksort)
index=jit_sort_func(index_arange)
return index
def compare(a,b):
x = arrr[a]
y = arrr[b]
i = 0
j = 0
while(i<size):
if((j==size-1)or(y[j]<x[i])):
return False
elif(x[i]<y[j]):
return True
i+=1
j+=1
return (j!=size-1)
arrr=np.random.randint(-9,10,(7900000,8))
size=arrr.shape[0]
index=custom_sorting(compare)
This gives 3.85s for the generated testdata. But the speed of a sorting algorithm heavily depends on the data....
Simple Example
import numpy as np
import numba as nb
from numba.targets import quicksort
#simple reverse sort
def compare(a,b):
return a > b
#create some test data
arrr=np.array(np.random.rand(7900000)*10000,dtype=np.int32)
#we can pass the comparison function
quicksort_func=quicksort.make_jit_quicksort(lt=compare,is_argsort=True)
#compile the sorting function
jit_sort_func=nb.njit(quicksort_func.run_quicksort)
#get the result
ind_sorted=jit_sort_func(arrr)
This implementation is about 35% slower than np.argsort, but this is also common in using np.argsort in compiled code.

If I understand your code correctly then the order you have in mind is the standard order, only that it starts at 0 wraps around at +/-infinity and maxes out at -0. On top of that we have simple left-to-right lexicographic order.
Now, if your array dtype is integer, observe the following: Because of complement representation of negatives view-casting to unsigned int makes your order the standard order. On top of that, if we use big endian encoding, efficient lexicographic ordering can be achieved by view-casting to void dtype.
The code below shows that using a 10000x4x4 example that this method gives the same result as your Python code.
It also benchmarks it on a 7,900,000x4x4 example (using array, not dict). On my modest laptop this method takes 8 seconds.
import numpy as np
def compare(x, y):
# print('DD '+str(x[0]))
if(np.array_equal(x[1],y[1])==True):
return -1
a = x[1].flatten()
b = y[1].flatten()
idx = np.where( (a>b) != (a<b) )[0][0]
if a[idx]<0 and b[idx]>=0:
return 0
elif b[idx]<0 and a[idx]>=0:
return 1
elif a[idx]<0 and b[idx]<0:
if a[idx]>b[idx]:
return 0
elif a[idx]<b[idx]:
return 1
elif a[idx]<b[idx]:
return 1
else:
return 0
def cmp_to_key(mycmp):
class K:
def __init__(self, obj, *args):
self.obj = obj
def __lt__(self, other):
return mycmp(self.obj, other.obj)
return K
def custom_sort(a):
assert a.dtype==np.int64
b = a.astype('>i8', copy=False)
return b.view(f'V{a.dtype.itemsize * a.shape[1]}').ravel().argsort()
tblocks = np.random.randint(-9,10, (10000, 4, 4))
tblocks = dict(enumerate(tblocks))
tblocks_s = sorted(tblocks.items(),key=cmp_to_key(compare))
tblocksa = np.array(list(tblocks.values()))
tblocksa = tblocksa.reshape(tblocksa.shape[0], -1)
order = custom_sort(tblocksa)
tblocks_s2 = list(tblocks.items())
tblocks_s2 = [tblocks_s2[o] for o in order]
print(tblocks_s == tblocks_s2)
from timeit import timeit
data = np.random.randint(-9_999, 10_000, (7_900_000, 4, 4))
print(timeit(lambda: data[custom_sort(data.reshape(data.shape[0], -1))],
number=5) / 5)
Sample output:
True
7.8328493310138585

Inheriting list: Creating division by other lists, integers and floats

I wanted to be able to divide entire lists by integers, floats, and other lists of equal length in Python, so I wrote the following little script.
class divlist(list):
def __init__(self, *args, **kwrgs):
super(divlist, self).__init__(*args, **kwrgs)
self.__cont_ = args[0]
self.__len_ = len(args[0])
def __floordiv__(self, other):
""" Adds the ability to floor divide list's indices """
if (isinstance(other, int) or isinstance(other, float)):
return [self.__cont_[i] // other \
for i in xrange(self.__len_)]
elif (isinstance(other, list)):
return [self.__cont_[i] // other[i] \
for i in xrange(self.__len_)]
else:
raise ValueError('Must divide by list, int or float')
My question: How can I write this in a simpler way? Do I really need the lines self.__cont_ and self.__len_? I was looking through the list's 'magic' methods and I couldn't find one that readily held this information.
An example of calling this simple class:
>>> X = divlist([1,2,3,4])
[1, 2, 3, 4]
>>> X // 2
[0, 1, 1, 2]
>>> X // [1,2,3,4]
[1, 1, 1, 1]
>>> X // X
[1, 1, 1, 1]

How can I write this in a simpler way?
By using self[i] instead of self.__cont_[i].
Do I really need the lines self.__cont_ and self.__len_?
No. Just use the regular methods of referring to a list, for example: [] and len().
As an aside, you might choose to have .__floordiv__() return a divlist instead of a list, so that you can continue to operate on the result.
class divlist(list):
def __floordiv__(self, other):
""" Adds the ability to floor divide list's indices """
if (isinstance(other, int) or isinstance(other, float)):
return [i // other for i in self]
elif (isinstance(other, list)):
# DANGER: data loss if len(other) != len(self) !!
return [i // j for i,j in zip(self, other)]
else:
raise ValueError('Must divide by list, int or float')
X = divlist([1,2,3,4])
assert X == [1, 2, 3, 4]
assert X // 2 == [0, 1, 1, 2]
assert X // [1,2,3,4] == [1, 1, 1, 1]
assert X // X == [1, 1, 1, 1]

Instead of examining the explicit types of each argument, assume that either the second argument is iterable, or it is a suitable value as the denominator for //.
def __floordiv__(self, other):
try:
pairs = zip(self, other)
except TypeError:
pairs = ((x, other) for x in self)
return [x // y for (x, y) in pairs]
You may want to check that self and other have the same length if the zip succeeds.

Multiply scipy.lti transfer functions

I have the following scipy.lti object that is basically an object representing a Laplace transform of an LTI system:
G_s = lti([1], [1, 2])
How to multiply such a transfer function with another one, such as i.e.:
H_s = lti([2], [1, 2])
#I_s = G_s * H_s <---- How to multiply this properly?
I guess I could do
I_s = lti(np.polymul([1], [2]), np.polymul([1, 2], [1, 2]))
But what if I want to do:
#I_s = H_s / (1 + H_s) <---- Does not work since H_s is an lti object
Is there an easy way to do this with scipy?

Interestingly, Scipy does not seem to supply that functionality. An alternative is converting the LTI system into a Sympy rational function. Sympy allows you to easily expand and cancel polynomials:
from IPython.display import display
from scipy import signal
import sympy as sy
sy.init_printing() # LaTeX like pretty printing for IPython
def lti_to_sympy(lsys, symplify=True):
""" Convert Scipy's LTI instance to Sympy expression """
s = sy.Symbol('s')
G = sy.Poly(lsys.num, s) / sy.Poly(lsys.den, s)
return sy.simplify(G) if symplify else G
def sympy_to_lti(xpr, s=sy.Symbol('s')):
""" Convert Sympy transfer function polynomial to Scipy LTI """
num, den = sy.simplify(xpr).as_numer_denom() # expressions
p_num_den = sy.poly(num, s), sy.poly(den, s) # polynomials
c_num_den = [sy.expand(p).all_coeffs() for p in p_num_den] # coefficients
l_num, l_den = [sy.lambdify((), c)() for c in c_num_den] # convert to floats
return signal.lti(l_num, l_den)
pG, pH, pGH, pIGH = sy.symbols("G, H, GH, IGH") # only needed for displaying
# Sample systems:
lti_G = signal.lti([1], [1, 2])
lti_H = signal.lti([2], [1, 0, 3])
# convert to Sympy:
Gs, Hs = lti_to_sympy(lti_G), lti_to_sympy(lti_H)
print("Converted LTI expressions:")
display(sy.Eq(pG, Gs))
display(sy.Eq(pH, Hs))
print("Multiplying Systems:")
GHs = sy.simplify(Gs*Hs).expand() # make sure polynomials are canceled and expanded
display(sy.Eq(pGH, GHs))
print("Closing the loop:")
IGHs = sy.simplify(GHs / (1+GHs)).expand()
display(sy.Eq(pIGH, IGHs))
print("Back to LTI:")
lti_IGH = sympy_to_lti(IGHs)
print(lti_IGH)
The output is:

Depending on your definition of "easy", you should consider deriving your own class from lti, implementing the necessary algebraic operations on your transfer functions. This is probably the most elegant approach.
Here's my take on the subject:
from __future__ import division
from scipy.signal.ltisys import TransferFunction as TransFun
from numpy import polymul,polyadd
class ltimul(TransFun):
def __neg__(self):
return ltimul(-self.num,self.den)
def __floordiv__(self,other):
# can't make sense of integer division right now
return NotImplemented
def __mul__(self,other):
if type(other) in [int, float]:
return ltimul(self.num*other,self.den)
elif type(other) in [TransFun, ltimul]:
numer = polymul(self.num,other.num)
denom = polymul(self.den,other.den)
return ltimul(numer,denom)
def __truediv__(self,other):
if type(other) in [int, float]:
return ltimul(self.num,self.den*other)
if type(other) in [TransFun, ltimul]:
numer = polymul(self.num,other.den)
denom = polymul(self.den,other.num)
return ltimul(numer,denom)
def __rtruediv__(self,other):
if type(other) in [int, float]:
return ltimul(other*self.den,self.num)
if type(other) in [TransFun, ltimul]:
numer = polymul(self.den,other.num)
denom = polymul(self.num,other.den)
return ltimul(numer,denom)
def __add__(self,other):
if type(other) in [int, float]:
return ltimul(polyadd(self.num,self.den*other),self.den)
if type(other) in [TransFun, type(self)]:
numer = polyadd(polymul(self.num,other.den),polymul(self.den,other.num))
denom = polymul(self.den,other.den)
return ltimul(numer,denom)
def __sub__(self,other):
if type(other) in [int, float]:
return ltimul(polyadd(self.num,-self.den*other),self.den)
if type(other) in [TransFun, type(self)]:
numer = polyadd(polymul(self.num,other.den),-polymul(self.den,other.num))
denom = polymul(self.den,other.den)
return ltimul(numer,denom)
def __rsub__(self,other):
if type(other) in [int, float]:
return ltimul(polyadd(-self.num,self.den*other),self.den)
if type(other) in [TransFun, type(self)]:
numer = polyadd(polymul(other.num,self.den),-polymul(other.den,self.num))
denom = polymul(self.den,other.den)
return ltimul(numer,denom)
# sheer laziness: symmetric behaviour for commutative operators
__rmul__ = __mul__
__radd__ = __add__
This defines the ltimul class, which is lti plus addition, multiplication, division, subtraction, and negation; binary ones also defined for integers and floats as partners.
I tested it for the example of Dietrich:
G_s = ltimul([1], [1, 2])
H_s = ltimul([2], [1, 0, 3])
print(G_s*H_s)
print(G_s*H_s/(1+G_s*H_s))
While GH is nicely equal to
ltimul(
array([ 2.]),
array([ 1., 2., 3., 6.])
)
the final result for GH/(1+GH) is less pretty:
ltimul(
array([ 2., 4., 6., 12.]),
array([ 1., 4., 10., 26., 37., 42., 48.])
)
Since I'm not very familiar with transfer functions, I'm not sure how likely it is that this gives the same result as the sympy-based solution due to some simplifications missing from this one. I find it suspicious that already lti behaves unexpectedly: lti([1,2],[1,2]) doesn't simplify its arguments, even though I'd suspect this function to be constant 1. So I'd rather not guess the correctness of this final result.
Anyway, the main message is inheritance itself, so possible bugs in the above implementation hopefully pose only a minor inconvenience. I'm also quite unfamiliar with class definitions, so it's possible that I didn't follow best practices in the above.
I eventually rewrote the above after #ochurlaud pointed out, that my original only worked for Python 2. The reason is that the / operation is implemented by __div__/__rdiv__ in Python 2 (and is the ambiguous "classical division"). In Python 3, however, there is a distinction between / (true division) and // (floor division), and they call __truediv__ and __floordiv__ (and their "right" counterparts), respectively. The __future__ import first in the line of the above code triggers the proper Python 3 behaviour even on Python 2, so the above works on both Python versions. Since floor (integer) division doesn't make much sense for our class, we explicitly signal that it can't do anything with // (unless the other operand implements it).
One could also easily define the respective __iadd__, __idiv__ etc. in-place operations for +=, /= etc., respectively.

How to modify arithmetic operators (+,-,x)

I am currently writing a Linear Algebra module for Python 3.x wherein I deal with self-defined matrix objects.
Is there any way I can make the basic arithmetic operators like +, -, * adhere to my matrix objects? For example -
>>> A = matrix("1 2;3 4")
>>> B = matrix("1 0; 0 1")
>>> A + B
[2 2]
[3 5]
>>> A * A
[7 10]
[15 22]
Right now I have written separate functions for addition, multiplication, etc. but typing A.multiply(A) is much more cumbersome than simply A*A.

You are looking for special methods. Particularly at emulating numerical types section.
Also, as you're trying to implement matrices and matrices are containers, you may find useful to define custom container methods for your type.
UPDATE: Here is an example of custom objects using special methods to implement arithmetical operators:
class Value(object):
def __init__(self, x):
self.x = x
def __add__(self, other):
if not isinstance(other, Value):
raise TypeError
return Value(self.x + other.x)
def __mul__(self, other):
if not isinstance(other, Value):
raise TypeError
return Value(self.x * other.x)
assert (Value(2) + Value(3)).x == 5
assert (Value(2) * Value(3)).x == 6

Unless you're doing this specifically to learn or for practice you should look at numerical python, numpy, which is the de facto standard solution for basic linear algebra and matrices. It has a matrix class which does what you are looking for.

You can override the built in methods for numerical types.
class matrix:
def __init__(self, string_matrix):
self.matrix = string_matrix.split(";")
for i in range(len(self.matrix)):
self.matrix[i] = map(int, self.matrix[i].split())
def __add__(self, other):
return_matrix = [[i for i in row] for row in self.matrix]
for i in range(len(self.matrix)):
for j in range(len(self.matrix[i])):
return_matrix[i][j] = self.matrix[i][j] + other.matrix[i][j]
return list_to_matrix(return_matrix)
def __str__(self):
return repr(self.matrix)
def list_to_matrix(list_matrix):
string_form = ""
for row in list_matrix:
for item in row:
if (item != row[-1]): string_form += str(item) + " "
else: string_form += str(item)
if (row != list_matrix[-1]): string_form += ";"
return matrix(string_form)
You will probably want to include a couple of checks (such as adding matrices of different dimensions, or adding a matrix to something that is not a matrix, etc) but this works as a simple example. Also notice that I'm returning a matrix object using the list_to_matrix() function - if that isn't the desired functionality you can change it pretty readily. You would use a similar process for all other arithmetic functions you need to implement.
Output:
>>> a = matrix("3 4;1 4")
>>> b = matrix("2 3;0 0")
>>> print(a)
[[3, 4], [1, 4]]
>>> print(b)
[[2, 3], [0, 0]]
>>> print(a + b)
[[5, 7], [1, 4]]
As mentioned in one of the other answers, numpy might be a good resource to use for your matrix operations - a lot of this functionality is already built in.
>>> import numpy as np
>>> a = np.matrix("3 4;1 4")
>>> b = np.matrix("2 3;0 0")
>>> print(a)
[[3 4]
[1 4]]
>>> print(b)
[[2 3]
[0 0]]
>>> print(a + b)
[[5 7]
[1 4]]

If you define some special methods within the class, Python will call them for arithmetic operations. An example class that define addition, multiplication, division, and subtraction:
class motar:
def __add__(self, other): return "9999"
def __mul__(self, other): return "8888"
def __sub__(self, other): return "7777"
def __div__(self, other): return "6666"
m = [ motar() for x in range(2) ] # create two instances
# arithmetic operations on class instances will call
# ... the special methods defined above:
print m[0] + m[1]
print m[0] * m[1]
print m[0] - m[1]
print m[0] / m[1]
Gives:
9999
8888
7777
6666