I am currently working on the following problem:
I want to write a function, that accepts comparisons as arguments - but want to prevent, that these comparisons are evaluated during runtime (leading to potentially unexpected results).
To be more precise, I have the following challenge (first, original one in pySpark, then similar, more general one):
def test_func(*comparisons):
for comparison in comparison:
left_hand = comparison[0]
comparison = comparison[1]
right_hand = comparison[2]
Example 1:
test_func(F.col('a') == F.col('b'))
left_hand -> F.col('a')
right_hand -> F.col('b')
comparison -> ==
Example 2:
test_func(1 <=> 2)
left_hand -> 1
right_hand -> 2
comparison -> <=>
Right now, the equation/parameter is evaluated before it reaches the function - i.e., I have problems splitting the equation into it individual parts.
Is this even possible to like this?
The python operator module stores operators as functions
from operator import *
def test_func(*comparison):
left_hand = comparison[0]
comparison = comparison[1]
right_hand = comparison[2]
test_func(F.col('a'), eq, F.col('b'))
The variables would be (remember they would still be local to test_func):
left_hand = F.col('a')
comparison = eq -> operator.eq
right_hand = F.col('b')
As a quick proof of concept:
>>> import operator
>>> class Col:
... def __init__(self, col):
... self.col = col
...
... def __eq__(self, other):
... return self, operator.eq, other
...
>>> Col('a') == Col('b')
(<__main__.Col object at 0x11134d5b0>, <built-in function eq>, <__main__.Col object at 0x11147cbe0>)
>>> lh, comp, rh = Col('a') == Col('b')
>>> comp(lh.col, rh.col)
False
You'll need to overload all special methods for all operators you want to support, and return the equivalent operator function (or whatever you want, perhaps '==', or a custom object).
I want to overload the operator # in python for a class I have written. I know how to do operator overloading in general (i.e. by defining __add__ and __radd__ to overload +) but I could not find a way to overload #.
Why I know, that # can be overloaded: for numpy arrays, A#B gives the matrix product of A and B, while A*B gives the Hadamard (element-wise) product.
The methods you need to overload are the __matmul__ and __rmatmul__ methods. E.g. if you want to add the inner product to lists:
class Vector(list):
def __matmul__(self, other):
return sum(x * y for x, y in zip(self, other))
def __rmatmul__(self, other):
return self.__matmul__(other)
Suppose x is a tensor in Pytorch. One can either write:
x_lowerthanzero = x.lt(0)
or:
x_lowerthanzero = (x<0)
with seemingly the exact same results. Many other operations have Pytorch built-in equivalents: x.gt(0) for (x>0), x.neg() for -x, x.mul() etc.
Is there a good reason to use one form over the other?
They are equivalent. < is simply a more readable alias.
Python operators have canonical function mappings e.g:
Algebraic operations
Operation
Syntax
Function
Addition
a + b
add(a, b)
Subtraction
a - b
sub(a, b)
Multiplication
a * b
mul(a, b)
Division
a / b
truediv(a, b)
Exponentiation
a ** b
pow(a, b)
Matrix Multiplication
a # b
matmul(a, b)
Comparisons
Operation
Syntax
Function
Ordering
a < b
lt(a, b)
Ordering
a <= b
le(a, b)
Equality
a == b
eq(a, b)
Difference
a != b
ne(a, b)
Ordering
a >= b
ge(a, b)
Ordering
a > b
gt(a, b)
You can check that these are indeed mapped to the respectively named torch functions here e.g:
def __lt__(self, other):
return self.lt(other)
Usually there is no reason for using one over the other, they are mostly for convenience: Many of those methods do have for instance an out argument, which lets you specify a tensor in which to save the result, but you can just as well do that using the operators instead of the methods.
Is there any way to use infix operators (like +,-,*,/) as higher order functions in python without creating "wrapper" functions?
def apply(f,a,b):
return f(a,b)
def plus(a,b):
return a + b
# This will work fine
apply(plus,1,1)
# Is there any way to get this working?
apply(+,1,1)
You can use the operator module, which has the "wrapper" functions written for you already.
import operator
def apply(f,a,b):
return f(a,b)
print apply(operator.add,1,1)
Result:
2
You can also define the wrapper using lambda functions, which saves you the trouble of a standalone def:
print apply(lamba a,b: a+b, 1, 1)
Use operator module and a dictionary:
>>> from operator import add, mul, sub, div, mod
>>> dic = {'+':add, '*':mul, '/':div, '%': mod, '-':sub}
>>> def apply(op, x, y):
return dic[op](x,y)
...
>>> apply('+',1,5)
6
>>> apply('-',1,5)
-4
>>> apply('%',1,5)
1
>>> apply('*',1,5)
5
Note that you can't use +, -, etc directly as they are not valid identifiers in python.
You can use the operator module this way:
import operator
def apply(op, a, b):
return op(a, b)
print(apply(operator.add, 1, 2))
print(apply(operator.lt, 1, 2))
Output:
3
True
The other solution is to use a lambda function, but "there should be one -- and preferably only one -- obvious way to do it", so I prefer to use the operator module
you can use anonymous function : apply(lambda x,y : x + y, 1,1)
# Is there any way to get this working?
apply(+,1,1)
No. As others have already mentioned, there are function forms of all of the operators in the operator module. But, you can't use the operators themselves as that is a SyntaxError and there is no way to dynamically change python's core syntax. You can get close though using dictionaries and passing strings:
_mapping = {'+':operator.add}
def apply(op,*args):
return _mapping[op](*args)
apply('+',1,1)
It is possible to give the operators +, -, *, and / special behavior for a class using magic methods, you can read about this here: http://www.rafekettler.com/magicmethods.html
This isn't exactly what you were asking for because this still requires the creation of a method for each operator, but it does allow you to use the operators by symbol in your code. Note that I don't think this is better than the other methods, it is just an illustration of how you can define behavior for operators:
class Prefix(object):
def __add__(self, other):
""" Prefix() + (a, b) == a + b """
return other[0] + other[1]
def __sub__(self, other):
""" Prefix() - (a, b) == a - b """
return other[0] - other[1]
def __mul__(self, other):
""" Prefix() * (a, b) == a * b """
return other[0] * other[1]
def __div__(self, other):
""" Prefix() / (a, b) == a / b """
return other[0] / other[1]
And examples:
>>> prefix = Prefix()
>>> prefix + (12, 3)
15
>>> prefix - (12, 3)
9
>>> prefix * (12, 3)
36
>>> prefix / (12, 3)
4
Of course this method can't be used for a more complex prefix equation like * / 6 2 5 because there is no way to define behavior for adjacent operators, which will always give a SyntaxError (except for a few special cases where + or - are interpreted as making the next element positive or negative).
Why I can't redefine the __and__ operator?
class Cut(object):
def __init__(self, cut):
self.cut = cut
def __and__(self, other):
return Cut("(" + self.cut + ") && (" + other.cut + ")")
a = Cut("a>0")
b = Cut("b>0")
c = a and b
print c.cut()
I want (a>0) && (b>0), but I got b, that the usual behaviour of and
__and__ is the binary (bitwise) & operator, not the logical and operator.
Because the and operator is a short-circuit operator, it can't be implemented as a function. That is, if the first argument is false, the second argument isn't evaluated at all. If you try to implement that as a function, both arguments have to be evaluated before the function can be invoked.
because you cannot redefine a keyword (that's what and is) in Python. __add__ is used to do something else:
These methods are called to implement the binary arithmetic operations (...&...