What does the # symbol do in Python?
An # symbol at the beginning of a line is used for class and function decorators:
PEP 318: Decorators
Python Decorators
The most common Python decorators are:
#property
#classmethod
#staticmethod
An # in the middle of a line is probably matrix multiplication:
# as a binary operator.
Example
class Pizza(object):
def __init__(self):
self.toppings = []
def __call__(self, topping):
# When using '#instance_of_pizza' before a function definition
# the function gets passed onto 'topping'.
self.toppings.append(topping())
def __repr__(self):
return str(self.toppings)
pizza = Pizza()
#pizza
def cheese():
return 'cheese'
#pizza
def sauce():
return 'sauce'
print pizza
# ['cheese', 'sauce']
This shows that the function/method/class you're defining after a decorator is just basically passed on as an argument to the function/method immediately after the # sign.
First sighting
The microframework Flask introduces decorators from the very beginning in the following format:
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return "Hello World!"
This in turn translates to:
rule = "/"
view_func = hello
# They go as arguments here in 'flask/app.py'
def add_url_rule(self, rule, endpoint=None, view_func=None, **options):
pass
Realizing this finally allowed me to feel at peace with Flask.
In Python 3.5 you can overload # as an operator. It is named as __matmul__, because it is designed to do matrix multiplication, but it can be anything you want. See PEP465 for details.
This is a simple implementation of matrix multiplication.
class Mat(list):
def __matmul__(self, B):
A = self
return Mat([[sum(A[i][k]*B[k][j] for k in range(len(B)))
for j in range(len(B[0])) ] for i in range(len(A))])
A = Mat([[1,3],[7,5]])
B = Mat([[6,8],[4,2]])
print(A # B)
This code yields:
[[18, 14], [62, 66]]
This code snippet:
def decorator(func):
return func
#decorator
def some_func():
pass
Is equivalent to this code:
def decorator(func):
return func
def some_func():
pass
some_func = decorator(some_func)
In the definition of a decorator you can add some modified things that wouldn't be returned by a function normally.
What does the “at” (#) symbol do in Python?
In short, it is used in decorator syntax and for matrix multiplication.
In the context of decorators, this syntax:
#decorator
def decorated_function():
"""this function is decorated"""
is equivalent to this:
def decorated_function():
"""this function is decorated"""
decorated_function = decorator(decorated_function)
In the context of matrix multiplication, a # b invokes a.__matmul__(b) - making this syntax:
a # b
equivalent to
dot(a, b)
and
a #= b
equivalent to
a = dot(a, b)
where dot is, for example, the numpy matrix multiplication function and a and b are matrices.
How could you discover this on your own?
I also do not know what to search for as searching Python docs or Google does not return relevant results when the # symbol is included.
If you want to have a rather complete view of what a particular piece of python syntax does, look directly at the grammar file. For the Python 3 branch:
~$ grep -C 1 "#" cpython/Grammar/Grammar
decorator: '#' dotted_name [ '(' [arglist] ')' ] NEWLINE
decorators: decorator+
--
testlist_star_expr: (test|star_expr) (',' (test|star_expr))* [',']
augassign: ('+=' | '-=' | '*=' | '#=' | '/=' | '%=' | '&=' | '|=' | '^=' |
'<<=' | '>>=' | '**=' | '//=')
--
arith_expr: term (('+'|'-') term)*
term: factor (('*'|'#'|'/'|'%'|'//') factor)*
factor: ('+'|'-'|'~') factor | power
We can see here that # is used in three contexts:
decorators
an operator between factors
an augmented assignment operator
Decorator Syntax:
A google search for "decorator python docs" gives as one of the top results, the "Compound Statements" section of the "Python Language Reference." Scrolling down to the section on function definitions, which we can find by searching for the word, "decorator", we see that... there's a lot to read. But the word, "decorator" is a link to the glossary, which tells us:
decorator
A function returning another function, usually applied as a function transformation using the #wrapper syntax. Common
examples for decorators are classmethod() and staticmethod().
The decorator syntax is merely syntactic sugar, the following two
function definitions are semantically equivalent:
def f(...):
...
f = staticmethod(f)
#staticmethod
def f(...):
...
The same concept exists for classes, but is less commonly used there.
See the documentation for function definitions and class definitions
for more about decorators.
So, we see that
#foo
def bar():
pass
is semantically the same as:
def bar():
pass
bar = foo(bar)
They are not exactly the same because Python evaluates the foo expression (which could be a dotted lookup and a function call) before bar with the decorator (#) syntax, but evaluates the foo expression after bar in the other case.
(If this difference makes a difference in the meaning of your code, you should reconsider what you're doing with your life, because that would be pathological.)
Stacked Decorators
If we go back to the function definition syntax documentation, we see:
#f1(arg)
#f2
def func(): pass
is roughly equivalent to
def func(): pass
func = f1(arg)(f2(func))
This is a demonstration that we can call a function that's a decorator first, as well as stack decorators. Functions, in Python, are first class objects - which means you can pass a function as an argument to another function, and return functions. Decorators do both of these things.
If we stack decorators, the function, as defined, gets passed first to the decorator immediately above it, then the next, and so on.
That about sums up the usage for # in the context of decorators.
The Operator, #
In the lexical analysis section of the language reference, we have a section on operators, which includes #, which makes it also an operator:
The following tokens are operators:
+ - * ** / // % #
<< >> & | ^ ~
< > <= >= == !=
and in the next page, the Data Model, we have the section Emulating Numeric Types,
object.__add__(self, other)
object.__sub__(self, other)
object.__mul__(self, other)
object.__matmul__(self, other)
object.__truediv__(self, other)
object.__floordiv__(self, other)
[...]
These methods are called to implement the binary arithmetic operations (+, -, *, #, /, //, [...]
And we see that __matmul__ corresponds to #. If we search the documentation for "matmul" we get a link to What's new in Python 3.5 with "matmul" under a heading "PEP 465 - A dedicated infix operator for matrix multiplication".
it can be implemented by defining __matmul__(), __rmatmul__(), and
__imatmul__() for regular, reflected, and in-place matrix multiplication.
(So now we learn that #= is the in-place version). It further explains:
Matrix multiplication is a notably common operation in many fields of
mathematics, science, engineering, and the addition of # allows
writing cleaner code:
S = (H # beta - r).T # inv(H # V # H.T) # (H # beta - r)
instead of:
S = dot((dot(H, beta) - r).T,
dot(inv(dot(dot(H, V), H.T)), dot(H, beta) - r))
While this operator can be overloaded to do almost anything, in numpy, for example, we would use this syntax to calculate the inner and outer product of arrays and matrices:
>>> from numpy import array, matrix
>>> array([[1,2,3]]).T # array([[1,2,3]])
array([[1, 2, 3],
[2, 4, 6],
[3, 6, 9]])
>>> array([[1,2,3]]) # array([[1,2,3]]).T
array([[14]])
>>> matrix([1,2,3]).T # matrix([1,2,3])
matrix([[1, 2, 3],
[2, 4, 6],
[3, 6, 9]])
>>> matrix([1,2,3]) # matrix([1,2,3]).T
matrix([[14]])
Inplace matrix multiplication: #=
While researching the prior usage, we learn that there is also the inplace matrix multiplication. If we attempt to use it, we may find it is not yet implemented for numpy:
>>> m = matrix([1,2,3])
>>> m #= m.T
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: In-place matrix multiplication is not (yet) supported. Use 'a = a # b' instead of 'a #= b'.
When it is implemented, I would expect the result to look like this:
>>> m = matrix([1,2,3])
>>> m #= m.T
>>> m
matrix([[14]])
What does the “at” (#) symbol do in Python?
# symbol is a syntactic sugar python provides to utilize decorator,
to paraphrase the question, It's exactly about what does decorator do in Python?
Put it simple decorator allow you to modify a given function's definition without touch its innermost (it's closure).
It's the most case when you import wonderful package from third party. You can visualize it, you can use it, but you cannot touch its innermost and its heart.
Here is a quick example,
suppose I define a read_a_book function on Ipython
In [9]: def read_a_book():
...: return "I am reading the book: "
...:
In [10]: read_a_book()
Out[10]: 'I am reading the book: '
You see, I forgot to add a name to it.
How to solve such a problem? Of course, I could re-define the function as:
def read_a_book():
return "I am reading the book: 'Python Cookbook'"
Nevertheless, what if I'm not allowed to manipulate the original function, or if there are thousands of such function to be handled.
Solve the problem by thinking different and define a new_function
def add_a_book(func):
def wrapper():
return func() + "Python Cookbook"
return wrapper
Then employ it.
In [14]: read_a_book = add_a_book(read_a_book)
In [15]: read_a_book()
Out[15]: 'I am reading the book: Python Cookbook'
Tada, you see, I amended read_a_book without touching it inner closure. Nothing stops me equipped with decorator.
What's about #
#add_a_book
def read_a_book():
return "I am reading the book: "
In [17]: read_a_book()
Out[17]: 'I am reading the book: Python Cookbook'
#add_a_book is a fancy and handy way to say read_a_book = add_a_book(read_a_book), it's a syntactic sugar, there's nothing more fancier about it.
If you are referring to some code in a python notebook which is using Numpy library, then # operator means Matrix Multiplication. For example:
import numpy as np
def forward(xi, W1, b1, W2, b2):
z1 = W1 # xi + b1
a1 = sigma(z1)
z2 = W2 # a1 + b2
return z2, a1
Decorators were added in Python to make function and method wrapping (a function that receives a function and returns an enhanced one) easier to read and understand. The original use case was to be able to define the methods as class methods or static methods on the head of their definition. Without the decorator syntax, it would require a rather sparse and repetitive definition:
class WithoutDecorators:
def some_static_method():
print("this is static method")
some_static_method = staticmethod(some_static_method)
def some_class_method(cls):
print("this is class method")
some_class_method = classmethod(some_class_method)
If the decorator syntax is used for the same purpose, the code is shorter and easier to understand:
class WithDecorators:
#staticmethod
def some_static_method():
print("this is static method")
#classmethod
def some_class_method(cls):
print("this is class method")
General syntax and possible implementations
The decorator is generally a named object ( lambda expressions are not allowed) that accepts a single argument when called (it will be the decorated function) and returns another callable object. "Callable" is used here instead of "function" with premeditation. While decorators are often discussed in the scope of methods and functions, they are not limited to them. In fact, anything that is callable (any object that implements the _call__ method is considered callable), can be used as a decorator and often objects returned by them are not simple functions but more instances of more complex classes implementing their own __call_ method.
The decorator syntax is simply only a syntactic sugar. Consider the following decorator usage:
#some_decorator
def decorated_function():
pass
This can always be replaced by an explicit decorator call and function reassignment:
def decorated_function():
pass
decorated_function = some_decorator(decorated_function)
However, the latter is less readable and also very hard to understand if multiple decorators are used on a single function.
Decorators can be used in multiple different ways as shown below:
As a function
There are many ways to write custom decorators, but the simplest way is to write a function that returns a subfunction that wraps the original function call.
The generic patterns is as follows:
def mydecorator(function):
def wrapped(*args, **kwargs):
# do some stuff before the original
# function gets called
result = function(*args, **kwargs)
# do some stuff after function call and
# return the result
return result
# return wrapper as a decorated function
return wrapped
As a class
While decorators almost always can be implemented using functions, there are some situations when using user-defined classes is a better option. This is often true when the decorator needs complex parametrization or it depends on a specific state.
The generic pattern for a nonparametrized decorator as a class is as follows:
class DecoratorAsClass:
def __init__(self, function):
self.function = function
def __call__(self, *args, **kwargs):
# do some stuff before the original
# function gets called
result = self.function(*args, **kwargs)
# do some stuff after function call and
# return the result
return result
Parametrizing decorators
In real code, there is often a need to use decorators that can be parametrized. When the function is used as a decorator, then the solution is simple—a second level of wrapping has to be used. Here is a simple example of the decorator that repeats the execution of a decorated function the specified number of times every time it is called:
def repeat(number=3):
"""Cause decorated function to be repeated a number of times.
Last value of original function call is returned as a result
:param number: number of repetitions, 3 if not specified
"""
def actual_decorator(function):
def wrapper(*args, **kwargs):
result = None
for _ in range(number):
result = function(*args, **kwargs)
return result
return wrapper
return actual_decorator
The decorator defined this way can accept parameters:
>>> #repeat(2)
... def foo():
... print("foo")
...
>>> foo()
foo
foo
Note that even if the parametrized decorator has default values for its arguments, the parentheses after its name is required. The correct way to use the preceding decorator with default arguments is as follows:
>>> #repeat()
... def bar():
... print("bar")
...
>>> bar()
bar
bar
bar
Finally lets see decorators with Properties.
Properties
The properties provide a built-in descriptor type that knows how to link an attribute to a set of methods. A property takes four optional arguments: fget , fset , fdel , and doc . The last one can be provided to define a docstring that is linked to the attribute as if it were a method. Here is an example of a Rectangle class that can be controlled either by direct access to attributes that store two corner points or by using the width , and height properties:
class Rectangle:
def __init__(self, x1, y1, x2, y2):
self.x1, self.y1 = x1, y1
self.x2, self.y2 = x2, y2
def _width_get(self):
return self.x2 - self.x1
def _width_set(self, value):
self.x2 = self.x1 + value
def _height_get(self):
return self.y2 - self.y1
def _height_set(self, value):
self.y2 = self.y1 + value
width = property(
_width_get, _width_set,
doc="rectangle width measured from left"
)
height = property(
_height_get, _height_set,
doc="rectangle height measured from top"
)
def __repr__(self):
return "{}({}, {}, {}, {})".format(
self.__class__.__name__,
self.x1, self.y1, self.x2, self.y2
)
The best syntax for creating properties is using property as a decorator. This will reduce the number of method signatures inside of the class
and make code more readable and maintainable. With decorators the above class becomes:
class Rectangle:
def __init__(self, x1, y1, x2, y2):
self.x1, self.y1 = x1, y1
self.x2, self.y2 = x2, y2
#property
def width(self):
"""rectangle height measured from top"""
return self.x2 - self.x1
#width.setter
def width(self, value):
self.x2 = self.x1 + value
#property
def height(self):
"""rectangle height measured from top"""
return self.y2 - self.y1
#height.setter
def height(self, value):
self.y2 = self.y1 + value
Starting with Python 3.5, the '#' is used as a dedicated infix symbol for MATRIX MULTIPLICATION (PEP 0465 -- see https://www.python.org/dev/peps/pep-0465/)
# can be a math operator or a DECORATOR but what you mean is a decorator.
This code:
def func(f):
return f
func(lambda :"HelloWorld")()
using decorators can be written like:
def func(f):
return f
#func
def name():
return "Hello World"
name()
Decorators can have arguments.
You can see this GeeksforGeeks post: https://www.geeksforgeeks.org/decorators-in-python/
It indicates that you are using a decorator. Here is Bruce Eckel's example from 2008.
Python decorator is like a wrapper of a function or a class. It’s still too conceptual.
def function_decorator(func):
def wrapped_func():
# Do something before the function is executed
func()
# Do something after the function has been executed
return wrapped_func
The above code is a definition of a decorator that decorates a function.
function_decorator is the name of the decorator.
wrapped_func is the name of the inner function, which is actually only used in this decorator definition. func is the function that is being decorated.
In the inner function wrapped_func, we can do whatever before and after the func is called. After the decorator is defined, we simply use it as follows.
#function_decorator
def func():
pass
Then, whenever we call the function func, the behaviours we’ve defined in the decorator will also be executed.
EXAMPLE :
from functools import wraps
def mydecorator(f):
#wraps(f)
def wrapped(*args, **kwargs):
print "Before decorated function"
r = f(*args, **kwargs)
print "After decorated function"
return r
return wrapped
#mydecorator
def myfunc(myarg):
print "my function", myarg
return "return value"
r = myfunc('asdf')
print r
Output :
Before decorated function
my function asdf
After decorated function
return value
To say what others have in a different way: yes, it is a decorator.
In Python, it's like:
Creating a function (follows under the # call)
Calling another function to operate on your created function. This returns a new function. The function that you call is the argument of the #.
Replacing the function defined with the new function returned.
This can be used for all kinds of useful things, made possible because functions are objects and just necessary just instructions.
# symbol is also used to access variables inside a plydata / pandas dataframe query, pandas.DataFrame.query.
Example:
df = pandas.DataFrame({'foo': [1,2,15,17]})
y = 10
df >> query('foo > #y') # plydata
df.query('foo > #y') # pandas
Related
What does the # symbol do in Python?
An # symbol at the beginning of a line is used for class and function decorators:
PEP 318: Decorators
Python Decorators
The most common Python decorators are:
#property
#classmethod
#staticmethod
An # in the middle of a line is probably matrix multiplication:
# as a binary operator.
Example
class Pizza(object):
def __init__(self):
self.toppings = []
def __call__(self, topping):
# When using '#instance_of_pizza' before a function definition
# the function gets passed onto 'topping'.
self.toppings.append(topping())
def __repr__(self):
return str(self.toppings)
pizza = Pizza()
#pizza
def cheese():
return 'cheese'
#pizza
def sauce():
return 'sauce'
print pizza
# ['cheese', 'sauce']
This shows that the function/method/class you're defining after a decorator is just basically passed on as an argument to the function/method immediately after the # sign.
First sighting
The microframework Flask introduces decorators from the very beginning in the following format:
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return "Hello World!"
This in turn translates to:
rule = "/"
view_func = hello
# They go as arguments here in 'flask/app.py'
def add_url_rule(self, rule, endpoint=None, view_func=None, **options):
pass
Realizing this finally allowed me to feel at peace with Flask.
In Python 3.5 you can overload # as an operator. It is named as __matmul__, because it is designed to do matrix multiplication, but it can be anything you want. See PEP465 for details.
This is a simple implementation of matrix multiplication.
class Mat(list):
def __matmul__(self, B):
A = self
return Mat([[sum(A[i][k]*B[k][j] for k in range(len(B)))
for j in range(len(B[0])) ] for i in range(len(A))])
A = Mat([[1,3],[7,5]])
B = Mat([[6,8],[4,2]])
print(A # B)
This code yields:
[[18, 14], [62, 66]]
This code snippet:
def decorator(func):
return func
#decorator
def some_func():
pass
Is equivalent to this code:
def decorator(func):
return func
def some_func():
pass
some_func = decorator(some_func)
In the definition of a decorator you can add some modified things that wouldn't be returned by a function normally.
What does the “at” (#) symbol do in Python?
In short, it is used in decorator syntax and for matrix multiplication.
In the context of decorators, this syntax:
#decorator
def decorated_function():
"""this function is decorated"""
is equivalent to this:
def decorated_function():
"""this function is decorated"""
decorated_function = decorator(decorated_function)
In the context of matrix multiplication, a # b invokes a.__matmul__(b) - making this syntax:
a # b
equivalent to
dot(a, b)
and
a #= b
equivalent to
a = dot(a, b)
where dot is, for example, the numpy matrix multiplication function and a and b are matrices.
How could you discover this on your own?
I also do not know what to search for as searching Python docs or Google does not return relevant results when the # symbol is included.
If you want to have a rather complete view of what a particular piece of python syntax does, look directly at the grammar file. For the Python 3 branch:
~$ grep -C 1 "#" cpython/Grammar/Grammar
decorator: '#' dotted_name [ '(' [arglist] ')' ] NEWLINE
decorators: decorator+
--
testlist_star_expr: (test|star_expr) (',' (test|star_expr))* [',']
augassign: ('+=' | '-=' | '*=' | '#=' | '/=' | '%=' | '&=' | '|=' | '^=' |
'<<=' | '>>=' | '**=' | '//=')
--
arith_expr: term (('+'|'-') term)*
term: factor (('*'|'#'|'/'|'%'|'//') factor)*
factor: ('+'|'-'|'~') factor | power
We can see here that # is used in three contexts:
decorators
an operator between factors
an augmented assignment operator
Decorator Syntax:
A google search for "decorator python docs" gives as one of the top results, the "Compound Statements" section of the "Python Language Reference." Scrolling down to the section on function definitions, which we can find by searching for the word, "decorator", we see that... there's a lot to read. But the word, "decorator" is a link to the glossary, which tells us:
decorator
A function returning another function, usually applied as a function transformation using the #wrapper syntax. Common
examples for decorators are classmethod() and staticmethod().
The decorator syntax is merely syntactic sugar, the following two
function definitions are semantically equivalent:
def f(...):
...
f = staticmethod(f)
#staticmethod
def f(...):
...
The same concept exists for classes, but is less commonly used there.
See the documentation for function definitions and class definitions
for more about decorators.
So, we see that
#foo
def bar():
pass
is semantically the same as:
def bar():
pass
bar = foo(bar)
They are not exactly the same because Python evaluates the foo expression (which could be a dotted lookup and a function call) before bar with the decorator (#) syntax, but evaluates the foo expression after bar in the other case.
(If this difference makes a difference in the meaning of your code, you should reconsider what you're doing with your life, because that would be pathological.)
Stacked Decorators
If we go back to the function definition syntax documentation, we see:
#f1(arg)
#f2
def func(): pass
is roughly equivalent to
def func(): pass
func = f1(arg)(f2(func))
This is a demonstration that we can call a function that's a decorator first, as well as stack decorators. Functions, in Python, are first class objects - which means you can pass a function as an argument to another function, and return functions. Decorators do both of these things.
If we stack decorators, the function, as defined, gets passed first to the decorator immediately above it, then the next, and so on.
That about sums up the usage for # in the context of decorators.
The Operator, #
In the lexical analysis section of the language reference, we have a section on operators, which includes #, which makes it also an operator:
The following tokens are operators:
+ - * ** / // % #
<< >> & | ^ ~
< > <= >= == !=
and in the next page, the Data Model, we have the section Emulating Numeric Types,
object.__add__(self, other)
object.__sub__(self, other)
object.__mul__(self, other)
object.__matmul__(self, other)
object.__truediv__(self, other)
object.__floordiv__(self, other)
[...]
These methods are called to implement the binary arithmetic operations (+, -, *, #, /, //, [...]
And we see that __matmul__ corresponds to #. If we search the documentation for "matmul" we get a link to What's new in Python 3.5 with "matmul" under a heading "PEP 465 - A dedicated infix operator for matrix multiplication".
it can be implemented by defining __matmul__(), __rmatmul__(), and
__imatmul__() for regular, reflected, and in-place matrix multiplication.
(So now we learn that #= is the in-place version). It further explains:
Matrix multiplication is a notably common operation in many fields of
mathematics, science, engineering, and the addition of # allows
writing cleaner code:
S = (H # beta - r).T # inv(H # V # H.T) # (H # beta - r)
instead of:
S = dot((dot(H, beta) - r).T,
dot(inv(dot(dot(H, V), H.T)), dot(H, beta) - r))
While this operator can be overloaded to do almost anything, in numpy, for example, we would use this syntax to calculate the inner and outer product of arrays and matrices:
>>> from numpy import array, matrix
>>> array([[1,2,3]]).T # array([[1,2,3]])
array([[1, 2, 3],
[2, 4, 6],
[3, 6, 9]])
>>> array([[1,2,3]]) # array([[1,2,3]]).T
array([[14]])
>>> matrix([1,2,3]).T # matrix([1,2,3])
matrix([[1, 2, 3],
[2, 4, 6],
[3, 6, 9]])
>>> matrix([1,2,3]) # matrix([1,2,3]).T
matrix([[14]])
Inplace matrix multiplication: #=
While researching the prior usage, we learn that there is also the inplace matrix multiplication. If we attempt to use it, we may find it is not yet implemented for numpy:
>>> m = matrix([1,2,3])
>>> m #= m.T
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: In-place matrix multiplication is not (yet) supported. Use 'a = a # b' instead of 'a #= b'.
When it is implemented, I would expect the result to look like this:
>>> m = matrix([1,2,3])
>>> m #= m.T
>>> m
matrix([[14]])
What does the “at” (#) symbol do in Python?
# symbol is a syntactic sugar python provides to utilize decorator,
to paraphrase the question, It's exactly about what does decorator do in Python?
Put it simple decorator allow you to modify a given function's definition without touch its innermost (it's closure).
It's the most case when you import wonderful package from third party. You can visualize it, you can use it, but you cannot touch its innermost and its heart.
Here is a quick example,
suppose I define a read_a_book function on Ipython
In [9]: def read_a_book():
...: return "I am reading the book: "
...:
In [10]: read_a_book()
Out[10]: 'I am reading the book: '
You see, I forgot to add a name to it.
How to solve such a problem? Of course, I could re-define the function as:
def read_a_book():
return "I am reading the book: 'Python Cookbook'"
Nevertheless, what if I'm not allowed to manipulate the original function, or if there are thousands of such function to be handled.
Solve the problem by thinking different and define a new_function
def add_a_book(func):
def wrapper():
return func() + "Python Cookbook"
return wrapper
Then employ it.
In [14]: read_a_book = add_a_book(read_a_book)
In [15]: read_a_book()
Out[15]: 'I am reading the book: Python Cookbook'
Tada, you see, I amended read_a_book without touching it inner closure. Nothing stops me equipped with decorator.
What's about #
#add_a_book
def read_a_book():
return "I am reading the book: "
In [17]: read_a_book()
Out[17]: 'I am reading the book: Python Cookbook'
#add_a_book is a fancy and handy way to say read_a_book = add_a_book(read_a_book), it's a syntactic sugar, there's nothing more fancier about it.
If you are referring to some code in a python notebook which is using Numpy library, then # operator means Matrix Multiplication. For example:
import numpy as np
def forward(xi, W1, b1, W2, b2):
z1 = W1 # xi + b1
a1 = sigma(z1)
z2 = W2 # a1 + b2
return z2, a1
Decorators were added in Python to make function and method wrapping (a function that receives a function and returns an enhanced one) easier to read and understand. The original use case was to be able to define the methods as class methods or static methods on the head of their definition. Without the decorator syntax, it would require a rather sparse and repetitive definition:
class WithoutDecorators:
def some_static_method():
print("this is static method")
some_static_method = staticmethod(some_static_method)
def some_class_method(cls):
print("this is class method")
some_class_method = classmethod(some_class_method)
If the decorator syntax is used for the same purpose, the code is shorter and easier to understand:
class WithDecorators:
#staticmethod
def some_static_method():
print("this is static method")
#classmethod
def some_class_method(cls):
print("this is class method")
General syntax and possible implementations
The decorator is generally a named object ( lambda expressions are not allowed) that accepts a single argument when called (it will be the decorated function) and returns another callable object. "Callable" is used here instead of "function" with premeditation. While decorators are often discussed in the scope of methods and functions, they are not limited to them. In fact, anything that is callable (any object that implements the _call__ method is considered callable), can be used as a decorator and often objects returned by them are not simple functions but more instances of more complex classes implementing their own __call_ method.
The decorator syntax is simply only a syntactic sugar. Consider the following decorator usage:
#some_decorator
def decorated_function():
pass
This can always be replaced by an explicit decorator call and function reassignment:
def decorated_function():
pass
decorated_function = some_decorator(decorated_function)
However, the latter is less readable and also very hard to understand if multiple decorators are used on a single function.
Decorators can be used in multiple different ways as shown below:
As a function
There are many ways to write custom decorators, but the simplest way is to write a function that returns a subfunction that wraps the original function call.
The generic patterns is as follows:
def mydecorator(function):
def wrapped(*args, **kwargs):
# do some stuff before the original
# function gets called
result = function(*args, **kwargs)
# do some stuff after function call and
# return the result
return result
# return wrapper as a decorated function
return wrapped
As a class
While decorators almost always can be implemented using functions, there are some situations when using user-defined classes is a better option. This is often true when the decorator needs complex parametrization or it depends on a specific state.
The generic pattern for a nonparametrized decorator as a class is as follows:
class DecoratorAsClass:
def __init__(self, function):
self.function = function
def __call__(self, *args, **kwargs):
# do some stuff before the original
# function gets called
result = self.function(*args, **kwargs)
# do some stuff after function call and
# return the result
return result
Parametrizing decorators
In real code, there is often a need to use decorators that can be parametrized. When the function is used as a decorator, then the solution is simple—a second level of wrapping has to be used. Here is a simple example of the decorator that repeats the execution of a decorated function the specified number of times every time it is called:
def repeat(number=3):
"""Cause decorated function to be repeated a number of times.
Last value of original function call is returned as a result
:param number: number of repetitions, 3 if not specified
"""
def actual_decorator(function):
def wrapper(*args, **kwargs):
result = None
for _ in range(number):
result = function(*args, **kwargs)
return result
return wrapper
return actual_decorator
The decorator defined this way can accept parameters:
>>> #repeat(2)
... def foo():
... print("foo")
...
>>> foo()
foo
foo
Note that even if the parametrized decorator has default values for its arguments, the parentheses after its name is required. The correct way to use the preceding decorator with default arguments is as follows:
>>> #repeat()
... def bar():
... print("bar")
...
>>> bar()
bar
bar
bar
Finally lets see decorators with Properties.
Properties
The properties provide a built-in descriptor type that knows how to link an attribute to a set of methods. A property takes four optional arguments: fget , fset , fdel , and doc . The last one can be provided to define a docstring that is linked to the attribute as if it were a method. Here is an example of a Rectangle class that can be controlled either by direct access to attributes that store two corner points or by using the width , and height properties:
class Rectangle:
def __init__(self, x1, y1, x2, y2):
self.x1, self.y1 = x1, y1
self.x2, self.y2 = x2, y2
def _width_get(self):
return self.x2 - self.x1
def _width_set(self, value):
self.x2 = self.x1 + value
def _height_get(self):
return self.y2 - self.y1
def _height_set(self, value):
self.y2 = self.y1 + value
width = property(
_width_get, _width_set,
doc="rectangle width measured from left"
)
height = property(
_height_get, _height_set,
doc="rectangle height measured from top"
)
def __repr__(self):
return "{}({}, {}, {}, {})".format(
self.__class__.__name__,
self.x1, self.y1, self.x2, self.y2
)
The best syntax for creating properties is using property as a decorator. This will reduce the number of method signatures inside of the class
and make code more readable and maintainable. With decorators the above class becomes:
class Rectangle:
def __init__(self, x1, y1, x2, y2):
self.x1, self.y1 = x1, y1
self.x2, self.y2 = x2, y2
#property
def width(self):
"""rectangle height measured from top"""
return self.x2 - self.x1
#width.setter
def width(self, value):
self.x2 = self.x1 + value
#property
def height(self):
"""rectangle height measured from top"""
return self.y2 - self.y1
#height.setter
def height(self, value):
self.y2 = self.y1 + value
Starting with Python 3.5, the '#' is used as a dedicated infix symbol for MATRIX MULTIPLICATION (PEP 0465 -- see https://www.python.org/dev/peps/pep-0465/)
# can be a math operator or a DECORATOR but what you mean is a decorator.
This code:
def func(f):
return f
func(lambda :"HelloWorld")()
using decorators can be written like:
def func(f):
return f
#func
def name():
return "Hello World"
name()
Decorators can have arguments.
You can see this GeeksforGeeks post: https://www.geeksforgeeks.org/decorators-in-python/
It indicates that you are using a decorator. Here is Bruce Eckel's example from 2008.
Python decorator is like a wrapper of a function or a class. It’s still too conceptual.
def function_decorator(func):
def wrapped_func():
# Do something before the function is executed
func()
# Do something after the function has been executed
return wrapped_func
The above code is a definition of a decorator that decorates a function.
function_decorator is the name of the decorator.
wrapped_func is the name of the inner function, which is actually only used in this decorator definition. func is the function that is being decorated.
In the inner function wrapped_func, we can do whatever before and after the func is called. After the decorator is defined, we simply use it as follows.
#function_decorator
def func():
pass
Then, whenever we call the function func, the behaviours we’ve defined in the decorator will also be executed.
EXAMPLE :
from functools import wraps
def mydecorator(f):
#wraps(f)
def wrapped(*args, **kwargs):
print "Before decorated function"
r = f(*args, **kwargs)
print "After decorated function"
return r
return wrapped
#mydecorator
def myfunc(myarg):
print "my function", myarg
return "return value"
r = myfunc('asdf')
print r
Output :
Before decorated function
my function asdf
After decorated function
return value
To say what others have in a different way: yes, it is a decorator.
In Python, it's like:
Creating a function (follows under the # call)
Calling another function to operate on your created function. This returns a new function. The function that you call is the argument of the #.
Replacing the function defined with the new function returned.
This can be used for all kinds of useful things, made possible because functions are objects and just necessary just instructions.
# symbol is also used to access variables inside a plydata / pandas dataframe query, pandas.DataFrame.query.
Example:
df = pandas.DataFrame({'foo': [1,2,15,17]})
y = 10
df >> query('foo > #y') # plydata
df.query('foo > #y') # pandas
I am using this way of decorating all methods
import inspect
def decallmethods(decorator, prefix='test_'):
def dectheclass(cls):
for name, m in inspect.getmembers(cls, inspect.ismethod):
if name.startswith(prefix):
setattr(cls, name, decorator(m))
return cls
return dectheclass
#decallmethods(login_testuser)
class TestCase(object):
def setUp(self):
pass
def test_1(self):
print "test_1()"
def test_2(self):
print "test_2()"
This is working but it applies at the top , if i have other decorators.
I mean
Now the result is
#login_testuser
#other
def test_2(self):
print "test_2()"
But i want
#other
#login_testuser
def test_2(self):
print "test_2()"
This is most certainly a bad idea, but what you want to do can be done in some extent, and this is going to take a lot of time to explain. First off, rather than thinking of decorators as a syntax sugar, think of them as what they really are: a function (that is a closure) with a function that exist inside it. Now this is out of the way, supposed we have a function:
def operation(a, b):
print('doing operation')
return a + b
Simply it will do this
>>> hi = operation('hello', 'world')
doing operation
>>> print(hi)
helloworld
Now define a decorator that prints something before and after calling its inner function (equivalent to the other decorator that you want to decorator later):
def other(f):
def other_inner(*a, **kw):
print('other start')
result = f(*a, **kw)
print('other finish')
return result
return other_inner
With that, build a new function and decorator
#other
def o_operation(a, b):
print('doing operation')
return a + b
Remembering, this is basically equivalent to o_operation = other(operation)
Run this to ensure it works:
>>> r2 = o_operation('some', 'inner')
other start
doing operation
other finish
>>> print(r2)
someinner
Finally, the final decorator you want to call immediately before operation but not d_operation, but with your existing code it results in this:
def inject(f):
def injected(*a, **kw):
print('inject start')
result = f(*a, **kw)
print('inject finish')
return result
return injected
#inject
#other
def i_o_operation(a, b):
print('doing operation')
return a + b
Run the above:
>>> i_o_operation('hello', 'foo')
inject start
other start
doing operation
other finish
inject finish
'hellofoo'
As mentioned decorators are really closures and hence that's why it's possible to have items inside that are effectively instanced inside. You can reach them by going through the __closure__ attribute:
>>> i_o_operation.__closure__
(<cell at 0x7fc0eabd1fd8: function object at 0x7fc0eabce7d0>,)
>>> i_o_operation.__closure__[0].cell_contents
<function other_inner at 0x7fc0eabce7d0>
>>> print(i_o_operation.__closure__[0].cell_contents('a', 'b'))
other start
doing operation
other finish
ab
See how this effectively calls the function inside the injected closure directly, as if that got unwrapped. What if that closure can be replaced with the one that did the injection? For all of our protection, __closure__ and cell.cell_contents are read-only. What needs to be done is to construct completely new functions with the intended closures by making use of the FunctionType function constructor (found in the types module)
Back to the problem. Since what we have now is:
i_o_operation = inject(other(operation))
And what we want is
o_i_operation = other(inject(operation))
We effectively have to somehow strip the call to other from i_o_operation and somehow wrap it around with inject to produce o_i_operation. (Dragons follows after the break)
First, construct a function that effectively calls inject(operation) by taking the closure to level deep (so that f will contain just the original operation call) but mix it with the code produced by inject(f):
i_operation = FunctionType(
i_o_operation.__code__,
globals=globals(),
closure=i_o_operation.__closure__[0].cell_contents.__closure__,
)
Since i_o_operation is the result of inject(f) we can take that code to produce a new function. The globals is a formality that's required, and finally take the closure of the nested level, and the first part of the function is produced. Verify that the other is not called.
>>> i_operation('test', 'strip')
inject start
doing operation
inject finish
'teststrip'
Neat. However we still want the other to be wrapped outside of this to finally produce o_i_operation. We do need to somehow put this new function we produced in a closure, and a way to do this is to create a surrogate function that produce one
def closure(f):
def surrogate(*a, **kw):
return f(*a, **kw)
return surrogate
And simply use it to construct and extract our closure
o_i_operation = FunctionType(
i_o_operation.__closure__[0].cell_contents.__code__,
globals=globals(),
closure=closure(i_operation).__closure__,
)
Call this:
>>> o_i_operation('job', 'complete')
other start
inject start
doing operation
inject finish
other finish
'jobcomplete'
Looks like we finally got what we need. While this doesn't exactly answer your exact problem, this started down the right track but is already pretty hairy.
Now for the actual problem: a function that will ensure a decorator function be the most inner (final) callable before a given original, undecorated function - i.e. for a given target and a f(g(...(callable)), we want to emulate a result that gives f(g(...(target(callable)))). This is the code:
from types import FunctionType
def strip_decorators(f):
"""
Strip all decorators from f. Assumes each are functions with a
closure with a first cell being the target function.
"""
# list of not the actual decorator, but the returned functions
decorators = []
while f.__closure__:
# Assume first item is the target method
decorators.append(f)
f = f.__closure__[0].cell_contents
return decorators, f
def inject_decorator(decorator, f):
"""
Inject a decorator to the most inner function within the stack of
closures in `f`.
"""
def closure(f):
def surrogate(*a, **kw):
return f(*a, **kw)
return surrogate
decorators, target_f = strip_decorators(f)
result = decorator(target_f)
while decorators:
# pop out the last one in
decorator = decorators.pop()
result = FunctionType(
decorator.__code__,
globals=globals(),
closure=closure(result).__closure__,
)
return result
To test this, we use a typical example use-case - html tags.
def italics(f):
def i(s):
return '<i>' + f(s) + '</i>'
return i
def bold(f):
def b(s):
return '<b>' + f(s) + '</b>'
return b
def underline(f):
def u(s):
return '<u>' + f(s) + '</u>'
return u
#italics
#bold
def hi(s):
return s
Running the test.
>>> hi('hello')
'<i><b>hello</b></i>'
Our target is to inject the underline decorator (specifically the u(hi) callable) into the most inner closure. This can be done like so, with the function we have defined above:
>>> hi_u = inject_decorator(underline, hi)
>>> hi_u('hello')
'<i><b><u>hello</u></b></i>'
Works with undecorated functions:
>>> def pp(s):
... return s
...
>>> pp_b = inject_decorator(bold, pp)
>>> pp_b('hello')
'<b>hello</b>'
A major assumption was made for this first-cut version of the rewriter, which is that all decorators in the chain only have a closure length of one, that one element being the function being decorated with. Take this decorator for instance:
def prefix(p):
def decorator(f):
def inner(*args, **kwargs):
new_args = [p + a for a in args]
return f(*new_args, **kwargs)
return inner
return decorator
Example usage:
>>> #prefix('++')
... def prefix_hi(s):
... return s
...
>>> prefix_hi('test')
'++test'
Now try to inject a bold decorator like so:
>>> prefix_hi_bold = inject_decorator(bold, prefix_hi)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 18, in inject_decorator
ValueError: inner requires closure of length 2, not 1
This is simply because the closure formed by decorator within prefix has two elements, one being the prefix string p and the second being the actual function, and inner being nested inside that expects both those to be present inside its closure. Resolving that will require more code to analyse and reconstruct the details.
Anyway, this explanation took quite a bit of time and words, so I hope you understand this and maybe get you started on the actual right track.
If you want to turn inject_decorator into a decorator, and/or mix it into your class decorator, best of luck, most of the hard work is already done.
Let me first acknowledge that what I want to do may be considered anything from silly to evil, but I want to find out if I can do it in Python anyway.
Let's say I have a function decorator that takes keyword arguments defining variables, and I want to access those variables in the wrapped function. I might do something like this:
def more_vars(**extras):
def wrapper(f):
#wraps(f)
def wrapped(*args, **kwargs):
return f(extras, *args, **kwargs)
return wrapped
return wrapper
Now I can do something like:
#more_vars(a='hello', b='world')
def test(deco_vars, x, y):
print(deco_vars['a'], deco_vars['b'])
print(x, y)
test(1, 2)
# Output:
# hello world
# 1 2
The thing I don't like about this is that when you use this decorator, you have to change the call signature of the function, adding the extra variable in addition to slapping on the decorator. Also, if you look at the help for the function, you see an extra variable that you're not expected to use when calling the function:
help(test)
# Output:
# Help on function test in module __main__:
#
# test(deco_vars, x, y)
This makes it look like the user is expected to call the function with 3 parameters, but obviously that won't work. So you'd have to also add a message to the docstring indicating that the first parameter isn't part of the interface, it's just an implementation detail and should be ignored. That's kind of crappy, though. Is there any way to do this without hanging these variables on something in the global scope? Ideally, I'd like it to look like the following:
#more_vars(a='hello', b='world')
def test(x, y):
print(a, b)
print(x, y)
test(1, 2)
# Output:
# hello world
# 1 2
help(test)
# Output:
# Help on function test in module __main__:
#
# test(x, y)
I am content with a Python 3 only solution if one exists.
You could do this with some trickery that inserts the variables passed to the decorator into the function's local variables:
import sys
from functools import wraps
from types import FunctionType
def is_python3():
return sys.version_info >= (3, 0)
def more_vars(**extras):
def wrapper(f):
#wraps(f)
def wrapped(*args, **kwargs):
fn_globals = {}
fn_globals.update(globals())
fn_globals.update(extras)
if is_python3():
func_code = '__code__'
else:
func_code = 'func_code'
call_fn = FunctionType(getattr(f, func_code), fn_globals)
return call_fn(*args, **kwargs)
return wrapped
return wrapper
#more_vars(a="hello", b="world")
def test(x, y):
print("locals: {}".format(locals()))
print("x: {}".format(x))
print("y: {}".format(y))
print("a: {}".format(a))
print("b: {}".format(b))
if __name__ == "__main__":
test(1, 2)
Can you do this? Sure! Should you do this? Probably not!
(Code available here.)
EDIT: answer edited for readability. Latest answer is on top, original follows.
If I understand well
you want the new arguments to be defined as keywords in the #more_vars decorator
you want to use them in the decorated function
and you want them to be hidden to the normal users (the exposed signature should still be the normal signature)
Have a look at the #with_partial decorator in my library makefun. It provides this functionality out of the box:
from makefun import with_partial
#with_partial(a='hello', b='world')
def test(a, b, x, y):
"""Here is a doc"""
print(a, b)
print(x, y)
It yields the expected output and the docstring is modified accordingly:
test(1, 2)
help(test)
yields
hello world
1 2
Help on function test in module <...>:
test(x, y)
<This function is equivalent to 'test(x, y, a=hello, b=world)', see original 'test' doc below.>
Here is a doc
To answer the question in your comment, the function creation strategy in makefun is exactly the same than the one in the famous decorator library: compile + exec. No magic here, but decorator has been using this trick for years in real-world applications so it is quite solid. See def _make in the source code.
Note that the makefun library also provides a partial(f, *args, **kwargs) function if you want to create the decorator yourself for some reason (see below for inspiration).
If you wish to do this manually, this is a solution that should work as you expect, it relies on the wraps function provided by makefun, to modify the exposed signature.
from makefun import wraps, remove_signature_parameters
def more_vars(**extras):
def wrapper(f):
# (1) capture the signature of the function to wrap and remove the invisible
func_sig = signature(f)
new_sig = remove_signature_parameters(func_sig, 'invisible_args')
# (2) create a wrapper with the new signature
#wraps(f, new_sig=new_sig)
def wrapped(*args, **kwargs):
# inject the invisible args again
kwargs['invisible_args'] = extras
return f(*args, **kwargs)
return wrapped
return wrapper
You can test that it works:
#more_vars(a='hello', b='world')
def test(x, y, invisible_args):
a = invisible_args['a']
b = invisible_args['b']
print(a, b)
print(x, y)
test(1, 2)
help(test)
You can even make the decorator definition more compact if you use decopatch to remove the useless level of nesting:
from decopatch import DECORATED
from makefun import wraps, remove_signature_parameters
#function_decorator
def more_vars(f=DECORATED, **extras):
# (1) capture the signature of the function to wrap and remove the invisible
func_sig = signature(f)
new_sig = remove_signature_parameters(func_sig, 'invisible_args')
# (2) create a wrapper with the new signature
#wraps(f, new_sig=new_sig)
def wrapped(*args, **kwargs):
kwargs['invisible_args'] = extras
return f(*args, **kwargs)
return wrapped
Finally, if you rather do not want to depend on any external library, the most pythonic way to do it is to create a function factory (but then you cannot have this as a decorator):
def make_test(a, b, name=None):
def test(x, y):
print(a, b)
print(x, y)
if name is not None:
test.__name__ = name
return test
test = make_test(a='hello', b='world')
test2 = make_test(a='hello', b='there', name='test2')
I'm the author of makefun and decopatch by the way ;)
It sounds like your only problem is that help is showing the signature of the raw test as the signature of the wrapped function, and you don't want it to.
The only reason that's happening is that wraps (or, rather, update_wrapper, which wraps calls) explicitly copies this from the wrappee to the wrapper.
You can decide exactly what you do and don't want to copy. If what you want to do differently is simple enough, it's just a matter of filtering stuff out of the default WRAPPER_ASSIGNMENTS and WRAPPER_UPDATES. If you want to change other stuff, you may need to fork update_wrapper and use your own version—but functools is one of those modules that has a link to the source right at the top of the docs, because it's meant to be used as readable sample code.
In your case, it may just be a matter of wraps(f, updated=[]), or you may want to do something fancy, like use inspect.signature to get the signature of f, and modify it to remove the first parameter, and build a wrapper explicitly around that to fool even the inspect module.
I've found a solution to this problem, although the solution is by most standards almost certainly worse than the problem itself. With some clever rewriting of the decorated function's bytecode, you can redirect all references to variables of a given name to a new closure you can dynamically create for the function. This solution only works for the standard CPython, and I have only tested it with 3.7.
import inspect
from dis import opmap, Bytecode
from types import FunctionType, CodeType
def more_vars(**vars):
'''Decorator to inject more variables into a function.'''
def wrapper(f):
code = f.__code__
new_freevars = code.co_freevars + tuple(vars.keys())
new_globals = [var for var in code.co_names if var not in vars.keys()]
new_locals = [var for var in code.co_varnames if var not in vars.keys()]
payload = b''.join(
filtered_bytecode(f, new_freevars, new_globals, new_locals))
new_code = CodeType(code.co_argcount,
code.co_kwonlyargcount,
len(new_locals),
code.co_stacksize,
code.co_flags & ~inspect.CO_NOFREE,
payload,
code.co_consts,
tuple(new_globals),
tuple(new_locals),
code.co_filename,
code.co_name,
code.co_firstlineno,
code.co_lnotab,
code.co_freevars + tuple(vars.keys()),
code.co_cellvars)
closure = tuple(get_cell(v) for (k, v) in vars.items())
return FunctionType(new_code, f.__globals__, f.__name__, f.__defaults__,
(f.__closure__ or ()) + closure)
return wrapper
def get_cell(val=None):
'''Create a closure cell object with initial value.'''
# If you know a better way to do this, I'd like to hear it.
x = val
def closure():
return x # pragma: no cover
return closure.__closure__[0]
def filtered_bytecode(func, freevars, globals, locals):
'''Get the bytecode for a function with adjusted closed variables
Any references to globlas or locals in the bytecode which exist in the
freevars are modified to reference the freevars instead.
'''
opcode_map = {
opmap['LOAD_FAST']: opmap['LOAD_DEREF'],
opmap['STORE_FAST']: opmap['STORE_DEREF'],
opmap['LOAD_GLOBAL']: opmap['LOAD_DEREF'],
opmap['STORE_GLOBAL']: opmap['STORE_DEREF']
}
freevars_map = {var: idx for (idx, var) in enumerate(freevars)}
globals_map = {var: idx for (idx, var) in enumerate(globals)}
locals_map = {var: idx for (idx, var) in enumerate(locals)}
for instruction in Bytecode(func):
if instruction.opcode not in opcode_map:
yield bytes([instruction.opcode, instruction.arg or 0])
elif instruction.argval in freevars_map:
yield bytes([opcode_map[instruction.opcode],
freevars_map[instruction.argval]])
elif 'GLOBAL' in instruction.opname:
yield bytes([instruction.opcode,
globals_map[instruction.argval]])
elif 'FAST' in instruction.opname:
yield bytes([instruction.opcode,
locals_map[instruction.argval]])
This behaves exactly as I wanted:
In [1]: #more_vars(a='hello', b='world')
...: def test(x, y):
...: print(a, b)
...: print(x, y)
...:
In [2]: test(1, 2)
hello world
1 2
In [3]: help(test)
Help on function test in module __main__:
test(x, y)
This is almost certainly not ready for production use. I would be surprised if there weren't edge cases that behave unexpectedly, and possibly even segfault. I'd probably file this under the "educational curiosity" heading.
I have a class that need to make some magic with every operator, like __add__, __sub__ and so on.
Instead of creating each function in the class, I have a metaclass which defines every operator in the operator module.
import operator
class MetaFuncBuilder(type):
def __init__(self, *args, **kw):
super().__init__(*args, **kw)
attr = '__{0}{1}__'
for op in (x for x in dir(operator) if not x.startswith('__')):
oper = getattr(operator, op)
# ... I have my magic replacement functions here
# `func` for `__operators__` and `__ioperators__`
# and `rfunc` for `__roperators__`
setattr(self, attr.format('', op), func)
setattr(self, attr.format('r', op), rfunc)
The approach works fine, but I think It would be better if I generate the replacement operator only when needed.
Lookup of operators should be on the metaclass because x + 1 is done as type(x).__add__(x,1) instead of x.__add__(x,1), but it doesn't get caught by __getattr__ nor __getattribute__ methods.
That doesn't work:
class Meta(type):
def __getattr__(self, name):
if name in ['__add__', '__sub__', '__mul__', ...]:
func = lambda:... #generate magic function
return func
Also, the resulting "function" must be a method bound to the instance used.
Any ideas on how can I intercept this lookup? I don't know if it's clear what I want to do.
For those questioning why do I need to this kind of thing, check the full code here.
That's a tool to generate functions (just for fun) that could work as replacement for lambdas.
Example:
>>> f = FuncBuilder()
>>> g = f ** 2
>>> g(10)
100
>>> g
<var [('pow', 2)]>
Just for the record, I don't want to know another way to do the same thing (I won't declare every single operator on the class... that will be boring and the approach I have works pretty fine :). I want to know how to intercept attribute lookup from an operator.
Some black magic let's you achieve your goal:
operators = ["add", "mul"]
class OperatorHackiness(object):
"""
Use this base class if you want your object
to intercept __add__, __iadd__, __radd__, __mul__ etc.
using __getattr__.
__getattr__ will called at most _once_ during the
lifetime of the object, as the result is cached!
"""
def __init__(self):
# create a instance-local base class which we can
# manipulate to our needs
self.__class__ = self.meta = type('tmp', (self.__class__,), {})
# add operator methods dynamically, because we are damn lazy.
# This loop is however only called once in the whole program
# (when the module is loaded)
def create_operator(name):
def dynamic_operator(self, *args):
# call getattr to allow interception
# by user
func = self.__getattr__(name)
# save the result in the temporary
# base class to avoid calling getattr twice
setattr(self.meta, name, func)
# use provided function to calculate result
return func(self, *args)
return dynamic_operator
for op in operators:
for name in ["__%s__" % op, "__r%s__" % op, "__i%s__" % op]:
setattr(OperatorHackiness, name, create_operator(name))
# Example user class
class Test(OperatorHackiness):
def __init__(self, x):
super(Test, self).__init__()
self.x = x
def __getattr__(self, attr):
print "__getattr__(%s)" % attr
if attr == "__add__":
return lambda a, b: a.x + b.x
elif attr == "__iadd__":
def iadd(self, other):
self.x += other.x
return self
return iadd
elif attr == "__mul__":
return lambda a, b: a.x * b.x
else:
raise AttributeError
## Some test code:
a = Test(3)
b = Test(4)
# let's test addition
print(a + b) # this first call to __add__ will trigger
# a __getattr__ call
print(a + b) # this second call will not!
# same for multiplication
print(a * b)
print(a * b)
# inplace addition (getattr is also only called once)
a += b
a += b
print(a.x) # yay!
Output
__getattr__(__add__)
7
7
__getattr__(__mul__)
12
12
__getattr__(__iadd__)
11
Now you can use your second code sample literally by inheriting from my OperatorHackiness base class. You even get an additional benefit: __getattr__ will only be called once per instance and operator and there is no additional layer of recursion involved for the caching. We hereby circumvent the problem of method calls being slow compared to method lookup (as Paul Hankin noticed correctly).
NOTE: The loop to add the operator methods is only executed once in your whole program, so the preparation takes constant overhead in the range of milliseconds.
The issue at hand is that Python looks up __xxx__ methods on the object's class, not on the object itself -- and if it is not found, it does not fall back to __getattr__ nor __getattribute__.
The only way to intercept such calls is to have a method already there. It can be a stub function, as in Niklas Baumstark's answer, or it can be the full-fledged replacement function; either way, however, there must be something already there or you will not be able to intercept such calls.
If you are reading closely, you will have noticed that your requirement for having the final method be bound to the instance is not a possible solution -- you can do it, but Python will never call it as Python is looking at the class of the instance, not the instance, for __xxx__ methods. Niklas Baumstark's solution of making a unique temp class for each instance is as close as you can get to that requirement.
It looks like you are making things too complicated. You can define a mixin class and inherit from it. This is both simpler than using metaclasses and will run faster than using __getattr__.
class OperatorMixin(object):
def __add__(self, other):
return func(self, other)
def __radd__(self, other):
return rfunc(self, other)
... other operators defined too
Then every class you want to have these operators, inherit from OperatorMixin.
class Expression(OperatorMixin):
... the regular methods for your class
Generating the operator methods when they're needed isn't a good idea: __getattr__ is slow compared to regular method lookup, and since the methods are stored once (on the mixin class), it saves almost nothing.
If you want to achieve your goal without metaclasses, you can append the following to your code:
def get_magic_wrapper(name):
def wrapper(self, *a, **kw):
print('Wrapping')
res = getattr(self._data, name)(*a, **kw)
return res
return wrapper
_magic_methods = ['__str__', '__len__', '__repr__']
for _mm in _magic_methods:
setattr(ShowMeList, _mm, get_magic_wrapper(_mm))
It reroutes the methods in _magic_methods to the self._data object, by adding these attributes to the class iteratively. To check if it works:
>>> l = ShowMeList(range(8))
>>> len(l)
Wrapping
8
>>> l
Wrapping
[0, 1, 2, 3, 4, 5, 6, 7]
>>> print(l)
Wrapping
[0, 1, 2, 3, 4, 5, 6, 7]
What does the # symbol do in Python?
An # symbol at the beginning of a line is used for class and function decorators:
PEP 318: Decorators
Python Decorators
The most common Python decorators are:
#property
#classmethod
#staticmethod
An # in the middle of a line is probably matrix multiplication:
# as a binary operator.
Example
class Pizza(object):
def __init__(self):
self.toppings = []
def __call__(self, topping):
# When using '#instance_of_pizza' before a function definition
# the function gets passed onto 'topping'.
self.toppings.append(topping())
def __repr__(self):
return str(self.toppings)
pizza = Pizza()
#pizza
def cheese():
return 'cheese'
#pizza
def sauce():
return 'sauce'
print pizza
# ['cheese', 'sauce']
This shows that the function/method/class you're defining after a decorator is just basically passed on as an argument to the function/method immediately after the # sign.
First sighting
The microframework Flask introduces decorators from the very beginning in the following format:
from flask import Flask
app = Flask(__name__)
#app.route("/")
def hello():
return "Hello World!"
This in turn translates to:
rule = "/"
view_func = hello
# They go as arguments here in 'flask/app.py'
def add_url_rule(self, rule, endpoint=None, view_func=None, **options):
pass
Realizing this finally allowed me to feel at peace with Flask.
In Python 3.5 you can overload # as an operator. It is named as __matmul__, because it is designed to do matrix multiplication, but it can be anything you want. See PEP465 for details.
This is a simple implementation of matrix multiplication.
class Mat(list):
def __matmul__(self, B):
A = self
return Mat([[sum(A[i][k]*B[k][j] for k in range(len(B)))
for j in range(len(B[0])) ] for i in range(len(A))])
A = Mat([[1,3],[7,5]])
B = Mat([[6,8],[4,2]])
print(A # B)
This code yields:
[[18, 14], [62, 66]]
This code snippet:
def decorator(func):
return func
#decorator
def some_func():
pass
Is equivalent to this code:
def decorator(func):
return func
def some_func():
pass
some_func = decorator(some_func)
In the definition of a decorator you can add some modified things that wouldn't be returned by a function normally.
What does the “at” (#) symbol do in Python?
In short, it is used in decorator syntax and for matrix multiplication.
In the context of decorators, this syntax:
#decorator
def decorated_function():
"""this function is decorated"""
is equivalent to this:
def decorated_function():
"""this function is decorated"""
decorated_function = decorator(decorated_function)
In the context of matrix multiplication, a # b invokes a.__matmul__(b) - making this syntax:
a # b
equivalent to
dot(a, b)
and
a #= b
equivalent to
a = dot(a, b)
where dot is, for example, the numpy matrix multiplication function and a and b are matrices.
How could you discover this on your own?
I also do not know what to search for as searching Python docs or Google does not return relevant results when the # symbol is included.
If you want to have a rather complete view of what a particular piece of python syntax does, look directly at the grammar file. For the Python 3 branch:
~$ grep -C 1 "#" cpython/Grammar/Grammar
decorator: '#' dotted_name [ '(' [arglist] ')' ] NEWLINE
decorators: decorator+
--
testlist_star_expr: (test|star_expr) (',' (test|star_expr))* [',']
augassign: ('+=' | '-=' | '*=' | '#=' | '/=' | '%=' | '&=' | '|=' | '^=' |
'<<=' | '>>=' | '**=' | '//=')
--
arith_expr: term (('+'|'-') term)*
term: factor (('*'|'#'|'/'|'%'|'//') factor)*
factor: ('+'|'-'|'~') factor | power
We can see here that # is used in three contexts:
decorators
an operator between factors
an augmented assignment operator
Decorator Syntax:
A google search for "decorator python docs" gives as one of the top results, the "Compound Statements" section of the "Python Language Reference." Scrolling down to the section on function definitions, which we can find by searching for the word, "decorator", we see that... there's a lot to read. But the word, "decorator" is a link to the glossary, which tells us:
decorator
A function returning another function, usually applied as a function transformation using the #wrapper syntax. Common
examples for decorators are classmethod() and staticmethod().
The decorator syntax is merely syntactic sugar, the following two
function definitions are semantically equivalent:
def f(...):
...
f = staticmethod(f)
#staticmethod
def f(...):
...
The same concept exists for classes, but is less commonly used there.
See the documentation for function definitions and class definitions
for more about decorators.
So, we see that
#foo
def bar():
pass
is semantically the same as:
def bar():
pass
bar = foo(bar)
They are not exactly the same because Python evaluates the foo expression (which could be a dotted lookup and a function call) before bar with the decorator (#) syntax, but evaluates the foo expression after bar in the other case.
(If this difference makes a difference in the meaning of your code, you should reconsider what you're doing with your life, because that would be pathological.)
Stacked Decorators
If we go back to the function definition syntax documentation, we see:
#f1(arg)
#f2
def func(): pass
is roughly equivalent to
def func(): pass
func = f1(arg)(f2(func))
This is a demonstration that we can call a function that's a decorator first, as well as stack decorators. Functions, in Python, are first class objects - which means you can pass a function as an argument to another function, and return functions. Decorators do both of these things.
If we stack decorators, the function, as defined, gets passed first to the decorator immediately above it, then the next, and so on.
That about sums up the usage for # in the context of decorators.
The Operator, #
In the lexical analysis section of the language reference, we have a section on operators, which includes #, which makes it also an operator:
The following tokens are operators:
+ - * ** / // % #
<< >> & | ^ ~
< > <= >= == !=
and in the next page, the Data Model, we have the section Emulating Numeric Types,
object.__add__(self, other)
object.__sub__(self, other)
object.__mul__(self, other)
object.__matmul__(self, other)
object.__truediv__(self, other)
object.__floordiv__(self, other)
[...]
These methods are called to implement the binary arithmetic operations (+, -, *, #, /, //, [...]
And we see that __matmul__ corresponds to #. If we search the documentation for "matmul" we get a link to What's new in Python 3.5 with "matmul" under a heading "PEP 465 - A dedicated infix operator for matrix multiplication".
it can be implemented by defining __matmul__(), __rmatmul__(), and
__imatmul__() for regular, reflected, and in-place matrix multiplication.
(So now we learn that #= is the in-place version). It further explains:
Matrix multiplication is a notably common operation in many fields of
mathematics, science, engineering, and the addition of # allows
writing cleaner code:
S = (H # beta - r).T # inv(H # V # H.T) # (H # beta - r)
instead of:
S = dot((dot(H, beta) - r).T,
dot(inv(dot(dot(H, V), H.T)), dot(H, beta) - r))
While this operator can be overloaded to do almost anything, in numpy, for example, we would use this syntax to calculate the inner and outer product of arrays and matrices:
>>> from numpy import array, matrix
>>> array([[1,2,3]]).T # array([[1,2,3]])
array([[1, 2, 3],
[2, 4, 6],
[3, 6, 9]])
>>> array([[1,2,3]]) # array([[1,2,3]]).T
array([[14]])
>>> matrix([1,2,3]).T # matrix([1,2,3])
matrix([[1, 2, 3],
[2, 4, 6],
[3, 6, 9]])
>>> matrix([1,2,3]) # matrix([1,2,3]).T
matrix([[14]])
Inplace matrix multiplication: #=
While researching the prior usage, we learn that there is also the inplace matrix multiplication. If we attempt to use it, we may find it is not yet implemented for numpy:
>>> m = matrix([1,2,3])
>>> m #= m.T
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: In-place matrix multiplication is not (yet) supported. Use 'a = a # b' instead of 'a #= b'.
When it is implemented, I would expect the result to look like this:
>>> m = matrix([1,2,3])
>>> m #= m.T
>>> m
matrix([[14]])
What does the “at” (#) symbol do in Python?
# symbol is a syntactic sugar python provides to utilize decorator,
to paraphrase the question, It's exactly about what does decorator do in Python?
Put it simple decorator allow you to modify a given function's definition without touch its innermost (it's closure).
It's the most case when you import wonderful package from third party. You can visualize it, you can use it, but you cannot touch its innermost and its heart.
Here is a quick example,
suppose I define a read_a_book function on Ipython
In [9]: def read_a_book():
...: return "I am reading the book: "
...:
In [10]: read_a_book()
Out[10]: 'I am reading the book: '
You see, I forgot to add a name to it.
How to solve such a problem? Of course, I could re-define the function as:
def read_a_book():
return "I am reading the book: 'Python Cookbook'"
Nevertheless, what if I'm not allowed to manipulate the original function, or if there are thousands of such function to be handled.
Solve the problem by thinking different and define a new_function
def add_a_book(func):
def wrapper():
return func() + "Python Cookbook"
return wrapper
Then employ it.
In [14]: read_a_book = add_a_book(read_a_book)
In [15]: read_a_book()
Out[15]: 'I am reading the book: Python Cookbook'
Tada, you see, I amended read_a_book without touching it inner closure. Nothing stops me equipped with decorator.
What's about #
#add_a_book
def read_a_book():
return "I am reading the book: "
In [17]: read_a_book()
Out[17]: 'I am reading the book: Python Cookbook'
#add_a_book is a fancy and handy way to say read_a_book = add_a_book(read_a_book), it's a syntactic sugar, there's nothing more fancier about it.
If you are referring to some code in a python notebook which is using Numpy library, then # operator means Matrix Multiplication. For example:
import numpy as np
def forward(xi, W1, b1, W2, b2):
z1 = W1 # xi + b1
a1 = sigma(z1)
z2 = W2 # a1 + b2
return z2, a1
Decorators were added in Python to make function and method wrapping (a function that receives a function and returns an enhanced one) easier to read and understand. The original use case was to be able to define the methods as class methods or static methods on the head of their definition. Without the decorator syntax, it would require a rather sparse and repetitive definition:
class WithoutDecorators:
def some_static_method():
print("this is static method")
some_static_method = staticmethod(some_static_method)
def some_class_method(cls):
print("this is class method")
some_class_method = classmethod(some_class_method)
If the decorator syntax is used for the same purpose, the code is shorter and easier to understand:
class WithDecorators:
#staticmethod
def some_static_method():
print("this is static method")
#classmethod
def some_class_method(cls):
print("this is class method")
General syntax and possible implementations
The decorator is generally a named object ( lambda expressions are not allowed) that accepts a single argument when called (it will be the decorated function) and returns another callable object. "Callable" is used here instead of "function" with premeditation. While decorators are often discussed in the scope of methods and functions, they are not limited to them. In fact, anything that is callable (any object that implements the _call__ method is considered callable), can be used as a decorator and often objects returned by them are not simple functions but more instances of more complex classes implementing their own __call_ method.
The decorator syntax is simply only a syntactic sugar. Consider the following decorator usage:
#some_decorator
def decorated_function():
pass
This can always be replaced by an explicit decorator call and function reassignment:
def decorated_function():
pass
decorated_function = some_decorator(decorated_function)
However, the latter is less readable and also very hard to understand if multiple decorators are used on a single function.
Decorators can be used in multiple different ways as shown below:
As a function
There are many ways to write custom decorators, but the simplest way is to write a function that returns a subfunction that wraps the original function call.
The generic patterns is as follows:
def mydecorator(function):
def wrapped(*args, **kwargs):
# do some stuff before the original
# function gets called
result = function(*args, **kwargs)
# do some stuff after function call and
# return the result
return result
# return wrapper as a decorated function
return wrapped
As a class
While decorators almost always can be implemented using functions, there are some situations when using user-defined classes is a better option. This is often true when the decorator needs complex parametrization or it depends on a specific state.
The generic pattern for a nonparametrized decorator as a class is as follows:
class DecoratorAsClass:
def __init__(self, function):
self.function = function
def __call__(self, *args, **kwargs):
# do some stuff before the original
# function gets called
result = self.function(*args, **kwargs)
# do some stuff after function call and
# return the result
return result
Parametrizing decorators
In real code, there is often a need to use decorators that can be parametrized. When the function is used as a decorator, then the solution is simple—a second level of wrapping has to be used. Here is a simple example of the decorator that repeats the execution of a decorated function the specified number of times every time it is called:
def repeat(number=3):
"""Cause decorated function to be repeated a number of times.
Last value of original function call is returned as a result
:param number: number of repetitions, 3 if not specified
"""
def actual_decorator(function):
def wrapper(*args, **kwargs):
result = None
for _ in range(number):
result = function(*args, **kwargs)
return result
return wrapper
return actual_decorator
The decorator defined this way can accept parameters:
>>> #repeat(2)
... def foo():
... print("foo")
...
>>> foo()
foo
foo
Note that even if the parametrized decorator has default values for its arguments, the parentheses after its name is required. The correct way to use the preceding decorator with default arguments is as follows:
>>> #repeat()
... def bar():
... print("bar")
...
>>> bar()
bar
bar
bar
Finally lets see decorators with Properties.
Properties
The properties provide a built-in descriptor type that knows how to link an attribute to a set of methods. A property takes four optional arguments: fget , fset , fdel , and doc . The last one can be provided to define a docstring that is linked to the attribute as if it were a method. Here is an example of a Rectangle class that can be controlled either by direct access to attributes that store two corner points or by using the width , and height properties:
class Rectangle:
def __init__(self, x1, y1, x2, y2):
self.x1, self.y1 = x1, y1
self.x2, self.y2 = x2, y2
def _width_get(self):
return self.x2 - self.x1
def _width_set(self, value):
self.x2 = self.x1 + value
def _height_get(self):
return self.y2 - self.y1
def _height_set(self, value):
self.y2 = self.y1 + value
width = property(
_width_get, _width_set,
doc="rectangle width measured from left"
)
height = property(
_height_get, _height_set,
doc="rectangle height measured from top"
)
def __repr__(self):
return "{}({}, {}, {}, {})".format(
self.__class__.__name__,
self.x1, self.y1, self.x2, self.y2
)
The best syntax for creating properties is using property as a decorator. This will reduce the number of method signatures inside of the class
and make code more readable and maintainable. With decorators the above class becomes:
class Rectangle:
def __init__(self, x1, y1, x2, y2):
self.x1, self.y1 = x1, y1
self.x2, self.y2 = x2, y2
#property
def width(self):
"""rectangle height measured from top"""
return self.x2 - self.x1
#width.setter
def width(self, value):
self.x2 = self.x1 + value
#property
def height(self):
"""rectangle height measured from top"""
return self.y2 - self.y1
#height.setter
def height(self, value):
self.y2 = self.y1 + value
Starting with Python 3.5, the '#' is used as a dedicated infix symbol for MATRIX MULTIPLICATION (PEP 0465 -- see https://www.python.org/dev/peps/pep-0465/)
# can be a math operator or a DECORATOR but what you mean is a decorator.
This code:
def func(f):
return f
func(lambda :"HelloWorld")()
using decorators can be written like:
def func(f):
return f
#func
def name():
return "Hello World"
name()
Decorators can have arguments.
You can see this GeeksforGeeks post: https://www.geeksforgeeks.org/decorators-in-python/
It indicates that you are using a decorator. Here is Bruce Eckel's example from 2008.
Python decorator is like a wrapper of a function or a class. It’s still too conceptual.
def function_decorator(func):
def wrapped_func():
# Do something before the function is executed
func()
# Do something after the function has been executed
return wrapped_func
The above code is a definition of a decorator that decorates a function.
function_decorator is the name of the decorator.
wrapped_func is the name of the inner function, which is actually only used in this decorator definition. func is the function that is being decorated.
In the inner function wrapped_func, we can do whatever before and after the func is called. After the decorator is defined, we simply use it as follows.
#function_decorator
def func():
pass
Then, whenever we call the function func, the behaviours we’ve defined in the decorator will also be executed.
EXAMPLE :
from functools import wraps
def mydecorator(f):
#wraps(f)
def wrapped(*args, **kwargs):
print "Before decorated function"
r = f(*args, **kwargs)
print "After decorated function"
return r
return wrapped
#mydecorator
def myfunc(myarg):
print "my function", myarg
return "return value"
r = myfunc('asdf')
print r
Output :
Before decorated function
my function asdf
After decorated function
return value
To say what others have in a different way: yes, it is a decorator.
In Python, it's like:
Creating a function (follows under the # call)
Calling another function to operate on your created function. This returns a new function. The function that you call is the argument of the #.
Replacing the function defined with the new function returned.
This can be used for all kinds of useful things, made possible because functions are objects and just necessary just instructions.
# symbol is also used to access variables inside a plydata / pandas dataframe query, pandas.DataFrame.query.
Example:
df = pandas.DataFrame({'foo': [1,2,15,17]})
y = 10
df >> query('foo > #y') # plydata
df.query('foo > #y') # pandas