I'm having some issues with the self parameter, and some seemingly inconsistent behavior in Python is annoying me, so I figure I better ask some people in the know. I have a class, Foo. This class will have a bunch of methods, m1, through mN. For some of these, I will use a standard definition, like in the case of m1 below. But for others, it's more convinient to just assign the method name directly, like I've done with m2 and m3.
import os
def myfun(x, y):
return x + y
class Foo():
def m1(self, y, z):
return y + z + 42
m2 = os.access
m3 = myfun
f = Foo()
print f.m1(1, 2)
print f.m2("/", os.R_OK)
print f.m3(3, 4)
Now, I know that os.access does not take a self parameter (seemingly). And it still has no issues with this type of assignment. However, I cannot do the same for my own modules (imagine myfun defined off in mymodule.myfun). Running the above code yields the following output:
3
True
Traceback (most recent call last):
File "foo.py", line 16, in <module>
print f.m3(3, 4)
TypeError: myfun() takes exactly 2 arguments (3 given)
The problem is that, due to the framework I work in, I cannot avoid having a class Foo at least. But I'd like to avoid having my mymodule stuff in a dummy class. In order to do this, I need to do something ala
def m3(self,a1, a2):
return mymodule.myfun(a1,a2)
Which is hugely redundant when you have like 20 of them. So, the question is, either how do I do this in a totally different and obviously much smarter way, or how can I make my own modules behave like the built-in ones, so it does not complain about receiving 1 argument too many.
os.access() is a built-in function, in the sense that it's part of an extension module written in C. When the class is defined, Python doesn't recognize m2 as a method because it's the wrong type — methods are Python functions, not C functions. m3, however, is a Python function, so it's recognized as a method and handled as such.
In other words, it's m2 that's exceptional here, not m3.
One simple way to do what you want would be to make m3 a static method:
m3 = staticmethod(myfun)
Now the interpreter knows never to try and pass myfun() a self parameter when it's called as the m3 method of a Foo object.
I just want to add that the behaviour is not inconsistent as already Luke hinted.
Just try the following
print Foo.__dict__
{'__doc__': None,
'__module__': '__main__',
'm1': <function m1 at 0x02861630>,
'm2': <built-in function access>,
'm3': <function myfun at 0x028616F0>}
Here you can see that Python can't distinguish between m1 and m2.
That's why both are evaluated to a bound-method.
A bound method is something like a method with an additional first argument pointing to an object: self.m(1, 2) -> m(self, 1, 2)
This binding behaviour is only implemented for User-defined methods. That explains why self.m2("/", os.R_OK) is not evaluated to m2(self, "/", os.R_OK).
One last demo:
print Foo.m1
<unbound method Foo.m1>
print Foo.m2
<built-in function access>
print f.m1
<bound method Foo.m1 of <__main__.Foo instance at 0x02324418>>
print f.m2
<built-in function access>
Further information about the different function types can be found here:
http://docs.python.org/reference/datamodel.html
And as mentioned before this binding mechanism can also be prevented by using a staticmethod descriptor:
http://docs.python.org/library/functions.html#staticmethod
I think you're looking for staticmethod(). See docs here.
m2 = staticmethod(os.access)
m3 = staticmethod(myfun)
As to why m2 worked in your example and m3 didn't, that's not clear to me. Printing f.m2 and f.m3 in your original example reveals that f.m2 is a direct reference to built-in function os.access, while f.m3 is a bound method (bound to the instance f).
You should use the staticmethod function in this case. When writing static class methods, you can use it as a decorator:
class A:
def printValue(self,value):
print value
#staticmethod
def staticPrintValue(value):
print value
>>> A.printValue(5)
Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
A.printValue(5)
TypeError: unbound method printValue() must be called with A instance as first argument (got int instance instead)
>>> A.staticPrintValue(5)
5
One way would be to manually apply the static method decorator:
class Foo(object):
m3 = staticmethod(mymodule.myfun)
Related
In cpython this code would work:
import inspect
from types import FunctionType
def f(a, b): # line 5
print(a, b)
f_clone = FunctionType(
f.__code__,
f.__globals__,
closure=f.__closure__,
name=f.__name__
)
f_clone.__annotations__ = {'a': int, 'b': int}
f_clone.__defaults__ = (1, 2)
print(inspect.signature(f_clone)) # (a: int = 1, b: int = 2)
print(inspect.signature(f)) # (a, b)
f_clone() # 1 2
f(1, 2) # 1 2
try:
f()
except TypeError as e:
print(e) # f() missing 2 required positional arguments: 'a' and 'b'
However in cython when calling f_clone, I get:
XXX lineno: 5, opcode: 0
Traceback (most recent call last):
...
File "test.py", line 5, in f # line of f definitio
SystemError: unknown opcode
I need this to create a copy of class __init__ method on each class creation and and modify its signature, but keep original __init__ signature untouched.
Edit:
Changes made to signature of copied object must not affect runtime calls and needed only for inspection purposes.
I am relatively convinced this is never going to work well. If I were you I'd modify your code to fail elegantly for unclonable functions (maybe by just using the original __init__ and not replacing it, since this seems to be a purely cosmetic approach to generate prettier docstrings). After that you could submit an issue to the Cython issue tracker - however the maintainers of Cython know that full-introspection compatibility with Python is very challenging, so may not be hugely interested.
One of the main reasons I think you should just handle the error rather than find a workaround is that Cython is not the only method to accelerate Python. For example Numba can generate classes containing JIT accelerated code, or people can write their own functions in C (either as a C-API function, or perhaps wrapped with Ctypes or CFFI). These are all situations where your rather fragile introspection approach is likely to break. Handling the error fixes it for all of these; while you're likely to need an individual workaround for each one, plus all the methods I haven't thought of, plus any that are developed in the future.
Some details about Cython functions: at the moment a Cython has a compilation option called binding that can generate functions in two different modes:
With binding=False functions have the type builtin_function_or_method, which has minimum introspection capacities, and so no __code__, __globals__, __closure__ (or most other) attributes.
With binding=True functions have the type cython_function_or_method. This has improved introspection capacity, so does provide most of the expected annotations. However some of them are nonsense defaults - specifically __code__. The __code__ attribute is expected to be Python bytecode, however Cython doesn't use Python bytecode (since it's compiled to C). Therefore it just provides a dummy attribute.
It looks like Cython defaults to binding=True when compiling a .py file and when compiling a regular (non-cdef) class, giving the behaviour you report. However, when compiling a .pyx file it currently defaults to binding=False. It's possible you may also want to handle the binding=False case in some circumstances too.
Having established that trying to create a regular Python function object with the __code__ attribute of a cython_function_or_method isn't going to work, let's look at a few other options:
>>> print(f)
<cyfunction f at 0x7f08a1c63550>
>>> type(f)()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: cannot create 'cython_function_or_method' instances
So you can't create your own cython_function_or_method and populate it from Python - the type does not have a user callable constructor.
copy.copy appears to work, but doesn't actually create a new instance:
>>> import copy
>>> copy.copy(f)
<cyfunction f at 0x7f08a1c63550>
Note, however, that this has exactly the same address - it isn't a copy:
>>> copy.copy(f) is f
True
At which point I'm out of ideas.
What I don't quite get is why you don't use functools.wraps?
#functools.wraps(f):
def wrapper(*args, **kwargs):
return f(*args, **kwargs)
This updates wrapper with most of the relevant introspection attributes from f, works for both types of Cython function (to an extent - the binding=False case doesn't provide much useful information), and should work for most other types of function too.
It's possible I'm missing something, but it seems a whole lot less fragile than your scheme of copying code objects.
I'm teaching myself how to code with the help of some online tutorials. I've encountered "decorators", I can seem to understand how it works but something bothers me. Here's the code given:
def decor(func):
def wrap():
print("-----------")
func()
print("-----------")
return wrap
def print_text():
print("Hello World")
decorated = decor(print_text)
decorated()
output:
-----------
Hello World
-----------
The things that I want to understand are:
Why do you have to call "return wrap" instead of "return wrap()"? Whereas if you don't you'll get a "TypeError: 'NoneType' object is not callable.
When I assigned the value of decorated variable. How come I also had to use "print_text" rather than "print_text()" whereas it'll raise the same TypeError if I do?
When I used the variable "decorated". Why did I have to call it like a function (adding () at the end). When I call it using "decorated" or "print(decorated)" it says something completely different?
Sorry for the dumb questions. But I'm just starting out so please bear with me. Also please make your responses beginner-friendly. Thank you
In Python, just about everything is an object. Functions are objects too. You can reference them by their name:
>>> def print_text():
... print("Hello World")
...
>>> print_text # **no** call here, this is just referencing the object
<function print_text at 0x10e3f1c80>
>>> print_text() # With a call, so now we *run* the function
Hello World
Adding () to the name told Python to call the function, which caused it to actually execute the function body, without the call, it is just showing you what the name references.
You can assign function objects to other names too. Those other names can still be called, invoking the function:
>>> other_name = print_text
>>> other_name
<function print_text at 0x10e3f1c80>
>>> other_name()
Hello World
So other_name is just another reference to the same object, and adding () (a call expression) causes the function object to be executed. print_text() and other_name() do the exact same thing, run the code inside the function.
That's what name func inside of decor() refers to; it is a reference to the same function object. You passed it in with decor(print_text). Only later on, inside wrapper() the expression func() calls that function object. If you passed in print_text() instead, you'd pass in the None object that function returned, and None can't be called:
>>> return_value = print_text()
Hello World
>>> return_value is None
True
>>> return_value()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object is not callable
Next, return wrapper returns the newly created wrapper function object to the caller. If you did return wrapper(), you'd return the result of the function, not the function object itself.
The whole point of a decorator is to replace the original function object with a new object that does extra stuff, which is why a decorator returns that replacement object (in your example wrapper) so that in future when you call decorated(), you call that wrapper function doing something extra before and after calling the original function (via the func name, which references print_text()).
So what decor(some_function) does is return a new function object, one that'll print something, call the function object that was passed in, then print something else. That new function object can then be used to replace the old function object.
This question already has an answer here:
Python: subscript a module
(1 answer)
Closed 7 years ago.
So it's quite a simple question. how do I add __getitem__ to a Python module. I mostly just want it as an ease of use, however it's confusing why it wouldn't let me 'just set it'. Below is a simple example of __getitem__ semi-working, however I wish for the other['test'] to work.
Here's the full output:
hello
hello
Traceback (most recent call last):
File "main.py", line 4, in <module>
print other['test']
TypeError: 'module' object has no attribute '__getitem__'
main.py
import other
print other.get('test')
print other.__getitem__('test')
print other['test']
other.py
test = 'hello'
def __getitem__(name):
return globals()[name]
get = __getitem__
I've tried to set __getitem__ using globals() aswell, globals()['__getitem__'] = __getitem__. It didn't work. And I tried to set it in main.py. So I'm confused as to why it's so adamant in not allowing me to use other['test'].
If it's impossible, then a short reason would be good.
Special methods are looked up on the type, not on an instance. Python looks for type(other).__getitem__() and that isn't available. You'd have to add the __getitem__ method to the module type; you can't in Python.
You'd have to replace the whole module instance in sys.modules with an instance of your own class to achieve what you want:
class MyModule(object):
def __init__(self, namespace):
self.__dict__.update(namespace)
def __getitem__(name):
return self.__dict__[name]
import other
import sys
sys.modules[other.__name__] = MyModule(other.__dict__)
This limitation doesn't just apply for modules, it applies for anything such that the type is not object or some subclass of object, or something with a metaclass that never bottoms out with object in the mro.
For example, you can also see this happening with type type:
In [32]: class Foo(type):
....: pass
....:
In [33]: type(Foo)
Out[33]: type
In [34]: Foo.__getitem__ = lambda x, y: x.__dict__.get(y)
In [35]: Foo.foo = "hello"
In [36]: Foo['foo']
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-38-e354ca231ddc> in <module>()
----> 1 Foo['foo']
TypeError: 'type' object has no attribute '__getitem__'
In [37]: Foo.__dict__.get('foo')
Out[37]: 'hello'
The reason is that at the C-API level, both module and type are particular instances of PyTypeObject which don't implement the required protocol for inducing the same search mechanism that the PyTypeObject implementation of object and friends does implement.
To change this aspect of the language itself, rather than hacking a replacement of sys.modules, you would need to change the C source definitions for PyModule_Type and PyType_Type such that there were C functions created for __getitem__ and added to the appropriate location in the C-API big PyTypeObject struct-o-magic-functions (a lot of which is expanded by the macro PyObject_HEAD) instead of 0 (which is the sentinel for does not exist), and recompile Python itself with these modified implementations of module and type.
In python I can create a class without class statement:
MyClass = type('X', (object,), dict(a=1))
Is there a way to create a function without 'def'?
Thats as far as i got...
d={} # func from string
exec'''\
def synthetics(s):
return s*s+1
''' in d
>>> d.keys()
['__builtins__', 'synthetics']
>>> d['synthetics']
<function synthetics at 0x00D09E70>
>>> foo = d['synthetics']
>>> foo(1)
2
Technically, yes, this is possible. The type of a function is, like all other types, a constructor for instances of that type:
FunctionType = type(lambda: 0)
help(FunctionType)
As you can see from the help, you need at minimum code and globals. The former is a compiled bytecode object; the latter is a dictionary.
To make the code object, you can use the code type's constructor:
CodeType = type((lambda: 0).func_code)
help(CodeType)
The help says this is "not for the faint of heart" and that's true. You need to pass bytecode and a bunch of other stuff to this constructor. So the easiest way to get a code object is from another function, or using the compile() function. But it is technically possible to generate code objects completely synthetically if you understand Python bytecode well enough. (I have done this, on a very limited basis, to construct signature-preserving wrapper functions for use in decorators.)
PS -- FunctionType and CodeType are also available via the types module.
There might be a more direct way than the following, but here's a full-blown function without def. First, use a trivial lambda expression to get a function object:
>>> func = lambda: None
Then, compile some source code to get a code object and use that to replace the lambda's code:
>>> func.__code__ = compile("print('Hello, world!')", "<no file>", "exec")
>>> func()
Hello, world!
I want to get the line number of a python function in the source code.
What i have in runtime is module, class, method objects
Had a look at inspect
inspect.getsourcelines(object)
which also gives line number in result.
I see that for methods with decorators, line no. returned from above inspect function points to the actual decorator's source code rather than desired function's source code.
So any way to workaround this?
(i understand that interpreter does something like wrapping function inside decorator in runtime, but i might be wrong)
There is no easy solution in the general case.
A decorator is a function that given a function returns a function, normally by "wrapping" it in a closure that performs the operation for which the decorator has been designed.
The file and line number information are not however in the function object itself and you cannot "fix" them by copying this information from the wrapped function to the wrapper. That data is instead contained in the code object of the function (available with .func_code), and it is shared among all closures you are going to create.
>>> def bar(x):
... def foo():
... return x
... return foo
...
>>> f1 = bar(1)
>>> f2 = bar(2)
>>> f1()
1
>>> f2()
2
>>> f1.func_code is f2.func_code
True
>>>
The wrapt module solves this problem by allowing you to write decorators which preserve the necessary metadata to find the source of a function as well as perform other introspection. It's like an improved functools.wraps.