Suppose I have a Python program that runs slow- after profiliing and I have identified the bottleneck. One particular function from a 3rd party module I imported is particularly slow.
For this particular case, I know that function is implemented in Python (Used Eclipse and it's easy to jump to the function definition). So I know that I can convert that function into Cython as a speed-up option. (If it is already implemented in C, there is no point in writing it in Cython...).
If I don't have an IDE, what would be an easy option to determine this?
I know that I can go to the directory where the module is installed and infer that it is in C if the module is in .so. But is there any alternative?
Thanks
Check whether it is an instance of types.FunctionType:
>>> import types
>>> isinstance(len, types.FunctionType)
False
>>> def mylen(): pass
...
>>> isinstance(mylen, types.FunctionType)
True
Probably you'd be safer to check for isinstance(X, (types.FunctionType, types.LambdaType).
C functions are instances of builtin_function_or_method:
>>> len.__class__
<type 'builtin_function_or_method'>
>>> np.vdot.__class__
<type 'builtin_function_or_method'>
You can access this type as types.BuiltinFunctionType/types.BuiltinMethodType.
Alternatively you can check whether the function has a __code__ attribute. Since C functions do not have bytecode, they can't have __code__.
Note sometimes what seems like a function is actually a class, e.g. enumerate but some 3rd party library may do the same. This means that you should also check whether a class is implemented in C or not. This one is harder since all classes are instances of type. A way may be to check whether the class has a __dict__ in its dir, and if it doesn't have you should check for __slots__.
Something like the following should be pretty accurate:
def is_implemented_in_c(obj):
if isinstance(obj, (types.FunctionType, types.LambdaType)):
return False
elif isinstance(obj, type):
if '__dict__' in dir(obj): return False
return not hasattr(obj, '__slots__')
# We accept also instances of classes.
# Return True for instances of C classes, False for python classes.
return not isinstance(obj, types.InstanceType)
Example usage:
>>> is_implemented_in_c(enumerate)
True
>>> is_implemented_in_c(len)
True
>>> is_implemented_in_c(np.vdot)
True
>>> is_implemented_in_c(lambda x: True)
False
>>> is_implemented_in_c(object)
True
>>> class A(object):
... __slots__ = ('a', 'b')
...
>>> is_implemented_in_c(A)
False
Related
I was looking at the PEP 484 section on Forward References and noticed the statement:
...that definition may be expressed as a string literal, to be resolved later.
And that got me wondering, when is "later" and by what? The interpreter doesn't try to resolve it as a literal later, so what does? Is it just if a third party tool is written to do that?
Small example to demonstrate the interpreter result:
class A:
def test(self, a: 'A') -> None:
pass
class B:
def test(self, a: A) -> None:
pass
>>> A().test.__annotations__
{'a': 'A', 'return': None}
>>> B().test.__annotations__
{'a': <class '__main__.A'>, 'return': None}
If my understanding of function annotations and type hints is correct, Python doesn't really do anything with them at runtime to improve performance, but rather the introspective use allows strictly third party applications such as linters, IDEs and static analysis tools (such as mypy) to take advantage of their availability. So would those tools try to resolve the type hint of 'A' rather than having that be a job given to the interpreter and if so, how do they accomplish this?
Update:
By using the typing module, user code can perform the following:
>>> typing.get_type_hints(A().test)
{'a': <class '__main__.A'>, 'return': <class 'NoneType'>}
>>> typing.get_type_hints(B().test)
{'a': <class '__main__.A'>, 'return': <class 'NoneType'>}
However, my question is aimed at whether or not Python has any responsibility in updating the __annotations__ of a function from a string literal, that is to say at runtime change:
>>> A().test.__annotations__
{'a': 'A', 'return': None}
to...
>>> A().test.__annotations__
{'a': <class '__main__.A'>, 'return': None}
If Python doesn't do it, then why would I want a string literal as a type hint other than for self-documented code? What value does the first form give to me, a user or a third party tool?
Consider the following code:
class Foo:
def bar(self) -> Foo:
return Foo()
This program will actually crash at runtime if you try running it with Python: when the interpreter sees the definition of bar, the definition of Foo is not yet finished. So, since Foo has not yet been added to the global namespace, we can't use it as a type hint yet.
Similarly, consider this program:
class Foo:
def bar(self) -> Bar:
return Bar()
class Bar:
def foo(self) -> Foo:
return Foo()
This mutually dependent definition suffers from the same problem: while we're evaluating Foo, Bar hasn't been evaluated yet so the interpreter throws an exception.
There are three solutions to this problem. The first is to make some of your type hints strings, effectively "forward declaring" them:
class Foo:
def bar(self) -> "Foo":
return Foo()
This satisfies the Python interpreter, and won't disrupt third party tools like mypy: they can just remove the quotes before parsing the type. The main disadvantage is that this syntax looks sort of ugly and clunky.
The second solution is to use type comments syntax:
class Foo:
def bar(self):
# type: () -> Foo
return Foo()
This has the same benefits and disadvantages as the first solution: it satisfies the interpreter and tooling, but looks hacky and ugly. It also has the additional benefit that it keeps your code backwards-compatibile with Python 2.7.
The third solution is Python 3.7+ only -- use the from __future__ import annotations directive:
from __future__ import annotations
class Foo:
def bar(self) -> Foo:
return Foo()
This will automatically make all annotations be represented as strings. So we get the benefit of the first solution, but without the ugliness.
This behavior will eventually become the default in future versions of Python.
It also turns out that automatically making all annotations strings can come with some performance improvements. Constructing types like List[Dict[str, int]] can be surprisingly expensive: they're just regular expressions at runtime and evaluated as if they were written as List.__getitem__(Dict.__getitem__((str, int)).
Evaluating this expression is somewhat expensive: we end up performing two method calls, constructing a tuple, and constructing two objects. This isn't counting any additional work that happens in the __getitem__ methods themselves, of course -- and the work that happens in those methods ends up being non-trivial out of necessity.
(In short, they need to construct special objects that ensure types like List[int] can't be used in inappropriate ways at runtime -- e.g. in isinstance checks and the like.)
I have a variable f. How can I determine its type? Here is my code, typed into a Python interpreter, showing that I get an error using the successful pattern of the many examples I have found with Google. (Hint: I am very new to Python.)
>>> i=2; type(i) is int
True
>>> def f():
... pass
...
>>> type(f)
<class 'function'>
>>> type(i)
<class 'int'>
>>> type(f) is function
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'function' is not defined
>>> f=3
>>> type(f) is int
True
The pythonic way to check the type of a function is using isinstance builtin.
i = 2
type(i) is int # not recommended
isinstance(i, int) # recommended
Python includes a types module for checking functions among other things.
It also defines names for some object types that are used by the
standard Python interpreter, but not exposed as builtins like int or
str are.
So, to check if an object is a function, you can use the types module as follows
def f():
print("test")
import types
type(f) is types.FunctionType # Not recommended but it does work
isinstance(f, types.FunctionType) # Recommended.
However, note that it will print false for builtin functions. If you wish to include those as well, then check as follows
isinstance(f, (types.FunctionType, types.BuiltinFunctionType))
However, use the above if you only want specifically functions. Lastly, if you only care about checking if it is one of function, callable or method, then just check if it behaves like a callable.
callable(f)
Since types.BuiltinFunctionType is exactly the same thing as types.BuiltinMethodType:
>>> import types
>>> types.BuiltinFunctionType is types.BuiltinMethodType
True
it isn't possible to distinguish functions from methods that are implemented in C:
>>> isinstance(sorted, types.BuiltinMethodType)
True
>>> isinstance(int.from_bytes, types.BuiltinFunctionType)
True
And yet, python clearly knows that sorted is a function, and int.from_bytes is a method, which we can see in their repr:
>>> sorted
<built-in function sorted>
>>> int.from_bytes
<built-in method from_bytes of type object at 0x7fded4c9d860>
Why, then, are BuiltinFunctionType and BuiltinMethodType the same thing? Is there or isn't there a difference between functions and methods defined in C?
In python I can create a class without class statement:
MyClass = type('X', (object,), dict(a=1))
Is there a way to create a function without 'def'?
Thats as far as i got...
d={} # func from string
exec'''\
def synthetics(s):
return s*s+1
''' in d
>>> d.keys()
['__builtins__', 'synthetics']
>>> d['synthetics']
<function synthetics at 0x00D09E70>
>>> foo = d['synthetics']
>>> foo(1)
2
Technically, yes, this is possible. The type of a function is, like all other types, a constructor for instances of that type:
FunctionType = type(lambda: 0)
help(FunctionType)
As you can see from the help, you need at minimum code and globals. The former is a compiled bytecode object; the latter is a dictionary.
To make the code object, you can use the code type's constructor:
CodeType = type((lambda: 0).func_code)
help(CodeType)
The help says this is "not for the faint of heart" and that's true. You need to pass bytecode and a bunch of other stuff to this constructor. So the easiest way to get a code object is from another function, or using the compile() function. But it is technically possible to generate code objects completely synthetically if you understand Python bytecode well enough. (I have done this, on a very limited basis, to construct signature-preserving wrapper functions for use in decorators.)
PS -- FunctionType and CodeType are also available via the types module.
There might be a more direct way than the following, but here's a full-blown function without def. First, use a trivial lambda expression to get a function object:
>>> func = lambda: None
Then, compile some source code to get a code object and use that to replace the lambda's code:
>>> func.__code__ = compile("print('Hello, world!')", "<no file>", "exec")
>>> func()
Hello, world!
The "dir()" function in python retrieves all attributes for a class. I was wondering if there was a similar function that returns only user defined functions? Thanks!
If you want to tell a builtin from a user-defined function I'd use the types module.
For instance:
>>> def hello():
... print("hi")
...
>>> import types
>>> type(hello) is types.BuiltinFunctionType
False
>>> type(hello) is types.FunctionType
True
Then it depends on what you want to do.You could use list comprehensions to check all attributes of a class and keep only those that turn out to be true.
[ x for x in dir(yourclass) if (type(x) is types.FunctionType) ]
Hope it helps.