I am writing a small application that takes users' input to give them a set of optimal parameters to use. (Each of these sets are ranked and the user can select whichever one they want to use)
To be able to do this, I select one function from an array of choices(depending on the context), partially fill the function using functools.partial then return the partial object to another module which in turn calls a C++ library (dlib) which has a python interface.
Up until today, I wasn't using functools.partial to fill the function and faced no issues. But to make the code less repetitive and easier to understand, I added that in. After adding that part, I get the following error :
AttributeError: 'functools.partial' object has no attribute '__code__'
I read a few posts and realized that this is an issue with partial objects as they often lack attributes like __name__, __module__ etc but I am not sure how to resolve this issue.
PS: I am using python 3.7
EDIT
I am adding a small code that reproduces the error
from functools import partial
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.datasets import load_breast_cancer
from dlib import find_max_global
def objective_calculator(*args, X, y):
args = args[0]
model = LogisticRegression()
model.set_params(**{'class_weight': args[0], 'max_iter':args[1]})
model.fit(train['data'], train['target'])
predictions = model.predict(X)
return accuracy_score(y, predictions)
train = load_breast_cancer()
obj_calc = partial(objective_calculator, X=train['data'], y=train['target'])
lower_bounds = [0.1, 10] # class_weight, max_iter
upper_bounds = [0.5, 200] # class_weight, max_iter
is_integer_variable = [False, True]
find_max_global(f=obj_calc,
bound1=lower_bounds,
bound2=upper_bounds,
num_function_calls=2,
is_integer_variable=is_integer_variable,
solver_epsilon=1,)
Running the above code results in the following error
AttributeError: 'functools.partial' object has no attribute '__code__'
Is it advisable to manually add the __code__ attribute to the partial object?
AttributeError: 'functools.partial' object has no attribute '__code__'
for solving this error we can use
wraps
functools.WRAPPER_ASSIGNMENTS to update attributes,
which defaults to ('module', 'name', 'doc')
in python 2.7.6
or,
we can update only present attributes...
import functools
import itertools
def wraps_safely(obj, attr_names=functools.WRAPPER_ASSIGNMENTS):
return wraps(obj, assigned=itertools.ifilter(functools.partial(hasattr, obj), attr_names))
`>>> def foo():
`... ` ` """ Ubiquitous foo function ...."""`
...
>>> functools.wraps(partial(foo))(foo)()
`Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File ```"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/functools.py", ``line 33, in update_wrapper`
setattr(wrapper, attr, getattr(wrapped, attr))
AttributeError: 'functools.partial' object has no attribute '__module__'
>>> wraps_safely(partial(foo))(foo)()
>>> `
`
just filter out all those attribute which aren't present.
second approach would be:-
*strictly deal with partial objects only.
fold wraps with singledispatch and it creates wrapped partial object and its attribute would be taken from the function "fun" attribute.
import functools
def wraps_partial(wrapper, *args, **kwargs):
` """ Creates a callable object whose attributes will be set from the partials nested func attribute ..."""
` ` ` wrapper = wrapper.func``
while isinstance(wrapper, functools.partial):
wrapper = wrapper.func
return functools.wraps(wrapper, *args, **kwargs)
# after returning functools.wraps
def foo():
""" Foo function.
:return: None """
pass
>>> wraps_partial(partial(partial(foo)))(lambda : None).__doc__
' Foo Function, returns None '
>>> wraps_partial(partial(partial(foo)))(lambda : None).__name__
'foo'
>>> wraps_partial(partial(partial(foo)))(lambda : None)()
>>> pfoo = partial(partial(foo))
>>> #wraps_partial(pfoo)
... def not_foo():
...
""" Not Foo function ... """
...
>>> not_foo.__doc__
' Foo Function, returns None '
>>> not_foo.__name__
'foo'
>>>
now we can get the original functions docs.
python(CPython) 3.
from functools import wraps, partial, WRAPPER_ASSIGNMENTS
try:
wraps(partial(wraps))(wraps)
except AttributeError:
#wraps(wraps)
def wraps(obj, attr_names=WRAPPER_ASSIGNMENTS, wraps=wraps):
return wraps(obj, assigned=(name for name in attr_names if hasattr(obj, name)))
*we define a new wraps function only if we fail to wrap a partial,*
*use the original wraps to copy the docs*
also,
In (Python 3.5) we have a reference to the original function, and it is maintained in the partial. You can access it as .func:
from functoolsimport partial
def a(b):
print(b)
In[20]: c=partial(a,5)
In[21]: c.func.__module__
Out[21]: '__main__'
In[22]: c.func.__name__
Out[22]: 'a'
I would not dare adding a __code__ attribute to a partial object. The __code__ attribute allows a low level access to the compiled Python code. It is normally never used in common scripts and is probably used here to interface it with the underlying C++ library.
The bullet proof way is to define a new function. In Python def is an executable statement, and it is possible to iterately redefine a function:
def objective_calculator(*args, X, y):
...
for X, y in ...:
def obj_calc(*args):
return objective_calculator(*args, X, y)
...
find_max_global(f=obj_calc, ...)
obj_calc is now a true Python function and it will have its own __code__ attribute.
If the dlib library supports it, it could be possible to use a lambda:
find_max_global(f=lambda *args: objective_calculator(*args, X, y), ...)
A lambda is almost a true function and has indeed a __code__ attribute, but it is defined as a separate object class in Python, so depending on the dlib library requirements (I could not find any reference on it) it could work or not.
Related
Suppose I have a python object x and a string s, how do I set the attribute s on x? So:
>>> x = SomeObject()
>>> attr = 'myAttr'
>>> # magic goes here
>>> x.myAttr
'magic'
What's the magic? The goal of this, incidentally, is to cache calls to x.__getattr__().
setattr(x, attr, 'magic')
For help on it:
>>> help(setattr)
Help on built-in function setattr in module __builtin__:
setattr(...)
setattr(object, name, value)
Set a named attribute on an object; setattr(x, 'y', v) is equivalent to
``x.y = v''.
However, you should note that you can't do that to a "pure" instance of object. But it is likely you have a simple subclass of object where it will work fine. I would strongly urge the O.P. to never make instances of object like that.
Usually, we define classes for this.
class XClass( object ):
def __init__( self ):
self.myAttr= None
x= XClass()
x.myAttr= 'magic'
x.myAttr
However, you can, to an extent, do this with the setattr and getattr built-in functions. However, they don't work on instances of object directly.
>>> a= object()
>>> setattr( a, 'hi', 'mom' )
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'object' object has no attribute 'hi'
They do, however, work on all kinds of simple classes.
class YClass( object ):
pass
y= YClass()
setattr( y, 'myAttr', 'magic' )
y.myAttr
let x be an object then you can do it two ways
x.attr_name = s
setattr(x, 'attr_name', s)
Also works fine within a class:
def update_property(self, property, value):
setattr(self, property, value)
If you want a filename from an argument:
import sys
filename = sys.argv[1]
file = open(filename, 'r')
contents = file.read()
If you want an argument to show on your terminal (using print()):
import sys
arg = sys.argv[1]
arg1config = print(arg1config)
I want to be able to get the length of the _fields member of a namedtuple from another module. However, it is flagged as protected.
The workaround I have is as follows:
MyTuple = namedtuple(
'MyTuple',
'a b'
)
"""MyTuple description
Attributes:
a (float): A descrip
b (float): B descrip
"""
NUM_MY_TUPLE_FIELDS = len(MyTuple._fields)
Then I import NUM_MY_TUPLE_FIELDS from the external module.
I was trying to find a way to make the functionality part of the class, such as to extend the namedtuple with a __len__ method. Is there a more pythonic way to get the number of fields in a namedtuple from an external module?
Updated to show the autodoc comments. The protected warning is seen in PyCharm. Originally, in the external module I simply imported MyTuple, then used:
x = len(MyTuple._fields)
I tried the following suggestion and thought it was going to work, but I get the following: TypeError: object of type 'type' has no len().
class MyTuple(typing.MyTuple):
a: float
b: float
"""MyTuple doc
Attributes:
a (float): A doc
b (float): B doc
"""
def __len__(self) -> int:
return len(self._fields)
fmt_str = f"<L {len(MyTuple)}f" # for struct.pack usage
print(fmt_str)
you can use inheritance:
class MyTuple(namedtuple('MyTuple', 'a b c d e f')):
"""MyTuple description
Attributes:
a (float): A description
...
"""
#property
def fields(self):
# _fields is a class level attribute and available via
# MyTuple._fields from external modules
return self._fields
def __len__(self):
# your implementation if you need it
return len(self._fields)
or use typing.NamedTuple if you are using python 3.5+
class MyTuple(typing.NamedTuple):
a: int
# other fields
One way is to use inspect.signature and just count how many parameters the __new__ method requires:
import inspect
n_fields = len(inspect.signature(NTClass).parameters)
This works because typing.NamedTuple disallows overriding the __new__ method, and that is unlikely to change due to the way it is implemented:
>>> import inspect
>>> from typing import NamedTuple
>>> class NTClass(NamedTuple):
... x: int
... y: float
...
>>> len(inspect.signature(NTClass).parameters)
2
It also works for the old collections.namedtuple:
>>> from collections import namedtuple
>>> NTClass = namedtuple("NTClass", "x y")
>>> len(inspect.signature(NTClass).parameters)
2
I don't understand why this code doesn't work:
import numpy as np
class Normalizer:
def __init__(self,x):
self.x = x
def mean(self):
return np.sum(self.x)/np.size(self.x)
def mean_zero(self):
return self.x - self.x.mean()
def new_calc(self):
return self.x.mean_zero()
a = np.random.randint(150,200,(5,8))
heights = Normalizer(a)
print(a)
print(heights.mean())
print(heights.mean_zero())
print(heights.mean_zero().mean())
print(heights.new_calc())
It executes heghts.mean_zero() correctly but in the method def new_calc(self) it doesn't execute it. It would be great if someone could explain that to me. Thanks!
I don't understand why this code doesn't work:
if you run the following code it will throw an error:
AttributeError: 'numpy.ndarray' object has no attribute 'mean_zero'
locate the problem, the only place where mean_zero has been called is new_calc method. So, first step done.
analyze, if you look at Normalize class it has one attribute x which is of the type numpy.ndarray. If you carefully read the error message it says that ndarray type doesn't have the attribute mean_zero. On the other hand you have mean_zero method defined in your class and that is the one you should call.
These two steps leads to conclusion that the problem is in new_calc method:
def new_calc(self):
return self.mean_zero() #(wrong)return self.x.mean_zero()
I am not sure what x from the __init__ is but it is very likely that you actually want to call mean_zero in new_calc in the context of self variable (the same object):
def new_calc(self):
return self.mean_zero()
Insted of self.x.mean_zero() write self.mean_zero()
import numpy as np
class Normalizer:
def __init__(self,x):
self.x = x
def mean(self):
return np.sum(self.x)/np.size(self.x)
def mean_zero(self):
return self.x - self.mean()
def new_calc(self):
return self.mean_zero()
a = np.random.randint(150,200,(5,8))
heights = Normalizer(a)
print(a)
print(heights.mean())
print(heights.mean_zero())
print(heights.mean_zero().mean())
print(heights.new_calc())
You're initializing Normalizer with 'a', which is the output of np.random.randint, which returns a numpy.ndarray object.
In the new_calc method you are attempting to call the mean_zero method of the ndarray object, but ndarray has no such method. mean_zero is a method on Normalizer, but self.x is not of type Normalizer.
I'm not sure what this code new_calc is supposed to do. If you can make that clearer, I may be able to provide more assistance.
The culprit:
def new_calc(self):
return self.x.mean_zero()
The reason:
self.x is an attribute of the Normalizer class. So if heights is an instance of the Normalizer class, then heights.x is the self.x.
The answer:
def new_calc(self):
return self.mean_zero()
The justification:
AttributeError: 'numpy.ndarray' object has no attribute 'mean_zero'
ndarray has no such method. mean_zero is a method of Normalizer
Suppose I have a python object x and a string s, how do I set the attribute s on x? So:
>>> x = SomeObject()
>>> attr = 'myAttr'
>>> # magic goes here
>>> x.myAttr
'magic'
What's the magic? The goal of this, incidentally, is to cache calls to x.__getattr__().
setattr(x, attr, 'magic')
For help on it:
>>> help(setattr)
Help on built-in function setattr in module __builtin__:
setattr(...)
setattr(object, name, value)
Set a named attribute on an object; setattr(x, 'y', v) is equivalent to
``x.y = v''.
However, you should note that you can't do that to a "pure" instance of object. But it is likely you have a simple subclass of object where it will work fine. I would strongly urge the O.P. to never make instances of object like that.
Usually, we define classes for this.
class XClass( object ):
def __init__( self ):
self.myAttr= None
x= XClass()
x.myAttr= 'magic'
x.myAttr
However, you can, to an extent, do this with the setattr and getattr built-in functions. However, they don't work on instances of object directly.
>>> a= object()
>>> setattr( a, 'hi', 'mom' )
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'object' object has no attribute 'hi'
They do, however, work on all kinds of simple classes.
class YClass( object ):
pass
y= YClass()
setattr( y, 'myAttr', 'magic' )
y.myAttr
let x be an object then you can do it two ways
x.attr_name = s
setattr(x, 'attr_name', s)
Also works fine within a class:
def update_property(self, property, value):
setattr(self, property, value)
If you want a filename from an argument:
import sys
filename = sys.argv[1]
file = open(filename, 'r')
contents = file.read()
If you want an argument to show on your terminal (using print()):
import sys
arg = sys.argv[1]
arg1config = print(arg1config)
This following code works fine, and shows a way to create attributes and methods in execution time:
class Pessoa:
pass
p = Pessoa( )
p.nome = 'fulano'
if hasattr(p, 'nome'):
print(p)
p.get_name = lambda self:'Sr.{}'.format(self.nome)
But, I think my way to create methods is not correct. There are another way to create a method dynamically ?
[Although this has really been answered in Steven Rumbalski's comment, pointing to two independent questions, I'm adding a short combined answer here.]
Yes, you're right that this does not correctly define a method.
>>> class C:
... pass
...
>>> p = C()
>>> p.name = 'nickie'
>>> p.get_name = lambda self: 'Dr. {}'.format(self.name)
>>> p.get_name()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: <lambda>() takes exactly 1 argument (0 given)
Here's how you can call the function that is stored in object p's attribute called get_name:
>>> p.get_name(p)
'Dr. nickie'
For properly defining an instance method dynamically, take a look at the answers to a relevant question.
If you want to define a class method dynamically, you have to define it as:
>>> C.get_name = lambda self: 'Dr. {}'.format(self.name)
Although the method will be added to existing objects, this will not work for p (as it already has its own attribute get_name). However, for a new object:
>>> q = C()
>>> q.name = 'somebody'
>>> q.get_name()
'Dr. somebody'
And (obviously), the method will fail for objects that don't have a name attribute:
>>> r = C()
>>> r.get_name()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 1, in <lambda>
AttributeError: C instance has no attribute 'name'
There are two ways to dynamically create methods in Python 3:
create a method on the class itself: just assign a function to a member ; it is made accessible to all objects of the class, even if they were created before the method was created:
>>> class A: # create a class
def __init__(self, v):
self.val = v
>>> a = A(1) # create an instance
>>> def double(self): # define a plain function
self.val *= 2
>>> A.double = double # makes it a method on the class
>>> a.double() # use it...
>>> a.val
2
create a method on an instance of the class. It is possible in Python 3 thanks to the types module:
>>> def add(self, x): # create a plain function
self.val += x
>>> a.add = types.MethodType(add, a) # make it a method on an instance
>>> a.add(2)
>>> a.val
4
>>> b = A(1)
>>> b.add(2) # chokes on another instance
Traceback (most recent call last):
File "<pyshell#55>", line 1, in <module>
b.add(2)
AttributeError: 'A' object has no attribute 'add'
>>> type(a.add) # it is a true method on a instance
<class 'method'>
>>> type(a.double)
<class 'method'>
A slight variation on method 1 (on class) can be used to create static or class methods:
>>> def static_add(a,b):
return a+b
>>> A.static_add = staticmethod(static_add)
>>> a.static_add(3,4)
7
>>> def show_class(cls):
return str(cls)
>>> A.show_class = classmethod(show_class)
>>> b.show_class()
"<class '__main__.A'>"
Here is how I add methods to classes imported from a library. If I modified the library I would lose the changes at the next library upgrade. I can't create a new derived class because I can't tell the library to use my modified instance. So I monkey patch the existing classes by adding the missing methods:
# Import the standard classes of the shapely library
import shapely.geometry
# Define a function that returns the points of the outer
# and the inner polygons of a Polygon
def _coords_ext_int_polygon(self):
exterior_coords = [self.exterior.coords[:]]
interior_coords = [interior.coords[:] for interior in self.interiors]
return exterior_coords, interior_coords
# Define a function that returns the points of the outer
# and the inner polygons of a MultiPolygon
def _coords_ext_int_multi_polygon(self):
if self.is_empty:
return [], []
exterior_coords = []
interior_coords = []
for part in self:
i, e = part.coords_ext_int()
exterior_coords += i
interior_coords += e
return exterior_coords, interior_coords
# Define a function that saves outer and inner points to a .pt file
def _export_to_pt_file(self, file_name=r'C:\WizardTemp\test.pt'):
'''create a .pt file in the format that pleases thinkdesign'''
e, i = self.coords_ext_int()
with open(file_name, 'w') as f:
for rings in (e, i):
for ring in rings:
for x, y in ring:
f.write('{} {} 0\n'.format(x, y))
# Add the functions to the definition of the classes
# by assigning the functions to new class members
shapely.geometry.Polygon.coords_ext_int = _coords_ext_int_polygon
shapely.geometry.Polygon.export_to_pt_file = _export_to_pt_file
shapely.geometry.MultiPolygon.coords_ext_int = _coords_ext_int_multi_polygon
shapely.geometry.MultiPolygon.export_to_pt_file = _export_to_pt_file
Notice that the same function definition can be assigned to two different classes.
EDIT
In my example I'm not adding methods to a class of mine, I'm adding methods to shapely, an open source library I installed.
In your post you use p.get_name = ... to add a member to the object instance p. I first define a funciton _xxx(), then I add it to the class definition with class.xxx = _xxx.
I don't know your use case, but usually you add variables to instances and you add methods to class definitions, that's why I am showing you how to add methods to the class definition instead of to the instance.
Shapely manages geometric objects and offers methods to calculate the area of the polygons, to add or subtract polygons to each other, and many other really cool things.
My problem is that I need some methods that shapely doesn't provide out of the box.
In my example I created my own method that returns the list of points of the outer profile and the list of points of the inner profiles. I made two methods, one for the Polygon class and one for the MultiPolygon class.
I also need a method to export all the points to a .pt file format. In this case I made only one method that works with both the Polygon and the MultiPolygon classes.
This code is inside a module called shapely_monkeypatch.py (see monkey patch). When the module is imported the functions with the name starting by _ are defined, then they are assigned to the existing classes with names without _. (It is a convention in Python to use _ for names of variables or functions intended for internal use only.)
I shall be maligned, pilloried, and excoriated, but... here is one way I make a keymap for an alphabet of methods within __init__(self).
def __init__(this):
for c in "abcdefghijklmnopqrstuvwxyz":
this.keymap[ord(c)] = eval(f"this.{c}")
Now, with appropriate code, I can press a key in pygame to execute the mapped method.
It is easy enough to use lambdas so one does not even need pre-existing methods... for instance, if __str__(this) is a method, capital P can print the instance string representation using this code:
this.keymap[ord('P')] = lambda: print(this)
but everyone will tell you that eval is bad.
I live to break rules and color outside the boundaries.