I am trying to achieve the following:
class A:
username = None
username = get_username()
def get_username(self):
if username is None:
try:
uname = os.environ["USER"]
except:
printf("Couldn't find a user name")
return uname
return username
Not sure how to achieve this. I'm sure I'm missing some "self." prefixes but this is the first time I'm working with python and static members.
In a sense I want a class with some members and functions to calculate values for these members but I don't want recalculations. I would also like these to be static functions and data members.
The problem is that the line "username = get_username()" the function hasn't already been defined. If I put username after the function then it's not
First, there's no reason to assign None to username if you're just going to reassign it immediately after.
Second, if you want the method to be a static method, you can't give it a self argument. And if you want a real static method, you have to declare it explicitly.
#staticmethod
def get_username():
if username is None:
...
Otherwise, you need an instance of the class (that self) to call it on, and you don't have one yet.
In Python 3, any regular method acts like a static method when called on the class, like an instance method when called on an instance. So, if you're sure you're never going to want to call a.get_username() on an instance a, you can skip the decorator. But you still need to get rid of the self parameter.
I think what you're actually trying to do is use a class variable to memoize the result of a static method. You can't do that, but you can use a class variable to memoize the result of a class method, which may be close enough. That would look like this:
class A:
username = None
#classmethod
def get_username(cls):
if cls.username is None:
try:
uname = os.environ["USER"]
except:
print("Couldn't find a user name")
else:
cls.username = uname
return cls.username
On the other hand, there's no good reason username has to be a class member. You can memoize by adding a member to the function, by passing a mutable default variable, or in various other ways which don't require infecting the class, and which allow you to leave get_username as a static method instead of a class method.
But really, the best solution is to find a memoization library on PyPI, in ActiveState's recipe list, etc., so you can just write this:
class A:
#memoize
#staticmethod
def get_username():
try:
return os.environ["USER"]
except:
print("Couldn't find a user name")
return None
Again, you can drop the #staticmethod if you're sure nobody's ever going to try to create an instance of A and call get_username on it.
if you don't want to lose the ability to refer to A.username, you can use class property with a little bit of metaclass:
class A(type):
def __new__(cls, name, bases, attrs):
# this allows B().username to also work
attrs['username'] = property(lambda s: s.__class__.username)
return type.__new__(cls, name, bases, attrs)
#property
def username(self):
if not hasattr(self, '_username'):
self._username = 'bar'
return self._username
class B(object):
__metaclass__ = A
print B.username
print B().username
Related
My problem, and why
I'm trying to write a decorator for a class method, #cachedproperty. I want it to behave so that when the method is first called, the method is replaced with its return value. I also want it to behave like #property so that it doesn't need to be explicitly called. Basically, it should be indistinguishable from #property except that it's faster, because it only calculates the value once and then stores it. My idea is that this would not slow down instantiation like defining it in __init__ would. That's why I want to do this.
What I tried
First, I tried to override the fget method of the property, but it's read-only.
Next, I figured I'd try to implement a decorator that does needs to be called the first time but then caches the values. This isn't my final goal of a property-type decorator that never needs to be called, but I thought this would be a simpler problem to tackle first. In other words, this is a not-working solution to a slightly simpler problem.
I tried:
def cachedproperty(func):
""" Used on methods to convert them to methods that replace themselves
with their return value once they are called. """
def cache(*args):
self = args[0] # Reference to the class who owns the method
funcname = inspect.stack()[0][3] # Name of the function, so that it can be overridden.
setattr(self, funcname, func()) # Replace the function with its value
return func() # Return the result of the function
return cache
However, this doesn't seem work. I tested this with:
>>> class Test:
... #cachedproperty
... def test(self):
... print "Execute"
... return "Return"
...
>>> Test.test
<unbound method Test.cache>
>>> Test.test()
but I get an error about how the class didn't pass itself to the method:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unbound method cache() must be called with Test instance as first argument (got nothing instead)
At this point, me and my limited knowledge of deep Python methods are very confused, and I have no idea where my code went wrong or how to fix it. (I've never tried to write a decorator before)
The question
How can I write a decorator that will return the result of calling a class method the first time it's accessed (like #property does), and be replaced with a cached value for all subsequent queries?
I hope this question isn't too confusing, I tried to explain it as well as I could.
If you don't mind alternative solutions, I'd recommend lru_cache
for example
from functools import lru_cache
class Test:
#property
#lru_cache(maxsize=None)
def calc(self):
print("Calculating")
return 1
Expected output
In [2]: t = Test()
In [3]: t.calc
Calculating
Out[3]: 1
In [4]: t.calc
Out[4]: 1
First of all Test should be instantiated
test = Test()
Second, there is no need for inspect cause we can get the property name from func.__name__
And third, we return property(cache) to make python to do all the magic.
def cachedproperty(func):
" Used on methods to convert them to methods that replace themselves\
with their return value once they are called. "
def cache(*args):
self = args[0] # Reference to the class who owns the method
funcname = func.__name__
ret_value = func(self)
setattr(self, funcname, ret_value) # Replace the function with its value
return ret_value # Return the result of the function
return property(cache)
class Test:
#cachedproperty
def test(self):
print "Execute"
return "Return"
>>> test = Test()
>>> test.test
Execute
'Return'
>>> test.test
'Return'
>>>
"""
With Python 3.8 or later you can use functools.cached_property().
It works similar as the previously proposed lru_cache solution.
Example usage:
import functools
class Test:
#functools.cached_property
def calc(self):
print("Calculating")
return 1
Test output:
In [2]: t = Test()
In [3]: t.calc
Calculating
Out[3]: 1
In [4]: t.calc
Out[4]: 1
I think you're better off with a custom descriptor, since this is exactly the kind of thing descriptors are for. Like so:
class CachedProperty:
def __init__(self, name, get_the_value):
self.name = name
self.get_the_value = get_the_value
def __get__(self, obj, typ):
name = self.name
while True:
try:
return getattr(obj, name)
except AttributeError:
get_the_value = self.get_the_value
try:
# get_the_value can be a string which is the name of an obj method
value = getattr(obj, get_the_value)()
except AttributeError:
# or it can be another external function
value = get_the_value()
setattr(obj, name, value)
continue
break
class Mine:
cached_property = CachedProperty("_cached_property ", get_cached_property_value)
# OR:
class Mine:
cached_property = CachedProperty("_cached_property", "get_cached_property_value")
def get_cached_property_value(self):
return "the_value"
EDIT: By the way, you don't even actually need a custom descriptor. You could just cache the value inside of your property function. E.g.:
#property
def test(self):
while True:
try:
return self._test
except AttributeError:
self._test = get_initial_value()
That's all there is to it.
However, many would consider this a bit of an abuse of property, and to be an unexpected way of using it. And unexpected usually means you should do it another, more explicit way. A custom CachedProperty descriptor is very explicit, so for that reason I would prefer it to the property approach, though it requires more code.
Django's version of this decorator does exactly what you describe and is simple, so besides my comment I'll just copy it here:
class cached_property(object):
"""
Decorator that converts a method with a single self argument into a
property cached on the instance.
Optional ``name`` argument allows you to make cached properties of other
methods. (e.g. url = cached_property(get_absolute_url, name='url') )
"""
def __init__(self, func, name=None):
self.func = func
self.__doc__ = getattr(func, '__doc__')
self.name = name or func.__name__
def __get__(self, instance, type=None):
if instance is None:
return self
res = instance.__dict__[self.name] = self.func(instance)
return res
(source).
As you can see, it uses func.name to determine the name of the function (no need to fiddle with inspect.stack) and it replaces the method with its result by mutating instance.__dict__. So subsequent "calls" are just an attribute lookup and there is no need for any caches, et cetera.
You can use something like this:
def cached(timeout=None):
def decorator(func):
def wrapper(self, *args, **kwargs):
value = None
key = '_'.join([type(self).__name__, str(self.id) if hasattr(self, 'id') else '', func.__name__])
if settings.CACHING_ENABLED:
value = cache.get(key)
if value is None:
value = func(self, *args, **kwargs)
if settings.CACHING_ENABLED:
# if timeout=None Django cache reads a global value from settings
cache.set(key, value, timeout=timeout)
return value
return wrapper
return decorator
When adding to the cache dictionary it generates keys based on the convention class_id_function in case you are caching entities and the property could possibly return a different value for each one.
It also checks a settings key CACHING_ENABLED in case you want to turn it off temporarily when doing benchmarks.
But it does not encapsulate the standard property decorator so you should still call it like a function, or you can use it like this (why restrict it to properties only):
#cached
#property
def total_sales(self):
# Some calculations here...
pass
Also it may be worth noting that in case you are caching a result from lazy foreign key relationships, there are times depending on your data where it would be faster to simply run an aggregate function when doing your select query and fetching everything at once, than visiting the cache for every record in your result-set. So use some tool like django-debug-toolbar for your framework to compare what performs best in your scenario.
#functools.lru_cache()
def func(....):
....
Reference: #functools.lru_cache() | Python
Have u tried djangos built in:
from django.utils.functional import cached_property
please don't use lru_cache as suggested by multiple people as it opens up a host of possible memory leak issues
In python code I often see the use of #property.
If I understand correctly, with the property function a getter setter and deleter can be defined.
Why would one use #property if the setter and deleter are not defined (#x.setter, #x.deleter)? Isn't this the same as not using #property at all?
It creates an API that does not allow a value to be set. This is similar in other languages to a constant.
Defining a property with a getter function but without a setter can be very useful in certain scenarios. Lets say you have a model as below in django; a model is essentially a database table with entries called fields. The property hostname is computed from one or more fields in the model from the database. This circumvents needing another entry in the database table that has to be changed everytime the relevant fields are changed.
The true benefit of using a property is calling object.hostname() vs. object.hostname. The latter is passed along with the object automatically so when we go to a place like a jinja template we can call object.hostname but calling object.hostname() will raise an error.
The example below is a virtualmachine model with a name field and an example of the jinja code where we passed a virtualmachine object.
# PYTHON CODE
class VirtualMachine(models.Model):
name = models.CharField(max_length=128, unique=True)
#property
def hostname(self):
return "{}-{}.{}".format(
gethostname().split('.')[0],
self.name,
settings.EFFICIENT_DOMAIN
)
# JINJA CODE
...start HTML...
Name: {{ object.name }}
# fails
Hostname: {{ object.hostname() }}
# passes
Hostname: {{ object.hostname }}
...end HTML...
This is a good answer. Additionally, you can also modify the value of your property based on other kwargs and do this within the same method declaration. If you create a self._hostname instance variable, you can also modify the value based on other kwargs or variables. You can also obtain the value from your property and use it within other methods as self.scheme (see below) is syntactically pleasing and simple :).
class Neo4j(Database):
def __init__(self, label, env, username, password, hostname, port=None, routing_context=False, policy=None, scheme=None, certificate=None):
super().__init__(label, env)
self.username = username
self._password = password
self.hostname = hostname
self.port = port # defaults, 7687
self._scheme = scheme # example - neo4j, bolt
self.routing_context = routing_context # self.policy = policy policy=None,
self.policy = policy # Examples, europe, america
self.certificate = certificate # examples, None, +s, +ssc
#property
def scheme(self):
if not self.certificate:
return f'{self._scheme}'
return f'{self._scheme}+{self.certificate}'
def __repr__(self) -> str:
return f'<{self.scheme}://{self.hostname}:{self.port}>' #if self.ro
db = Neo4j(label='test', env='dec', username='jordan', password='pass', hostname='localhost', port=7698, scheme='neo4j', certificate='ssc')
print(db.scheme) >>> neo4j+ssc
TL;DR
So if you have heavy logic in the #property function, be aware that it will be running the entire logic each time you access the property. In this case I would suggest using a getter with a setter
Verbose
Another aspect which I don't feel has been explored is that the #property which is a getter, could be and most likely will be called multiple times where as the setter will most likely be called once when you instantiate the object.
IMO, this model should be used if the #property function is not doing too much heavy lifting. In the example below, we are just concatenating some strings to generate an email address.
class User:
DOMAIN = "boulder.com"
def __init__(self, first_name: str, last_name: str) -> None:
self.first_name = first_name
self.last_name = last_name
#property
def email(self) -> str:
return "{}_{}#{}".format(self.first_name, self.last_name, self.DOMAIN)
But if you are going to add some extended or heavy logic to the function, then I would recommend creating a getter for it so that it is only run once. For example, lets say we need to check whether the email is unique, this logic would be better served in a getter other wise you will run the logic to check for uniqueness of the email each time you want to access it.
class User:
DOMAIN = "boulder.com"
def __init__(self, first_name: str, last_name: str) -> None:
self.first_name = first_name
self.last_name = last_name
#property
def email(self) -> str:
return self._email
#email.setter
def email(self) -> None:
proposed_email = "{}_{}#{}".format(self.first_name, self.last_name, self.DOMAIN)
if is_unique_email(proposed_email):
self._email = proposed_email
else:
random_suffix = get_random_suffix()
self._email = "{}_{}_{}#{}".format(
self.first_name, self.last_name, random_suffix, self.DOMAIN
)
If I have a class such as below (only with many more properties), is there are clean way to note which fields are required before calling a particular method?
class Example():
def __init__(self):
pass
#property
"""Have to use property methods to have docstrings..."""
def prop1(self):
return self._prop1
#prop1.setter
def task(self, value):
# validation logic..
self._prop1 = value
def method(self):
# check all required properties have been added
I could write an array by hand of all required propeties and loop through them in a method, but I was wondering if there is a cleaner way for example by implementing a #requiredProperty descriptor.
The class is used to generate a POST request for a web API. The request has 25+ parameters, some of which are required and some optional.
Rather than on the method calling the request having to loop through an array such as:
required_props = ['prop1','prop2',....]
I was hoping there was a way in Python of adding a required decorator to properties so I wouldn't have to keep track by hand. E.g.
#property, #required
def prop1(self):
return self._prop1
Would it not be best to make sure that all the attributes are supplied when an object is initialised? Then all your properties will be defined when you try to acces them.
For example,
class Example(object):
def __init__(self, prop1, prop2):
self.prop1 = prop1
self.prop2 = prop2
Also, note from PEP8:
For simple public data attributes, it
is best to expose just the attribute
name, without complicated
accessor/mutator methods.
So why use properties?
This should work the same way as in any OO language: A required property must be set during construction time. Calling the objects methods must never leave the object in a "bad" state, so that method can be called on any constructed object.
If the above doesn't hold true, you should think about refactoring your code.
Of course it is always possible to alter a python object to not be valid anymore by poking around in its guts. You don't do that unless you have a good reason. Don't bother checking for this, as your program should just blow up in your face whenever you do something stupid so you learn and stop.
It's hard to tell from your example what problem you are actually trying to solve, but I'm not convinced properties are the answer.
If you just want to check that an instance variable exists, you could use the special attribute __dict__, thus:
% cat ./test.py
#!/usr/bin/env python
class Example():
def __init__(self):
self.foo = None
def method(self):
assert 'foo' in self.__dict__
assert 'bar' in self.__dict__
Example().method()
% ./test.py
Traceback (most recent call last):
File "./test.py", line 12, in <module>
Example().method()
File "./test.py", line 10, in method
assert 'bar' in self.__dict__
AssertionError
But remember... EAFP: Easier to ask for forgiveness than permission.
As others have suggested, I suspect you are over-engineering. However, you could use a decorator to define 'required' attributes. Something along the lines of:
import functools
class MissingAttributeError(Exception):
pass
def requires(*required_attrs):
def wrapper(method):
#functools.wraps(method)
def inner_wrapper(self, *args, **kargs):
if not all(hasattr(self, attr) for attr in required_attrs):
raise MissingAttributeError()
return method(self, *args, **kargs)
return inner_wrapper
return wrapper
class Test(object):
def __init__(self, spam, eggs):
self.spam = spam
self.eggs = eggs
#requires('spam', 'eggs', 'ham')
def something(self):
return 'Done'
t = Test('fu', 'bar')
t.something() ## fails
t.ham = 'nicer than spam'
t.something() ## succeeds
Although defining attribute dependencies this way has a certain neatness to it, I'm not sure I recommend it.
I'm using Django and want to be able to store classes in a database for things like forms and models so that I can easily make them creatable through a user interface since they are just stored in the database as opposed to a regular file. I don't really know a whole lot about this and am not sure if this is a situation where I need to use exec in python or if there is some other way. My searches on this aren't turning up much of anything.
Basically, it would just be where I do a database call and get the contents of a class, then I want to instantiate it. Any advice is appreciated on how to best do this sort of thing.
EDIT: In response to the idea of a malicious __init__ in the class, these are only for things like forms or models where it is tightly controlled through validation what goes in the class, there would never be an __init__ in the class and it would be basically impossible, since I would validate everything server side, to put anything malicious in the class.
Do not store code in the database!!!
Imagine a class with a malicious __init__ method finding it's way in your "class repository" in the database. This means whoever has write access to those database tables has the ability to read any file from your web server and even nuke it's file system, since they have the ability to execute any python code on it.
Don't store the class itself, store the import path as a string in the database (e.g. 'django.forms.CharField')
I started doing this same thing for another project, and saved off the code in my local repository. To address the security concerns I was going to add an argument to the field constructor of allowed base classes. If you do implement this, let me know, I'd love to have it.
helpers.py
def get_class_from_concrete_classpath(class_path):
# Unicode will throw errors in the __import__ (at least in 2.6)
class_path = str(class_path)
mod_list = class_path.split('.')
module_path = '.'.join(mod_list[:-1])
class_name = mod_list[-1]
base_mod = __import__(module_path, fromlist=[class_name,])
return getattr(base_mod, class_name)
def get_concrete_name_of_class(klass):
"""Given a class return the concrete name of the class.
klass - The reference to the class we're interested in.
Raises a `TypeError` if klass is not a class.
"""
if not isinstance(klass, (type, ClassType)):
raise TypeError('The klass argument must be a class. Got type %s; %s' %
(type(klass), klass))
return '%s.%s' % (klass.__module__, klass.__name__)
fields.py
class ClassFormField(forms.Field):
def to_python(self, value):
return get_concrete_name_of_class(value)
class ClassField(models.CharField):
__metaclass__ = models.SubfieldBase
"""Field used for storing a class as a string for later retrieval"""
MAX_LENGTH = 255
default_error_messages = {
'invalid': _(u'Enter a valid class variable.'),
}
def __init__(self, *args, **kwargs):
kwargs['max_length'] = kwargs.get('max_length', ClassField.MAX_LENGTH)
super(ClassField, self).__init__(*args, **kwargs)
def get_prep_value(self, value):
if isinstance(value, (basestring, NoneType)):
return value
return get_concrete_name_of_class(value)
def to_python(self, value):
if isinstance(value, basestring):
return get_class_from_concrete_classpath(value)
return value
def formfield(self, **kwargs):
defaults = {'form_class' : ClassFormField}
defaults.update(kwargs)
return super(ClassField, self).formfield(**defaults)
I want to do a one time callback registration within Observer. I don't want to do the registration inside init or other function. I don't know if there is a class level equivalent for init
class Observer:
#classmethod
def on_new_user_registration(new_user):
#body of handler...
# first I try
NewUserRegistered().subscribe \
(Observer.on_new_user_registration) #gives NameError for Observer
#so I try
NewUserRegistered().subscribe(on_new_user_registration) #says not callable
#neither does this work
NewUserRegistered().subscribe(__metaclass__.on_new_user_registration)
class BaseEvent(object):
_subscriptions = {}
def __init__(self, event_info = None):
self.info = event_info
def fire(self):
for callback in self._subscriptions[event_type]:
callback(event_info)
def subscribe(self, callback):
if not callable(callback):
raise Exception(str(callback) + 'is not callable')
existing = self._subscriptions.get(self.__class__, None)
if not existing:
existing = set()
self._subscriptions[self.__class__] = existing
existing.add(callback)
class NewUserRegistered(BaseEvent):
pass
I suggest to cut down on the number of classes -- remember that Python isn't Java. Every time you use #classmethod or #staticmethod you should stop and think about it since these keywords are quite rare in Python.
Doing it like this works:
class BaseEvent(object):
def __init__(self, event_info=None):
self._subscriptions = set()
self.info = event_info
def fire(self, data):
for callback in self._subscriptions:
callback(self.info, data)
def subscribe(self, callback):
if not callable(callback):
raise ValueError("%r is not callable" % callback)
self._subscriptions.add(callback)
return callback
new_user = BaseEvent()
#new_user.subscribe
def on_new_user_registration(info, username):
print "new user: %s" % username
new_user.fire("Martin")
If you want an Observer class, then you can do it like this:
class Observer:
#staticmethod
#new_user.subscribe
def on_new_user_registration(info, username):
print "new user: %s" % username
But note that the static method does not have access to the protocol instance, so this is probably not very useful. You can not subscribe a method bound to an object instance like this since the object wont exist when the class definition is executed.
But you can of course do this:
class Observer:
def on_new_user_registration(self, info, username):
print "new user: %s" % username
o = Observer()
new_user.subscribe(o.on_new_user_registration)
where we use the bound o.on_new_user_registration as argument to subscribe.
I've come to accept that python isn't very intuitive when it comes to functional programming within class definitions. See this question. The problem with the first method is that Observer doesn't exist as a namespace until the class has been built. The problem with the second is that you've made a class method that doesn't really do what it's supposed to until after the namespace has been created. (I have no idea why you're trying the third.) In both case neither of these things occurs until after the class definition of Observer has been populated.
This might sound like a sad constraint, but it's really not so bad. Just register after the class definition. Once you realize that it's not bad style to perform certain initialization routines on classes in the body of the module but outside the body of the class, python becomes a lot friendlier. Try:
class Observer:
# Define the other classes first
class Observer:
#classmethod
def on_new_user_registration(new_user):
#body of handler...
NewUserRegistered().subscribe(Observer.on_new_user_registration)
Because of the way modules work in python, you are guaranteed that this registration will be performed once and only once (barring process forking and maybe some other irrelevant boundary cases) wherever Observer is imported.
oops. sorry about that.
All I had to do was to move the subscription outside the class definition
class Observer:
#classmethod
def on_new_user_registration(new_user):
#body of handler...
#after end of class
NewUserRegistered().subscribe(Observer.on_new_user_registration)
Guess it is a side-effect of too much Java that one doesn't immediately think of this.
What you're doing should work:
>>> class foo:
... #classmethod
... def func(cls):
... print 'func called!'
...
>>> foo.func()
func called!
>>> class foo:
... #classmethod
... def func(cls):
... print 'func called!'
... foo.func()
...
func called!
One thing to note though, class methods take a cls argument instead of a self argument. Thus, your class definition should look like this:
class Observer:
#classmethod
def on_new_user_registration(cls, new_user):
#body of handler...