I'm using Flask with template caching on a Redis server:
TIMEOUT = 60 * 60
cache = Cache(app.server, config={
'CACHE_TYPE': 'redis',
'CACHE_REDIS_HOST': "myredis",
'CACHE_DEFAULT_TIMEOUT': TIMEOUT,
'CACHE_REDIS_PORT': 6379,
})
# to disable caching
#app.config["CACHE_TYPE"] = "null"
and then with the #cache decorator like
#cache.memoize(timeout=TIMEOUT)
def update_date():
return manager.getData()
The problem is that when manager.getData() has errors or no data the decorator will cache the response anyways. How to avoid it?
[UPDATE]
I have tried using the unless parameter, that according to the docs it should be
unless – Default None. Cache will always execute the caching facilities unelss this callable is true. This will bypass the caching entirely.
so used like
#cache.memoize(timeout=TIMEOUT unless=DataLoader.instance.hasData)
def update_date():
return manager.getData()
where DataLoader is a Singleton instance and hasData method will return None if has no data or True if it has data, so the method getData would compute the data and return instance variable self.data that holds always last computed data or None.
class DataLoader(SingletonMixin):
def __init__(self):
self.data=None
def hasData(self):
if self.data is Not None:
return True
else:
return None
def getData(self):
# calculate data
res = self.computeData()
if res is not None:
self.data=res
return self.data
but it seems it does not work as expected.
The problem is that when manager.getData() has errors or no data the decorator will cache the response anyways. How to avoid it?
Have you checked that? If you look at the source code (I assume you are using flask-caching because flask-cache is not maintained for over 4 years) if you get None from cache (rv value) you don't use it, you call your f function. If f function raises exception nothing is saved to cache.
Related
I have a function than depends on db connection. This function has a lot of return statements, of this kind:
def do_something_with_data(db: Session, data: DataClass):
db = Session()
if condition1:
db.do_something1()
db.close()
return
if condition2:
db.do_something2()
db.close()
return
if condition3:
db.do_something3()
db.close()
return
...
After executing the function, I need to run db.close(), but because of the structure of the function this entry will have to be duplicated many times for each return as shown above.
So I made a decorator that passes the created session to the function and closes the session at the end of the execution of the function instead.
def db_depends(function_that_depends_on_db):
def inner(*args, **kwargs):
db = Session()
result = function_that_depends_on_db(db, *args, **kwargs)
db.close()
return result
return inner
#db_depends
def do_something_with_data(db: Session, data: DataClass):
if condition1:
db.do_something1()
return
if condition2:
db.do_something2()
return
if condition3:
db.do_something3()
return
...
All works great, but the fact, that user see two required arguments in definition, however there is only one (data) seems kinda dirty.
Is it possible to do the same thing without misleading people who will read the code or IDE hints?
Just have the function accept the Session parameter normally:
def do_something_with_data(db: Session, data: DataClass):
if condition1:
db.do_something1()
return
if condition2:
db.do_something2()
return
if condition3:
db.do_something3()
return
This allows the user to specify a Session explicitly, for example to reuse the same Session to do multiple things.
Yes, that doesn't close the Session. Because that is the responsibility of the calling code, since that's where the Session came from in the first place. After all, if the calling code wants to reuse a Session, then it shouldn't be closed.
If you want a convenience method to open a new, temporary Session for the call, you can easily do that using the existing decorator code:
do_something_in_new_session = db_depends(do_something_with_data)
But if we don't need to apply this logic to multiple functions, then "simple is better than complex" - just write an ordinary wrapper:
def do_something_in_new_session(data: DataClass):
db = Session()
result = do_something_with_data(db, data)
db.close()
return result
Either way, it would be better to write the closing logic using a with block, assuming your library supports it:
def do_something_in_new_session(data: DataClass):
with Session() as db:
return do_something_with_data(db, data)
Among other things, this ensures that .close is called even if an exception is raised in do_something_with_data.
If your DB library doesn't support that (i.e., the Session class isn't defined as a context manager - although that should only be true for very old libraries now), that's easy to fix using contextlib.closing from the standard library:
from contextlib import closing
def do_something_in_new_session(data: DataClass):
with closing(Session()) as db:
return do_something_with_data(db, data)
(And of course, if you don't feel the need to make a wrapper like that, you can easily use such a with block directly at the call site.)
I'm not sure if I used the right terms in the title. This maybe a known way to program interface functions for a subsystem or module but because I don't know the keywords, I'm not finding the results in my search queries.
I want to create a function whose intention can be clearly described in the functions name but the parameters are flexible. I want to write the function to be generic enough so that the function can complete the intention with whatever parameters it receives from whichever caller.
Let's take a function do_foo.
do_foo can take in some_obj whose attributes allows do_foo to do its work. Additionally, do_foo can just take in the individual attributes it cares about like obj_attr0 or obj_attr1 and perform the same work. In both cases, the expected result is the same as well.
So this would look something like this:
Class SomeObj():
def __init__(self, obj_attr0, obj_attr1, obj_attrN):
self.obj_attr0 = None
self.obj_attr1 = None
self.obj_attrN = None # denotes an N number of attributes
def do_foo(params)
# unpack params. do_foo requires obj_attr0 and obj_attr1 and so its searching it in the params iterable
# determine which data is passed in
# execute the work the same way regardless of what form the data is passed in
pass
obj_attr0 = None
obj_attr1 = None
obj_attrN = None
some_obj = SomeObj(obj_attr0, obj_attr1, obj_attrN)
# One can either call with a subset of attributes that would make up SomeObj or SomeObj itself if all the information is there. E.g.:
params = (some_obj)
do_foo(params)
# OR
params = (obj_att0, obj_attr1)
do_foo(params)
I know python offers *args and **kwargs facilities that offer the flexibility above. I'm looking for some examples of where the implementation lends itself to reducing pitfalls. What is a good way to implement the above? And if there are any resources out there what are examples/articles/or terms that describe the above style of programming? Clearly, I'm trying to write my interface functions to be generic and usable in multiple logic paths where the users has its data in different forms where sticking to a specific parameter list is limiting.
Short answer:
You can use function decorators to do this
Long answer:
I have a concrete example for you. It might not be the prettiest code but it does something similar to what you are asking for.
Mini HTTP Testing library
I made a mini HTTP testing library because I make my REST http tests in python, and I realized that I always write the same code again and again. So I made a more general setup
The core
The core is kind of ugly and this is the part I don't want to write again and again.
Just skip this part quick and check how it is used in the interface section.
Then if you like it you can go back and try to understand how it is all tied together.
# base.py
import json, requests, inspect
# This function drops invallid parameters
def request(*args, **kwargs):
allowed = inspect.signature(requests.Session.request).parameters
return {k:v for (k,v) in kwargs.items() if k in allowed}
def response(r, code):
if r.status_code != code:
print(r.text)
return
data = r.json()
if data:
print(json.dumps(data, indent=2, ensure_ascii=False))
return data
# This is the core function it is not pretty but it creates all the abstaction in multiple levels of decorations.
def HTTP(base_url):
def outer(func_one):
def over(*args_one, **kwargs_one):
req, url, code = func_one(*args_one, **kwargs_one)
url = base_url + url
def inner(func_two):
def under(*args_two, **kwargs_two):
allowed = inspect.signature(func_two).parameters
kwparams = {k:v for (k,v) in kwargs_two.items() if k in allowed}
from_inner = func_two(*args_two, **kwparams)
u = url.format(id=kwargs_two.pop('_id')) if '{id}' in url else url
r = req(u, **request(**kwargs_two, **from_inner))
return response(r, code)
return under
return inner
return over
return outer
The interface
The interface functions are all each decorated by the HTTP function which makes them a HTTP caller function, it is still abstract since it will return a function.
Note: interface is just what I call it but it is really just functions which returns functions based on the HTTP decorator
BASE_URL = "https://example.com"
#HTTP(BASE_URL)
def POST(url, code=200): return requests.post, url, code
#HTTP(BASE_URL)
def PUT(url, code=200): return requests.put, url, code
#HTTP(BASE_URL)
def DELETE(url, code=200): return requests.delete, url, code
#HTTP(BASE_URL)
def GET(url, code=200): return requests.get, url, code
A middleware function
When one of the interface functions are decorated with this one then they need a token.
def AUTH(func):
def inner(token, *args, **kwargs):
headers = {'Authorization': f'bearer {token}'}
return func(*args, **kwargs, headers=headers)
return inner
The implementation
The interface can be used for many implementations.
Here I use the interface of POST, PUT, GET and DELETE for the user model.
This is the final decoration, and the functions returned will actually return content instead of other functions.
# users.py
from httplib.base import (
POST,
GET,
DELETE,
PUT,
AUTH,
request
)
#POST('/users',200)
def insert(user):
return request(json=user)
#AUTH
#GET('/users')
def find(_filter={}):
return request(params=_filter)
#AUTH
#GET('/users/{id}')
def find_one(_id):
return request()
#AUTH
#DELETE('/users/{id}')
def delete(_id):
return request()
#AUTH
#PUT('/users/{id}')
def update(_id, updates={}):
return request(json=updates)
Operation
Here you can see how the users delete insert and find functions work.
from httplib import users
def create_and_delete_users(token, n): return [
users.delete(token, _id=x['user']['id'])
for x in [
users.insert(user={
'username' : f'useruser{str(i).zfill(2)}',
'password' : 'secretpassword',
'email' : f'useruser{str(i).zfill(2)}#mail.com',
'gender' : 'male',
}) for i in range(n)]
]
def find_all_and_then_find_each(token): return [
users.find_one(token, _id=x['id'])
for x in users.find(token)['data']
]
I hope this was helpful.
I have an object that is used for fetching information from another service which is very simple. Since the object is simple and the initialization method could be easily patched I thought I would try to write my code to be super reusable and extendable. But alas, I cannot figure out how to make it work. The code below is pretty well sudo code and is super simplified but it should get the point across.
class SimpleClient:
def __init__(self):
pass
def read(self, key, path='some/path'):
return value_from_get_on_another_service
I then have a request handler object that initializes a client via get_client() (seen below)
def get_client():
return SimpleClient()
Then a method on the request handler uses the client.read() method a few times with different parameters (2nd dependent upon the 1st).
For my tests, I thought I could "patch" the get_client method to return my own simple object that could then be used "regularly" and eliminate the dependence on the third party service and actually use the values retrieved from the method execution. I was disappointed to find it was not that easy and clean. The test pattern is seen below.
class MockClient:
def __init__(self, addr='someAddr', token='someToken'):
pass
def read(self, value, prefix):
data = {}
if prefix == 'path/1':
data = self.p1_lookup(value)
elif prefix == 'path/2':
data = self.p2_lookup(value)
return self.response_wrapper(data)
def p2_lookup(self, key):
data = {
'key1': {
'sub_key': {"55B3FE7D-9F43-4DD4-9090-9D89330C918A": "Dev2",
"7A1C2F4B-E91C-4659-A33E-1B18B0BEE2B3": "Dev"}
}
}
return data.get(key, {})
#mock.patch('a.module.get_client')
def test_authorize_valid_request_no_body(mock_get_client):
request = RequestMock()
request.body = None
handler = RequestHandler(Application(), request=request, logging_level='INFO')
mock_get_client.return_value = MockClient()
handler.authorize_request()
assert handler.verified_headers is None
assert handler.verified_body is None
assert handler.user_authenticated is False
I have seen where I can mock the responses for the actual client.read() to return multiple values with a list. But this just seems like I will be doing lots of copy and paste and have to do the same thing over and over for each little test. Forgive me if this is simple, sadly I am just learning the art of testing. Is there a way to accomplish what I am trying to do? Maybe there is something super simple I am missing. Or maybe I am just totally on the wrong track for no good reason. Help?!
After a sleep, with fresh eyes I was able to figure this out relatively quickly thanks to a couple other similar questions/answers that I had not found before. Primarily this one, Python Mock Object with Method called Multiple Times.
Rather than needing to rebuild the module object completely I need to let mock do that for me and then override the specific method on it with the side_effect attribute. So below is what sanitized version of the code looks like.
def read_override(value, prefix):
lookup_data1 = {"lookup1": {'key1': 'value1'}}
lookup_data2 = {'some_id': {'akey': {'12345678': 'DEV'}}
data = {}
if prefix == 'path1/1a':
data = lookup_data1.get(value, {})
elif prefix == 'path2/2a':
data = lookup_data2.get(value, {})
return {'data': data}
# Create a true Mock of the entire LookupClient Object
VAULT_MOCK = mock.Mock(spec=LookupClient)
# make the read method work the way I want it to with an "override" of sorts
VAULT_MOCK.read.side_effect = vault_read_override
Then the test simply looked like this...
#mock.patch('a.module.get_client')
def test_authorize_valid_request_no_body(get_client):
get_client.return_value = VAULT_MOCK
request = RequestMock()
request.body = None
handler = RequestHandler(Application(), request=request, logging_level='INFO')
handler.authorize_request()
assert handler.verified_headers is None
assert handler.verified_body is None
assert handler.user_authenticated is False
Currently working on python and redis. I have Flask as my framework and working on Blueprints.
Looking into implementing cache with redis for my APIs, I have tried Flask-Cache and redis-simple-cache.
Downside are Flask-Cache saves the function even when you change the parameter of the function. It only saves it once per function.
As per redis-simple-cache, it save its keys as SimpleCache-<key name> which not advisable on my end.
So my question is, how can you create a decorator which save and retrieve or check if there is a key existing for the specific key.
I know a save decorator is possible. But is a retrieve or check decorator possible?? Please correct me if I am wrong. Thank you.
You seem to not have read the Flask-Cache documentation very closely. Caching does not ignore parameters and the cache key is customisable. The decorators the project supplies already give you the functionality you seek.
From the Caching View Functions section:
This decorator will use request.path by default for the cache_key.
So the default cache key is request.path, but you can specify a different key. Since Flask view functions get their arguments from path elements, the default request.path makes a great key.
From the #cached() decorator documentation:
cached(timeout=None, key_prefix='view/%s', unless=None)
By default the cache key is view/request.path. You are able to use this decorator with any function by changing the key_prefix. If the token %s is located within the key_prefix then it will replace that with request.path. You are able to use this decorator with any function by changing the key_prefix.
and
key_prefix – [...] Can optionally be a callable which takes no arguments but returns a string that will be used as the cache_key.
So you can set key_prefix to a function, and it'll be called (without arguments) to produce the key.
Moreover:
The returned decorated function now has three function attributes assigned to it. These attributes are readable/writable:
[...]
make_cache_key
A function used in generating the cache_key used.
This function is passed the same arguments the view function is passed. In all, this allows you to produce any cache key you want; either use key_prefix and pull out more information from the request or g or other sources, or assign to view_function.make_cache_key and access the same arguments the view function receives.
Then there is the #memoize() decorator:
memoize(timeout=None, make_name=None, unless=None)
Use this to cache the result of a function, taking its arguments into account in the cache key.
So this decorator caches return values purely based on the arguments passed into the function. It too supports a make_cache_key function.
I've used both decorators to make a Google App Engine Flask project scale to double-digit millions of views per month for a CMS-backed site, storing results in the Google memcached structure. Doing this with Redis would only require setting a configuration option.
You can use this cache decorator, the cache object you create will have to be a flask cache object instead of django one i.e. should support cache.get and cache.set methods, this is pretty flexible based on how you want to create cache keys, i.e.
based on what parameters being passed to the method
in what cases to cache/not cache the result
Use same cache even if kwarg order is changed, i.e. same cache for my_method(a=1,b=2) and my_method(b=2,a=1) call.
"
__author__ = 'Dhruv Pathak'
import cPickle
import logging
import time
from functools import wraps
from django.conf import settings
import traceback
"""following imports are from datautils.py in this repo, datautils
also contains many other useful data utility methods/classes
"""
from datautils import mDict, mList, get_nested_ordered_dict, nested_path_get
"""following import is specific to django framework, and can be altered
based on what type of caching library your code uses"""
from django.core.cache import cache, caches
logger = logging.getLogger(__name__)
def cache_result(cache_key=None, cache_kwarg_keys=None, seconds=900, cache_filter=lambda x: True, cache_setup = "default"):
def set_cache(f):
#wraps(f)
def x(*args, **kwargs):
if settings.USE_CACHE is not True:
result = f(*args, **kwargs)
return result
try:
# cache_conn should be a cache object supporting get,set methods
# can be from python-memcached, pylibmc, django, django-redis-cache, django-redis etc
cache_conn = caches[cache_setup]
except Exception, e:
result = f(*args, **kwargs)
return result
final_cache_key = generate_cache_key_for_method(f, kwargs, args, cache_kwarg_keys, cache_key)
try:
result = cache_conn.get(final_cache_key)
except Exception, e:
result = None
if settings.DEBUG is True:
raise
else:
logger.exception("Cache get failed,k::{0},ex::{1},ex::{2}".format(final_cache_key, str(e), traceback.format_exc()))
if result is not None and cache_filter(result) is False:
result = None
logger.debug("Cache had invalid result:{0},not returned".format(result))
if result is None: # result not in cache
result = f(*args, **kwargs)
if isinstance(result, (mDict, mList)):
result.ot = int(time.time())
if cache_filter(result) is True:
try:
cache_conn.set(final_cache_key, result, seconds)
except Exception, e:
if settings.DEBUG is True:
raise
else:
logger.exception("Cache set failed,k::{0},ex::{1},ex::{2},dt::{3}".format(final_cache_key, str(e), traceback.format_exc(), str(result)[0:100],))
#else:
# logger.debug("Result :{0} failed,not cached".format(result))
else: # result was from cache
if isinstance(result, (mDict, mList)):
result.src = "CACHE_{0}".format(cache_setup)
return result
return x
return set_cache
def generate_cache_key_for_method(method, method_kwargs, method_args, cache_kwarg_keys=None, cache_key=None):
if cache_key is None:
if cache_kwarg_keys is not None and len(cache_kwarg_keys) > 0:
if len(method_args) > 0:
raise Exception("cache_kwarg_keys mode needs set kwargs,args should be empty")
method_kwargs = get_nested_ordered_dict(method_kwargs)
cache_kwarg_values = [nested_path_get(method_kwargs, path_str=kwarg_key, strict=False) for kwarg_key in cache_kwarg_keys]
if any([kwarg_value is not None for kwarg_value in cache_kwarg_values]) is False:
raise Exception("all cache kwarg keys values are not set")
final_cache_key = "{0}::{1}::{2}".format(str(method.__module__), str(method.__name__), hash(cPickle.dumps(cache_kwarg_values)))
else:
final_cache_key = "{0}::{1}".format(str(method.__module__), str(method.__name__))
final_cache_key += "::{0}".format(str(hash(cPickle.dumps(method_args, 0)))) if len(method_args) > 0 else ''
final_cache_key += "::{0}".format(str(hash(cPickle.dumps(method_kwargs, 0)))) if len(method_kwargs) > 0 else ''
else:
final_cache_key = "{0}::{1}::{2}".format(method.__module__, method.__name__, cache_key)
return final_cache_key
2-3 utility methods are imported from this file in same repo, you can just put them in same file.
I came across the __getattr__ built-in and was wondering when it would be used. I had a hard time thinking of a practical use from the documentation
http://docs.python.org/reference/datamodel.html#. What would be an actual example of how it could be used and useful in code?
One example is to use object notation with dictionaries. For example, consider a dictionary
myDict = {'value': 1}
Typically in Python one accesses the 'value' variable as
myDict['value']
which will print 1 at the Python interpreter. However, one may wish to use the myDict.value notation. This may be achieved by using the following class:
class DictAsMember(dict):
def __getattr__(self, name):
value = self[name]
if isinstance(value, dict):
value = DictAsMember(value)
return value
my_dict = DictAsMember()
my_dict['property'] = {'sub_property': 1}
print(my_dict.property.sub_property) # 1 will be printed
An example usage would be to create a simple wrapper around some object. In order, for example, to log the calls, or modify its behavior without inheriting from it, and without having to implement the whole interface of the object.
There is several good documented examples out there, like, for example, http://western-skies.blogspot.fr/2008/02/complete-example-of-getattr-in-python.html.
Since __getattr__ is only called when an attribute is not found, it can be a useful way to define an alternate place to look up an attribute, or to give default values, similar to a defaultdict.
You could also emulate a base class higher than all the others in an object's MRO, by delegating all the lookups here to another object (though doing this you could potentially have an infinite loop if the other object is delegating the attribute back).
There is also __getattribute__, which is related in that it is called anytime any attribute is looked up on the object.
Edit: This is about the built-in function getattr, not the __getattr__ method.
I needed to do this for a REST client using bearer tokens. I wrapped Requests's Session object into my own interface so I could always send the auth header, and (more relevantly) make HTTP requests to the same site, just using the URL's path.
class RequestsWrapper():
def __init__(self, base_url):
self.client = requests.Session(
headers={'Authorization':'myauthtoken'}
)
self.base_url = base_url
def _make_path_request(self, http_method, path, **kwargs):
"""
Use the http_method string to find the requests.Session instance's
method.
"""
method_to_call = getattr(self.client, http_method.lower())
return method_to_call(self.base_url + path, **kwargs)
def path_get(self, path, **kwargs):
"""
Sends a GET request to base_url + path.
"""
return self._make_path_request('get', path, **kwargs)
def path_post(self, path, **kwargs):
"""
Sends a POST request to base_url + path.
"""
return self._make_path_request('post', path, **kwargs)
def path_put(self, path, **kwargs):
"""
Sends a PUT request to base_url + path.
"""
return self._make_path_request('put', path, **kwargs)
def path_delete(self, path, **kwargs):
"""
Sends a DELETE request to base_url + path.
"""
return self._make_path_request('delete', path, **kwargs)
Then, I could just make a request based on the path:
# Initialize
myclient = RequestsWrapper("http://www.example.com")
# Make a get request to http://www.example.com/api/spam/eggs
response = myclient.path_get("/api/spam/eggs")
# Print the response JSON data
if response.ok:
print response.json