Develop Reference

Python Generic functions with generic parameters

I'm not sure if I used the right terms in the title. This maybe a known way to program interface functions for a subsystem or module but because I don't know the keywords, I'm not finding the results in my search queries.
I want to create a function whose intention can be clearly described in the functions name but the parameters are flexible. I want to write the function to be generic enough so that the function can complete the intention with whatever parameters it receives from whichever caller.
Let's take a function do_foo.
do_foo can take in some_obj whose attributes allows do_foo to do its work. Additionally, do_foo can just take in the individual attributes it cares about like obj_attr0 or obj_attr1 and perform the same work. In both cases, the expected result is the same as well.
So this would look something like this:
Class SomeObj():
def __init__(self, obj_attr0, obj_attr1, obj_attrN):
self.obj_attr0 = None
self.obj_attr1 = None
self.obj_attrN = None # denotes an N number of attributes
def do_foo(params)
# unpack params. do_foo requires obj_attr0 and obj_attr1 and so its searching it in the params iterable
# determine which data is passed in
# execute the work the same way regardless of what form the data is passed in
pass
obj_attr0 = None
obj_attr1 = None
obj_attrN = None
some_obj = SomeObj(obj_attr0, obj_attr1, obj_attrN)
# One can either call with a subset of attributes that would make up SomeObj or SomeObj itself if all the information is there. E.g.:
params = (some_obj)
do_foo(params)
# OR
params = (obj_att0, obj_attr1)
do_foo(params)
I know python offers *args and **kwargs facilities that offer the flexibility above. I'm looking for some examples of where the implementation lends itself to reducing pitfalls. What is a good way to implement the above? And if there are any resources out there what are examples/articles/or terms that describe the above style of programming? Clearly, I'm trying to write my interface functions to be generic and usable in multiple logic paths where the users has its data in different forms where sticking to a specific parameter list is limiting.

Short answer:
You can use function decorators to do this
Long answer:
I have a concrete example for you. It might not be the prettiest code but it does something similar to what you are asking for.
Mini HTTP Testing library
I made a mini HTTP testing library because I make my REST http tests in python, and I realized that I always write the same code again and again. So I made a more general setup
The core
The core is kind of ugly and this is the part I don't want to write again and again.
Just skip this part quick and check how it is used in the interface section.
Then if you like it you can go back and try to understand how it is all tied together.
# base.py
import json, requests, inspect
# This function drops invallid parameters
def request(*args, **kwargs):
allowed = inspect.signature(requests.Session.request).parameters
return {k:v for (k,v) in kwargs.items() if k in allowed}
def response(r, code):
if r.status_code != code:
print(r.text)
return
data = r.json()
if data:
print(json.dumps(data, indent=2, ensure_ascii=False))
return data
# This is the core function it is not pretty but it creates all the abstaction in multiple levels of decorations.
def HTTP(base_url):
def outer(func_one):
def over(*args_one, **kwargs_one):
req, url, code = func_one(*args_one, **kwargs_one)
url = base_url + url
def inner(func_two):
def under(*args_two, **kwargs_two):
allowed = inspect.signature(func_two).parameters
kwparams = {k:v for (k,v) in kwargs_two.items() if k in allowed}
from_inner = func_two(*args_two, **kwparams)
u = url.format(id=kwargs_two.pop('_id')) if '{id}' in url else url
r = req(u, **request(**kwargs_two, **from_inner))
return response(r, code)
return under
return inner
return over
return outer
The interface
The interface functions are all each decorated by the HTTP function which makes them a HTTP caller function, it is still abstract since it will return a function.
Note: interface is just what I call it but it is really just functions which returns functions based on the HTTP decorator
BASE_URL = "https://example.com"
#HTTP(BASE_URL)
def POST(url, code=200): return requests.post, url, code
#HTTP(BASE_URL)
def PUT(url, code=200): return requests.put, url, code
#HTTP(BASE_URL)
def DELETE(url, code=200): return requests.delete, url, code
#HTTP(BASE_URL)
def GET(url, code=200): return requests.get, url, code
A middleware function
When one of the interface functions are decorated with this one then they need a token.
def AUTH(func):
def inner(token, *args, **kwargs):
headers = {'Authorization': f'bearer {token}'}
return func(*args, **kwargs, headers=headers)
return inner
The implementation
The interface can be used for many implementations.
Here I use the interface of POST, PUT, GET and DELETE for the user model.
This is the final decoration, and the functions returned will actually return content instead of other functions.
# users.py
from httplib.base import (
POST,
GET,
DELETE,
PUT,
AUTH,
request
)
#POST('/users',200)
def insert(user):
return request(json=user)
#AUTH
#GET('/users')
def find(_filter={}):
return request(params=_filter)
#AUTH
#GET('/users/{id}')
def find_one(_id):
return request()
#AUTH
#DELETE('/users/{id}')
def delete(_id):
return request()
#AUTH
#PUT('/users/{id}')
def update(_id, updates={}):
return request(json=updates)
Operation
Here you can see how the users delete insert and find functions work.
from httplib import users
def create_and_delete_users(token, n): return [
users.delete(token, _id=x['user']['id'])
for x in [
users.insert(user={
'username' : f'useruser{str(i).zfill(2)}',
'password' : 'secretpassword',
'email' : f'useruser{str(i).zfill(2)}#mail.com',
'gender' : 'male',
}) for i in range(n)]
]
def find_all_and_then_find_each(token): return [
users.find_one(token, _id=x['id'])
for x in users.find(token)['data']
]
I hope this was helpful.

Use methods on Mock object

I have an object that is used for fetching information from another service which is very simple. Since the object is simple and the initialization method could be easily patched I thought I would try to write my code to be super reusable and extendable. But alas, I cannot figure out how to make it work. The code below is pretty well sudo code and is super simplified but it should get the point across.
class SimpleClient:
def __init__(self):
pass
def read(self, key, path='some/path'):
return value_from_get_on_another_service
I then have a request handler object that initializes a client via get_client() (seen below)
def get_client():
return SimpleClient()
Then a method on the request handler uses the client.read() method a few times with different parameters (2nd dependent upon the 1st).
For my tests, I thought I could "patch" the get_client method to return my own simple object that could then be used "regularly" and eliminate the dependence on the third party service and actually use the values retrieved from the method execution. I was disappointed to find it was not that easy and clean. The test pattern is seen below.
class MockClient:
def __init__(self, addr='someAddr', token='someToken'):
pass
def read(self, value, prefix):
data = {}
if prefix == 'path/1':
data = self.p1_lookup(value)
elif prefix == 'path/2':
data = self.p2_lookup(value)
return self.response_wrapper(data)
def p2_lookup(self, key):
data = {
'key1': {
'sub_key': {"55B3FE7D-9F43-4DD4-9090-9D89330C918A": "Dev2",
"7A1C2F4B-E91C-4659-A33E-1B18B0BEE2B3": "Dev"}
}
}
return data.get(key, {})
#mock.patch('a.module.get_client')
def test_authorize_valid_request_no_body(mock_get_client):
request = RequestMock()
request.body = None
handler = RequestHandler(Application(), request=request, logging_level='INFO')
mock_get_client.return_value = MockClient()
handler.authorize_request()
assert handler.verified_headers is None
assert handler.verified_body is None
assert handler.user_authenticated is False
I have seen where I can mock the responses for the actual client.read() to return multiple values with a list. But this just seems like I will be doing lots of copy and paste and have to do the same thing over and over for each little test. Forgive me if this is simple, sadly I am just learning the art of testing. Is there a way to accomplish what I am trying to do? Maybe there is something super simple I am missing. Or maybe I am just totally on the wrong track for no good reason. Help?!

After a sleep, with fresh eyes I was able to figure this out relatively quickly thanks to a couple other similar questions/answers that I had not found before. Primarily this one, Python Mock Object with Method called Multiple Times.
Rather than needing to rebuild the module object completely I need to let mock do that for me and then override the specific method on it with the side_effect attribute. So below is what sanitized version of the code looks like.
def read_override(value, prefix):
lookup_data1 = {"lookup1": {'key1': 'value1'}}
lookup_data2 = {'some_id': {'akey': {'12345678': 'DEV'}}
data = {}
if prefix == 'path1/1a':
data = lookup_data1.get(value, {})
elif prefix == 'path2/2a':
data = lookup_data2.get(value, {})
return {'data': data}
# Create a true Mock of the entire LookupClient Object
VAULT_MOCK = mock.Mock(spec=LookupClient)
# make the read method work the way I want it to with an "override" of sorts
VAULT_MOCK.read.side_effect = vault_read_override
Then the test simply looked like this...
#mock.patch('a.module.get_client')
def test_authorize_valid_request_no_body(get_client):
get_client.return_value = VAULT_MOCK
request = RequestMock()
request.body = None
handler = RequestHandler(Application(), request=request, logging_level='INFO')
handler.authorize_request()
assert handler.verified_headers is None
assert handler.verified_body is None
assert handler.user_authenticated is False

Detect if route is not outermost / wrong "order" of decorators in Flask

Since the #route decorator has to register the view with the current callback given to the decorator, it has to be the outermost decorator to receive the correct function to invoke when handling a request.
This creates a possible situation where a view has been decorated, but since the decorators are in the wrong order, the decorated function is not invoked. If used for decorating views that require the user to be logged in, have a certain role or have a specific flag, the check will be left out silently.
Our current fix is to have the standard action be to deny access to the resource, then requiring a decorator to allow access. In that case, if the decorator isn't invoked when the request is being handled, the request will fail.
But there are use-cases where this becomes cumbersome since it requires you to decorate all views, except for those few that should be exempt. For a pure hierarchical layout this may work, but for checking single flags the structure can get complicated.
Is there a proper way to detect that we're being invoked in a useful place in the decoratory hierarchy? I.e. can we detect that there hasn't already been a route decorator applied to function we get to wrap?
# wrapped in wrong order - #require_administrator should be after #app.route
#require_administrator
#app.route('/users', methods=['GET'])
Implemented as:
def require_administrator(func):
#functools.wraps(func)
def has_administrator(*args, **kwargs):
if not getattr(g, 'user') or not g.user.is_administrator:
abort(403)
return func(*args, **kwargs)
return has_administrator
Here I'd like to detect if my custom decorator is being wrapped after #app.route, and thus, never will be invoked when the request is handled.
Using functools.wraps replaces the wrapped function with the new one in all ways, so looking at __name__ of the function to be wrapped will fail. This also happens at each step of the decorator wrapping process.
I've tried looking at both traceback and inspect, but haven't found any decent way of determining if the sequence is correct.
Update
My currently best solution is to check the called function name against the set of registered endpoints. But since a Route() decorator can change the name of the endpoint, I'll have to support that for my decorator as well in that case, and it'll silently pass if a different function has used the same endpoint name as the current function.
It also have to iterate the set of registered endpoints since I weren't able to find a simple way to check if just the endpoint name exists (possibly more efficient by attempting to build an URL with it and catch the exception).
def require_administrator_checked(func):
for rule in app.url_map.iter_rules():
if func.__name__ == rule.endpoint:
raise DecoratorOrderError(f"Wrapped endpoint '{rule.endpoint}' has already been registered - wrong order of decorators?")
# as above ..

Update 2: See my other answer for a more reusable, less hack-y solution.
Update:
Here is an decidedly less hack-y solution. However, it requires you to use a
custom function instead of app.route. It takes an arbitrary number of decorators, and applies them in the order given, and then makes sure that app.route is called as the final function.
This requires that you use only this decorator for every function.
def safe_route(rule, app, *decorators, **options):
def _route(func):
for decorator in decorators:
func = decorator(func)
return app.route(rule, **options)(func)
return _route
You can then use it like this:
def require_administrator(func):
#functools.wraps(func)
def has_administrator(*args, **kwargs):
print("Would check admin now")
return func(*args, **kwargs)
return has_administrator
#safe_route("/", app, require_administrator, methods=["GET"])
def test2():
return "foo"
test2()
print(test2.__name__)
This prints:
Would check admin now
foo
test2
So if all supplied decorators use functools.wraps, this also conserves the test2 name.
Old answer:
If you are OK with an admittedly hack-y solution, you can roll your own inspection by reading the file line by line. Here is a very rough function that does this. You can refine this quite a bit, e.g. at the moment it relies on the app being called "app",
function definitions having at least one empty line before them (normal PEP-8 behavior, but still might be an issue), ...
Here is the complete code I used to test it.
import flask
import functools
from itertools import groupby
class DecoratorOrderError(TypeError):
pass
app = flask.Flask(__name__)
def require_administrator(func):
#functools.wraps(func)
def has_administrator(*args, **kwargs):
print("Would check admin now")
return func(*args, **kwargs)
return has_administrator
#require_administrator # Will raise a custom exception
#app.route("/", methods=["GET"])
def test():
return "ok"
def check_route_is_topmost_decorator():
# Read own source
with open(__file__) as f:
content = [line.strip() for line in f.readlines()]
# Split source code on line breaks
split_by_lines = [list(group) for k, group in groupby(content, lambda x: x == "") if not k]
# Find consecutive decorators
decorator_groups = dict()
for line_group in split_by_lines:
decorators = []
for line in line_group:
if line.startswith("#"):
decorators.append(line)
elif decorators:
decorator_groups[line] = decorators
break
else:
break
# Check if app.route is the last one (if it exists)
for func_def, decorators in decorator_groups.items():
is_route = [dec.startswith("#app.route") for dec in decorators]
if sum(is_route) > 1 or (sum(is_route) == 1 and not decorators[0].startswith("#app.route")):
raise DecoratorOrderError(f"#app.route is not the topmost decorator for '{func_def}'")
check_route_is_topmost_decorator()
This snippet will give you the following error:
Traceback (most recent call last):
File "/home/vXYZ/test_sso.py", line 51, in <module>
check_route_is_topmost_decorator()
File "/home/vXYZ/test_sso.py", line 48, in check_route_is_topmost_decorator
raise DecoratorOrderError(f"#app.route is not the topmost decorator for '{func_def}'")
__main__.DecoratorOrderError: #app.route is not the topmost decorator for 'def test():'
If you switch the order of the decorator for the test() function, it simply does nothing.
One downside is that you have to call this method explicitly in every file.
I don't exactly know how reliable this is, I admit it is pretty ugly, and I won't take any responsibility if it breaks, but it's a start! I am sure there must be a better way.

I am adding another answer, because now I have something that is the least amount of hacky (read: I am using inspect to read to source code of a given function instead of reading the whole file myself), works across modules, and can be reused for any other decorators that should always be the last one. You also do not have to use a different syntax for app.route as in the update of my other answer.
Here is how to do this (Warning: This is quite a closure-inception):
import flask
import inspect
class DecoratorOrderError(TypeError):
pass
def assert_last_decorator(final_decorator):
"""
Converts a decorator so that an exception is raised when it is not the last decorator to be used on a function.
This only works for decorator syntax, not if somebody explicitly uses the decorator, e.g.
final_decorator = some_other_decorator(final_decorator) will still work without an exception.
:param final_decorator: The decorator that should be made final.
:return: The same decorator, but it checks that it is the last one before calling the inner function.
"""
def check_decorator_order(func):
# Use inspect to read the code of the function
code, _ = inspect.getsourcelines(func)
decorators = []
for line in code:
if line.startswith("#"):
decorators.append(line)
else:
break
# Remove the "#", function calls, and any object calls, such as "app.route". We just want the name of the decorator function (e.g. "route")
decorator_names_only = [dec.replace("#", "").split("(")[0].split(".")[-1] for dec in decorators]
is_final_decorator = [final_decorator.__name__ == name for name in decorator_names_only]
num_finals = sum(is_final_decorator)
if num_finals > 1 or (num_finals == 1 and not is_final_decorator[0]):
raise DecoratorOrderError(f"'{final_decorator.__name__}' is not the topmost decorator of function '{func.__name__}'")
return func
def handle_arguments(*args, **kwargs):
# Used to pass the arguments to the final decorator
def handle_function(f):
# Which function should be decorated by the final decorator?
return final_decorator(*args, **kwargs)(check_decorator_order(f))
return handle_function
return handle_arguments
You can now replace the app.route function with this function, applied to the app.route function. This is important and has to be done before any use of the app.route decorator, so I suggest to just do it when creating the app.
app = flask.Flask(__name__)
app.route = assert_last_decorator(app.route)
def require_administrator(func):
#functools.wraps(func)
def has_administrator(*args, **kwargs):
print("Would check admin now")
return func(*args, **kwargs)
return has_administrator
#app.route("/good", methods=["GET"]) # Works
#require_administrator
def test_good():
return "ok"
#require_administrator
#app.route("/bad", methods=["GET"]) # Raises an Exception
def test_bad():
return "not ok"
I believe this is pretty much what you wanted in your question.

Verify Tornado arguments

I have a server which can accept 0, 1 or many of the following url arguments:
/api/cases?id={id}&name={name}&owner={owner}&status={status}
So these, amongst other, are correct:
/api/cases?owner=me
/api/cases
/api/cases?name=bob&status=waiting
Currently, my code looks like this
routes = [(r'/cases?([^/]+)', MyHandler)]
tornado.web.Application.__init__(self, routes, settings={})
class MyHandler(APIHandler):
ACCEPTED_URL_ARGS = ["id", "name", "owner", "status"]
def get(self, i):
for key in self.request.arguments:
if key not in self.ACCEPTED_URL_ARGS:
# error
Is there a better way to check for the url arguments?

What you have is correct. In Tornado there is no other way to verify that you only got the arguments that you expect than to iterate over self.request.arguments.

As Ben says, this is the correct way to do it in Tornado. That being said, the better place for your test would be the prepare method; also, a strictly more "pythonic" approach would be to use sets:
class MyHandler(APIHandler):
ACCEPTED_URL_ARGS = {"id", "name", "owner", "status"}
def prepare(self):
unwanted_args = self.ACCEPTED_URL_ARGS - set(self.request.arguments)
if unwanted_args:
# error

Loop using app.route on Python

I'm trying to create several URLs on my serv thanks to a loop . The issue is that each function I create in a app.route can't have the same name than the others . And I don't know how to create different function names ...
Here is the code :
json_tweets = []
for line in open('C:\Users\Benjamin\Desktop\DashboardProject\last_rated_set.json',"r"):
json_tweets.append(json.loads(line,"ISO-8859-1"))
cashtag_tab = []
for tweet in json_tweets:
if not(tweet['cashtag'] in cashtag_tab) :
cashtag_tab.append(tweet['cashtag'])
for i in range(0,(len(cashtag_tab)-1)) :
var=cashtag_tab[i]
#app.route("/"+var)
def company(var) :
finance=Share(var)
datas = finance.get_historical('2014-01-01', '2014-12-31')
datas = json.dumps(datas, default=json_util.default)
return datas
I'm getting the error AssertionError : View function mapping is overwritting an existing endpoint function : company

This fails because Flask derives the endpoint name from the function by default, but it would anyway fail later because the function company requires an argument var and the route is not parameterised. The simplest option would be just checking the value inside the handler:
#api.route('/<var>')
def company(var):
if var not in cashtag_tab:
abort(404)
If you want all the routes to be in the routing map for any reason, I once needed a similar thing and came up with something like this:
def url_family(source, methods=('GET',)):
def decorator(f):
for entry in source:
# create a handler that delegates to your function
def view_func(entry=entry, **kwargs):
return f(entry, **kwargs)
endpoint = '{0}_{1}'.format(f.__name__, entry)
url = '/{0}'.format(entry)
api.add_url_rule(url,
methods=methods,
endpoint=endpoint,
view_func=view_func)
return decorator
Then you register the handlers as:
#url_family(cashtag_tab)
def company(var):
...

Assuming that you are using flask now, you should consider Custom URL Converter. Check links below
http://flask.pocoo.org/docs/0.10/api/#flask.Flask.url_map - url_map UrlConverter API
https://exploreflask.com/views.html#url-converters - example url converter
https://stackoverflow.com/a/5872904/3451543 - RegexConverter by Philip Southam
Anyway, specifying more details on your question is always helpful to get accurate answer :)