Develop Reference

Python Generic functions with generic parameters

I'm not sure if I used the right terms in the title. This maybe a known way to program interface functions for a subsystem or module but because I don't know the keywords, I'm not finding the results in my search queries.
I want to create a function whose intention can be clearly described in the functions name but the parameters are flexible. I want to write the function to be generic enough so that the function can complete the intention with whatever parameters it receives from whichever caller.
Let's take a function do_foo.
do_foo can take in some_obj whose attributes allows do_foo to do its work. Additionally, do_foo can just take in the individual attributes it cares about like obj_attr0 or obj_attr1 and perform the same work. In both cases, the expected result is the same as well.
So this would look something like this:
Class SomeObj():
def __init__(self, obj_attr0, obj_attr1, obj_attrN):
self.obj_attr0 = None
self.obj_attr1 = None
self.obj_attrN = None # denotes an N number of attributes
def do_foo(params)
# unpack params. do_foo requires obj_attr0 and obj_attr1 and so its searching it in the params iterable
# determine which data is passed in
# execute the work the same way regardless of what form the data is passed in
pass
obj_attr0 = None
obj_attr1 = None
obj_attrN = None
some_obj = SomeObj(obj_attr0, obj_attr1, obj_attrN)
# One can either call with a subset of attributes that would make up SomeObj or SomeObj itself if all the information is there. E.g.:
params = (some_obj)
do_foo(params)
# OR
params = (obj_att0, obj_attr1)
do_foo(params)
I know python offers *args and **kwargs facilities that offer the flexibility above. I'm looking for some examples of where the implementation lends itself to reducing pitfalls. What is a good way to implement the above? And if there are any resources out there what are examples/articles/or terms that describe the above style of programming? Clearly, I'm trying to write my interface functions to be generic and usable in multiple logic paths where the users has its data in different forms where sticking to a specific parameter list is limiting.

Short answer:
You can use function decorators to do this
Long answer:
I have a concrete example for you. It might not be the prettiest code but it does something similar to what you are asking for.
Mini HTTP Testing library
I made a mini HTTP testing library because I make my REST http tests in python, and I realized that I always write the same code again and again. So I made a more general setup
The core
The core is kind of ugly and this is the part I don't want to write again and again.
Just skip this part quick and check how it is used in the interface section.
Then if you like it you can go back and try to understand how it is all tied together.
# base.py
import json, requests, inspect
# This function drops invallid parameters
def request(*args, **kwargs):
allowed = inspect.signature(requests.Session.request).parameters
return {k:v for (k,v) in kwargs.items() if k in allowed}
def response(r, code):
if r.status_code != code:
print(r.text)
return
data = r.json()
if data:
print(json.dumps(data, indent=2, ensure_ascii=False))
return data
# This is the core function it is not pretty but it creates all the abstaction in multiple levels of decorations.
def HTTP(base_url):
def outer(func_one):
def over(*args_one, **kwargs_one):
req, url, code = func_one(*args_one, **kwargs_one)
url = base_url + url
def inner(func_two):
def under(*args_two, **kwargs_two):
allowed = inspect.signature(func_two).parameters
kwparams = {k:v for (k,v) in kwargs_two.items() if k in allowed}
from_inner = func_two(*args_two, **kwparams)
u = url.format(id=kwargs_two.pop('_id')) if '{id}' in url else url
r = req(u, **request(**kwargs_two, **from_inner))
return response(r, code)
return under
return inner
return over
return outer
The interface
The interface functions are all each decorated by the HTTP function which makes them a HTTP caller function, it is still abstract since it will return a function.
Note: interface is just what I call it but it is really just functions which returns functions based on the HTTP decorator
BASE_URL = "https://example.com"
#HTTP(BASE_URL)
def POST(url, code=200): return requests.post, url, code
#HTTP(BASE_URL)
def PUT(url, code=200): return requests.put, url, code
#HTTP(BASE_URL)
def DELETE(url, code=200): return requests.delete, url, code
#HTTP(BASE_URL)
def GET(url, code=200): return requests.get, url, code
A middleware function
When one of the interface functions are decorated with this one then they need a token.
def AUTH(func):
def inner(token, *args, **kwargs):
headers = {'Authorization': f'bearer {token}'}
return func(*args, **kwargs, headers=headers)
return inner
The implementation
The interface can be used for many implementations.
Here I use the interface of POST, PUT, GET and DELETE for the user model.
This is the final decoration, and the functions returned will actually return content instead of other functions.
# users.py
from httplib.base import (
POST,
GET,
DELETE,
PUT,
AUTH,
request
)
#POST('/users',200)
def insert(user):
return request(json=user)
#AUTH
#GET('/users')
def find(_filter={}):
return request(params=_filter)
#AUTH
#GET('/users/{id}')
def find_one(_id):
return request()
#AUTH
#DELETE('/users/{id}')
def delete(_id):
return request()
#AUTH
#PUT('/users/{id}')
def update(_id, updates={}):
return request(json=updates)
Operation
Here you can see how the users delete insert and find functions work.
from httplib import users
def create_and_delete_users(token, n): return [
users.delete(token, _id=x['user']['id'])
for x in [
users.insert(user={
'username' : f'useruser{str(i).zfill(2)}',
'password' : 'secretpassword',
'email' : f'useruser{str(i).zfill(2)}#mail.com',
'gender' : 'male',
}) for i in range(n)]
]
def find_all_and_then_find_each(token): return [
users.find_one(token, _id=x['id'])
for x in users.find(token)['data']
]
I hope this was helpful.

How to skip Flask cache on template errors

I'm using Flask with template caching on a Redis server:
TIMEOUT = 60 * 60
cache = Cache(app.server, config={
'CACHE_TYPE': 'redis',
'CACHE_REDIS_HOST': "myredis",
'CACHE_DEFAULT_TIMEOUT': TIMEOUT,
'CACHE_REDIS_PORT': 6379,
})
# to disable caching
#app.config["CACHE_TYPE"] = "null"
and then with the #cache decorator like
#cache.memoize(timeout=TIMEOUT)
def update_date():
return manager.getData()
The problem is that when manager.getData() has errors or no data the decorator will cache the response anyways. How to avoid it?
[UPDATE]
I have tried using the unless parameter, that according to the docs it should be
unless – Default None. Cache will always execute the caching facilities unelss this callable is true. This will bypass the caching entirely.
so used like
#cache.memoize(timeout=TIMEOUT unless=DataLoader.instance.hasData)
def update_date():
return manager.getData()
where DataLoader is a Singleton instance and hasData method will return None if has no data or True if it has data, so the method getData would compute the data and return instance variable self.data that holds always last computed data or None.
class DataLoader(SingletonMixin):
def __init__(self):
self.data=None
def hasData(self):
if self.data is Not None:
return True
else:
return None
def getData(self):
# calculate data
res = self.computeData()
if res is not None:
self.data=res
return self.data
but it seems it does not work as expected.

The problem is that when manager.getData() has errors or no data the decorator will cache the response anyways. How to avoid it?
Have you checked that? If you look at the source code (I assume you are using flask-caching because flask-cache is not maintained for over 4 years) if you get None from cache (rv value) you don't use it, you call your f function. If f function raises exception nothing is saved to cache.

is this possible to create sub-methods in python?

I want to write a method to parse a site with requests library, the method should take a part of url having base_url in it and perform the get request on this, the main problem is that I do not know how to make it better;
What I have in mind now is:
import requests
class Response:
# ...
def site_parser(self, atom):
base_url="https://example.com/"
def category1(self):
return requests.get(base_url + category1/ + atom).text
def category2(self):
return requests.get(base_url + category2/ + atom).text
if __name == "__main__":
def main():
result = Response()
result.site_parser.category1("atom")
result.site_parser.category2("atom")
so needed data has the same base url but different dirs to get into, and I need to gen each dir if only the method was called afterwards. is there a way of doing this properly? I wouuld like to avoid making base url global variable

It seems to me that what you need is another class.
class Response:
# ... Some dark magic here ...
def site_parser(self, atom):
return ResponseParser(self, atom)
class ResponseParser:
def __init__(self, res, atom):
self.atom = atom
self.res = res
self.base_url = "https://example.com/"
def category1(self):
# ... Do stuff ...
def category2(self):
# ... Do stuff ...
Then you call it with
result = Response()
result.site_parser("atom").category1()
If you really insist on getting rid of the parentheses on the site_parser call, you could move the "atom" bit to the categoryN methods and turn site_parser into a property, but IMO that would probably just confuse people more than anything.
As a functional programmer, I love nested functions and closures as much as the next guy, but it seems to me that, based on the limited example you've given, having a second helper class is probably going to be the more readable way to go about this in this case.

pytest fixture to introspect calling function

I have a test class and a setup function that looks like this:
#pytest.fixture(autouse=True, scope='function')
def setup(self, request):
self.client = MyClass()
first_patcher = patch('myclass.myclass.function_to_patch')
first_mock = first_patcher.start()
first_mock.return_value = 'foo'
value_to_return = getattr(request, 'value_name', None)
second_patcher = patch('myclass.myclass.function_two')
second_mock = second_patcher.start()
second_mock.return_value = value_to_return
#could clean up my mocks here, but don't care right now
I see in the documentation for pytest, that introspection can be done for a module level value:
val = getattr(request.module, 'val_name', None)
But, I want to be able to specify different values to return based on the test I am in. So I am looking for a way to introspect the test_function not the test_module.
http://pytest.org/latest/fixture.html#fixtures-can-introspect-the-requesting-test-context

You can use request.function to get to the test function. Just follow the link on the b wepage you referenced to see what is available on the test request object :)

Maybe the documentation has changed since the time of the accepted answer.
At least for me it was not clear how to
Just follow the link
So I thought I'd update this thread with the link itself:
https://pytest.org/en/6.2.x/reference.html#request
Edit December 2021
Even when the link is correct now I think this statement from the pytest documentation is just not correct:
Fixture functions can accept the request object to introspect the “requesting” test function ...
While I found some examples for getting attributes of the module I did not find a single working example of introspecting the test function that requests the fixture. May be related to collection and runtime order.
What really helped me to get the desired behavior was to use the factory idiom a little further down in the pytest documentation:
https://pytest.org/en/6.2.x/fixture.html#factories-as-fixtures
Set up the fixture factory
#pytest.fixture(scope='function')
def getQueryResult() -> object:
def _impl(_mrId: int = 7622):
return QueryResult(_mrId)
return _impl
Usage
# Concrete value
def test_foo(getQueryResult):
queryResult = getQueryResult(4711)
...
# Default value
def test_bar(getQueryResult):
queryResult = getQueryResult()
...

Patterns - Event Dispatcher without else if?

I'm creating a Python wrapper for the Detours library. One piece of the tool is a dispatcher to send all of the hooked API calls to various handlers.
Right now my code looks like this:
if event == 'CreateWindowExW':
# do something
elif event == 'CreateProcessW':
# do something
elif ...
This feels ugly. Is there a pattern to create an event dispatcher without my having to create an elif branch for each Windows API function?

One nice way to do this is to define a class which has methods equating to the relevant API function names, plus a dispatch method which dispatches to the correct method. For example:
class ApiDispatcher(object):
def handle_CreateWindowExW(self):
# do whatever
def handle_CreateProcessW(self):
# do this one
def dispatch(self, event):
method = getattr(self, 'handle_%s' % event)
method()

Those if's will eventually have to go somewhere. Why not do it like this:
handler = get_handler(event)
handler.process()
and in the get_handler you'd have your ifs, each returning an object which does its work in the process method.
An alternative would be a map to callables, like this:
def react_to_create_window_exw():
# do something with event here
pass
handlers = {
"CreateWindowExW" : react_to_create_window_exw
}
and you would use it like this:
handler = handlers[event]
handler()
This way you would not use any if/else conditions.

You can use the dispatch dict method.
def handle_CreateWindowExW():
print "CreateWindowExW"
#do something
events = {
"CreateWindowExW": handle_CreateWindowExW
}
events[event]()
This way, you can just add events without having to add different if statements.

Usually in such cases when you have a predefined list of actions to take, use a map e.g.
def CreateWindowExW():
print 'CreateWindowExW'
def CreateProcessW():
print 'CreateProcessW'
action_map = {
'CreateWindowExW': CreateWindowExW,
'CreateProcessW': CreateProcessW
}
for action in ['CreateWindowExW', 'UnkownAction']:
try:
action_map[action]()
except KeyError:
print action, "Not Found"
Output:
CreateWindowExW
UnkownAction Not Found
so using a map you can create a very powerful dispatcher

I didn't find anything that was as graceful as it could be in this area, so I wrote something that let's you do:
from switcheroo import Switch, default
switch = Switch({
'foo': lambda x: x+1,
default: lambda x: x-1,
})
>>> switch['foo'](1)
2
>>> switch['bar'](1)
0
There are some other flavours; docs are here, code is on github, package is on pypi.