Develop Reference

Python Generic functions with generic parameters

I'm not sure if I used the right terms in the title. This maybe a known way to program interface functions for a subsystem or module but because I don't know the keywords, I'm not finding the results in my search queries.
I want to create a function whose intention can be clearly described in the functions name but the parameters are flexible. I want to write the function to be generic enough so that the function can complete the intention with whatever parameters it receives from whichever caller.
Let's take a function do_foo.
do_foo can take in some_obj whose attributes allows do_foo to do its work. Additionally, do_foo can just take in the individual attributes it cares about like obj_attr0 or obj_attr1 and perform the same work. In both cases, the expected result is the same as well.
So this would look something like this:
Class SomeObj():
def __init__(self, obj_attr0, obj_attr1, obj_attrN):
self.obj_attr0 = None
self.obj_attr1 = None
self.obj_attrN = None # denotes an N number of attributes
def do_foo(params)
# unpack params. do_foo requires obj_attr0 and obj_attr1 and so its searching it in the params iterable
# determine which data is passed in
# execute the work the same way regardless of what form the data is passed in
pass
obj_attr0 = None
obj_attr1 = None
obj_attrN = None
some_obj = SomeObj(obj_attr0, obj_attr1, obj_attrN)
# One can either call with a subset of attributes that would make up SomeObj or SomeObj itself if all the information is there. E.g.:
params = (some_obj)
do_foo(params)
# OR
params = (obj_att0, obj_attr1)
do_foo(params)
I know python offers *args and **kwargs facilities that offer the flexibility above. I'm looking for some examples of where the implementation lends itself to reducing pitfalls. What is a good way to implement the above? And if there are any resources out there what are examples/articles/or terms that describe the above style of programming? Clearly, I'm trying to write my interface functions to be generic and usable in multiple logic paths where the users has its data in different forms where sticking to a specific parameter list is limiting.

Short answer:
You can use function decorators to do this
Long answer:
I have a concrete example for you. It might not be the prettiest code but it does something similar to what you are asking for.
Mini HTTP Testing library
I made a mini HTTP testing library because I make my REST http tests in python, and I realized that I always write the same code again and again. So I made a more general setup
The core
The core is kind of ugly and this is the part I don't want to write again and again.
Just skip this part quick and check how it is used in the interface section.
Then if you like it you can go back and try to understand how it is all tied together.
# base.py
import json, requests, inspect
# This function drops invallid parameters
def request(*args, **kwargs):
allowed = inspect.signature(requests.Session.request).parameters
return {k:v for (k,v) in kwargs.items() if k in allowed}
def response(r, code):
if r.status_code != code:
print(r.text)
return
data = r.json()
if data:
print(json.dumps(data, indent=2, ensure_ascii=False))
return data
# This is the core function it is not pretty but it creates all the abstaction in multiple levels of decorations.
def HTTP(base_url):
def outer(func_one):
def over(*args_one, **kwargs_one):
req, url, code = func_one(*args_one, **kwargs_one)
url = base_url + url
def inner(func_two):
def under(*args_two, **kwargs_two):
allowed = inspect.signature(func_two).parameters
kwparams = {k:v for (k,v) in kwargs_two.items() if k in allowed}
from_inner = func_two(*args_two, **kwparams)
u = url.format(id=kwargs_two.pop('_id')) if '{id}' in url else url
r = req(u, **request(**kwargs_two, **from_inner))
return response(r, code)
return under
return inner
return over
return outer
The interface
The interface functions are all each decorated by the HTTP function which makes them a HTTP caller function, it is still abstract since it will return a function.
Note: interface is just what I call it but it is really just functions which returns functions based on the HTTP decorator
BASE_URL = "https://example.com"
#HTTP(BASE_URL)
def POST(url, code=200): return requests.post, url, code
#HTTP(BASE_URL)
def PUT(url, code=200): return requests.put, url, code
#HTTP(BASE_URL)
def DELETE(url, code=200): return requests.delete, url, code
#HTTP(BASE_URL)
def GET(url, code=200): return requests.get, url, code
A middleware function
When one of the interface functions are decorated with this one then they need a token.
def AUTH(func):
def inner(token, *args, **kwargs):
headers = {'Authorization': f'bearer {token}'}
return func(*args, **kwargs, headers=headers)
return inner
The implementation
The interface can be used for many implementations.
Here I use the interface of POST, PUT, GET and DELETE for the user model.
This is the final decoration, and the functions returned will actually return content instead of other functions.
# users.py
from httplib.base import (
POST,
GET,
DELETE,
PUT,
AUTH,
request
)
#POST('/users',200)
def insert(user):
return request(json=user)
#AUTH
#GET('/users')
def find(_filter={}):
return request(params=_filter)
#AUTH
#GET('/users/{id}')
def find_one(_id):
return request()
#AUTH
#DELETE('/users/{id}')
def delete(_id):
return request()
#AUTH
#PUT('/users/{id}')
def update(_id, updates={}):
return request(json=updates)
Operation
Here you can see how the users delete insert and find functions work.
from httplib import users
def create_and_delete_users(token, n): return [
users.delete(token, _id=x['user']['id'])
for x in [
users.insert(user={
'username' : f'useruser{str(i).zfill(2)}',
'password' : 'secretpassword',
'email' : f'useruser{str(i).zfill(2)}#mail.com',
'gender' : 'male',
}) for i in range(n)]
]
def find_all_and_then_find_each(token): return [
users.find_one(token, _id=x['id'])
for x in users.find(token)['data']
]
I hope this was helpful.

Best practices for sharing a python dictionary between functions in an imported module

Relatively new to python and I'm curious as to the best method to pull off sharing/modifying a dictionary between functions within a module. Example:
some_module.py
import requests
my_dict = { 'url': 'www.difficult.com', 'keys': None, 'params': None }
def set_keys(keys):
my_dict['keys']=keys #these are needed for every request/
def set_limit_param(param):
my_dict["params"]["limit"] = param #not needed for every request
def make_request(added):
r = requests.get(my_dict["url"] + added,headers=my_dict["keys"],params=my_dict["params"])
def do_thing1():
make_request("/thing1")
def do_thing2():
set_limit_param("75,000")
make_request("/thing2")
In my use case some_module.py is imported within another script.
Calling some_module.do_thing2() modifies my dictionary with values I don't really want present when I call some_module.do_thing1().
However I want the data added when calling some_module.set_keys(key="blahblah") to persist within the dictionary.
I've experimented a bit with my_dict.copy() and copy.deepcopy() but it seems cumbersome to do that within very function that modifies my_dict. Any guidance would be appreciated.

As you are modifying the dict, I would recommend an object (from a standard class):
import requests
class Requester():
def __init__(self):
self.config = { 'url': 'www.difficult.com', 'keys': None, 'params': None }
def set_keys(self, keys):
self.config['keys']=keys #these are needed for every request/
def set_limit_param(self, param):
self.config["params"]["limit"] = param #not needed for every request
def make_request(self, added):
r = requests.get(
self.config["url"] + added,
headers=self.config["keys"],
params=self.config["params"]
)
def do_thing1(self):
make_request("/thing1")
def do_thing2(self):
set_limit_param("75,000")
make_request("/thing2")
# create
my_requester = Requester()
Then, generally in your code you use this object instead of a module.
Even if there will be only one instance of such class, objects are generally best to store some data along with some methods to these data.
Another approach would be leave the methods as they are and use a plain dict (as you did), but don't store it in the methods' module. Tread your module just like a collection of stateless methods and store/define the data (dict) separately:
# requester.py
import requests
def set_keys(my_dict, keys):
my_dict['keys']=keys #these are needed for every request/
def set_limit_param(my_dict, param):
my_dict["params"]["limit"] = param #not needed for every request
def make_request(my_dict, added):
r = requests.get(my_dict["url"] + added,headers=my_dict["keys"],params=my_dict["params"])
def do_thing1(my_dict):
make_request(my_dict, "/thing1")
def do_thing2(my_dict):
set_limit_param("75,000")
make_request(my_dict, "/thing2")
# my_app.py
import requester
my_dict = { 'url': 'www.difficult.com', 'keys': None, 'params': None }
requester.set_keys(my_dict, 1)
requester.do_thing1(my_dict)
Calling some_module.do_thing2() modifies my dictionary with values I don't really want present when I call some_module.do_thing1().
This sounds like you need 2 different dicts - to avoid modifying one by another. Depending on what you do in do_thing1 and do_thing2 functions I would also consider separate them to different classes/modules.

is this possible to create sub-methods in python?

I want to write a method to parse a site with requests library, the method should take a part of url having base_url in it and perform the get request on this, the main problem is that I do not know how to make it better;
What I have in mind now is:
import requests
class Response:
# ...
def site_parser(self, atom):
base_url="https://example.com/"
def category1(self):
return requests.get(base_url + category1/ + atom).text
def category2(self):
return requests.get(base_url + category2/ + atom).text
if __name == "__main__":
def main():
result = Response()
result.site_parser.category1("atom")
result.site_parser.category2("atom")
so needed data has the same base url but different dirs to get into, and I need to gen each dir if only the method was called afterwards. is there a way of doing this properly? I wouuld like to avoid making base url global variable

It seems to me that what you need is another class.
class Response:
# ... Some dark magic here ...
def site_parser(self, atom):
return ResponseParser(self, atom)
class ResponseParser:
def __init__(self, res, atom):
self.atom = atom
self.res = res
self.base_url = "https://example.com/"
def category1(self):
# ... Do stuff ...
def category2(self):
# ... Do stuff ...
Then you call it with
result = Response()
result.site_parser("atom").category1()
If you really insist on getting rid of the parentheses on the site_parser call, you could move the "atom" bit to the categoryN methods and turn site_parser into a property, but IMO that would probably just confuse people more than anything.
As a functional programmer, I love nested functions and closures as much as the next guy, but it seems to me that, based on the limited example you've given, having a second helper class is probably going to be the more readable way to go about this in this case.

Use methods on Mock object

I have an object that is used for fetching information from another service which is very simple. Since the object is simple and the initialization method could be easily patched I thought I would try to write my code to be super reusable and extendable. But alas, I cannot figure out how to make it work. The code below is pretty well sudo code and is super simplified but it should get the point across.
class SimpleClient:
def __init__(self):
pass
def read(self, key, path='some/path'):
return value_from_get_on_another_service
I then have a request handler object that initializes a client via get_client() (seen below)
def get_client():
return SimpleClient()
Then a method on the request handler uses the client.read() method a few times with different parameters (2nd dependent upon the 1st).
For my tests, I thought I could "patch" the get_client method to return my own simple object that could then be used "regularly" and eliminate the dependence on the third party service and actually use the values retrieved from the method execution. I was disappointed to find it was not that easy and clean. The test pattern is seen below.
class MockClient:
def __init__(self, addr='someAddr', token='someToken'):
pass
def read(self, value, prefix):
data = {}
if prefix == 'path/1':
data = self.p1_lookup(value)
elif prefix == 'path/2':
data = self.p2_lookup(value)
return self.response_wrapper(data)
def p2_lookup(self, key):
data = {
'key1': {
'sub_key': {"55B3FE7D-9F43-4DD4-9090-9D89330C918A": "Dev2",
"7A1C2F4B-E91C-4659-A33E-1B18B0BEE2B3": "Dev"}
}
}
return data.get(key, {})
#mock.patch('a.module.get_client')
def test_authorize_valid_request_no_body(mock_get_client):
request = RequestMock()
request.body = None
handler = RequestHandler(Application(), request=request, logging_level='INFO')
mock_get_client.return_value = MockClient()
handler.authorize_request()
assert handler.verified_headers is None
assert handler.verified_body is None
assert handler.user_authenticated is False
I have seen where I can mock the responses for the actual client.read() to return multiple values with a list. But this just seems like I will be doing lots of copy and paste and have to do the same thing over and over for each little test. Forgive me if this is simple, sadly I am just learning the art of testing. Is there a way to accomplish what I am trying to do? Maybe there is something super simple I am missing. Or maybe I am just totally on the wrong track for no good reason. Help?!

After a sleep, with fresh eyes I was able to figure this out relatively quickly thanks to a couple other similar questions/answers that I had not found before. Primarily this one, Python Mock Object with Method called Multiple Times.
Rather than needing to rebuild the module object completely I need to let mock do that for me and then override the specific method on it with the side_effect attribute. So below is what sanitized version of the code looks like.
def read_override(value, prefix):
lookup_data1 = {"lookup1": {'key1': 'value1'}}
lookup_data2 = {'some_id': {'akey': {'12345678': 'DEV'}}
data = {}
if prefix == 'path1/1a':
data = lookup_data1.get(value, {})
elif prefix == 'path2/2a':
data = lookup_data2.get(value, {})
return {'data': data}
# Create a true Mock of the entire LookupClient Object
VAULT_MOCK = mock.Mock(spec=LookupClient)
# make the read method work the way I want it to with an "override" of sorts
VAULT_MOCK.read.side_effect = vault_read_override
Then the test simply looked like this...
#mock.patch('a.module.get_client')
def test_authorize_valid_request_no_body(get_client):
get_client.return_value = VAULT_MOCK
request = RequestMock()
request.body = None
handler = RequestHandler(Application(), request=request, logging_level='INFO')
handler.authorize_request()
assert handler.verified_headers is None
assert handler.verified_body is None
assert handler.user_authenticated is False

How to deal with globals in modules?

I try to make a non blocking api calls for OpenWeatherMap, but my problem is:
When i was doing tests on the file, and run it, the global api was taking effect, but when importing the function, global dont work anymore, and api dident change: api = ""?
Just after declaring the function i put global api, and then when I use print 'The API link is: ' + api I get the exact api, but global dident took effect!
Here is the code: https://github.com/abdelouahabb/tornadowm/blob/master/tornadowm.py#L62
What am I doing wrong?
When I import the file:
from tornadowm import *
forecast('daily', q='london', lang='fr')
The API link is: http://api.openweathermap.org/data/2.5/forecast/daily?lang=fr&q=london
api
Out[5]: ''
When executing the file instead of importing it:
runfile('C:/Python27/Lib/site-packages/tornadowm.py', wdir='C:/Python27/Lib/site-packages')
forecast('daily', q='london', lang='fr')
The API link is: http://api.openweathermap.org/data/2.5/forecast/daily?lang=fr&q=london
api
Out[8]: 'http://api.openweathermap.org/data/2.5/forecast/daily?lang=fr&q=london'
Edit: here is the code, if the Git got updated:
from tornado.httpclient import AsyncHTTPClient
import json
import xml.etree.ElementTree as ET
http_client = AsyncHTTPClient()
url = ''
response = ''
args = []
link = 'http://api.openweathermap.org/data/2.5/'
api = ''
result = {}
way = ''
def forecast(way, **kwargs):
global api
if way in ('weather', 'forecast', 'daily', 'find'):
if way == 'daily':
way = 'forecast/daily?'
else:
way += '?'
for i, j in kwargs.iteritems():
args.append('&{0}={1}'.format(i, j))
a = ''.join(set(args))
api = (link + way + a.replace(' ', '+')).replace('?&', '?')
print 'The API link is: ' + api
def handle_request(resp):
global response
if resp.error:
print "Error:", resp.error
else:
response = resp.body
http_client.fetch(api, handle_request)
else:
print "please put a way: 'weather', 'forecast', 'daily', 'find' "
def get_result():
global result
if response.startswith('{'):
print 'the result is JSON, stored in the variable result'
result = json.loads(response)
elif response.startswith('<'):
print 'the result is XML, parse the result variable to work on the nodes,'
print 'or, use response to see the raw result'
result = ET.fromstring(response)
else:
print '''Sorry, no valid response, or you used a parameter that is not compatible with the way!\n please check http://www.openweathermap.com/api for more informations''

It's the side effect of using global.
When you do from tornadowm import * your forecast() function is, we could say metaphorically, "on its own" and is not "hard-linked" to your global space anymore.
Why? Because any effect you make on your global api will "end" with your function, and the definition of api = "" in your global space will take precedence.
Also, as a side note, it's not considered a good practice to use from something import *. You should do from tornadowm import forecast or even better, import tornadown and then use tornadowm.forecast().
OR
Even better, I just noticed your forecast() function doesn't return anything. Which technically makes it not a function anymore, but a procedure (a procedure is like a function but it returns nothing, it just "does" stuff).
Instead of using a global, you should define api in this function and then return api from it. Like this:
def forecast(blablabla):
api = "something"
blablabla
return api
And then
import tornadowm
api = tornadown.forecast(something)
And you're done.

Globals are global only to the module they're defined in. So, normally, you would expect tornadowm.api to be changed when you call forecast, but not api in some other namespace.
The import * is contributing to your understanding of the problem. This imports api (among other names) into the importing namespace. This means that api and tornadowm.api initially point to the same object. But these two names are not linked in any way, and so calling forecast() changes only tornadowm.api and now the two names point to different objects.
To avoid this, don't use import *. It is bad practice anyway and this is just one of the reasons. Instead, import tornadowm and access the variable in the importing module as tornadowm.api.

I'm afraid this is because global is coupled within module, by the time you from tornadowm import * you have imported the api name, but the global api won't take any effects within another module.