Develop Reference

Python test patch is never called

I am trying to test with my code by mocking the PyGithub library.
I want to create a repository for an organization. So first I need to get it and on the "Organization" returned object, I need to make another call.
It fails when trying to assert that my second method was called.
I am very new to python and I am guessing that there is a missing connection between the mocks but I cannot figure out what.
class GithubService:
def __init__(self, token: str) -> None:
self.__github__ = Github(token)
self.__token__ = token
def create_repo_extra(self, repo_name, description, organization_name, team_name):
try:
organization = self.__github__.get_organization(organization_name)
repo = organization.create_repo(name=repo_name,
description=description,
private=True,
has_issues=False,
has_wiki=False,
has_projects=False,
allow_merge_commit=False)
# do other things with the returned repo.....
return True
except GithubException as ex:
print(ex.data)
return False
Here is the test:
import unittest
from unittest.mock import patch, MagicMock, ANY
from github.Organization import Organization
from github.Repository import Repository
from src.github_service import GithubService
class TestGithubService(unittest.TestCase):
#patch('github.Organization.Organization.create_repo',
side_effect=MagicMock(return_value=Repository(ANY, {}, {}, True)))
#patch('github.MainClass.Github.get_organization',
return_value=MagicMock(return_value=Organization(ANY, {}, {}, True)))
def test_create_repo_returns_true(self, get_organization, create_repo):
sut = GithubService("token")
actual = sut.create_repo_extra('repo-name', 'description', 'organization-name', 'team-name')
get_organization.assert_called() # ok
create_repo.assert_called() # failed
self.assertTrue(actual)

Since you mock your Github.get_organization you can use the MagicMock it returns directly rather than trying to mock another layer.
In this, I patch the same Github.get_organization, but avoid giving it a side effect or return value, and therefore pass it as an arg (like you did).
Then I create a convenience mock_organization and it will be the return value of the patched Github.get_organization.
Finally, the patch is checked like you did, and through the convenience mock_organization I check the create_repo method is called as well.
class TestGithubService(unittest.TestCase):
#patch("github.MainClass.Github.get_organization")
def test_create_repo_returns_true(self, mock_get_organization):
mock_organization = MagicMock()
mock_get_organization.return_value = mock_organization
sut = GithubService("token")
actual = sut.create_repo_extra(
"repo-name", "description", "organization-name", "team-name"
)
mock_get_organization.assert_called() # ok
mock_organization.create_repo.assert_called() # ok
self.assertTrue(actual)
Without seeing more of your code I am not sure why patching Organization did not work, but this is simpler, cleaner and just as effective.

Mocking inside a unit test to test missing a line coverage in python

How would I test this function by mocking everything out so that I can assert that place_members is called.
I'm able to get all the other assertions but this one is giving me a lot of trouble with MagicMock().
from .data import INFORMATION
def check(members, status, **kwargs):
placement = kwargs.get("placement", INFORMATION)
place_people(member, status, places=placement)
if kwargs.get("place_people_in_rooms", True):
if not members["already_placed"].get_status(1):
members_placed = place_members(members, status=status)
logger.info(f"Placed members {len(memebers)}")
else:
logger.info("Members already placed")

Python Generic functions with generic parameters

I'm not sure if I used the right terms in the title. This maybe a known way to program interface functions for a subsystem or module but because I don't know the keywords, I'm not finding the results in my search queries.
I want to create a function whose intention can be clearly described in the functions name but the parameters are flexible. I want to write the function to be generic enough so that the function can complete the intention with whatever parameters it receives from whichever caller.
Let's take a function do_foo.
do_foo can take in some_obj whose attributes allows do_foo to do its work. Additionally, do_foo can just take in the individual attributes it cares about like obj_attr0 or obj_attr1 and perform the same work. In both cases, the expected result is the same as well.
So this would look something like this:
Class SomeObj():
def __init__(self, obj_attr0, obj_attr1, obj_attrN):
self.obj_attr0 = None
self.obj_attr1 = None
self.obj_attrN = None # denotes an N number of attributes
def do_foo(params)
# unpack params. do_foo requires obj_attr0 and obj_attr1 and so its searching it in the params iterable
# determine which data is passed in
# execute the work the same way regardless of what form the data is passed in
pass
obj_attr0 = None
obj_attr1 = None
obj_attrN = None
some_obj = SomeObj(obj_attr0, obj_attr1, obj_attrN)
# One can either call with a subset of attributes that would make up SomeObj or SomeObj itself if all the information is there. E.g.:
params = (some_obj)
do_foo(params)
# OR
params = (obj_att0, obj_attr1)
do_foo(params)
I know python offers *args and **kwargs facilities that offer the flexibility above. I'm looking for some examples of where the implementation lends itself to reducing pitfalls. What is a good way to implement the above? And if there are any resources out there what are examples/articles/or terms that describe the above style of programming? Clearly, I'm trying to write my interface functions to be generic and usable in multiple logic paths where the users has its data in different forms where sticking to a specific parameter list is limiting.

Short answer:
You can use function decorators to do this
Long answer:
I have a concrete example for you. It might not be the prettiest code but it does something similar to what you are asking for.
Mini HTTP Testing library
I made a mini HTTP testing library because I make my REST http tests in python, and I realized that I always write the same code again and again. So I made a more general setup
The core
The core is kind of ugly and this is the part I don't want to write again and again.
Just skip this part quick and check how it is used in the interface section.
Then if you like it you can go back and try to understand how it is all tied together.
# base.py
import json, requests, inspect
# This function drops invallid parameters
def request(*args, **kwargs):
allowed = inspect.signature(requests.Session.request).parameters
return {k:v for (k,v) in kwargs.items() if k in allowed}
def response(r, code):
if r.status_code != code:
print(r.text)
return
data = r.json()
if data:
print(json.dumps(data, indent=2, ensure_ascii=False))
return data
# This is the core function it is not pretty but it creates all the abstaction in multiple levels of decorations.
def HTTP(base_url):
def outer(func_one):
def over(*args_one, **kwargs_one):
req, url, code = func_one(*args_one, **kwargs_one)
url = base_url + url
def inner(func_two):
def under(*args_two, **kwargs_two):
allowed = inspect.signature(func_two).parameters
kwparams = {k:v for (k,v) in kwargs_two.items() if k in allowed}
from_inner = func_two(*args_two, **kwparams)
u = url.format(id=kwargs_two.pop('_id')) if '{id}' in url else url
r = req(u, **request(**kwargs_two, **from_inner))
return response(r, code)
return under
return inner
return over
return outer
The interface
The interface functions are all each decorated by the HTTP function which makes them a HTTP caller function, it is still abstract since it will return a function.
Note: interface is just what I call it but it is really just functions which returns functions based on the HTTP decorator
BASE_URL = "https://example.com"
#HTTP(BASE_URL)
def POST(url, code=200): return requests.post, url, code
#HTTP(BASE_URL)
def PUT(url, code=200): return requests.put, url, code
#HTTP(BASE_URL)
def DELETE(url, code=200): return requests.delete, url, code
#HTTP(BASE_URL)
def GET(url, code=200): return requests.get, url, code
A middleware function
When one of the interface functions are decorated with this one then they need a token.
def AUTH(func):
def inner(token, *args, **kwargs):
headers = {'Authorization': f'bearer {token}'}
return func(*args, **kwargs, headers=headers)
return inner
The implementation
The interface can be used for many implementations.
Here I use the interface of POST, PUT, GET and DELETE for the user model.
This is the final decoration, and the functions returned will actually return content instead of other functions.
# users.py
from httplib.base import (
POST,
GET,
DELETE,
PUT,
AUTH,
request
)
#POST('/users',200)
def insert(user):
return request(json=user)
#AUTH
#GET('/users')
def find(_filter={}):
return request(params=_filter)
#AUTH
#GET('/users/{id}')
def find_one(_id):
return request()
#AUTH
#DELETE('/users/{id}')
def delete(_id):
return request()
#AUTH
#PUT('/users/{id}')
def update(_id, updates={}):
return request(json=updates)
Operation
Here you can see how the users delete insert and find functions work.
from httplib import users
def create_and_delete_users(token, n): return [
users.delete(token, _id=x['user']['id'])
for x in [
users.insert(user={
'username' : f'useruser{str(i).zfill(2)}',
'password' : 'secretpassword',
'email' : f'useruser{str(i).zfill(2)}#mail.com',
'gender' : 'male',
}) for i in range(n)]
]
def find_all_and_then_find_each(token): return [
users.find_one(token, _id=x['id'])
for x in users.find(token)['data']
]
I hope this was helpful.

Flask-Rebar Rule Validation

I need to validate a Flask rule in a REST API that is built using Flask-Rebar. I've tried the following method:
#registry.handles(
rule='/horses/<int:horse_id>',
method='GET',
marshal_schema=GetHorseByIdSchema()
)
def getHorseById(horse_id):
return {"horse_id": horse_id}
This is using the <int:my_var> syntax as specified here. When entering an integer value as the horse_id everything works correctly. The issue comes from entering a non-integer value, i.e. a; this throws a 404 Not Found status code when I was expecting a 400 Bad Request.
I don't think I can use any of the marshalling schemas for this so I'm unsure as to what to try next, any help is appreciated.
Thanks,
Adam

This is how Flask/Werkzeug behaves, so its a bit beyond Flask-Rebar's control. That is, the following will also return a 404 for /horses/a:
app = Flask(__name__)
#app.route('/horses/<int:horse_id>')
def getHorseById(horse_id):
return str(horse_id)
With that, here are some workarounds:
(1) Custom URL converter: http://flask.pocoo.org/docs/1.0/api/#flask.Flask.url_map
This would look something like:
import flask
from werkzeug.routing import BaseConverter
class StrictIntegerConverter(BaseConverter):
def to_python(self, value):
try:
return int(value)
except ValueError:
flask.abort(400)
app = flask.Flask(__name__)
app.url_map.converters['strict_integer'] = StrictIntegerConverter
#registry.handles(
rule='/horses/<strict_integer:horse_id>',
method='GET',
marshal_schema=GetHorseByIdSchema()
)
def getHorseById(horse_id):
return {'horse_id': horse_id}
However, routing is done outside of application context, so we can't use flask.jsonify nor Flask-Rebar's errors to raise nice JSON errors.
(2) Check the type inside the handler function
from flask_rebar.errors import BadRequest
#registry.handles(
rule='/horses/<horse_id>',
method='GET',
marshal_schema=GetHorseByIdSchema()
)
def getHorseById(horse_id):
try:
horse_id = int(horse_id)
except ValueError:
raise BadRequest('horse_id must be an integer')
return {'horse_id': horse_id}
This is a little less elegant, but it works.
The Swagger document will default to a string type for the horse_id parameter, but we can work around that as well:
from werkzeug.routing import UnicodeConverter
class StrDocumentedAsIntConverter(UnicodeConverter):
pass
app.url_map.converters['str_documented_as_int'] = StrDocumentedAsIntConverter
registry.swagger_generator.register_flask_converter_to_swagger_type('str_documented_as_int', 'int')
#registry.handles(rule='/horses/<str_documented_as_int:horse_id>')
...
(3) Accept Flask/Werkzeug behavior
Depending on how badly you need 400 instead of 404, it might be most practical to do nothing and just give in to how Flask/Werkzeug do it.

Python - Monkey Patching weird bug

My Fake Mock Class looks like following:
class FakeResponse:
method = None #
url = None # static class variables
def __init__(self, method, url, data):#, response):
self.status_code = 200 # always return 200 OK
FakeResponse.method = method #
FakeResponse.url = url #
#staticmethod
def check(method, url, values):
""" checks method and URL.
"""
print "url fake: ", FakeResponse.url
assert FakeResponse.method == method
assert FakeResponse.url == url
I have another decorator which is applicable over all the test cases:
#pytest.fixture(autouse=True)
def no_requests(monkeypatch):
monkeypatch.setattr('haas.cli.do_put',
lambda url,data: FakeResponse('PUT', url, data))
monkeypatch.setattr("haas.cli.do_post",
lambda url,data: FakeResponse('POST', url, data))
monkeypatch.setattr("haas.cli.do_delete",
lambda url: FakeResponse('DELETE', url, None))
I am using Py.test to test the code.
Some example test cases are:
class Test:
#test case passes
def test_node_connect_network(self):
cli.node_connect_network('node-99','eth0','hammernet')
FakeResponse.check('POST','http://abc:5000/node/node-99/nic/eth0/connect_network',
{'network':'hammernet'})
# test case fails
def test_port_register(self):
cli.port_register('1') # This make a indirect REST call to the original API
FakeResponse.check('PUT','http://abc:5000/port/1', None)
# test case fails
def test_port_delete(self):
cli.port_delete('port', 1)
FakeResponse.check('DELETE','http://abc:5000/port/1', None)
A sample error message which I get:
method = 'PUT', url = 'http://abc:5000/port/1', values = None
#staticmethod
def check(method, url, values):
""" checks method and URL.
'values': if None, verifies no data was sent.
if list of (name,value) pairs, verifies that each pair is in 'values'
"""
print "url fake: ", FakeResponse.url
> assert FakeResponse.method == method
E assert 'POST' == 'PUT'
E - POST
E + PUT
haas/tests/unit/cli_v1.py:54: AssertionError
--------------------------------------------- Captured stdout call -------------------------------------
port_register <port>
Register a <port> on a switch
url fake: http://abc:5000/node/node-99/nic/eth0/connect_network
--------------------------------------------- Captured stderr call -------------------------------------
Wrong number of arguements. Usage:
Whereas if I call the second test case in the following way considering the
check function takes "self" argument and #staticmethod is not used then the test case works:
def test_port_register(self):
cli.port_register('1')
fp = FakeResponse('PUT','http://abc:5000/port/1', None) #Create a FakeResponse class instance
fp.check('PUT','http://abc:5000/port/1', None) # Just call the check function with the same
arguments
Questions:
Are there any side effects of using monkey patching and #staticmethod
How is the url defined for a previous test function being used in the next function call.
Should'nt there be a scoping of argument to disallow the above unwanted behavior.
Is there a better way to monkey patch.
Sorry for the long post, I have been trying to resolve this for a week and wanted some perspective
of other programmers.

The issue was not having the right signature for one of the functions. It was resolved by changing the argument passed to the MonkeyPatch function as en empty dictionary {} instead of 'None' value which is kind of specific to my code.
The reason the I was initially hitting the issue was, as the current function's call(cli.port_register) was failing when the parameters where passed to port_register itself, so it picked up the argument values from a previous state and doing the assert with the FakeResponse call.