Develop Reference

Python test patch is never called

I am trying to test with my code by mocking the PyGithub library.
I want to create a repository for an organization. So first I need to get it and on the "Organization" returned object, I need to make another call.
It fails when trying to assert that my second method was called.
I am very new to python and I am guessing that there is a missing connection between the mocks but I cannot figure out what.
class GithubService:
def __init__(self, token: str) -> None:
self.__github__ = Github(token)
self.__token__ = token
def create_repo_extra(self, repo_name, description, organization_name, team_name):
try:
organization = self.__github__.get_organization(organization_name)
repo = organization.create_repo(name=repo_name,
description=description,
private=True,
has_issues=False,
has_wiki=False,
has_projects=False,
allow_merge_commit=False)
# do other things with the returned repo.....
return True
except GithubException as ex:
print(ex.data)
return False
Here is the test:
import unittest
from unittest.mock import patch, MagicMock, ANY
from github.Organization import Organization
from github.Repository import Repository
from src.github_service import GithubService
class TestGithubService(unittest.TestCase):
#patch('github.Organization.Organization.create_repo',
side_effect=MagicMock(return_value=Repository(ANY, {}, {}, True)))
#patch('github.MainClass.Github.get_organization',
return_value=MagicMock(return_value=Organization(ANY, {}, {}, True)))
def test_create_repo_returns_true(self, get_organization, create_repo):
sut = GithubService("token")
actual = sut.create_repo_extra('repo-name', 'description', 'organization-name', 'team-name')
get_organization.assert_called() # ok
create_repo.assert_called() # failed
self.assertTrue(actual)

Since you mock your Github.get_organization you can use the MagicMock it returns directly rather than trying to mock another layer.
In this, I patch the same Github.get_organization, but avoid giving it a side effect or return value, and therefore pass it as an arg (like you did).
Then I create a convenience mock_organization and it will be the return value of the patched Github.get_organization.
Finally, the patch is checked like you did, and through the convenience mock_organization I check the create_repo method is called as well.
class TestGithubService(unittest.TestCase):
#patch("github.MainClass.Github.get_organization")
def test_create_repo_returns_true(self, mock_get_organization):
mock_organization = MagicMock()
mock_get_organization.return_value = mock_organization
sut = GithubService("token")
actual = sut.create_repo_extra(
"repo-name", "description", "organization-name", "team-name"
)
mock_get_organization.assert_called() # ok
mock_organization.create_repo.assert_called() # ok
self.assertTrue(actual)
Without seeing more of your code I am not sure why patching Organization did not work, but this is simpler, cleaner and just as effective.

Python Generic functions with generic parameters

I'm not sure if I used the right terms in the title. This maybe a known way to program interface functions for a subsystem or module but because I don't know the keywords, I'm not finding the results in my search queries.
I want to create a function whose intention can be clearly described in the functions name but the parameters are flexible. I want to write the function to be generic enough so that the function can complete the intention with whatever parameters it receives from whichever caller.
Let's take a function do_foo.
do_foo can take in some_obj whose attributes allows do_foo to do its work. Additionally, do_foo can just take in the individual attributes it cares about like obj_attr0 or obj_attr1 and perform the same work. In both cases, the expected result is the same as well.
So this would look something like this:
Class SomeObj():
def __init__(self, obj_attr0, obj_attr1, obj_attrN):
self.obj_attr0 = None
self.obj_attr1 = None
self.obj_attrN = None # denotes an N number of attributes
def do_foo(params)
# unpack params. do_foo requires obj_attr0 and obj_attr1 and so its searching it in the params iterable
# determine which data is passed in
# execute the work the same way regardless of what form the data is passed in
pass
obj_attr0 = None
obj_attr1 = None
obj_attrN = None
some_obj = SomeObj(obj_attr0, obj_attr1, obj_attrN)
# One can either call with a subset of attributes that would make up SomeObj or SomeObj itself if all the information is there. E.g.:
params = (some_obj)
do_foo(params)
# OR
params = (obj_att0, obj_attr1)
do_foo(params)
I know python offers *args and **kwargs facilities that offer the flexibility above. I'm looking for some examples of where the implementation lends itself to reducing pitfalls. What is a good way to implement the above? And if there are any resources out there what are examples/articles/or terms that describe the above style of programming? Clearly, I'm trying to write my interface functions to be generic and usable in multiple logic paths where the users has its data in different forms where sticking to a specific parameter list is limiting.

Short answer:
You can use function decorators to do this
Long answer:
I have a concrete example for you. It might not be the prettiest code but it does something similar to what you are asking for.
Mini HTTP Testing library
I made a mini HTTP testing library because I make my REST http tests in python, and I realized that I always write the same code again and again. So I made a more general setup
The core
The core is kind of ugly and this is the part I don't want to write again and again.
Just skip this part quick and check how it is used in the interface section.
Then if you like it you can go back and try to understand how it is all tied together.
# base.py
import json, requests, inspect
# This function drops invallid parameters
def request(*args, **kwargs):
allowed = inspect.signature(requests.Session.request).parameters
return {k:v for (k,v) in kwargs.items() if k in allowed}
def response(r, code):
if r.status_code != code:
print(r.text)
return
data = r.json()
if data:
print(json.dumps(data, indent=2, ensure_ascii=False))
return data
# This is the core function it is not pretty but it creates all the abstaction in multiple levels of decorations.
def HTTP(base_url):
def outer(func_one):
def over(*args_one, **kwargs_one):
req, url, code = func_one(*args_one, **kwargs_one)
url = base_url + url
def inner(func_two):
def under(*args_two, **kwargs_two):
allowed = inspect.signature(func_two).parameters
kwparams = {k:v for (k,v) in kwargs_two.items() if k in allowed}
from_inner = func_two(*args_two, **kwparams)
u = url.format(id=kwargs_two.pop('_id')) if '{id}' in url else url
r = req(u, **request(**kwargs_two, **from_inner))
return response(r, code)
return under
return inner
return over
return outer
The interface
The interface functions are all each decorated by the HTTP function which makes them a HTTP caller function, it is still abstract since it will return a function.
Note: interface is just what I call it but it is really just functions which returns functions based on the HTTP decorator
BASE_URL = "https://example.com"
#HTTP(BASE_URL)
def POST(url, code=200): return requests.post, url, code
#HTTP(BASE_URL)
def PUT(url, code=200): return requests.put, url, code
#HTTP(BASE_URL)
def DELETE(url, code=200): return requests.delete, url, code
#HTTP(BASE_URL)
def GET(url, code=200): return requests.get, url, code
A middleware function
When one of the interface functions are decorated with this one then they need a token.
def AUTH(func):
def inner(token, *args, **kwargs):
headers = {'Authorization': f'bearer {token}'}
return func(*args, **kwargs, headers=headers)
return inner
The implementation
The interface can be used for many implementations.
Here I use the interface of POST, PUT, GET and DELETE for the user model.
This is the final decoration, and the functions returned will actually return content instead of other functions.
# users.py
from httplib.base import (
POST,
GET,
DELETE,
PUT,
AUTH,
request
)
#POST('/users',200)
def insert(user):
return request(json=user)
#AUTH
#GET('/users')
def find(_filter={}):
return request(params=_filter)
#AUTH
#GET('/users/{id}')
def find_one(_id):
return request()
#AUTH
#DELETE('/users/{id}')
def delete(_id):
return request()
#AUTH
#PUT('/users/{id}')
def update(_id, updates={}):
return request(json=updates)
Operation
Here you can see how the users delete insert and find functions work.
from httplib import users
def create_and_delete_users(token, n): return [
users.delete(token, _id=x['user']['id'])
for x in [
users.insert(user={
'username' : f'useruser{str(i).zfill(2)}',
'password' : 'secretpassword',
'email' : f'useruser{str(i).zfill(2)}#mail.com',
'gender' : 'male',
}) for i in range(n)]
]
def find_all_and_then_find_each(token): return [
users.find_one(token, _id=x['id'])
for x in users.find(token)['data']
]
I hope this was helpful.

Mocking urllib.request.urlopen's read function returns MagicMock signature

I'm trying to mock out urllib.request.urlopen's Read method on Python 3:
Function Code:
try:
with request.urlopen(_webhook_url, json.dumps(_message).encode('utf-8')) as _response:
_response_body = _response.read()
return _response_body
Test Code:
with mock.patch('urllib.request.urlopen') as mock_urlopen:
response_mock = MagicMock()
response_mock.read.return_value = 'ok'
mock_urlopen.return_value = response_mock
with self.stubber:
_response = NotifySlack.lambda_handler(_event)
self.assertEqual('ok', _response)
If I call response_mock.read() I get the 'ok' value returned, however when I assert the return value I get a mock signature:
Expected :ok
Actual :<MagicMock name='urlopen().__enter__().read()' id='2148156925992'>
Any ideas on why the mock isn't returning the value assigned to read()?

To follow #jonrsharpe's comment and the Python: Mocking a context manager thread, to properly mock the context manager in this case, you would need this interestingly looking line:
mock_urlopen.return_value.__enter__.return_value.read.return_value = 'ok'
#^^^^^^context manager to return response^^^^^^^|^^^read method^^^

mock xmlrpc.client method python

I am in the process of learning unit testing, however I am struggling to understand how to mock functions for unit testing. I have reviewed many how-to's and examples but the concept is not transferring enough for me to use it on my code. I am hoping getting this to work on a actual code example I have will help.
In this case I am trying to mock isTokenValid.
Here is example code of what I want to mock.
<in library file>
import xmlrpc.client as xmlrpclib
class Library(object):
def function:
#...
AuthURL = 'https://example.com/xmlrpc/Auth'
auth_server = xmlrpclib.ServerProxy(AuthURL)
socket.setdefaulttimeout(20)
try:
if pull == 0:
valid = auth_server.isTokenValid(token)
#...
in my unit test file I have
import library
class Tester(unittest.TestCase):
#patch('library.xmlrpclib.ServerProxy')
def test_xmlrpclib(self, fake_xmlrpclib):
assert 'something'
How would I mock the code listed in 'function'? Token can be any number as a string and valid would be a int(1)

First of all, you can and should mock xmlrpc.client.ServerProxy; your library imports xmlrpc.client as a new name, but it is still the same module object so both xmlrpclib.ServerProxy in your library and xmlrpc.client.ServerProxy lead to the same object.
Next, look at how the object is used, and look for calls, the (..) syntax. Your library uses the server proxy like this:
# a call to create an instance
auth_server = xmlrpclib.ServerProxy(AuthURL)
# on the instance, a call to another method
valid = auth_server.isTokenValid(token)
So there is a chain here, where the mock is called, and the return value is then used to find another attribute that is also called. When mocking, you need to look for that same chain; use the Mock.return_value attribute for this. By default a new mock instance is returned when you call a mock, but you can also set test values.
So to test your code, you'd want to influence what auth_server.isTokenValid(token) returns, and test if your code works correctly. You may also want to assert that the correct URL is passed to the ServerProxy instance.
Create separate tests for different outcomes. Perhaps the token is valid in one case, not valid in another, and you'd want to test both cases:
class Tester(unittest.TestCase):
#patch('xmlrpc.client.ServerProxy')
def test_valid_token(self, mock_serverproxy):
# the ServerProxy(AuthURL) return value
mock_auth_server = mock_serverproxy.return_value
# configure a response for a valid token
mock_auth_server.isTokenValid.return_value = 1
# now run your library code
return_value = library.Library().function()
# and make test assertions
# about the server proxy
mock_serverproxy.assert_called_with('some_url')
# and about the auth_server.isTokenValid call
mock_auth_server.isTokenValid.assert_called_once()
# and if the result of the function is expected
self.assertEqual(return_value, 'expected return value')
#patch('xmlrpc.client.ServerProxy')
def test_invalid_token(self, mock_serverproxy):
# the ServerProxy(AuthURL) return value
mock_auth_server = mock_serverproxy.return_value
# configure a response; now testing for an invalid token instead
mock_auth_server.isTokenValid.return_value = 0
# now run your library code
return_value = library.Library().function()
# and make test assertions
# about the server proxy
mock_serverproxy.assert_called_with('some_url')
# and about the auth_server.isTokenValid call
mock_auth_server.isTokenValid.assert_called_once()
# and if the result of the function is expected
self.assertEqual(return_value, 'expected return value')

There are many mock attributes to use, and you can change your patch decorator usage a little as follows:
class Tester(unittest.TestCase):
def test_xmlrpclib(self):
with patch('library.xmlrpclib.ServerProxy.isTokenValid') as isTokenValid:
self.assertEqual(isTokenValid.call_count, 0)
# your test code calling xmlrpclib
self.assertEqual(isTokenValid.call_count, 1)
token = isTokenValid.call_args[0] # assume this token is valid
self.assertEqual(isTokenValid.return_value, 1)
You can adjust the code above to satisfy your requirements.

Python Tornado get URL arguments

I'm trying to inspect a request's argument before the get() is invoked. I have a route which is described as so:
user_route = r"/users/key=(?P<key>\w+)"
app = web.Application([
web.URLSpec(user_route, user_manager.UserHandler), ..])
Next, (in the handler) prepare() is used to inspect the request before get().
def prepare(self):
# inspect request arguments
print(self.request.arguments) # prints "{}"
The problem I'm having is that I cannot access the arguments from prepare(). The last statement prints an empty dict. My get() successfully uses the arguments as they are passed in the function like this:
def get(self, key):
print(key) #works
How do I access arguments in prepare()? I have also tried self.argument('key') which gives an error "400 GET .... Missing argument key", but requested URL does have a key argument in it.

In your code key is not a GET-argument, it's a part of a path. tornado.we.URLSpec passes any capturing groups in the regex into the handler’s get/post/etc methods as arguments.
tornado.web.RequestHandler has RequestHandler.path_args and RequestHandler.path_kwargs which contain the positional and keyword arguments from URLSpec. Those are available in prepare method:
def prepare(self):
# inspect request arguments
print(self.path_kwargs) # prints {"key": "something"}

As Gennady Kandaurov mentioned, you passed the key as a part of the we.URLSpec path and you can access it using Tornado's self.path_kwargs. If you wanted to pass it as an argument you could used RequestHandler.get_argument to get the argument on your get method and use self.request.arguments on your prepare method to access it as your initial intention.
Your code could be as follow:
class Application(tornado.web.Application):
def __init__(self):
user_route = r"/users"
app = tornado.web.Application([
tornado.web.url(user_route, user_manager.UserHandler), ..])
class UserHandler(tornado.web.RequestHandler):
def get(self):
key = self.get_argument('key')
print(key)
def prepare(self):
# inspect request arguments
print(self.request.arguments)
Please let me know if you have any further question.

It's generally bad to use a character like = in a URL path fragment, since they are generally used for query arguments. Either don't use it:
`r"/users/(?P<key>\w+)"`
or turn it into a proper query argument
`r"/users/\?key=(?P<key>\w+)"`
Otherwise it's confusing for a maintainer to try to figure out which scheme you intended to use (did you really want to route a path fragment called /key%3D\w+? Or did you really mean you wanted a query arg and forgot the ??)
In any case, for URL path fragment matching ("slug-matching"), using argument unpacking can let you access them in the handler too, without having to invoke path_kwargs:
# `r"/users/(?P<key>\w+)"`
class Handler(RequestHandler):
def get(self, **kwargs):
key = kwargs.get('key')
# I prefer dict.get() here, since if you change the `+` to a `*`,
# it's possible that no key was supplied, and kwargs['key']
# will throw a KeyError exception
If you intended to use a query argument for key, then #afxentios's answer is appropriate. (You can also use self.get_query_argument('key') which will explicitly only look for query arguments in the URL (whereas get_argument also checks in the request BODY for a www-url-encoded argument (such as if you POST)).