Develop Reference

mock on function with optional arguments

I want to write a test for "A" function which calls another "B" function in it. B function looks like this:
def Bfuncion(self, city: Optional[str], name: Optional[str]):
in A function I use B function two times, first time I use it in this way:
cities = self.Bfunction(name=None, city="ny", page=page)
and the second time in this way:
cities = self.Bfunction(name="Tom", city=None, page=page)
Now I want to write a test of A function, I would try this if there was only one use of B function:
mocker.patch(
"Path.To.My.Function",
return_value=[
{"someReturnValue"},
{"someReturnValue"},
{"someReturnValue"},
],
How do I write call mocker(), as I use B function with different arguments in each call.

Use the side_effect keyword argument, not the return_value keyword argument.
>>> from unittest.mock import Mock
>>> m = Mock(return_value=[1,2,3])
>>> m()
[1, 2, 3]
>>> m = Mock(side_effect=[1,2,3])
>>> m()
1
>>> m()
2
>>> m()
3
(patch, of course, simply passes keyword arguments it does not itself recognize to Mock to configure the object that is used by the patched name.)
This requires you to know ahead of time the order of calls that will be made. If you need more flexibility, patch the name with a custom function that behaves the way you want, instead of a Mock object.
def my_func(self, name, city, page):
...
mocker.patch(
"Path.To.My.Function",
new=my_func
)

I try to suggest you a solution for your test; I don't know if it is suited for your need because I have tried to guess some details that you have omitted in your question.
The production code
I suppose that your production code is contained in a file called functions.py and the content of functions.py is the following:
from typing import Optional
class MyClass:
def Bfunction(self, city: Optional[str], name: Optional[str]):
return city
def Afunction(self):
cities = self.Bfunction(name=None, city="ny", page='page')
print(cities)
cities = self.Bfunction(name="Tom", city=None, page='page')
print(cities)
Note that in functions.py:
I have defined a class because I have seen that your definition of the Bfunction contains the self argument.
The Afunction executes the two calls of Bfunction that you have write in your question.
The test file
The test file (which doesn't use pytest but only the module unittest) is the following:
import unittest
from unittest import mock
import functions
class MyTestCase(unittest.TestCase):
def test_something(self):
sut = functions.MyClass()
with mock.patch.object(sut, 'Bfunction') as mock_b:
# set values returned by Bfunction in the first and in the second call
mock_b.side_effect = ['cityA', 'cityB']
sut.Afunction()
# here I verify that Bfunction is called 2 times
self.assertEqual(2, mock_b.call_count)
mock_b.assert_called_with(city=None, name='Tom', page='page')
mock_b.assert_called_with(name="Tom", city=None, page='page')
if __name__ == '__main__':
unittest.main()
The output of test file execution
The output of the test file execution is:
cityA
cityB
.
----------------------------------------------------------------------
Ran 1 test in 0.002s
OK
The output showes:
Afunction calls exactly two times the Bfunction
the test verify the arguments passed to Bfunction
the presence of cityA and cityB in the output show you how to select different values returned by Bfunction (you have to use side_effect and not return_value).
I hope that this answer is useful for you.

Python: is it possible to compare function import origin?

I have 4 python files, the first two is the function itself, the second is functions dictionary, and the third is kind of a 'definition' parser
function_1
def increment(obj):
return obj+1
#another function
function_2
def decrement(obj):
return obj-1
#another function
function_dictionary
import fucntion1
import function2
FUNC_DICT = {
'increment': function1.increment,
'decrement': function2.decrement,
#another function
}
definition_parser
from function_dictionary import FUNC_DICT
def get_definition():
result = {}
for key, value in FUNC_DICT.items():
#check if value is from function_1 or function_2
#result[key] = 'function_1' or 'function_2', depends on its origin
return result
is it possible to compare function import? I tried it with is_in_function_1 = value is in function_1, doesn't work.
if it is not, what are the way around without much repetition?

You can get the module of functions via the __module__ property.
from function_dictionary import FUNC_DICT
def get_definition():
result = {}
for key, value in FUNC_DICT.items():
result[key] = value.__module__
return result
The output would look like the following:
{
'increment': 'function_1',
'decrement': 'function_2'
}

You could use the inspect module like so:
import inspect
print(inspect.getmodule(SequenceMatcher))
for example, if I inspect SequenceMatcher, the output is:
<module 'difflib' from 'C:\ProgramData\Anaconda2\lib\difflib.py'>
So to compare the origin of two functions, you could simply do this:
if inspect.getmodule(increment) == inspect.getmodule(decrement):
do stuff

Decorator to define function-local statics - fine details of AST-munging

I am trying to produce a better answer to the frequently-asked question "How do I do function-local static variables in Python?" (1, 2, 3, ...) "Better" means completely encapsulated in a decorator, that can be used in any context where a function definition may appear. In particular, it must DTRT when applied to methods and nested functions; it must play nice with other decorators applied to the same function (in any order); it must accept arbitrary initializers for the static variables, and it must not modify the formal parameter list of the decorated function. Basically, if this were to be proposed for inclusion in the standard library, nobody should be able to object on quality-of-implementation grounds.
Ideal surface syntax would be
#static_vars(a=0, b=[])
def test():
b.append(a)
a += 1
sys.stdout.write(repr(b) + "\n")
I would also accept
#static_vars(a=0, b=[])
def test():
static.b.append(static.a)
static.a += 1
sys.stdout.write(repr(static.b) + "\n")
or similar, as long as the namespace for the static variables is not the name of the function! (I intend to use this in functions that may have very long names.)
A slightly more motivated example involves precompiled regular expressions that are only relevant to one function:
#static_vars(encode_re = re.compile(
br'[\x00-\x20\x7F-\xFF]|'
br'%(?!(?:[0-9A-Fa-f]{2}|u[0-9A-Fa-f]{4}))')
def encode_nonascii_and_percents(segment):
segment = segment.encode("utf-8", "surrogateescape")
return encode_re.sub(
lambda m: "%{:02X}".format(ord(m.group(0))).encode("ascii"),
segment).decode("ascii")
Now, I already have a mostly-working implementation. The decorator rewrites each function definition as if it had read like so (using the first example):
def _wrap_test_():
a = 0
b = 1
def test():
nonlocal a, b
b.append(a)
a += 1
sys.stdout.write(repr(b) + "\n")
test = _wrap_test_()
del _wrap_test_
It seems that the only way to accomplish this is to munge the AST. I have code that works for simple cases (see below) but I strongly suspect it is wrong in more complicated cases. For instance, I think it will break if applied to a method definition, and of course it also breaks in any situation where inspect.getsource() fails.
So the question is, first, what should I do to make it work in more cases, and second, is there a better way to define a decorator with the same black-box effects?
Note 1: I only care about Python 3.
Note 2: Please assume that I have read all of the proposed solutions in all of the linked questions and found all of them inadequate.
#! /usr/bin/python3
import ast
import functools
import inspect
import textwrap
def function_skeleton(name, args):
"""Return the AST of a function definition for a function named NAME,
which takes keyword-only args ARGS, and does nothing. Its
.body field is guaranteed to be an empty array.
"""
fn = ast.parse("def foo(*, {}): pass".format(",".join(args)))
# The return value of ast.parse, as used here, is a Module object.
# We want the function definition that should be the Module's
# sole descendant.
assert isinstance(fn, ast.Module)
assert len(fn.body) == 1
assert isinstance(fn.body[0], ast.FunctionDef)
fn = fn.body[0]
# Remove the 'pass' statement.
assert len(fn.body) == 1
assert isinstance(fn.body[0], ast.Pass)
fn.body.clear()
fn.name = name
return fn
class static_vars:
"""Decorator which provides functions with static variables.
Usage:
#static_vars(foo=1, bar=2, ...)
def fun():
foo += 1
return foo + bar
The variables are implemented as upvalues defined by a wrapper
function.
Uses introspection to recompile the decorated function with its
context changed, and therefore may not work in all cases.
"""
def __init__(self, **variables):
self._variables = variables
def __call__(self, func):
if func.__name__ in self._variables:
raise ValueError(
"function name {} may not be the same as a "
"static variable name".format(func.__name__))
fname = inspect.getsourcefile(func)
lines, first_lineno = inspect.getsourcelines(func)
mod = ast.parse(textwrap.dedent("".join(lines)), filename=fname)
# The return value of ast.parse, as used here, is a Module
# object. Save that Module for use later and extract the
# function definition that should be its sole descendant.
assert isinstance(mod, ast.Module)
assert len(mod.body) == 1
assert isinstance(mod.body[0], ast.FunctionDef)
inner_fn = mod.body[0]
mod.body.clear()
# Don't apply decorators twice.
inner_fn.decorator_list.clear()
# Fix up line numbers. (Why the hell doesn't ast.parse take a
# starting-line-number argument?)
ast.increment_lineno(inner_fn, first_lineno - inner_fn.lineno)
# Inject a 'nonlocal' statement declaring the static variables.
svars = sorted(self._variables.keys())
inner_fn.body.insert(0, ast.Nonlocal(svars))
# Synthesize the wrapper function, which will take the static
# variableas as arguments.
outer_fn_name = ("_static_vars_wrapper_" +
inner_fn.name + "_" +
hex(id(self))[2:])
outer_fn = function_skeleton(outer_fn_name, svars)
outer_fn.body.append(inner_fn)
outer_fn.body.append(
ast.Return(value=ast.Name(id=inner_fn.name, ctx=ast.Load())))
mod.body.append(outer_fn)
ast.fix_missing_locations(mod)
# The new function definition must be evaluated in the same context
# as the original one. FIXME: supply locals if appropriate.
context = func.__globals__
exec(compile(mod, filename="<static-vars>", mode="exec"),
context)
# extract the function we just defined
outer_fn = context[outer_fn_name]
del context[outer_fn_name]
# and call it, supplying the static vars' initial values; this
# returns the adjusted inner function
adjusted_fn = outer_fn(**self._variables)
functools.update_wrapper(adjusted_fn, func)
return adjusted_fn
if __name__ == "__main__":
import sys
#static_vars(a=0, b=[])
def test():
b.append(a)
a += 1
sys.stdout.write(repr(b) + "\n")
test()
test()
test()
test()

Isn't this what classes are for?
import sys
class test_class:
a=0
b=[]
def test(self):
test_class.b.append(test_class.a)
test_class.a += 1
sys.stdout.write(repr(test_class.b) + "\n")
t = test_class()
t.test()
t.test()
[0]
[0, 1]
Here is a version of your regexp encoder:
import re
class encode:
encode_re = re.compile(
br'[\x00-\x20\x7F-\xFF]|'
br'%(?!(?:[0-9A-Fa-f]{2}|u[0-9A-Fa-f]{4}))')
def encode_nonascii_and_percents(self, segment):
segment = segment.encode("utf-8", "surrogateescape")
return encode.encode_re.sub(
lambda m: "%{:02X}".format(ord(m.group(0))).encode("ascii"),
segment).decode("ascii")
e = encode()
print(e.encode_nonascii_and_percents('foo bar'))
foo%20bar
There is always the singleton class.
Is there a simple, elegant way to define Singletons in Python?

Does an equivalent of override exist for nested functions?

If I have this function, what should I do to replace the inner function with my own custom version?
def foo():
def bar():
# I want to change this
pass
# here starts a long list of functions I want to keep unchanged
def baz():
pass
Using classes this would be easily done overriding the method. Though, I can't figure out how to do that with nested functions. Changing foo to be a class (or anything else) is not an option because it comes from a given imported module I can't modify.

Here's one way of doing it, creating a new foo that "does the right thing" by hacking the function internals. ( As mentioned by #DSM ). Unfortunately we cant just jump into the foo function and mess with its internals, as they're mostly marked read only, so what we have to do is modify a copy we construct by hand.
# Here's the original function
def foo():
def bar():
print(" In bar orig")
def baz():
print(" Calling bar from baz")
bar()
print("Foo calling bar:")
bar()
print("Foo calling baz:")
baz()
# Here's using it
foo()
# Now lets override the bar function
import types
# This is our replacement function
def my_bar():
print(" Woo hoo I'm the bar override")
# This creates a new code object used by our new foo function
# based on the old foo functions code object.
foocode = types.CodeType(
foo.func_code.co_argcount,
foo.func_code.co_nlocals,
foo.func_code.co_stacksize,
foo.func_code.co_flags,
foo.func_code.co_code,
# This tuple is a new version of foo.func_code.co_consts
# NOTE: Don't get this wrong or you will crash python.
(
foo.func_code.co_consts[0],
my_bar.func_code,
foo.func_code.co_consts[2],
foo.func_code.co_consts[3],
foo.func_code.co_consts[4]
),
foo.func_code.co_names,
foo.func_code.co_varnames,
foo.func_code.co_filename,
foo.func_code.co_name,
foo.func_code.co_firstlineno,
foo.func_code.co_lnotab,
foo.func_code.co_freevars,
foo.func_code.co_cellvars )
# This is the new function we're replacing foo with
# using our new code.
foo = types.FunctionType( foocode , {})
# Now use it
foo()
I'm pretty sure its not going to catch all cases. But it works for the example (for me on an old python 2.5.1 )
Ugly bits that could do with some tidy up are:
The huge argument list being passed to CodeType
The ugly tuple constructed from co_consts overriding only one member. All the info is in co_consts to determine which to replace - so a smarter function could do this. I dug into the internals by hand using print( foo.func_code.co_consts ).
You can find some information about the CodeType and FunctionType by using the interpreter
command help( types.CodeType ).
UPDATE:
I thought this was too ugly so I built a helper function to make it prettier. With the helper you can write:
# Use our function to get a new version of foo with "bar" replaced by mybar
foo = monkey_patch_fn( foo, "bar", my_bar )
# Check it works
foo()
Here's the implementation of monkey_patch_fn:
# Returns a copy of original_fn with its internal function
# called name replaced with new_fn.
def monkey_patch_fn( original_fn, name, new_fn ):
#Little helper function to pick out the correct constant
def fix_consts(x):
if x==None: return None
try:
if x.co_name == name:
return new_fn.func_code
except AttributeError, e:
pass
return x
original_code = original_fn.func_code
new_consts = tuple( map( fix_consts, original_code.co_consts ) )
code_type_args = [
"co_argcount", "co_nlocals", "co_stacksize", "co_flags", "co_code",
"co_consts", "co_names", "co_varnames", "co_filename", "co_name",
"co_firstlineno", "co_lnotab", "co_freevars", "co_cellvars" ]
new_code = types.CodeType(
*[ ( getattr(original_code,x) if x!="co_consts" else new_consts )
for x in code_type_args ] )
return types.FunctionType( new_code, {} )

You can pass it in as an optional parameter
def foo(bar=None):
def _bar():
# I want to change this
pass
if bar is None:
bar = _bar

Go through a number of functions in Python

I have an unknown number of functions in my python script (well, it is known, but not constant) that start with site_...
I was wondering if there's a way to go through all of these functions in some main function that calls for them.
something like:
foreach function_that_has_site_ as coolfunc
if coolfunc(blabla,yada) == true:
return coolfunc(blabla,yada)
so it would go through them all until it gets something that's true.
thanks!

The inspect module, already mentioned in other answers, is especially handy because you get to easily filter the names and values of objects you care about. inspect.getmembers takes two arguments: the object whose members you're exploring, and a predicate (a function returning bool) which will accept (return True for) only the objects you care about.
To get "the object that is this module" you need the following well-known idiom:
import sys
this_module = sys.modules[__name__]
In your predicate, you want to select only objects which are functions and have names that start with site_:
import inspect
def function_that_has_site(f):
return inspect.isfunction(f) and f.__name__.startswith('site_')
With these two items in hand, your loop becomes:
for n, coolfunc in inspect.getmembers(this_module, function_that_has_site):
result = coolfunc(blabla, yada)
if result: return result
I have also split the loop body so that each function is called only once (which both saves time and is a safer approach, avoiding possible side effects)... as well as rewording it in Python;-)

Have you tried using the inspect module?
http://docs.python.org/library/inspect.html
The following will return the methods:
inspect.getmembers
Then you could invoke with:
methodobjToInvoke = getattr(classObj, methodName)
methodobj("arguments")

This method goes through all properties of the current module and executes all functions it finds with a name starting with site_:
import sys
import types
for elm in dir():
f = getattr(sys.modules[__name__], elm)
if isinstance(f, types.FunctionType) and f.__name__[:5] == "site_":
f()
The function-type check is unnecessary if only functions are have names starting with site_.

def run():
for f_name, f in globals().iteritems():
if not f_name.startswith('site_'):
continue
x = f()
if x:
return x

It's best to use a decorator to enumerate the functions you care about:
_funcs = []
def enumfunc(func):
_funcs.append(func)
return func
#enumfunc
def a():
print 'foo'
#enumfunc
def b():
print 'bar'
#enumfunc
def c():
print 'baz'
if __name__ == '__main__':
for f in _funcs:
f()

Try dir(), globals() or locals(). Or inspect module (as mentioned above).
def site_foo():
pass
def site_bar():
pass
for name, f in globals().items():
if name.startswith("site_"):
print name, f()