Develop Reference

Is it possible to hook C module function in python?

I have this code where "clr" and "Assembly" are parts of C library in Linux:
import clr
from System.Reflection import Assembly
import functools
def prefix_function(function, prefunction):
#functools.wraps(function)
def run(*args, **kwargs):
prefunction(*args, **kwargs)
return function(*args, **kwargs)
return run
def this_is_a_function():
print("Hook function called instead of the original!")
# This gives me error "attribute is read-only"
Assembly.GetEntryAssembly = prefix_function(
Assembly.GetEntryAssembly, this_is_a_function)
# Should print:
# "Hook function called instead of the original!"
Assembly.GetEntryAssembly()
Is it possible to somehow hook calling to static function "GetEntryAssembly()" as well as calls to any functions in C libraries ? I need to modify logic and return result of the original "GetEntryAssembly()". So, in the example above I wish to print "Hook function called" when call to GetEntryAssembly() is made.
I can't subclass, but could accept overwrite kind of "pointers" like in C code.
Also I tried this answer, but read-only attribute error occurs.

Obtained source code of python function is wrong

I would like to obtain the source code of a Python function. However, the functions I am dealing with have decorators that are instantiated with functions themselves. It seems like inspect or dill have problems with that. In particular, I tried the following MWE:
class Decorator(object):
def __init__(self, item):
self.item = item
self.func = None
def __call__(self, func):
self.func = func
return self
#Decorator(42)
def fun_1():
pass
#Decorator(lambda x: 42)
def fun_2():
pass
#Decorator(fun_2.item)
def fun_3():
pass
import inspect
print("Inspect results")
print(inspect.getsource(fun_1.func))
print(inspect.getsource(fun_2.func))
print(inspect.getsource(fun_3.func))
import dill
print("Dill results")
print(dill.source.getsource(fun_1.func))
print(dill.source.getsource(fun_2.func))
print(dill.source.getsource(fun_3.func))
For functions fun_1 and fun_3, this gives the expected results. For fun_2 (although it is essentially the same as fun_3), the source code as returned by inspect or dill is just wrong - it only gives me the decorator line. I would expect to get
#Decorator(lambda x: 42)
def fun_2():
pass
but all I get is
#Decorator(lambda x: 42)
I can imagine that both modules just search for the first function declaration. Is this assumption correct and is there a good workaround besides reimplementing inspect or dill?
Edit: I am using Python 2.7, but experienced the same behavior using Python 3.4.

I'm the dill author. I think you've found a bug in inspect, with regard to parsing code blocks containing a lambda as an argument in a decorator.
>>> f = lambda x: 42
>>> #Decorator(f)
... def fun_4():
... pass
...
>>>
>>> print dill.source.getsource(fun_4.func)
#Decorator(f)
def fun_4():
pass
>>>
Note that the above works, when you simply use a pointer to the lambda object. So that's your workaround.
I believe this corner case is a bug in inspect, which dill is using under the covers... so it's a bug in dill also.
I've opened a ticket on the dill GitHub page, and I'll investigate it further to see (1) if there's an easy workaround that can be implemented in dill, and (2) if indeed it's a bug in inspect (I'm guessing it is) -- and if so, it should be populated up to the python developers. Here's the link to the ticket: https://github.com/uqfoundation/dill/issues/221

Override a "private" method in a python module

I want to test a function in python, but it relies on a module-level "private" function, that I don't want called, but I'm having trouble overriding/mocking it. Scenario:
module.py
_cmd(command, args):
# do something nasty
function_to_be_tested():
# do cool things
_cmd('rm', '-rf /')
return 1
test_module.py
import module
test_function():
assert module.function_to_be_tested() == 1
Ideally, in this test I dont want to call _cmd. I've looked at some other threads, and I've tried the following with no luck:
test_function():
def _cmd(command, args):
# do nothing
pass
module._cmd = _cmd
although checking module._cmd against _cmd doesn't give the correct reference. Using mock:
from mock import patch
def _cmd_mock(command, args):
# do nothing
pass
#patch('module._cmd', _cmd_mock)
test_function():
...
gives the correct reference when checking module._cmd, although `function_to_be_tested' still uses the original _cmd (as evidenced by it doing nasty things).
This is tricky because _cmd is a module-level function, and I dont want to move it into a module

[Disclaimer]
The synthetic example posted in this question works and the described issue become from specific implementation in production code. Maybe this question should be closed as off topic because the issue is not reproducible.
[Note] For impatient people Solution is at the end of the answer.
Anyway that question given to me a good point to thought: how we can patch a method reference when we cannot access to the variable where the reference is?
Lot of times I found some issue like this. There are lot of ways to meet that case and the commons are
Decorators: the instance we would like replace is passed as decorator argument or used in decorator static implementation
What we would like to patch is a default argument of a method
In both cases maybe refactor the code is the best way to play with that but what about if we are playing with some legacy code or the decorator is a third part decorator?
Ok, we have the back on the wall but we are using python and in python nothing is impossible. What we need is just the reference of the function/method to patch and instead of patching its reference we can patch the __code__: yes I'm speaking about patching the bytecode instead the function.
Get a real example. I'm using default parameter case that is simple, but it works either in decorator case.
def cmd(a):
print("ORIG {}".format(a))
def cmd_fake(a):
print("NEW {}".format(a))
def do_work(a, c=cmd):
c(a)
do_work("a")
cmd=cmd_fake
do_work("b")
Output:
ORIG a
ORIG b
Ok In this case we can test do_work by passing cmd_fake but there some cases where is impossible do it: for instance what about if we need to call something like that:
def what_the_hell():
list(map(lambda a:do_work(a), ["c","d"]))
what we can do is patch cmd.__code__ instead of _cmd by
cmd.__code__ = cmd_fake.__code__
So follow code
do_work("a")
what_the_hell()
cmd.__code__ = cmd_fake.__code__
do_work("b")
what_the_hell()
Give follow output:
ORIG a
ORIG c
ORIG d
NEW b
NEW c
NEW d
Moreover if we want to use a mock we can do it by add follow lines:
from unittest.mock import Mock, call
cmd_mock = Mock()
def cmd_mocker(a):
cmd_mock(a)
cmd.__code__=cmd_mocker.__code__
what_the_hell()
cmd_mock.assert_has_calls([call("c"),call("d")])
print("WORKS")
That print out
WORKS
Maybe I'm done... but OP still wait for a solution of his issue
from mock import patch, Mock
cmd_mock = Mock()
#A closure for grabbing the right function code
def cmd_mocker(a):
cmd_mock(a)
#patch.object(module._cmd,'__code__', new=cmd_mocker.__code__)
test_function():
...
Now I should say never use this trick unless you are with the back on the wall. Test should be simple to understand and to debug ... try to debug something like this and you will become mad!

Pythonic way of replacing real return values and implementation of functions with mock ones

My flask+python app pulls json from native binaries via a third party module nativelib.
Essentially, my functions look like this
def list_users():
return nativelib.all_users()
However, strong dependency on this third party module and a huge native component is proving to be a huge impediment in rapid development.
What I would like to do is mock the return value of my list_users function.
Additionally, I should be able to switch back to 'real data' by simply toggling some boolean.
This boolean could be some attribute within code or some command line argument.
The solution I have currently devised looks something like this:
#mockable('users_list')
def list_users():
return nativelib.all_users()
I have implemented the aforementioned mockable as a class:
class mockable:
mock = False
# A dictionary that contains a mapping between the api method
# ..and the file which contains corresponding the mock json response
__mock_json_resps = {'users_list':'/var/mock/list_user.json', 'user_by_id': '/var/mock/user_1.json'}
def __init__(self, api_method):
self.api_method = api_method
def __call__(self, fn):
#wraps(fn)
def wrapper():
if mock:
# If mocking is enabled,read mock data from json file
mock_resp = __mock_json_resps[self.api_method]
with open(mock_resp) as json_file:
return json.load(json_file)
else:
return self.fn()
return wrapper
In my entry point module I can then enable mocking using
mockable.mock = True
Now although this might work, I am interested in knowing if this is the 'pythonic' way of doing it.
If not, what is the best way to achieve this?

Personally, I prefer using the mock library, despite it being an additional dependency. Also, mocking is used in my unittests only, so the actual code is not mixed with code for testing. So, in your place, I'd leave the function you have as it is:
def list_users():
return nativelib.all_users()
And then in my unittest, I would patch the native library "temporarily":
def test_list_users(self):
with mock.patch.object(nativelib,
'all_users',
return_value='json_data_here')):
result = list_users()
self.assertEqual(result, 'expected_result')
In addition, for writing multiple tests, I usually turn the mocking code into a decorator, like:
def mock_native_all_users(return_value):
def _mock_native_all_users(func):
def __mock_native_all_users(self, *args, **kwargs):
with mock.patch.object(nativelib,
'all_users',
return_value=return_value):
return func(self, *args, **kwargs)
return __mock_native_all_users
return _mock_native_all_users
and then use it:
#mock_native_all_users('json_data_to_return')
def test_list_users(self):
result = list_users()
self.assertEqual(result, 'expected_result')

You set mockable.mock in your entry code, that is you don't dynamically switch it on and off at different times during a single run. So I'm inclined to think of this as a dependency configuration issue. This isn't necessarily better, just a slightly different way of framing the problem.
If you aren't bothered about inspection on nativelib.all_users then you don't need to touch list_users, just replace its dependency. For example as a quick one-shot if nativelib is only imported in one place:
if mock:
class NativeLibMock:
def all_users(self):
return whatever # maybe configure the mock with init params
nativelib = NativeLibMock()
else:
import nativelib
If you are bothered about inspection then there's a potential problem with the extra self parameter and so on. In practice you'd probably not want to duplicate my code above or have different instances of NativeLibMock kicking around in different modules that depend on nativelib. If you're defining a module anyway to contain your mock object then you might as well just implement a mock module.
So you can of course re-implement the module to return the mock data and either adjust PYTHONPATH (or sys.path) according to whether you want the mock, or do import mocklib as nativelib. Which you choose depends on whether you want to mock all uses of nativelib or just this one. Basically, the same thing you'd do during rapid development if the reason that you want to mock nativelib is because it hasn't been written yet ;-)
Consider how important it is that the specification of 'users_list' as the tag for the mock data for list_users, is located with list_users. If it's useful then decorating list_users is a benefit, but you still might want to just provide a different definition of the decorator in the mock vs non-mock cases. The non-mock "decorator" could just return fn instead of wrapping it.

Python: Using `copyreg` to define reducers for types that already have reducers

(Keep in mind I'm working in Python 3, so a solution needs to work in Python 3.)
I would like to use the copyreg module to teach Python how to pickle functions. When I tried to do it, the _Pickler object would still try to pickle functions using the save_global function. (Which doesn't work for unbound methods, and that's the motivation for doing this.)
It seems like _Pickler first tries to look in its own dispatch for the type of the object that you want to pickle before looking in copyreg.dispatch_table. I'm not sure if this is intentional.
Is there any way for me to tell Python to pickle functions with the reducer that I provide?

The following hack seems to work in Python 3.1...:
import copyreg
def functionpickler(f):
print('pickling', f.__name__)
return f.__name__
ft = type(functionpickler)
copyreg.pickle(ft, functionpickler)
import pickle
pickle.Pickler = pickle._Pickler
del pickle.Pickler.dispatch[ft]
s = pickle.dumps(functionpickler)
print('Result is', s)
Out of this, the two hackish lines are:
pickle.Pickler = pickle._Pickler
del pickle.Pickler.dispatch[ft]
You need to remove the dispatch entry for functions' type because otherwise it preempts the copyreg registration; and I don't think you can do that on the C-coded Pickler so you need to set it to the Python-coded one.
It would be a bit less of a hack to subclass _Pickler with a class of your own which makes its own dispatch (copying the parent's and removing the entry for the function type), and then use your subclass specifically (and its dump method) rather than pickle.dump; however it would also be a bit less convenient that this monkeypatching of pickle itself.