Develop Reference

Write Module with decorated functions and imports

i have this code in a python file:
from dec import my_decorator
import asyncio
#my_decorator
async def simple_method(bar): # , x, plc_name, var_name):
print("Henlo from simple_method\npartent:{}".format(parent))
return
#my_decorator
async def other_simple_meth(bar, value):
print("Henlo from other_simple_meth:\t Val:{}".format(value))
return
async def main():
print("Start Module-Export")
open('module_functions.py', 'a').close()
# Write all decorated functions to modue_functions.py
print("Functions in module_functions.py exported")
while True:
asyncio.sleep(2)
print("z...z...Z...")
My goal is to write all decorated functions (inc. the import dependencies) into a second module file (here "module_functions.py"). My 'module_functions.py' file should look like this:
from dec import my_decorator
import asyncio
#my_decorator
async def simple_method(bar): # , x, plc_name, var_name):
print("Henlo from simple_method\npartent:{}".format(parent))
return
#my_decorator
async def other_simple_meth(bar, value):
print("Henlo from other_simple_meth:\t Val:{}".format(value))
return
I know how to get references and names of a function, but not how to "copy/paste" the functioncode (incl. decorator and all dependencies) into a seperated file. Is this even possible?
EDIT: I know that pickle and dill exist, but this may not fullfill the goal. The problem is, that someone else may not know the order of the dumped file and loading them back may/will cause problem. As well it seems to be not possible to edit such loaded functions again.

I found a (not ideal, but ok) solution for my problems.
I) Find and write functions, coroutines etc. into a file (works):
Like #MisterMiyagi suspected, is the inspect module a good way to go. For the common stuff, it is possible with inspect.getsource() to get the code and write them into a file:
# List of wanted stuff
func_list = [simple_method, meth_with_input, meth_with_input_and_output, func_myself]
with open('module_functions.py', 'a') as module_file:
for func in func_list:
try:
module_file.write(inspect.getsource(func))
module_file.write("\n")
except:
print("Error :( ")
II) But what about decorated stuff(seems to work)?
I) will not work for decorated stuff, it is just ignored without throwing an exception. What seems to be used is from functools import wraps.
In many examples the #wraps decorator is added into the decorator class. This was not possible for me, but there is a good workaround:
#wraps(lambda: simple_method) #<---add wraps-decorator here
#my_decorator
async def simple_method(parent): # , x, plc_name, var_name):
print("Henlo from simple_method\npartent:{}".format(parent))
return
Wraps can be placed above the original decorated method/class/function and it seems to behave like I want. Now we can add simple_methodinto the func_listof I).
III) What about the imports?
Well it seems to be quite tricky/impossible to actually read the dependencies of a function. My workaround is to drop all wanted imports into a class (sigh). This class can be throw into the func_listof I) and is written into the file.
EDIT:
There is a cleaner way, which may works, after some modification, with I) and II) as well. The magic module is ast.
I have overwritten following:
class ImportVisitor(ast.NodeVisitor):
def __init__(self, target):
super().__init__()
self.file_target = target
"pick these special nodes via overwriting: visit_classname." \
"classnames are listed in https://docs.python.org/3.6/library/ast.html#abstract-grammar"
def visit_Import(self, node):
"Overwrite func!"
"Write all statements just with import like - import ast into file_target"
str = 'import '+', '.join(alias.name for alias in node.names)
self.file_target.write(str+"\n")
def visit_ImportFrom(self, node):
"Overwrite func!"
"Write all statements with from ... import (like - from os.path import basename) into file_tagrget"
str = 'from '+ node.module+ ' import '+', '.join(alias.name for alias in node.names)
self.file_target.write(str+"\n")
Now I can parse my own script name and fill the module_file with the imports and from...imports it will find while visiting all nodes in this tree:
with open('module_functions.py', 'a') as module_file:
with open(basename(__file__), "rb") as f:
tree = ast.parse(f.read(), basename(__file__))
visitor = ImportVisitor(module_file)
visitor.visit(tree)
module_file.write("\n\n")

Class-level classmethod() can only be called on a method_descriptor or instance method

util.py:
import inspect
class Singleton(type):
_instances=[]
def __call__(cls, *args, **kwargs):
if cls not in cls._instances:
cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
class MetaResult(Singleton):
def __getattribute__(cls, name):
return super().__getattribute__(name)
class Result(metaclass=MetaResult):
#staticmethod
def res_decorator(func):
def funcwrap(*args, **kwargs):
sig = inspect.signature(func)
bound_sig = sig.bind(*args, **kwargs)
bound_sig.apply_defaults()
#additional code to extract function arguments
return funcwrap
check_params.py
from util import Result as _Result
from abc import ABCMeta as _ABCMeta
class paramparse(metaclass=_ABCMeta)
#classmethod
#_Result.res_decorator
def parse_flash_params(cls, flash_config_path):
#some code
Now, I cythonize the file check_params.py with following setup :
cythonize.py
import os as _os
from pathlib import Path as _Path
from distutils.core import setup as _setup
from Cython.Distutils import build_ext as _build_ext
files_to_compile = []
def cython_build(source_path):
for dirpath, _, fnames in os.walk(source_path):
for fname in [x for x in fnames if f.endswith('.py'):
fname = _Path(fname)
files_to_compile.append(fname)
for e in files_to_compile:
e.cython_directives = {'binding':True, 'language_level':3}
_setup(name="Proj1",cmdclass={'build_ext':_build_ext}, ext_modules=files_to_compile)
cythonized as:
python cythonize.py --path C:\directory_where_check_params_exist
generates a pyd file on which the following unit tests were attempted to run:
Now, coming to the usage, in the unit tests:
unit_test_check_params.py
from check_params import * #getting error here , details outside the code
# unit tests written here
check_params.pyx:112: in init check_params
???
E
TypeError: Class-level classmethod() can only be called on a method_descriptor or instance method.
So when i debug this, the error appears as being caused because of the classmethod descriptor over decorator (def parse_flash_params) in check_params.py
Please let me know if you need more information.

This still isn't a hugely helpful example in since the code you provide still doesn't actually work. However:
The case
#classmethod
#_Result.res_decorator
is definitely a Cython bug. In the function __Pyx_Method_ClassMethod Cython has a lot of type checks to ensure that the type is a method (a function defined in a class), while actually it only needs to be callable, and this should only really be checked at call time. As a quick workround you can edit the relevant internal Cython file (CythonFunction.c) to replace the lines
PyErr_SetString(PyExc_TypeError,
"Class-level classmethod() can only be called on "
"a method_descriptor or instance method.");
return NULL;
with
return PyClassMethod_New(method);
This seems to me closer to what Python does, where it accepts any object and only checks for callability when the function is actually called.
In the longer term you should report it as a bug in Cython with an example that actually works to demonstrate the problem. This way it can actually be fixed. I don't think you need Result and the staticclass bit - res_decorator as an isolated function should demonstrate the problem.
The second possible order
#_Result.res_decorator
#classmethod
doesn't work in unCythonized Python either since the direct result of a classmethod decorator isn't callable. It only becomes callable when it becomes a bound method, which happens later. Therefore this isn't a bug in Cython.
Final addendum:
A cleaner workaround is to force Cython to use the builtin classmethod, instead of its own version that's causing bugs
try:
myclassmethod = __builtins__.classmethod
except AttributeError:
myclassmethod = __builtins__['classmethod']
class paramparse(metaclass=_ABCMeta):
#myclassmethod
#_Result.res_decorator
def parse_flash_params(cls, flash_config_path):
pass
The try ... except block is because __builtins__ behaves slightly differently in Cython and in a Python module, which is fine because it's an implementation detail anyway.

AttributeError: while using monkeypatch of pytest

src/mainDir/mainFile.py
contents of mainFile.py
import src.tempDir.tempFile as temp
data = 'someData'
def foo(self):
ans = temp.boo(data)
return ans
src/tempDir/tempFile.py
def boo(data):
ans = data
return ans
Now I want to test foo() from src/tests/test_mainFile.py and I want to mock temp.boo(data) method in foo() method
import src.mainDir.mainFile as mainFunc
testData = 'testData'
def test_foo(monkeypatch):
monkeypatch.setattr('src.tempDir.tempFile', 'boo', testData)
ans = mainFunc.foo()
assert ans == testData
but I get error
AttributeError: 'src.tempDir.tempFile' has no attribute 'boo'
I expect ans = testData.
I would like to know if I am correctly mocking my tempDir.boo() method or I should use pytest's mocker instead of monkeypatch.

You're telling monkeypatch to patch the attribute boo of the string object you pass in.
You'll either need to pass in a module like monkeypatch.setattr(tempFile, 'boo', testData), or pass the attribute as a string too (using the two-argument form), like monkeypatch.setattr('src.tempDir.tempFile.boo', testData).

My use case was was slightly different but should still apply. I wanted to patch the value of sys.frozen which is set when running an application bundled by something like Pyinstaller. Otherwise, the attribute does not exist. Looking through the pytest docs, the raising kwarg controls wether or not AttributeError is raised when the attribute does not already exist. (docs)
Usage Example
import sys
def test_frozen_func(monkeypatch):
monkeypatch.setattr(sys, 'frozen', True, raising=False)
# can use ('fq_import_path.sys.frozen', ...)
# if what you are trying to patch is imported in another file
assert sys.frozen

Update: mocking function calls can be done with monkeypatch.setattr('package.main.slow_fun', lambda: False) (see answer and comments in https://stackoverflow.com/a/44666743/3219667) and updated snippet below
I don't think this can be done with pytest's monkeypatch, but you can use the pytest-mock package. Docs: https://github.com/pytest-dev/pytest-mock
Quick example with the two files below:
# package/main.py
def slow_fun():
return True
def main_fun():
if slow_fun():
raise RuntimeError('Slow func returned True')
# tests/test_main.py
from package.main import main_fun
# Make sure to install pytest-mock so that the mocker argument is available
def test_main_fun(mocker):
mocker.patch('package.main.slow_fun', lambda: False)
main_fun()
# UPDATE: Alternative with monkeypatch
def test_main_fun_monkeypatch(monkeypatch):
monkeypatch.setattr('package.main.slow_fun', lambda: False)
main_fun()
Note: this also works if the functions are in different files

Store the cache to a file functools.lru_cache in Python >= 3.2

I'm using #functools.lru_cache in Python 3.3. I would like to save the cache to a file, in order to restore it when the program will be restarted. How could I do?
Edit 1 Possible solution: We need to pickle any sort of callable
Problem pickling __closure__:
_pickle.PicklingError: Can't pickle <class 'cell'>: attribute lookup builtins.cell failed
If I try to restore the function without it, I get:
TypeError: arg 5 (closure) must be tuple

You can't do what you want using lru_cache, since it doesn't provide an API to access the cache, and it might be rewritten in C in future releases. If you really want to save the cache you have to use a different solution that gives you access to the cache.
It's simple enough to write a cache yourself. For example:
from functools import wraps
def cached(func):
func.cache = {}
#wraps(func)
def wrapper(*args):
try:
return func.cache[args]
except KeyError:
func.cache[args] = result = func(*args)
return result
return wrapper
You can then apply it as a decorator:
>>> #cached
... def fibonacci(n):
... if n < 2:
... return n
... return fibonacci(n-1) + fibonacci(n-2)
...
>>> fibonacci(100)
354224848179261915075L
And retrieve the cache:
>>> fibonacci.cache
{(32,): 2178309, (23,): 28657, ... }
You can then pickle/unpickle the cache as you please and load it with:
fibonacci.cache = pickle.load(cache_file_object)
I found a feature request in python's issue tracker to add dumps/loads to lru_cache, but it wasn't accepted/implemented. Maybe in the future it will be possible to have built-in support for these operations via lru_cache.

You can use a library of mine, mezmorize
import random
from mezmorize import Cache
cache = Cache(CACHE_TYPE='filesystem', CACHE_DIR='cache')
#cache.memoize()
def add(a, b):
return a + b + random.randrange(0, 1000)
>>> add(2, 5)
727
>>> add(2, 5)
727

Consider using joblib.Memory for persistent caching to the disk.
Since the disk is enormous, there's no need for an LRU caching scheme.

You are not supposed to touch anything inside the decorator implementation except for the public API so if you want to change its behavior you probably need to copy its implementation and add necessary functions yourself. Note that the cache is currently stored as a circular doubly linked list so you will need to take care when saving and loading it.

This is something that I wrote might be helpful devcache.
It's designed to help you speed up iterations for long running methods. It's configurable with a config file
#devcache(group='crm')
def my_method(a, b, c):
...
#devcache(group='db')
def another_method(a, b, c):
...
The cache can be refreshed or used with a yaml config file like:
refresh: false # refresh true will ignore use_cache and refresh all cached data
props:
1:
group: crm
use_cache: false
2:
group: db
use_cache: true
Would refresh the cache for my_method and use the cache for another_method.
It's not going to help you pickle the the callable but it does the caching part and would be straight forward to modify the code to add specialized serialization.

If your use-case is to cache the result of computationally intensive functions in your pytest test suites, pytest already has a file-based cache. See the docs for more info.
This being said, I had a few extra requirements:
I wanted to be able to call the cached function directly in the test instead of from a fixture
I wanted to cache complex python objects, not just simple python primitives/containers
I wanted an implementation that could refresh the cache intelligently (or be forced to invalidate only a single key)
Thus I came up with my own wrapper for the pytest cache, which you
can find below. The implementation is fully documented, but if you
need more info let me know and I'll be happy to edit this answer :)
Enjoy:
from base64 import b64encode, b64decode
import hashlib
import inspect
import pickle
from typing import Any, Optional
import pytest
__all__ = ['cached']
#pytest.fixture
def cached(request):
def _cached(func: callable, *args, _invalidate_cache: bool = False, _refresh_key: Optional[Any] = None, **kwargs):
"""Caches the result of func(*args, **kwargs) cross-testrun.
Cache invalidation can be performed by passing _invalidate_cache=True or a _refresh_key can
be passed for improved control on invalidation policy.
For example, given a function that executes a side effect such as querying a database:
result = query(sql)
can be cached as follows:
refresh_key = query(sql=fast_refresh_sql)
result = cached(query, sql=slow_or_expensive_sql, _refresh_key=refresh_key)
or can be directly invalidated if you are doing rapid iteration of your test:
result = cached(query, sql=sql, _invalidate_cache=True)
Args:
func (callable): Callable that will be called
_invalidate_cache (bool, optional): Whether or not to invalidate_cache. Defaults to False.
_refresh_key (Optional[Any], optional): Refresh key to provide a programmatic way to invalidate cache. Defaults to None.
*args: Positional args to pass to func
**kwargs: Keyword args to pass to func
Returns:
_type_: _description_
"""
# get debug info
# see https://stackoverflow.com/a/24439444/4442749
try:
func_name = getattr(func, '__name__', repr(func))
except:
func_name = '<function>'
try:
caller = inspect.getframeinfo(inspect.stack()[1][0])
except:
func_name = '<file>:<lineno>'
call_key = _create_call_key(func, None, *args, **kwargs)
cached_value = request.config.cache.get(call_key, {"refresh_key": None, "value": None})
value = cached_value["value"]
current_refresh_key = str(b64encode(pickle.dumps(_refresh_key)), encoding='utf8')
cached_refresh_key = cached_value.get("refresh_key")
if (
_invalidate_cache # force invalidate
or cached_refresh_key is None # first time caching this call
or current_refresh_key != cached_refresh_key # refresh_key has changed
):
print("Cache invalidated for '%s' # %s:%d" % (func_name, caller.filename, caller.lineno))
result = func(*args, **kwargs)
value = str(b64encode(pickle.dumps(result)), encoding='utf8')
request.config.cache.set(
key=call_key,
value={
"refresh_key": current_refresh_key,
"value": value
}
)
else:
print("Cache hit for '%s' # %s:%d" % (func_name, caller.filename, caller.lineno))
result = pickle.loads(b64decode(bytes(value, encoding='utf8')))
return result
return _cached
_args_marker = object()
_kwargs_marker = object()
def _create_call_key(func: callable, refresh_key: Any, *args, **kwargs):
"""Produces a hex hash str of the call func(*args, **kwargs)"""
# producing a key from func + args
# see https://stackoverflow.com/a/10220908/4442749
call_key = pickle.dumps(
(func, refresh_key) +
(_args_marker, ) +
tuple(args) +
(_kwargs_marker,) +
tuple(sorted(kwargs.items()))
)
# create a hex digest of the key for the filename
m = hashlib.sha256()
m.update(bytes(call_key))
return m.digest().hex()

How to mark a global as deprecated in Python?

I've seen decorators that let you mark a function a deprecated so that a warning is given whenever that function is used. I'd like to do the same thing but for a global variable, but I can't think of a way to detect global variable accesses. I know about the globals() function, and I could check its contents, but that would just tell me if the global is defined (which it still will be if the function is deprecated and not all out removed) not if it's actually being used. The best alternative I can think of is something like this:
# myglobal = 3
myglobal = DEPRECATED(3)
But besides the problem of how to get DEPRECATED to act exactly like a '3', I'm not sure what DEPRECATED could do that would let you detect every time it's accessed. I think the best it could do is iterate through all of the global's methods (since everything in Python is an object, so even '3' has methods, for converting to string and the like) and 'decorate' them to all be deprecated. But that's not ideal.
Any ideas? Has anyone else tackled this problem?

You can't do this directly, since theres no way of intercepting the module access. However, you can replace that module with an object of your choosing that acts as a proxy, looking for accesses to certain properties:
import sys, warnings
def WrapMod(mod, deprecated):
"""Return a wrapped object that warns about deprecated accesses"""
deprecated = set(deprecated)
class Wrapper(object):
def __getattr__(self, attr):
if attr in deprecated:
warnings.warn("Property %s is deprecated" % attr)
return getattr(mod, attr)
def __setattr__(self, attr, value):
if attr in deprecated:
warnings.warn("Property %s is deprecated" % attr)
return setattr(mod, attr, value)
return Wrapper()
oldVal = 6*9
newVal = 42
sys.modules[__name__] = WrapMod(sys.modules[__name__],
deprecated = ['oldVal'])
Now, you can use it as:
>>> import mod1
>>> mod1.newVal
42
>>> mod1.oldVal
mod1.py:11: UserWarning: Property oldVal is deprecated
warnings.warn("Property %s is deprecated" % attr)
54
The downside is that you are now performing two lookups when you access the module, so there is a slight performance hit.

You could make your module into a class (see e.g this SO question) and make that deprecated global into a property, so you can execute some of your code when it's accessed and provide the warning you desire. However, this does seem a bit of an overkill.

Behold:
Code
from types import *
def wrapper(f, warning):
def new(*args, **kwargs):
if not args[0].warned:
print "Deprecated Warning: %s" % warning
args[0].warned = True
return f(*args, **kwargs)
return new
class Deprecated(object):
def __new__(self, o, warning):
print "Creating Deprecated Object"
class temp(o.__class__): pass
temp.__name__ = "Deprecated_%s" % o.__class__.__name__
output = temp.__new__(temp, o)
output.warned = True
wrappable_types = (type(int.__add__), type(zip), FunctionType)
unwrappable_names = ("__str__", "__unicode__", "__repr__", "__getattribute__", "__setattr__")
for method_name in dir(temp):
if not type(getattr(temp, method_name)) in wrappable_types: continue
if method_name in unwrappable_names: continue
setattr(temp, method_name, wrapper(getattr(temp, method_name), warning))
output.warned = False
return output
Output
>>> a=Deprecated(1, "Don't use 1")
Creating Deprecated Object
>>> a+9
Deprecated Warning: Don't use 1
10
>>> a*4
4
>>> 2*a
2
This can obviously be refined, but the gist is there.

This is one of the main rationale for PEP 562 (implemented in Python 3.7):
Typical workarounds are assigning __class__ of a module object to a
custom subclass of types.ModuleType or replacing the sys.modules item
with a custom wrapper instance. It would be convenient to simplify
this procedure by recognizing __getattr__ defined directly in a module
that would act like a normal __getattr__ method, except that it will
be defined on module instances. For example:
# lib.py
from warnings import warn
deprecated_names = ["old_function", ...]
def _deprecated_old_function(arg, other):
...
def __getattr__(name):
if name in deprecated_names:
warn(f"{name} is deprecated", DeprecationWarning)
return globals()[f"_deprecated_{name}"]
raise AttributeError(f"module {__name__} has no attribute {name}")
# main.py
from lib import old_function # Works, but emits the warning