I have two files file1.py and file2.py.
I execute a function from file2.py which executes a function from file1.py.
# tree/file1.py
def transfer(.., token="Y"):
...
# tree/file2.py
from .file1 import transfer
def secure(..):
...
transfer(..) #the `token` argument does not need to be called here.
def main(..):
...
secure(..)
The transfer function from file1.py is used in several other files in the tree.
Could I, from outside the tree, set the token variable for the transfer function in my file1.py? Which would be applied for all these executions.
# outside the tree folder
from tree.file2 import main
token = ?
# choose the token variable and apply it to the transfer function
main(..)
This would save me from overloading the code and having to put a token argument to all the functions. I want the user to be able to choose his token. Avoid this:
from tree.file2 import main
token = "X"
# choose the token variable and apply it to the transfer function
main(.., token=token)
# and be forced to put the token argument to all functions..
My suggestion would be that file1.py should be left alone because if you ever want to use a different token the function is already defined to take a token: def transfer(.., token):
One possibility is to make an intermediary:
# file1a.py
from .file1 import transfer as tr
def transfer(...):
token = "X"
tr(..., token)
Now all other modules need to have this: from .file1a import transfer
(if you have hundreds of files and you don't want to change the import, you could swap the contents of file1.py for file1a.py)
You can also effectively mock out the actual function and make all functions automatically call your own function which supplies the required token:
from tree.file1 import transfer as tr
def transfer(...):
token = "X" # this function substitutes a different token
return tr(..., token) # call to the real function
import tree.file1 as file1
file1.transfer = transfer # this sets up the mock from here
from tree.file2 import main
main(...)
What follows is a whirlwind tour of maintaining mutable state between function calls, of which the default argument is but one example.
If you want a "configurable" default value, use a sentinel to detect when no argument is passed, then assign a module global to the parameter.
transfer_default = "Y"
def transfer(.., token=None):
if token is None:
token = transfer_token
...
You can technically alter a preset default value (by replacing all the defaults at once, since they are stored in an immutable tuple):
# Assuming token is the first/only parameter with a default
transfer.__defaults__ = ("N",)
but I don't recommend it.
You can also use a closure to allow the caller to create their own version of transfer with whatever default they want:
def make_transfer(default):
def _(..., token=default):
...
return _
transfer1 = make_transfer("Y")
transfer2 = make_transfer("N")
While you could always use make_transfer("Y")(...) on demand, instead of reusing transfer1(...), keep in mind that make_transfer has to define a new function every time it is called.
Using a closure leads to the dual of a closure, a class.
class Transferrer:
def __init__(self, token="Y"):
self.token = token
def transfer(self):
...
Transferrer().transfer() # with Y
Transferrer("N").transfer() # with N
As with closures, you probably want to reuse the same Transferrer object multiple times, rather than always creating a new one each time you want to call its transfer method.
Related
I have a Python (3) structure like following:
main_script.py
util_script.py
AccessClass.py
The main script is calling a function in util with following signature:
def migrate_entity(project, name, access=AccessClass.AccessClass()):
The call itself in the main script is:
migrate_entity(project_from_file, name_from_args, access=access_object)
All objects do have values when the call is done.
However, As soon as the main script is executed the AccessClass in the function parameters defaults is initialized, even though it is never used. For example this main script __init__ will create the default class in the function signature:
if __name__ == "__main__":
argparser = argparse.ArgumentParser(description='Migrate support data')
argparser.add_argument('--name', dest='p_name', type=str, help='The entity name to migrate')
load_dotenv()
fileConfig('logging.ini')
# Just for the sake of it
quit()
# The rest of the code...
# ...and then
migrate_entity(project_from_file, name_from_args, access=access_object)
Even with the quit() added the AccessClass is created. And if I run the script with ./main_script.py -h the AccessClass in the function signature is created. And even though the only call to the function really is with an access object I can see that the call is made to the AccessClass.__init__.
If I replace the default with None and instead check the parameter inside the function and then create it, everything is working as expected, i.e. the AccessClass is not created if not needed.
Can someone please enlighten me why this is happening and how defaults are expected to work?
Are parameter defaults always created in advance in Python?
Basically the mutable objects are initialized the moment you declare the function, not when you invoke it. That's why it's widely discouraged to use mutable types as defaults. You can use None as you mentioned and inside the body do the check if something is None and then initialize it properly.
def foo_bad(x = []): pass # This is bad
foo_bad() # the list initialized during declaration used
foo_bad([1,2]) # provided list used
foo_bad() # again the list initialized during declaration used
def foo_good(x = None):
if x is None:
x=[]
... # further logic
AccessClass is being created because you've set it as a default parameter, so it it's in the scope of the file itself and will be initialised when the file is first imported. This is also why it's not recommended to use lists or dicts as default parameters.
This is a much safer way of defining a default value if nothing is provided:
def migrate_entity(project, name, access=None):
if access is None:
access = AccessClass.AccessClass()
You could also use type hinting to demonstrate what type access should be:
def migrate_entity(project, name, access: Optional[AccessClass.AccessClass] = None): ...
I am wondering how I can send data to a callback function defined in an outer scope, from within an inner scope.
Here is a minimum example as it is easier to describe with code:
from somelib import CreateWebserver, Collector
def collector_callback(code):
# I want to affect some change to the server from this function
# i.e. server.close()
print(code)
def main():
server = CreateWebserver()
# The signature for this callback was defined by the library and I can't change it
collector = Collector(collector_callback)
# Blocks forever
collector.listen()
if __name__ == "__main__":
main()
In the above example I create my program in a main function.
I am using a library with a Collector class. The class takes a callback that is called when an event happens and the signature of this function is defined by the library (I can't change it).
Let's say the callback is called with a random number, and I want to do something to the server if the number is bigger than 5.
How do I access the server object in the collector_callback function?
It seems like the only way to do this would be to make the server a global variable, which I do not want to do because I want to keep things encapsulated.
Is there a better way to reach the server object in this instance, given than the collector_callback will only be called with the random number?
You can use functools.partial to create a function where one of the parameters is set and is not in the resulting function's signature
def collector_callback(server, code):
server.close()
print(code)
def main():
server = CreateWebserver()
callback = functools.partial(collector_callback, server)
# callback is now a function with only one parameter "code"
# server is always passed as the first parameter to collector_callback
# when it is called
collector = Collector(callback)
I am stuck in a situation where I need to pass some value (which is always be random/different) returned from a function to another function, and the sequence in which the functions will be called is undefined as it will be figured at run-time based on user inputs.
For example,
def func1(some_value):
# Use some_value for whatever purpose
# Some code
return some_random_value
def func2(some_value):
# Use some_value for whatever purpose
# Some code
return some_random_value
def func3(some_value):
# Use some_value for whatever purpose
# Some code
return some_random_value
So let's assume if func2 is called first, any initial/default value is passed as parameter some_value and the function will return some_random_value. Now, I don't know which function will be called next, but whatever function is called the some_random_value returned from the previous function (in this case func2) should be passed as parameter some_value to the next called function (let it be func1). And this process goes on and on.
What could be the recommended way to achieve this? Should this be done using a global variable whose value is amended each time a function runs to store the function's return value? If yes, then how?
More specifically
A CLI will allow the user to choose some action and an appropriate function will be called according to this action. The last returned value from a function should be in the memory till the application ends. After a function performs it's task, it'll return a value. That value is required when any other function is called using CLI action. Again, the next function will process some data using the last function's return value, and then return some processed value, which later will be used by the next function or CLI action.
I was thinking like instead of returning the value from any of those functions, create a global variable with the default value:
common_data = 'some string'
And then in every function definition, add:
global common_data
common_data = 'new processed string'
This will ensure any next function call will be passed the value last saved in common_data by the previous function.
But this seems to be a non-recommend solution, at least I think so.
Please allow me to edit or elaborate this question if I am unable to explain my situation properly.
Thank you
I will deliver on this recursion error. ^^
from random import choice
from random import randint
def get_fs(f):
return [x for x in (func1, func2, func3) if x != f]
def func1(some_value, fs):
# Use some_value for whatever purpose
# Some code
f = choice(fs)
print("func1", f.__name__)
return f(randint(1,10), get_fs(func1))
def func2(some_value, fs):
# Use some_value for whatever purpose
# Some code
f = choice(fs)
print("func2", f.__name__)
return f(randint(1,10), get_fs(func2))
def func3(some_value, fs):
# Use some_value for whatever purpose
# Some code
f = choice(fs)
print("func3", f.__name__)
return f(randint(1,10), get_fs(func3))
def main():
functions = [func2, func3]
func1(randint(1,10), functions)
if __name__ == '__main__':
main()
I have a utilities.py file for my python project. It contains only util functions, for example is_float(string), is_empty(file), etc.
Now I want to have a function is_valid(number), which has to:
read from a file, valid.txt, which contains all numbers which are valid, and load them onto a map/set.
check the map for the presence of number and return True or False.
This function is called often, and running time should be as small as possible. I don't want to read open and read valid.txt everytime the function is called. The only solution I have come up with is to use a global variable, valid_dict, which is loaded once from valid.txt when utilities.py is imported. The loading code is written as main in utilities.py.
My question is how do I do this without using a global variable, as it is considered bad practice? What is a good design pattern for doing such a task without using globals? Also note again that this is a util file, so there should ideally be no main as such, just functions.
The following is a simple example of a closure. The dictionary, cache, is encapsulated within the outer function (load_func), but remains in scope of the inner, even when it is returned. Notice that load_func returns the inner function as an object, it does not call it.
In utilities.py:
def _load_func(filename):
cache = {}
with open(filename) as fn:
for line in fn:
key, value = line.split()
cache[int(key)] = value
def inner(number):
return number in cache
return inner
is_valid = _load_func('valid.txt')
In __main__:
from utilities import is_valid # or something similar
if is_valid(42):
print(42, 'is valid')
else:
print(42, 'is not valid')
The dictionary (cache) creation could have been done using a dictionary comprehension, but I wanted you to concentrate on the closure.
The variable valid_dict would not be global but local to utilities.py. It would only become global if you did something like from utilities import *. Now that is considered bad practice when you're developing a package.
However, I have used a trick in cases like this that essentially requires a static variable: Add an argument valid_dict={} to is_valid(). This dictionary will be instantiated only once and each time the function is called the same dict is available in valid_dict.
def is_valid(number, valid_dict={}):
if not valid_dict:
# first call to is_valid: load valid.txt into valid_dict
# do your check
Do NOT assign to valid_dict in the if-clause but only modify it: e.g., by setting keys valid_dict[x] = y or using something like valid_dict.update(z).
(PS: Let me know if this is considered "dirty" or "un-pythonic".)
I'd like to modify the arguments passed to a method in a module, as opposed to replacing its return value.
I've found a way around this, but it seems like something useful and has turned into a lesson in mocking.
module.py
from third_party import ThirdPartyClass
ThirdPartyClass.do_something('foo', 'bar')
ThirdPartyClass.do_something('foo', 'baz')
tests.py
#mock.patch('module.ThirdPartyClass.do_something')
def test(do_something):
# Instead of directly overriding its return value
# I'd like to modify the arguments passed to this function.
# change return value, no matter inputs
do_something.return_value = 'foo'
# change return value, based on inputs, but have no access to the original function
do_something.side_effect = lambda x, y: y, x
# how can I wrap do_something, so that I can modify its inputs and pass it back to the original function?
# much like a decorator?
I've tried something like the following, but not only is it repetitive and ugly, it doesn't work. After some PDB introspection.. I'm wondering if it's simply due to however this third party library works, as I do see the original functions being called successfully when I drop a pdb inside the side_effect.
Either that, or some auto mocking magic I'm just not following that I'd love to learn about.
def test():
from third_party import ThirdPartyClass
original_do_something = ThirdPartyClass.do_something
with mock.patch('module.ThirdPartyClass.do_something' as mocked_do_something:
def side_effect(arg1, arg2):
return original_do_something(arg1, 'overridden')
mocked_do_something.side_effect = side_effect
# execute module.py
Any guidance is appreciated!
You may want to use parameter wraps for the mock call. (Docs for reference.) This way the original function will be called, but it will have everything from Mock interface.
So for changing parameters called to original function you may want to try it like that:
org.py:
def func(x):
print(x)
main.py:
from unittest import mock
import org
of = org.func
def wrapped(a):
of('--{}--'.format(a))
with mock.patch('org.func', wraps=wrapped):
org.func('x')
org.func.assert_called_with('x')
result:
--x--
The trick is to pass the original underlying function that you still want to access as a parameter to the function.
Eg, for race condition testing, have tempfile.mktemp return an existing pathname:
def mock_mktemp(*, orig_mktemp=tempfile.mktemp, **kwargs):
"""Ensure mktemp returns an existing pathname."""
temp = orig_mktemp(**kwargs)
open(temp, 'w').close()
return temp
Above, orig_mktemp is evaluated when the function is declared, not when it is called, so all invocations will have access to the original method of tempfile.mktemp via orig_mktemp.
I used it as follows:
#unittest.mock.patch('tempfile.mktemp', side_effect=mock_mktemp)
def test_retry_on_existing_temp_path(self, mock_mktemp):
# Simulate race condition: creation of temp path after tempfile.mktemp
...