*This post has been edited to reflect suggestions relating to question clarity and intent
*I have conceptualized a solution to this problem, and I am currently developing the code. I will update this post when the solution is complete. This comment will be removed when I post my solution.
My goal is to implement a modular framework that meets the following requirements:
There is a single instance that manages all of the modular components; I will refer to it as the Engine, and the modular components as Models. The Engine must provide the following services:
Direct information to each Models subscribed functions; I will refer to these as callbacks.
Run functions belonging to an instance of a Model, if the model has requested that the function be run; I will refer to these as updates.
A Model can be defined and instantiated without an Engine. The Model has no dependencies on the Engine (although without an Engine, the model is mostly useless).
There should be some attribute applied to each callback and update in a Model so that, when added to the Engine, the Engine can determine the purpose of each function in the Model.
Each update is defined with update criteria, consisting of Boolean logic and a priority. An update will only run successfully if the update criteria are True. If the update criteria are False, the update will raise an Exception
updates are called by the Engine, highest priority first.
Information can only be passed to an Engine by returning a value in a update. Information passed to the Engine is distributed immediately to callbacks.
Each callback is defined with a topic. The callback will only be called by the Engine, if the information topic matches the callback topic.
It is my current understanding that decorators would be effective in implementing the desired behavior. I think we could do the following:
Create a decorator named callback, that takes a parameter tag which is set as an attribute on the decorated function. tag denotes the type of information that the function should receive.
Create a decorator named 'update', that takes parameters logic and priority which are set as attributes on the decorated function. logic is a callable that returns a Boolean. priority denotes the priority of the function.
Using decorators would allow me to define Models without Engine dependency. The Engine can utilize inspect to get functions in each Model that have the relevant attributes.
After implementing various attempts, I have the following concerns:
Decorators are applied at instantiation, and as a result cannot accept self as an argument.
I'm still unclear about the source of the arguments, why you are attaching flags to functions and where this exception should be used...
This seems more like an XY-problem than anything else.
Nonetheless, decorators are cool so I wrote this which might point you in the right direction.
import typing as T
from datetime import date
def testmonday() -> bool:
return date.today().weekday() == 0 # when I wrote this it was Monday
def testfalse() -> bool:
return False
def runonany(*tests: T.Callable[[], bool]):
def wrapper(func):
if any(t() for t in tests):
return func
else:
return lambda: None
return wrapper
#runonany(testmonday, testfalse, lambda: False)
def runfunc():
return "running"
#runonany(testfalse, lambda: True)
def dontrunfunc():
return "how did you get this ?"
# Or if flags are a group of booleans
FLAGS = [True, False, True, False]
def runonanyflag():
def wrapper(func):
if any(FLAGS):
return func
else:
return lambda: None
return wrapper
#runonanyflag
def runfuncflag():
return "also running"
print(runfunc())
print(dontrunfunc()) # None
print(runfuncflag())
Seems that your Model is nothing but a set of some Callbacks, somehow tagged, with the only intention to be called by the Engine.
In that case it is enough to have simple abstract class defining all known accepted callbacks by their rightful names. It’s easy to know if a given callback is implemented: one deadly stupid option is to catch the NotImplemented on the Engine side.
Further more: your Engine could expose a way to register expected Callback upfront, in which case design is approaching the standard proven by Time, waterproof and bulletproof Observer pattern.
I would suggest to take a really careful look into so called “reactive extensions”, especially in JavaScript (rxjs), and it’s core interfaces: Observer and Observable.
Related
I have a Python class that requires some data in order to be initialized. This data is usually obtained using a function from another module, which makes calls to an API. One of the parameters my class' initializer takes is the same ID that can be used to obtain the resource with the API.
Calling the API from inside the initializer, and obtaining the data it needs would make for shorter (and cleaner?) initialization. But I am concerned this could make the class harder to test, and introduce a dependency deep inside the code.
I'm trying to devise the best way to implement this in a maintainable and testable way.
Would it be bad to call the API module directly from within the initializer, and obtain the data it needs to complete initialization? Or is it better to just call the API from outside and pass the data to the initializer?
The "normal" way(1) is the pass the dependent function, module, or class, into the constructor itself.
Then, in your production code, pass in the real thing. In your test code, pass in a dummy one that will behave exactly as you desire for the specific test case.
That's actually a half-way measure between the two things you posit.
In other words, something like:
def do_something_with(something_generator):
something = something_generator.get()
print(something)
# Real code.
do_something_with(ProductionGenerator())
# Test code.
class TestGenerator:
def get(self):
return 42
do_something_with(TestGenerator())
If you're reticent to always pass in a dependency, you can get around that with something like a default value and creating it inside the function if not given:
def do_something(something_generator=None):
if something_generator is None:
local_gen = ProductionGenerator()
else:
local_gen = something_generator
something = something_generator.get()
print(something)
# Real code.
do_something()
# Test code.
class TestGenerator:
def get(self):
return 42
do_something(TestGenerator())
(1) Defined, of course, as the way I do it :-)
Suppose that I have a function in my Python application that define some kind of context - a user_id for example. This function call other functions that do not take this context as a function argument. For example:
def f1(user, operation):
user_id = user.id
# somehow define user_id as a global/context variable for any function call inside this scope
f2(operation)
def f2(operation):
# do something, not important, and then call another function
f3(operation)
def f3(operation):
# get user_id if there is a variable user_id in the context, get `None` otherwise
user_id = getcontext("user_id")
# do something with user_id and operation
My questions are:
Can the Context Variables of Python 3.7 be used for this? How?
Is this what these Context Variables are intended for?
How to do this with Python v3.6 or earlier?
EDIT
For multiple reasons (architectural legacy, libraries, etc) I can't/won't change the signature of intermediary functions like f2, so I can't just pass user_id as arguments, neither place all those functions inside the same class.
You can use contextvars in Python 3.7 for what you're asking about. It's usually really easy:
import contextvars
user_id = contextvars.ContextVar("user_id")
def f1(user, operation):
user_id.set(user.id)
f2()
def f2():
f3()
def f3():
print(user_id.get(default=None)) # gets the user_id value, or None if no value is set
The set method on the ContextVar returns a Token instance, which you can use to reset the variable to the value it had before the set operation took place. So if you wanted f1 to restore things the way they were (not really useful for a user_id context variable, but more relevant for something like setting the precision in the decimal module), you can do:
token = some_context_var.set(value)
try:
do_stuff() # can use value from some_context_var with some_context_var.get()
finally:
some_context_var.reset(token)
There's more to the contextvars module than this, but you almost certainly don't need to deal with the other stuff. You probably only need to be creating your own contexts and running code in other contexts if you're writing your own asynchronous framework from scratch.
If you're just using an existing framework (or writing a library that you want to play nice with asynchronous code), you don't need to deal with that stuff. Just create a global ContextVar (or look up one already defined by your framework) and get and set values on it as shown above, and you should be good to go.
A lot of contextvars use is probably going to be in the background, as an implementation detail of various libraries that want to have a "global" state that doesn't leak changes between threads or between separate asynchronous tasks within a single thread. The example above might make more sense in this kind of situation: f1 and f3 are part of the same library, and f2 is a user-supplied callback passed into the library somewhere else.
Essentially what you're looking for is a way to share a state between a set of function. The canonical way to do so in an object oriented language is to use a class:
class Foo(object):
def __init__(self, operation, user=None):
self._operation = operation
self._user_id = user.id if user else None
def f1(self):
print("in f1 : {}".format(self._user_id))
self.f2()
def f2(self):
print("in f2 : {}".format(self._user_id))
self.f3()
def f3(self):
print("in f3 : {}".format(self._user_id))
f = Foo(operation, user)
f.f1()
With this solution, your class instances (here f) are "the context" in which the functions are executed - each instance having it's own dedicated context.
The functional programing equivalent would be to use closures, I'm not going to give an example here since while Python supports closures, it's still first and mainly an object language so the OO solution is the most obvious.
And finally, the clean procedural solution is to pass this context (which can be expressed as a dict or any similar datatype) all along the call chain, as shown in DFE's answer.
As a general rule : relying on global variables or some "magic" context that could - or not - be set by you-dont-who-nor-where-nor-when makes for code that is hard if not impossible to reason about, and that can break in the most unpredictable ways (googling for "globals evil" will yield an awful lot of litterature on the topic).
You can use kwargs in your function calls in order to pass
def f1(user, operation):
user_id = user.id
# somehow define user_id as a global/context variable for any function call inside this scope
f2(operation, user_id=user_id)
def f2(operation, **kwargs):
# do something, not important, and then call another function
f3(operation, **kwargs)
def f3(operation, **kwargs):
# get user_id if there is a variable user_id in the context, get `None` otherwise
user_id = kwargs.get("user_id")
# do something with user_id and operation
the kwargs dict is the equivalent to what you are looking at in context variables, but limited at a call stack. It is the same memory element passed (through pointer-like) in each function and not duplicates variables in memory.
In my opinion, but I would like to see what you all think, context variables is an elegant way to authorize globals variables and to control it.
I am following this explanation, and I don't quite get how Python interpreter arrives at the following. In the first example, is Python seeing #implementer(IAmericanSocket) is not implemented by UKSocket, then it decides to make it a AdaptToAmericanSocket because that is the only implementation of IAmericanSocket with one argument? What if there is another class instance implementing IAmericanSocket with one argument? In the second example, why is IAmericanSocket not overriding AmericanSocket's voltage method?
>>> IAmericanSocket(uk)
<__main__.AdaptToAmericanSocket instance at 0x1a5120>
>>> IAmericanSocket(am)
<__main__.AmericanSocket instance at 0x36bff0>
with the code below:
from zope.interface import Interface, implementer
from twisted.python import components
class IAmericanSocket(Interface):
def voltage():
"""
Return the voltage produced by this socket object, as an integer.
"""
#implementer(IAmericanSocket)
class AmericanSocket:
def voltage(self):
return 120
class UKSocket:
def voltage(self):
return 240
#implementer(IAmericanSocket)
class AdaptToAmericanSocket:
def __init__(self, original):
self.original = original
def voltage(self):
return self.original.voltage() / 2
components.registerAdapter(
AdaptToAmericanSocket,
UKSocket,
IAmericanSocket)
You can see the full documentation for zope.interface here: http://docs.zope.org/zope.interface/ - it may provide a more thorough introduction than Twisted's quick tutorial.
To answer your specific question, the registerAdapter call at the end there changes the behavior of calling IAmericanSocket.
When you call an Interface, it first checks to see if its argument provides itself. Since the class AmericanSocket implements IAmericanSocket, instances of AmericanSocket provide IAmericanSocket. This means that when you call IAmercianSocket with an argument of an AmericanSocket instance, you just get the instance back.
However, when the argument does not provide the interface already, the interface then searches for adapters which can convert something that the argument does provide to the target interface. ("Searches for adapters" is a huge oversimplification, but Twisted's registerAdapter exists specifically to allow for this type of simplification.)
So when IAmericanSocket is called with an instance of a UKSocket, it finds a registered adapter from instances of UKSocket. The adapter itself is a 1-argument callable that takes an argument of the type being adapted "from" (UKSocket) and returns a value of the type being adapted "to" (provider of IAmericanSocket). AdaptToAmericanSocket is a class, but classes are themselves callable, and since its constructor takes a UKSocket, it fits the contract of thing-that-takes-1-argument-of-type-UKSocket-and-returns-an-IAmericanSocket.
The existence of another class would not make a difference, unless it were registered as an adapter. If you register two adapters which might both be suitable their interactions are complicated, but since they both do the job, in theory you shouldn't care which one gets used.
Case. I want to modify and add the following behavior to the code below (it's a context processor):
After checking if a user is authenticated check the last time the balance was updated (cookie maybe) if it was updated in the last 5 mins do nothing, else get the new balance as normal.
def get_balance(request):
if request.user.is_authenticated():
balance = Account.objects.get(user=request.user).balance
else:
balance = 0
return {'account_balance': balance}
HOWEVER:
I want to learn a little more about OOP in Django/Python can some modify the example to achieve my goal include the use of:
Property: I come from Java, I want to set and get, it makes more sense to me. get balance if does not exist else create new one.
Constructor method: In Python I think I have to change this to a class and use init right?
UPDATE:
To use a construct I first think I need to create a class, I'm assuming this is ok using as a context processor in Django to do something like this:
class BalanceProcessor(request):
_balance = Account.objects.get(user=request.user).balance
#property
def get_balance(self):
return return {'account_balance': _balance}
#setter???
Python is not Java. In Python you don't create classes for no reason. Classes are for when you have data you want to encapsulate with code. In this case, there is no such thing: you simply get some data and return it. A class would be of no benefit here whatsoever.
In any case, even if you do create a class, once again Python is not Java, and you don't create getters and setters on properties unless you actually need to do some processing when you get and set. If you just want to access an instance attribute, then you simply access it.
Finally, your proposed code will not work for two reasons. Firstly, you are trying to inherit from request. That makes no sense: you should inherit from object unless you are subclassing something. Secondly, how are you expecting your class to be instantiated? Context processors are usually functions, and that means Django is expecting a callable. If you give the class as the context processor, then calling it will instantiate it: but then there's nothing that will call the get_balance method. And your code will fail because Django will pass the request into the instantation (as it is expecting to do with a function) and your __init__ doesn't expect that parameter.
It's fine to experiment with classes in Python, but a context processor is not the place for it.
I have a POST method which calls a few tasklets. These tasklets do have yields in them, and I do have some x.put_async() in my code. So I don't want it to return before all the async stuff is done. So I decorated all my tasklets, which are just small functions with #ndb.tasklet. Also, on top of my POST method, I have:
#ndb.toplevel
def post(self):
However, in the documentation it states:
But if a handler method uses yield, that method still needs to be
wrapped in another decorator, #ndb.synctasklet; otherwise, it will
stop executing at the yield and not finish.
Indeed my method has a yield. It's already wrapped in #ndb.tasklet. Do I replace this with #ndb.synctasklet or do I use both (if so how would I use both)?
Also, see this thread which has some relevance. I too noticed an issue where my request would return without any output, but is un-reproducible. It happens every 15 minutes or so of constant use. I had app = ndb.toplevel(webapp2.WSGIApplication([..]) only, but now I've added #ndb.toplevel to the main POST methods, but the issue still persists.
Should I put #ndb.tasklet on top of methods that have just put_async()'s too? (Should I put it on top of every method just to be safe? What are the downsides to this?)
Regarding the handler and using #ndb.toplevel and #ndb.synctasklet:
The way I understood it was that you need to use both #ndb.synctasklet and #ndb.toplevel on the handler. All the sub-tasklets only need the #ndb.tasklet decorator. e.g.
class Foo(ndb.Model):
name = ndb.StringProperty()
#ndb.tasklet
def my_async(self):
....
#do something else that yields
raise ndb.Return("some result")
#ndb.toplevel
#ndb.synctasklet
def post(self):
foo = Foo(name="baz")
yield foo.put_async()
yield foo.my_async()
....
However. looking at the source, it appears that #ndb.toplevel is actually a synctasklet anyway:
def toplevel(func):
"""A sync tasklet that sets a fresh default Context.
Use this for toplevel view functions such as
webapp.RequestHandler.get() or Django view functions.
"""
Running a small test with yields in the handler and decorated with #ndb.toplevel still seems to work, and appears that you can remove #ndb.synctasklet from the handler.
Regarding whether you should include #ndb.tasklet on methods that call put_async():
If you're not yielding on the put_async(), then you don't need to include #ndb.tasklet on the surrounding method (#ndb.toplevel will handle getting the results from the put_async())