Develop Reference

How to define a "limit test context" using TDD?

How to I can define the "limit context" of my tests?
I say this because of mocks, where my service has many other libs integrated, like: rabbit-mq, redis, etc, and instances from some one classes. Then, the greather part of time I'm writing test code, creating "complex" mocks to the service be possible to test.
Is possible defines this limit? Should be a possible way "no test" theses "expensive" method, and just test the "simple related method", that are simple to input the paramters, like a sum(a, b).
Is obvious that the more test coverage the better, but is many, many time writing some tests with a questionable useful result.
When is using TDD to well defined methods, like the sum(a,b), it is very useful, but when this method will receive instances or use instance from another integrated services, the mocks receive the greather part of attemption, not the "method objective".
Imagine this service/method to test:
class ClientService:
def ok_here_is_the_discount(self, some_args):
# Ok, receive the discount to thi signature, now is possible
# calcula the it, and response to the broker
self.calculate_user_discount_based_on_signature_type(some_args.discount_value)
def calculate_user_discount_based_on_signature_type(self, some_args):
# here will send the "result" to the broker
some_message_borker_publisher(
some_args.value - some_args.discount_signature
)
def request_the_signature_type_discount_in_another_service(self, some_args):
# Ok, I receive the client that has the signature type,
# now is need request what is the value to this signature
# in signature service
some_message_borker_publisher(
queue='siganture.service.what_discount_for_this_signature',
signature_type=some_args.client.singature_type
)
#Ok, this message goes to the broker, and the signature.service receive it
def method_that_receive_messages(self, some_args):
# Here we receives that want to calculate discount
# but just receive the client(dict/class/instance) with the signature type,
# we still dont know the value of theses discount, because when is responsible
# to manage the siganture, is the signature server
if some_args.message_type == 'please_calculate_discount':
self.request_the_signature_type_discount_in_another_service(some_args.client)
if some_args.message_type == 'message_from_signature_discount':
self.ok_here_is_the_discount(some_args.value)
1 - it receive a message 'please_calculate_discount' and call the method
self.request_the_signature_type_discount_in_another_service(some_args.client)
But it still not have the discount value, because this is signature service. (message to the broker)
2 - The supose that the signature server response to 'message_from_signature_discount', then it call the method:
self.ok_here_is_the_discount(some_args.value)
3 - ok, the method ok_here_is_the_discount receive discount and call method
calculate_user_discount_based_on_signature_type()
That now has the values to calculate to the discount and send these value to the broker.
Understand the complexity of these tests (TDD)? I need tests the "method_that_receive_messages" mocking all related nexted actions, or just test the related method, like "calculate_user_discount_based_on_signature_type"?
In this case is better uses a really broker to be possible test it?

Well it is easy to mock things in Python but it still comes with some overhead of course. What I have tended towards over many years now is to set up integration tests with mocks that tests the happy path through the system and then design the system with side effect free functions as much as possible and throw unit tests at them. Then you can start your TDDing with setting up the overall test for the happy path and then unit test particulars in easy to test functions.
It is still useless when the thing behind the mock changes though but it gives a great sense of security when refactoring.
The mock libraries in Python are fine but sometimes it is easier to just write your own and replace the real thing with patch. Pretty sweet actually.

Is it possible to create a Tornado application from instances of Request/Web Handlers instead of class definitions?

The apparent requirement to provide class definitions instead of instances causes very difficult problems. I have two different classes and one of them needs a reference to the other
app = tornado.web.Application([
(r"/fusion.*", FusionListener),
(r"/admin.*", AdminListener),
])
. The AdminListener needs a reference to the FusionListener since there are internal items needing to be managed. Sending messages is an unacceptable additional complexity here. The current mechanism does not seem to afford that possibility.
What kind of pattern can get around this shortcoming in Tornado?

For my use-case there are both persistent and in-memory state. We have spark and postgres repositories for the former. For the latter I had already designed and written the application to have instance-level data structures. But I have gathered that instance attributes on Tornado launched RequestHandler / WebHandler subclasses are not persistent.
The latter wants to live in a class managing the state: but I am compelled to significantly redraw the boundaries due to this design ot Tornado. Instead it will be necessary to push everything to global variables. Few would argue this were a preferred design. I will be dumping tornado as soon as I can get the time.
Not sure what will be the alternative: I already reverted from cherrypy due to significant limitations of its own: here are a couple of my questions on it
404 for path served by Cherrypy
How to specify the listening server instances using cherrypy tree.mount?
I got through those with some scars but still in one piece. There were additional issues that knocked me out: url's were not being served, and there was no clear end to the mole whacking. It also generally does not get a lot of attention and has confusing outdated or incomplete documentation. There is plenty of docs - that's why I got started on it: but the holes make for a series of rabbit-chasing episodes.
Flask and django have their own issues. It seems finding a functionally adequate but not super heavy weight web server in python is an illusory target. Not certain yet which framework has the least gotchas.

Posting this as answer in order to benefit from proper code formatting.
The paradigm I used for keeping track of existing instances of a RequestHandler is very simple:
class MyHandler(RequestHandler):
_instances = set()
def get(self):
if needs_to_be_added(self.request): # some conditions can be used here
if len(MyHandler._instances) > THRESHOLD: # careful with memory usage
return self.finish("some_error")
MyHandler._instances.add(self)
...
def post(self):
if needs_to_be_removed(self.request):
MyHandler._instances.discard(self)
...
Of course you might need to change when to add / discard elements.
Depending on how you want to refer to existing instances in the future (by some key for example) you could use a dict for keeping track of them.
I don't think you can use something weak references (as in classes from the weakref module) because those will only track live instances which won't work with the way request handlers instances are created and destroyed.

User authentication: prepare vs get_current_user in tornado

I need to authenticate a user from a cookie in an application running on Tornado. I need to parse the cookie and load the user from the DB using the cookie content. On checking out the Tornado RequestHandler documentation, there are 2 ways of doing it:
by overriding prepare() method of RequestHandler class.
by overriding get_current_user() method of RequestHandler class.
I'm confused with the following statement:
Note that prepare() may be a coroutine while get_current_user() may
not, so the latter form is necessary if loading the user requires
asynchronous operations.
I don't understand 2 things in it:
What does the doc mean by saying that get_current_user() may not be a coroutine? What does may not mean here? Either it can be a coroutine, or it can't.
Why is the latter form, i.e. get_current_user(), required if async operation is required? If prepare() can be a coroutine and get_current_user() may not, then shouldn't prepare() be used for async operations?
I would really appreciate any help with this.

Here, "may not be a coroutine" means "is not allowed to be a coroutine" or "must not be a coroutine". The language used is confusing and it should probably be changed to say "must not".
Again, the docs are confusing: in this sentence prepare() is mentioned first, but before this sentence are two examples and get_current_user is first. "Latter" refers to the second example which uses prepare().
So in summary, it always works to override prepare() and set self.current_user, whether you need a coroutine or not. If you don't need a coroutine to get the current user, you can override get_current_user() instead and it will be called automatically the first time self.current_user is accessed. It doesn't really matter which one you choose; you can use whichever feels more natural to you. (The reason we have two different methods is that get_current_user() is older but we had to use a different method for coroutines)

1
The recommended way of getting current user is to use RequestHandler.current_user property. This property is actually a function, it returns RequestHandler._current_user if set, otherwise it tries to set it with call of get_current_user.
Because of current_user is a property - it can't be yielded, therefore get_current_user can't be coroutine function.
Of course you could, read cookie and call db, authenticate user in get_current_user, but only in blocking (sync) manner.
2
In the the doc you quoted, the latter example is the one with prepare.

"Local" publish-subscriber pattern for use in MVC wxpython

Using wxpython in MVC, I looked for a way to let the models tell the controllers about changes. I found (py)pubsub, which implements a global notification mechanism: Messages are sent to one place (the pubsub Publisher), which sends them to all subscribers. Each subscriber checks whether the message is interesting, and does what is needed.
From Smalltalk times, I know a more "local" approach: Each model object keeps a list of interested controllers, and only sends change notifications to these. No global publisher is involved. This can be implemented as part of the Model class, and works in much the same way, except it's local to the model and the controller.
Now is there a reason to use the global approach (which seems much less performant to me, and might be prone to all the issues related to global approaches)? Is there another package implementing a local observer?
Thanks!

I'm not really seeing the subtle difference here. As far as I know, pubsub is the way to go. It's included in wxPython in wx.lin.pubsub or you can download it from http://pubsub.sourceforge.net/. You can put the listeners just in the models and the publisher(s) just in the controller or however you need to. Here are a couple links to get you started:
http://www.blog.pythonlibrary.org/2010/06/27/wxpython-and-pubsub-a-simple-tutorial/
http://wiki.wxpython.org/WxLibPubSub

I've been playing around for a while to do MVC with wxpython and i know what you mean about pubsub being global.
The latest idea i've come up with is each view and model have there own observer.
The observers have weak references to their handlers and it all works in a separate thread so as to not block the GUI. To call back to the GUI thread I'm using wxAnyThread Gui method decorator.
There are 3 types of signal that get sent, for the model you can set which attributes are observed they automatically send out a signal when they are changed. then on both the model and the view you can send a message signal or a keyword signal. Each of the three signal types have to be unique per view or model as they are used to make a tuple that identify them.
model attributes
controller handlers are decorated with
#onAttr('attributeName')
def onModelAttributeName(self, attributeName)
When you bind to a method that handlers attributes it straight away calls the handler with its current value and then continues to observe changes.
Sending messages
Use the method
view/model.notify('Your message'):
The controller callback is decorated with
#onNotify('Your message')
def onYourMessage(self):
Sending keywords
Use the method
view/model.notifyKw(valid=True, value='this)
The controller callback is decorated with
#onNotifyKw('valid', 'value')
def onValidValueKw(self, valid, value)
The GUI is left knowing nothing about the models the only thing you add to the GUI is the view signaler, the controller attaches it self to this, so if you don't add a controller the view will just happily fire off messages to no one.
I've uploaded what i have so far on github
https://github.com/Yoriz/Y_Signal
https://github.com/Yoriz/Y_Mvc
Both have unit test which should give a bit of an example of what it does, but i will create some wxpython examples.
I'm using python version 2.7 and the Ysignals module requires
https://pypi.python.org/pypi/futures/2.1.3 for the threading.
Please take a look ill be interested in what someone else thinks of this way of approaching mvc or to point out something i seriously overlooked.

Adding REST behavior to a class with flask, case for blueprints?

I am dealing with a python application that consists of multiple distributed lightweight components that communicate using RabbitMQ & Kombu.
A component listens on two queues and can receive multiple message types on each queue. Subclasses can override how each message type is processed by registering custom handlers.
All this works fine.
I now have the added requirement that each component must have a basic REST/HTML interface. The idea being you point your browser at the running component and get realtime information on what it is currently doing (what messages it is processing, cpu usage, state info, log, etc.)
It needs to be lightweight, so after some research I have settled on Flask (but am open to suggestions). In pseudocode this means taking:
class Component:
Queue A
Queue B
...
def setup(..):
# connect to the broker & other initialization
def start(..):
# start the event loop and wait for work
def handle_msg_on_A(self,msg):
# dispatch a msg to a handler depending on the msg type
def handle_msg_on_B(self,msg):
...
...
and adding a number of view methods:
#app.route('/')
def web_ui(self):
# render to a template
#app.route('/state')
def get_state(self):
# REST method to return some internal state info as JSON
...
However, bolting a web UI onto a class like this breaks SOLID principles and brings problems with inheritance (a subclass may want to display more/less information). Decorators are not inherited so every view method would need to be explicitly overridden and redecorated. Maybe using a mixin + reflection could work somehow but it feels hackish.
Instead, using composition could work: put the web stuff in a separate class that delegates the url routes to a fixed, predefined set of polymorphic methods on the nested component.
This way components remain unaware of Flask at the cost of some loss in flexibility (the set of available methods is fixed).
I have now discovered Flask blueprints and Application Dispatching and it looks like they could bring a better, more extensible solution. However, I have yet to wrap my head around them.
I feel like I am missing a design pattern here and hopefully somebody with more flask-fu or experience with this type of problem can comment.

Something else was quietly introduced in Flask 0.7 that might be of interest to you - Pluggable Views. These are class based rather than function based endpoints - so you can use the dispatch_request method to manage your state transitions (only overriding it when needed).
The benefit of doing it this way, as opposed to using Application Dispatching, is that you get url_for support all across your application (as opposed to having to hard code in URLs that cross application boundaries.) You'll have to decide if this is something that is likely to be an issue for your application.
In pseudo-code:
# File: Components.py
from flask.views import View
class Component(View):
# Define your Component-specific application logic here
dispatch_request(self, *url_args, **url_kwargs):
# Define route-specific logic that all Components should have here.
# Call Component-specific methods as necessary
class Tool_1(Component):
pass
class Tool_2(Component):
# Override methods here
# File: app.py
from flask import Flask
from yourapplication import Tool_1, Tool_2
app = Flask()
# Assuming you want to pass all additional parameters as one argument
app.add_url_rule("/tool_1/<path:options>", "tool1", view_func=Tool_1.as_view())
# Assuming you want to pass additional parameters separately
tool_2_view = Tool_2.as_view()
app.add_url_rule("/tool_2/", "tool2", view_func=tool_2_view )
app.add_url_rule("/tool_2/<option>", "tool2", view_func=tool_2_view)
app.add_url_rule("/tool_2/<option>/<filter>", "tool2", view_func=tool_2_view)
You can add blueprints to the mix if you have a series of components that are all logically connected together and you don't want to have to remember to put /prefix in front of each one's add_url_rule call. But if you just have a series of components that are mostly independent of each other, this is the pattern I'd use*.
*. On the other hand, if they need to be isolated from each other I'd use the Application Dispatch pattern recommended in the docs.