Develop Reference

Reusing a single session for routing connections with SQLAlchemy between master & read replicas

We needed to route our database requests to either a writer master database or a set of read replicas.
We found a blog post by Mike Bayer suggesting how to do so using SQLAlchemy. We replicated the solution but that did not work out with our existing tests due to various reasons.
We went on with the following below. This will reuse one session rather than creating new ones that will stack altogether:
class ExplicitRoutingSession(SignallingSession):
_name = None
def get_bind(self, mapper=None, clause=None):
# If reader and writer binds are not configured,
# connect using the default SQLALCHEMY_DATABASE_URI
if not self.binds_setup:
return super().get_bind(mapper, clause)
return self.load_balance(mapper, clause)
def load_balance(self, mapper=None, clause=None):
# Use the explicit name if present
if self._name and not self._flushing:
bind = self._name
self._name = None
self.app.logger.debug(f"Connecting -> {bind}")
return get_state(self.app).db.get_engine(self.app, bind=bind)
# Everything else goes to the writer engine
else:
self.app.logger.debug("Connecting -> writer")
return get_state(self.app).db.get_engine(self.app, bind='writer')
def using_bind(self, name):
self._name = name
return self
#cached_property
def binds_setup(self):
binds = self.app.config['SQLALCHEMY_BINDS'] or {}
return all([k in binds for k in ['reader', 'writer']])
So far it works good for us. We assume we might lose some functionality such as db save points by not having stacked sessions... but we'd like to know if there are stability and unforeseen risks other than losing features with such an approach?
Notes:
We are also using flask-sqlalchemy.
This is from an open source notification platform and you can browse the code/branch yourself.

BDD behave Python need to create a World map to hold values

I'm not too familiar with Python but I have setup a BDD framework using Python behave, I now want to create a World map class that holds data and is retrievable throughout all scenarios.
For instance I will have a world class where I can use:
World w
w.key.add('key', api.response)
In one scenario and in another I can then use:
World w
key = w.key.get('key').
Edit:
Or if there is a built in way of using context or similar in behave where the attributes are saved and retrievable throughout all scenarios that would be good.
Like lettuce where you can use world http://lettuce.it/tutorial/simple.html
I've tried this between scenarios but it doesn't seem to be picking it up
class World(dict):
def __setitem__(self, key, item):
self.__dict__[key] = item
print(item)
def __getitem__(self, key):
return self.__dict__[key]
Setting the item in one step in scenario A: w.setitem('key', response)
Getting the item in another step in scenario B: w.getitem('key',)
This shows me an error though:
Traceback (most recent call last):
File "C:\Program Files (x86)\Python\lib\site-packages\behave\model.py", line 1456, in run
match.run(runner.context)
File "C:\Program Files (x86)\Python\lib\site-packages\behave\model.py", line 1903, in run
self.func(context, *args, **kwargs)
File "steps\get_account.py", line 14, in step_impl
print(w.__getitem__('appToken'))
File "C:Project\steps\world.py", line 8, in __getitem__
return self.__dict__[key]
KeyError: 'key'
It appears that the World does not hold values here between steps that are run.
Edit:
I'm unsure how to use environment.py but can see it has a way of running code before the steps. How can I allow my call to a soap client within environment.py to be called and then pass this to a particular step?
Edit:
I have made the request in environment.py and hardcoded the values, how can I pass variables to environment.py and back?

It's called "context" in the python-behave jargon. The first argument of your step definition function is an instance of the behave.runner.Context class, in which you can store your world instance. Please see the appropriate part of the tutorial.

Have you tried the
simple approach, using global var, for instance:
def before_all(context):
global response
response = api.response
def before_scenario(context, scenario):
global response
w.key.add('key', response)
Guess feature can be accessed from context, for instance:
def before_feature(context, feature):
feature.response = api.response
def before_scenario(context, scenario):
w.key.add('key', context.feature.response)
You are looking for:
Class variable: A variable that is shared by all instances of a class.
Your code in Q uses Class Instance variable.
Read about: python_classes_objects
For instance:
class World(dict):
__class_var = {}
def __setitem__(self, key, item):
World.__class_var[key] = item
def __getitem__(self, key):
return World.__class_var[key]
# Scenario A
A = World()
A['key'] = 'test'
print('A[\'key\']=%s' % A['key'] )
del A
# Scenario B
B = World()
print('B[\'key\']=%s' % B['key'] )
Output:
A['key']=test
B['key']=test
Tested with Python:3.4.2
Come back and Flag your Question as answered if this is working for you or comment why not.

Defining global var in before_all hook did not work for me.
As mentioned by #stovfl
But defining global var within one of my steps worked out.
Instead, as Szabo Peter mentioned use the context.
context.your_variable_name = api.response
and just use
context.your_variable_name anywhere the value is to be used.

For this I actually used a config file [config.py] I then added the variables in there and retrieved them using getattr. See below:
WSDL_URL = 'wsdl'
USERNAME = 'name'
PASSWORD = 'PWD'
Then retrieved them like:
import config
getattr(config, 'USERNAME ', 'username not found')

#EndpointsAliasProperty and #Model.query_method causes BadRequestError(Key path element must not be incomplete:...)

Hey so right now I'm developing backend api using Google ProtoRPC and Endpoints. I'm using the endpoints-proto-datastore library.
So strange things happen here, here is the EndpointsModel class
class AssetData(EndpointsModel):
type = msgprop.EnumProperty(AssetType, indexed=True)
def auth_id_set(self, value):
if ApplicationID.get_by_id(value) is None:
raise endpoints.UnauthorizedException('no auth_id')
self._auth_id = value
#EndpointsAliasProperty(required=True, setter=auth_id_set, property_type=messages.IntegerField)
def auth_id(self):
return self._auth_id
def app_id_set(self, value):
if ApplicationID.query(ApplicationID.app_id == value).get() is None:
raise endpoints.UnauthorizedException('wrong app_id')
self._app_id = value
if self.check_auth_app_id_pair(self.auth_id, value):
self._app_id = value
else:
raise endpoints.BadRequestException('auth_id and app_id mismatch')
#EndpointsAliasProperty(required=True, setter=app_id_set)
def app_id(self):
return self._app_id
#staticmethod
def check_auth_app_id_pair(authen_id, applic_id):
dat = ApplicationID.get_by_id(authen_id)
if dat.app_id != applic_id:
return False
else:
return True
and this is the API class
#endpoints.api(...)
class AssetDatabaseAPI(remote.Service):
#AssetData.query_method(query_fields=('limit', 'order', 'pageToken', 'type', 'auth_id', 'app_id'),
path='assets', http_method='GET', name='assets.getAssetMultiple')
def assets_get_multiple(self, query):
return query
When I deploy this, everytime I tried to access assets.getMultipleAssets it just gives me this error
raised BadRequestError(Key path element must not be incomplete: [ApplicationID: ]). Strangely enough this only happen to method using #Model.query_method, I have other methods using the same system but using #Model.method and it just runs ok.
If I tried it in development server, sometimes it just gives me RuntimeError: BadRequestError('missing key id/name',) then if I just re-save the .py file and retry it, it will work (sometimes not and another re-save can also make the error happens again).
Can anyone tell me my mistake?
Thanks

I think your problem is how you call this method - it's a static method, so you have to access it through class, not the instance (self):
if AssetData.check_auth_app_id_pair(self.auth_id, value):
self._app_id = value
else:
raise endpoints.BadRequestException('auth_id and app_id mismatch')

BLE subscribe to notification using gatttool or bluepy

I am writing a program using bluepy that listen for a characteristic sent by a bluetooth device. I can also use any library or language, the only constraint is to run on Linux and not in mobile environment (it seems is widely used only in mobile devices, no one use BLE with desktop).
Using bluepy I register the delegate and after trying to register for notification calling write('\x01\x00') as described in the bluetooth rfc.
But it doesn't work, any notification for the characteristic is received.
Maybe I am wrong in writing the message for subscribing.
Is there an error in the small snippet I wrote? Thank you so much.
class MyDelegate(btle.DefaultDelegate):
def __init__(self, hndl):
btle.DefaultDelegate.__init__(self)
self.hndl=hndl;
def handleNotification(self, cHandle, data):
if (cHandle==self.hndl):
val = binascii.b2a_hex(data)
val = binascii.unhexlify(val)
val = struct.unpack('f', val)[0]
print str(val) + " deg C"
p = btle.Peripheral("xx:xx:xx:xx", "random")
try:
srvs = (p.getServices());
chs=srvs[2].getCharacteristics();
ch=chs[1];
print(str(ch)+str(ch.propertiesToString()));
p.setDelegate(MyDelegate(ch.getHandle()));
# Setup to turn notifications on, e.g.
ch.write("\x01\x00");
# Main loop --------
while True:
if p.waitForNotifications(1.0):
continue
print "Waiting..."
finally:
p.disconnect();

I was struggling with this myself, and jgrant's comment really helped. I'd like to share my solution, if it could help anyone.
Note that I needed indication, hence the x02 rather than x01.
If it were possible to read the descriptors using bluepy, I would do that, but it doesn't seem to work (bluepy v 1.0.5). The method in the service class appears to be missing, and the method in the peripheral class gets stuck when I try to use it.
from bluepy import btle
class MyDelegate(btle.DefaultDelegate):
def __init__(self):
btle.DefaultDelegate.__init__(self)
def handleNotification(self, cHandle, data):
print("A notification was received: %s" %data)
p = btle.Peripheral(<MAC ADDRESS>, btle.ADDR_TYPE_RANDOM)
p.setDelegate( MyDelegate() )
# Setup to turn notifications on, e.g.
svc = p.getServiceByUUID( <UUID> )
ch = svc.getCharacteristics()[0]
print(ch.valHandle)
p.writeCharacteristic(ch.valHandle+1, "\x02\x00")
while True:
if p.waitForNotifications(1.0):
# handleNotification() was called
continue
print("Waiting...")
# Perhaps do something else here

It looks like the problem is that you're trying to write \x01\x00 to the characteristic itself. You need to write it to the Client Characteristic Configuration descriptor that proceeds it (0x2902). The handle is likely 1 greater than the characteristic (but you may want to confirm by reading the descriptors).
ch=chs[1]
cccd = ch.valHandle + 1
cccd.write("\x01\x00")

What was confusing for me was that in https://ianharvey.github.io/bluepy-doc/notifications.html
the part that enabled the notifications was in comments, so it didn't look obligatory to me.
the bare minimum (given you know the MAC-adress already an you included everything and declared the Delegateclass) for me is
p1 = Peripheral(<MAC>)
ch1 = p1.getCharacteristics()[3]
p1.setDelegate(MyDelegate())
p1.writeCharacteristic(ch1.valHandle + 1, b"\x01\x00")
Note that I already knew I wanted to get notifications from characteristic#3.
Also, without the 'b'-bytesprefix infront of "\x0\x00", it wouldn't work for me.

bluepy classes docs and samples are crazy, and not complete. To get more details, just checkout bluepy source (it is not big and easy to read)
But, as starting point you can use this notifications code sample, working with from Heart Rate Service (tested on bluepy 1.3.0)
Don't forget to replace device MAC to your own in Peripheral!
from bluepy import btle
from bluepy.btle import AssignedNumbers
import binascii
class MyDelegate(btle.DefaultDelegate):
def __init__(self, handle):
btle.DefaultDelegate.__init__(self)
self.handle = handle
print "Created delegate for handle", self.handle
# ... more initialise here
def handleNotification(self, cHandle, data):
if(cHandle == self.handle):
print "handleNotification for handle: ", cHandle, "; Raw data: ", binascii.b2a_hex(data)
#Found somewhere. Not tested is this working, but leave here as decode example
#val = binascii.b2a_hex(data)
#val = binascii.unhexlify(val)
#val = struct.unpack('f', val)[0]
#print str(val) + " deg C"
print "Connecting..."
dev = btle.Peripheral("c8:2b:96:a3:d4:76")
try:
print "Device services list:"
for svc in dev.services:
print str(svc)
HRService = dev.getServiceByUUID(AssignedNumbers.heartRate)
print "HRService", HRService
print "HRService characteristics list: "
for char in HRService.getCharacteristics():
print "HRService char[", char.getHandle(), "]: ", char
HRMeasurementChar = HRService.getCharacteristics(AssignedNumbers.heart_rate_measurement)[0] #Notice! Check is characteristic found before usage in production code!
print "HRMeasurementChar", HRMeasurementChar, HRMeasurementChar.propertiesToString();
# Assign delegate to target characteristic
dev.setDelegate(MyDelegate(HRMeasurementChar.getHandle()));
# We need to write into org.bluetooth.descriptor.gatt.client_characteristic_configuration descriptor to enabe notifications
# to do so, we must get this descriptor from characteristic first
# more details you can find in bluepy source (def getDescriptors(self, forUUID=None, hndEnd=0xFFFF))
desc = HRMeasurementChar.getDescriptors(AssignedNumbers.client_characteristic_configuration);
print "desc", desc
print "Writing \"notification\" flag to descriptor with handle: ", desc[0].handle
dev.writeCharacteristic(desc[0].handle, b"\x01\x00")# Notice! Do not use [0] in production. Check is descriptor found first!
print "Waiting for notifications..."
while True:
if dev.waitForNotifications(1.0):
# handleNotification() was called
continue
finally:
dev.disconnect();

Python class __init__ layout?

In python, is it bad form to write an __init__ definition like:
class someFileType(object):
def __init__(self, path):
self.path = path
self.filename = self.getFilename()
self.client = self.getClient()
self.date = self.getDate()
self.title = self.getTitle()
self.filetype = self.getFiletype()
def getFilename(self):
'''Returns entire file name without extension'''
filename = os.path.basename(self.path)
filename = os.path.splitext(filename)
filename = filename[0]
return filename
def getClient(self):
'''Returns client name associated with file'''
client = self.filename.split()
client = client[1] # Assuming filename is formatted "date client - docTitle"
return client
where the initialized variables are calls to functions returning strings? Or is it considered lazy coding? It's mostly to save me from writing something.filetype as something.getFiletype() whenever I want to reference some aspect of the file.
This code is to sort files into folders by client, then by document type, and other manipulations based on data in the file name.

Nope, I don't see why that would be bad form. Calculating those values only once when the instance is created can be a great idea, in fact.
You could also postpone the calculations until needed by using caching propertys:
class SomeFileType(object):
_filename = None
_client = None
def __init__(self, path):
self.path = path
#property
def filename(self):
if self._filename is None:
filename = os.path.basename(self.path)
self._filename = os.path.splitext(filename)[0]
return self._filename
#property
def client(self):
'''Returns client name associated with file'''
if self._client is None:
client = self.filename.split()
self._client = client[1] # Assuming filename is formatted "date client - docTitle"
return self._client
Now, accessing somefiletypeinstance.client will trigger calculation of self.filename as needed, as well as cache the result of it's own calculation.
In this specific case, you may want to make .path a property as well; one with a setter that clears the cached values:
class SomeFileType(object):
_filename = None
_client = None
def __init__(self, path):
self._path = path
#property
def path(self):
return self._path
#path.setter
def path(self, value):
# clear all private instance attributes
for key in [k for k in vars(self) if k[0] == '_']:
delattr(self, key)
self._path = value
#property
def filename(self):
if self._filename is None:
filename = os.path.basename(self.path)
self._filename = os.path.splitext(filename)[0]
return self._filename
#property
def client(self):
'''Returns client name associated with file'''
if self._client is None:
client = self.filename.split()
self._client = client[1] # Assuming filename is formatted "date client - docTitle"
return self._client
Because property-based caching does add some complexity overhead, you need to consider if it is really worth your while; for your specific, simple example, it probably is not. The calculation cost for your attributes is very low indeed, and unless you plan to create large quantities of these classes, the overhead of calculating the properties ahead of time is negligible, compared to the mental cost of having to maintain on-demand caching properties.

Your code is doing two different things:
a) Simplifying the class API by exposing certain computed attributes as variables, rather than functions.
b) Precomputing their values.
The first task is what properties are for; a straightforward use would make your code simpler, not more complex, and (equally important) would make the intent clearer:
class someFileType(object):
#property
def filename(self):
return os.path.basename(self.path)
You can then write var.filename and you will dynamically compute the filename from the path.
#Martijn's solution adds caching, which also takes care of part b (precomputation). In your example, at least, the calculations are cheap so I don't see any benefit in doing so.
On the contrary, caching or precomputation raises consistency issues. Consider the following snippet:
something = someFileType("/home/me/document.txt")
print something.filename # prints `document`
...
something.path = "/home/me/document-v2.txt"
print something.filename # STILL prints `document` if you cache values
What should the last statement print? If you cache your computations, you will still get document instead of document-v2! Unless you are certain that nobody will try to change the value of the basic variable, you need to either avoid caching, or take measures to ensure consistency. The easiest way is to prohibit modifications to path-- one of the things that properties are designed to do.
Conclusion: Use properties to simplify your interface. Don't cache computations, unless it's necessitated by performance reasons. If you cache, take measures to ensure consistency, e.g. by making the underlying value read-only.
PS. The issues are analogous to database normalization (non-normalized designs raise consistency issues), but in python you have more resources for keeping things in sync.