Develop Reference

Receiving "'hmset' with mapping of length 0" error

I want to store my session data on redis dataset. I have set SESSION_ENGINE = 'redis' in settings.py.
Code for redis.py
#redis.py
from django.contrib.sessions.backends.base import SessionBase
from django.utils.functional import cached_property
from redis import Redis
class SessionStore(SessionBase):
#cached_property
def _connection(self):
return Redis(
host='127.0.0.1',
port='6379',
db=0,
decode_responses=True
)
def load(self):
return self._connection.hgetall(self.session_key)
def exists(self, session_key):
return self._connection.exists(session_key)
def create(self):
# Creates a new session in the database.
self._session_key = self._get_new_session_key()
self.save(must_create=True)
self.modified = True
def save(self, must_create=False):
# Saves the session data. If `must_create` is True,
# creates a new session object. Otherwise, only updates
# an existing object and doesn't create one.
if self.session_key is None:
return self.create()
data = self._get_session(no_load=must_create)
session_key = self._get_or_create_session_key()
self._connection.hmset(session_key, data)
self._connection.expire(session_key, self.get_expiry_age())
def delete(self, session_key=None):
# Deletes the session data under the session key.
if session_key is None:
if self.session_key is None:
return
session_key = self.session_key
self._connection.delete(session_key)
#classmethod
def clear_expired(cls):
# There is no need to remove expired sessions by hand
# because Redis can do it automatically when
# the session has expired.
# We set expiration time in `save` method.
pass
I am receiving 'hmset' with mapping of length 0 error on accessing http://localhost:8000/admin in django.
After removing SESSION_ENGINE='redis' I am not receiving this error.

From redis documentation:
As per Redis 4.0.0, HMSET is considered deprecated. Please use HSET in new code.
I have replaced this line in save() method:
self._connection.hmset(session_key, data)
with:
self._connection.hset(session_key, 'session_key', session_key, data)
On making the change, it works as expected.

How to "inject" in-memory SQLAlchemy sqlite3 database into Flask test_client?

I have a Flask application that I am using with SQLAlchemy. I don't want to use the Flask-SQLAlchemy extension. If I use a "real" database for testing and just set up environment variables that point to the test instance of the database, everything works well.
However, I've run into a problem if I want to point to an in-memory sqlite database for testing. In that case I can set up the data in my test, but when I use the test_client to execute a given route in my application, it can't find my database tables on the "server side" as it were - i.e. in the hello.py code shown further down. There must be a way to configure the test_client for this to work, but I can't seem to quite figure out how to do it.
Here are the snippets of code that may be relevant (db.py):
import os
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session
from sqlalchemy.orm import sessionmaker
engine = create_engine(os.environ['SQLALCHEMY_URL'])
Session = scoped_session(sessionmaker(bind=engine))
Here I am setting up the scoped_session so that my database access will be thread-local.
The code that bootstraps my application (__init__.py):
from flask import Flask
from .db import Session
from .hello import hello_blueprint
app = Flask(__name__)
app.register_blueprint(hello_blueprint)
#app.teardown_appcontext
def cleanup(resp_or_exc):
Session.remove()
Here I'm setting up my app and registering the cleanup callback each time Flask pops the application context.
An example route in a blueprint (hello.py):
import json
from flask import Blueprint
from .db import Session
from .models import Message
hello_blueprint = Blueprint('hello', __name__)
#hello_blueprint.route('/messages')
def messages():
values = Session.query(Message).all()
results = []
for value in values:
results.append({ 'message': value.message })
return (json.dumps(results), 200, { 'content_type': 'application/json' })
Here I'm using the scoped session to get some data from the database.
The definition of my Message model is just vanilla SQLAlchemy (models.py):
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, String
Base = declarative_base()
class Message(Base):
__tablename__ = 'messages'
id = Column(Integer, primary_key=True)
message = Column(String)
def __repr__(self):
return "<Message(message='%s')>" % (self.message)
Below is a very raw pytest unit test just to demonstrate the problem all in one place (test_hello.py):
import os
import json
import pytest
import app
from .models import Message
#pytest.fixture
def client():
client = app.app.test_client()
return client
def test_hello(client):
response = client.get('/')
data = json.loads(response.data.decode('utf-8'))
assert data == { 'message': "Hello friend!" }
def test_messages(client):
with app.app.app_context():
from sqlalchemy import create_engine
from sqlalchemy import MetaData
engine = create_engine('sqlite://')
from .models import Base
Base.metadata.create_all(engine)
print('***metadata tables***')
print(Base.metadata.tables.keys())
from sqlalchemy.orm import scoped_session
from sqlalchemy.orm import sessionmaker
Session = scoped_session(sessionmaker(bind=engine))
message = Message(message='Hello there!')
Session.add(message)
Session.commit()
values = Session.query(Message).all()
results = []
for value in values:
results.append({ 'message': value.message })
# This works, prints : [{"message": "Hello there!"}]
print('*** result***')
print(json.dumps(results))
# The code below doesn't work. Flask's app.py throws an exception
# with the following at its root:
# sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table:
# messages [SQL: 'SELECT messages.id AS messages_id, messages.message AS messages_message, messages.new_field
# AS messages_new_field \nFROM messages'] (Background on this error at: http://sqlalche.me/e/e3q8)
response = client.get('/messages')
data =json.loads(response.data.decode('utf-8'))
assert data == [{'message': 'Hello there!'}]

I'm not quite sure what I was doing wrong before, but I'm managed to get my test passing.
The modified version of the unit test is below (test_hello.py):
import os
import json
import pytest
import app
from .db import engine
from .db import Session
from .models import Base
from .models import Message
#pytest.fixture
def client():
Base.metadata.drop_all(engine)
Base.metadata.create_all(engine)
client = app.app.test_client()
return client
def test_messages(client):
message = Message(message='Hello there!')
Session.add(message)
Session.commit()
response = client.get('/messages')
data = json.loads(response.data.decode('utf-8'))
assert data == [{'message': 'Hello there!'}]

How to separate Master Slave (DB read / writes) in Flask Sqlalchemy

I'm trying to separate the Read and write DB operations via Flask Sqlalchemy. I'm using binds to connect to the mysql databases. I would want to perform the write operation in Master and Reads from slaves. There does not seem to be an in built way to handle this.
I'm new to python and was surprised that a much needed functionality like this is not pre-built into flask-sqlalchemy already. Any help is appreciated. Thanks

There is no official support, but you can customize Flask-SQLalchemy session to use master slave connects
from functools import partial
from sqlalchemy import orm
from flask import current_app
from flask_sqlalchemy import SQLAlchemy, get_state
class RoutingSession(orm.Session):
def __init__(self, db, autocommit=False, autoflush=True, **options):
self.app = db.get_app()
self.db = db
self._bind_name = None
orm.Session.__init__(
self, autocommit=autocommit, autoflush=autoflush,
bind=db.engine,
binds=db.get_binds(self.app),
**options,
)
def get_bind(self, mapper=None, clause=None):
try:
state = get_state(self.app)
except (AssertionError, AttributeError, TypeError) as err:
current_app.logger.info(
'cant get configuration. default bind. Error:' + err)
return orm.Session.get_bind(self, mapper, clause)
# If there are no binds configured, use default SQLALCHEMY_DATABASE_URI
if not state or not self.app.config['SQLALCHEMY_BINDS']:
return orm.Session.get_bind(self, mapper, clause)
# if want to user exact bind
if self._bind_name:
return state.db.get_engine(self.app, bind=self._bind_name)
else:
# if no bind is used connect to default
return orm.Session.get_bind(self, mapper, clause)
def using_bind(self, name):
bind_session = RoutingSession(self.db)
vars(bind_session).update(vars(self))
bind_session._bind_name = name
return bind_session
class RouteSQLAlchemy(SQLAlchemy):
def __init__(self, *args, **kwargs):
SQLAlchemy.__init__(self, *args, **kwargs)
self.session.using_bind = lambda s: self.session().using_bind(s)
def create_scoped_session(self, options=None):
if options is None:
options = {}
scopefunc = options.pop('scopefunc', None)
return orm.scoped_session(
partial(RoutingSession, self, **options),
scopefunc=scopefunc,
)
Than the default session will be master, when you want to select from slave you can call it directly, here the examples:
In your app:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql:///master'
app.config['SQLALCHEMY_BINDS'] = {
'slave': 'postgresql:///slave'
}
db = RouteSQLAlchemy(app)
Select from master
session.query(User).filter_by(id=1).first()
Select from slave
session.using_bind('slave').query(User).filter_by(id=1).first()

Here is the documentation http://packages.python.org/Flask-SQLAlchemy/binds.html

Is having multiple SQLAlchemy sessions in the same controller okay, or should I put them all into one session?

So I have a controller that renders a page. In the controller, I call multiple functions from the model that create its own sessions. For example:
def page(request):
userid = authenticated_userid(request)
user = User.get_by_id(userid)
things = User.get_things()
return {'user': user, 'things': things}
Where in the model I have:
class User:
...
def get_by_id(self, userid):
return DBSession.query(User)...
def get_things(self):
return DBSession.query(Thing)...
My question is, is creating a new session for each function optimal, or should I start a session in the controller and use the same session throughout the controller (assuming I'm both querying as well as inserting into the database in the controller)? Ex.
def page(request):
session = DBSession()
userid = authenticated_userid(request)
user = User.get_by_id(userid, session)
things = User.get_things(session)
...
return {'user': user, 'things': things}
class User:
...
def get_by_id(self, userid, session=None):
if not session:
session = DBSession()
return session.query(User)...
def get_things(self, session=None):
if not session:
session = DBSession()
return session.query(Thing)...

Your first code is OK, if your DBSession is a ScopedSession. DBSession() is not a constructor then, but just an accessor function to thread-local storage. You might speed up things a bit by passing the session explicitly, but premature optimization is the root of all evil.

Unable to test persistent sessions in Django 1.1

As part of the registration process for my site I have several views that set session data. Later views depend on the session data being set. This all works fine in the browser, however when I try to test it the session data seems to be lost in between requests which makes it impossible to test. Here's a simple example of the problem I'm having. I would expect get_name to have access to session['name'] and return a 200, however the session data is being lost and get_name returns a 302.
>>> c = Client()
>>> r = c.post(reverse(set_name))
>>> r = c.post(reverse(get_name))
>>> r.status_code
200
def set_name(request):
request.session['name'] = 'name'
return HttpResponse()
def get_name(request):
try:
name = request.session['name']
except KeyError:
return redirect(reverse(set_name))
return HttpResponse(name)

Sessions are tested quite awkwardly in Django. You have to setup the session engine first.
class TestSession(TestCase):
"""A class for working with sessions - working.
http://groups.google.com/group/django-users/browse_thread/thread/5278e2f2b9e6da13?pli=1
To modify the session in the client do:
session = self.client.session
session['key'] = 'value'
session.save()
"""
def setUp(self):
"""Do the session preparation magic here"""
super(TestSession, self).setUp()
from django.conf import settings
from django.utils.importlib import import_module
engine = import_module(settings.SESSION_ENGINE)
store = engine.SessionStore()
store.save() # we need to make load() work, or the cookie is worthless
self.client.cookies[settings.SESSION_COOKIE_NAME] = store.session_key
Use this class as a base class for your test cases. Check out the link for more info.