Develop Reference

Elegant Way to Deal with Marshmallow Altering a SQLAlchemy Object

I have found myself in a situation that does not seem to have an elegant solution. Consider the following (pseudo-ish) REST API code
bp = Blueprint('Something', __name__, url_prefix='/api')
class SomeOutputSchema(ma.SQLAlchemyAutoSchema)
class Meta:
model = MyModel
#pre_dump(pass_many=False)
def resolveFilePath(self, ormObject, many):
# ormObject has a parent via a relationship
ormObject.filePath = os.path.join(ormObject.parent.rootDir, ormObject.filePath)
#bp.route("/someRoute")
class SomeClass(MethodView):
def put(self):
ormObject = MyModel(filePath = "/some/relative/path")
db.session.add(ormObject)
db.session.flush()
outputDump = SomeOutputSchema().dump(ormObject)
# Lots of other code that uses outputDump...
# Only commit here in case
# anything goes wrong above
db.session.commit()
return jsonify({"data": outputDump}), 201
I have
A PUT endpoint that will create a new resource, then return the dump of that resource.
An ORM object that has a filePath property. This must be stored as a relative path.
A Marshmallow schema. It has a #pre_dump method to resolve the file path by the use of another property (parent.rootDir)
So basically the process is
Create the new resource
Create a schema dump of that resource to use
Commit
Return the schema dump
So finally, the problem is: outputDump's #pre_dump actually alters ormObject, so that it is now a fully resolved path by the time db.session.commit() is called. My first instinct here was to create a deep copy of ormObject but that fails with
"Parent instance <MyModel at 0x7f31cdd44240> is not bound to a Session; lazy load operation of attribute 'parent' cannot proceed (Background on this error at: http://sqlalche.me/e/14/bhk3)"
It's not that this is a difficult thing to solve, but it seems to be difficult to solve elegantly with my current knowledge. I need the path to be relative for the database, and resolved otherwise.
My current solution is to tell the SomeOutputSchema to skip the #pre_dump in this case, then take the outputDump and then resolve the file paths just after the schema dump. But this feels really gross to me.
I would love to hear any ideas on this, as currently my code feels messy and I don't like the idea of just leaving it and pushing on.

Solved by using a #post_dump and using pass_original=True to get access to the original object
class SomeOutputSchema(ma.SQLAlchemyAutoSchema)
class Meta:
model = MyModel
#post_dump(pass_original=True)
def resolveFilePath(self, data, ormObject, many):
data['filePath'] = os.path.join(ormObject.parent.rootDir, ormObject.filePath)

GAE Python NDB Datastore - No need for memcache.set?

The documentation (https://cloud.google.com/appengine/docs/python/ndb/) states that
NDB uses Memcache as a cache service for "hot spots" in the data
I am now using memcache only as follows:
memcache.set(key=(id), value=params, time=0)
That expires (auto flushes) pretty often and so I would like to use NDB Datastore.
I thought I would have to always put the key-value in both NDB and Memcache, then check both.
Is this being done automatically by NDB?
Ie.
ancestor_key = ndb.Key("Book", guestbook_name or "*notitle*")
greetings = Greeting.query_book(ancestor_key).fetch(20)
Would that implicitly set Memcache ?
And when I read from NDB, would it implicitly try a memcache.get(key) first?
Thanks for your patience.
EDIT - What I tried:
As a test I tried something like this:
class Book(ndb.Model):
content = ndb.StringProperty()
class update(webapp2.RequestHandler):
def post(self):
p1='1'
p2='2'
p3='3'
p4='4'
p5='5'
id='test'
paramarray = (p1,p2,p3,p4,p5)
book = Book(name=id,value=paramarray)
# OR likes this - book = Book(ndb.Key(id),value=paramarray)
book.put()
Both versions error out.
Trying to get a key of the var id with the values of paramarray
EDIT 2 Daniel, Thank you for everything.
Have follow up formatting questions, will ask a new question.

Yes; see the full documentation on ndb caching. Basically, every write is cached both in a request-local in-context cache, and in the main memcached store; a get by key will look up in both caches first before falling back to the real datastore.
Edit I can't understand why you think your example would work. You defined a model with a content property, but then try to set name and value properties on it; naturally that will fail.
You should go through the ndb documentation, which gives a good introduction to using the model class.

JSONField getting saved as string django

I have a django model like below:
from jsonfield import JSONField
class SCUser(User):
address = JSONField(blank=True,null=True)
When I save a json in this address it gets saved as string.
Here is a code snippet:
appuser.address = {"state":""}
appuser.save()
Now if I try to retrieve appuser.address it gives me
>>>appuser.address
>>>u'{"state":""}'
>>>appuser.save()
>>>appuser.address
>>>u'"{\\"state\\":\\"\\"}"'
And it gets recursive.
What am I missing here?
Edit:
The AppUser inherits from SCUser model.

I met this problem when I am using a non-Autofield key as the model's primary key and I found some issues which is still open on github related to this problem.
https://github.com/dmkoch/django-jsonfield/issues/92
https://github.com/dmkoch/django-jsonfield/issues/101
I solved this problem by define a pk property in the model. I don't known is there any side effects by using this solution.
class SCUser(User):
....
#property
def pk(self):
return self.id # your pk

Please try:
appuser.address = {"state":""}
appuser.save()
appuser.get_data_json()

Django Unittesting adding a variable to the session

In my unittest I need to add a variable to the session, because that variable is used in the view which is being tested. The django documentation says this is possible in the following way (https://docs.djangoproject.com/en/1.3/topics/testing/#django.test.client.Client.session):
def test_something(self):
session = self.client.session
session['somekey'] = 'test'
session.save()
This code example actually doesn't work, because you will get the error that a dict doesn't have a function save. I also tried various other ways to change the contents inside the session dict, but haven't found a way to change it yet.
I know what it means, what i get back is a dict object so it doesnt have the save function. But the session dict also doesn't update when adding keys.
The documentation statues when using self.client.session it should return a SessionStore object instead of a dictionary.

Seems right now there is a bug in Django, so it doesn't work for unauthenticated users to change the session. This is the corresponding ticket: https://code.djangoproject.com/ticket/11475 .
A work around is to create a dummy view in which the session variables are set and calling that view with the Client.get('url_of_dummy_view').

The ticket referenced by #Sam Stoelinga has been closed as a duplicate. The corresponding ticket can be found here.
To save session data you will have to use an authenticated user. I found a solution here.
class BlogAdminTestCase(TestCase):
def setUp(self):
# Setup Test User
User.objects.create_user(
username='foo',
password='bar'
)
# Must login to modify session variables
self.client.login(username='foo', password='bar')
s = self.client.session
s['my_session_variable'] = 'Yay!'
s.save()

KindError in Google App Engine

I defined a simple class in GAE for keeping user profiles data like this:
class User(db.Model):
email = db.EmailProperty()
role = db.StringProperty(default=roles.USER)
first_name = db.StringProperty()
last_name = db.StringProperty()
...
I use memcache to keep session information. memcache data looks like this { 'key': 'agpjYW5kaXJhdGVzcgoLEgRVc2VyGCMM'}. I get session_id value from the cookie. When I try to get user info linked to that cookie like this:
session_id = request['session_id']
data = memcache.get(session_id)
user = User.get(data['key'])
I get KindError exception:
KindError: Kind 'User' is not a subclass of kind 'User'
I know this user exists, memcache exists. User class is defined only once in my project. Why this error occurs and how can I make it work?
UPDATE: I tried to use db.get() instead of User.get() and it worked. So, what's the problem there can be?

Model.get() does check whether the supplied key is of the correct kind, as defined in the documentation. If not of the correct kind it will throw a KindError.
db.get() does not do any type checking and therefore will succeed with the supplied value if it exists in the data store, but will not necessarily return a User entity.
So you need to check whether the key in your memcache is actually of the User kind. Are you sure it's not overwritten with the key of a different model at some point?

The App Engine framework defines a class called 'User' as part of the Users API. In addition, you have your own class by the same name. When the exception occurs, you're trying to use one, but getting the other.
To avoid this, rename your model. You should also be careful how you import modules in Python. Instead of:
from google.appengine.api.users import User
or worse:
from google.appengine.api.users import *
you should use:
from google.appengine.api import users
And then refer to users.User, which is unambiguous.

The problem, it seems to me, is more subtle than that. I was getting the error with this call to Model.get() (I'm retrieving a top-level singleton object, always there):
datastore = GDSDatastore.get(gds.Key.from_path(*path))
so I investigated with this code:
datastore = gds.get(gds.Key.from_path(*path))
if not(datastore is None or isinstance(datastore, GDSDatastore)):
logger.error("KindError isinstance(GDSDatastore)=%s class=%s" % (isinstance(datastore, GDSDatastore), datastore.__class__.__name__))
raise gds.KindError('Kind %r is not a GDSDatastore instance' %
(datastore.kind()))
The vast majority of the time I get no error, but today I got this interesting log:
KindError isinstance(GDSDatastore)=False class=GDSDatastore
Now, that strikes me as rather peculiar.
(Note: GDSDatastore is defined locally: class GDSDatastore(gds.Model))