Develop Reference

How to rollback in Google App Engine NDB after method fails

I have a project deployed on Google App Engine....I am facing rollback issue. I want to rollback just like Relational db where we do commit before return or end of method. How do we achieve this in ndb???
Please find below code snippet for clarity
class PersonTable(ndb.Model):
personId = ndb.StringProperty()
personName = ndb.StringProperty()
personAddress = ndb.StringProperty()
personOldReference = ndb.StringProperty()
scopeReference = ndb.StringProperty()
class ScopeTable(ndb.Model):
child = ndb.StringProperty()
isPerson = ndb.BooleanProperty()
parent = ndb.StringProperty()
#endpoints.method(ZGScopeRequest, ZGScopeResponse, name='scope', path='scope', http_method='POST')
def scope(self, request):
scope = request.scope
person = request.personName
list = scope.split('.')
length = len(list)
for entry in list:
newId = GenerateRandomId('SCOPE')
child = mobileappmodels.ScopeTable.query(mobileappmodels.ScopeTable.child.IN([entry])).get()
if child:
pass
else:
index = list.index(entry)
parent = list[index-1]
if entry == 'PersonName':
entry = person
fetchPerson = mobileappmodels.CommunityTable.gql("""WHERE personName = :1""",entry).fetch()
update = fetchCommunity[0].key.get()
update.scopeReference = newId
update.put()
save = mobileappmodels.ScopeTable(child=person, isCommunity = True, parent = parent, id=newId)
save.put()
elif entry == 'Global':
save = mobileappmodels.ScopeTable(child=entry, isCommunity = False, parent = '', id=newId)
save.put()
else:
save = mobileappmodels.ScopeTable(child=entry, isCommunity = False, parent = parent, id=newId)
save.put()
return ZGScopeResponse(message="Uploaded")
Suppose my function fails just before return and after 2nd put, how do I rollback???
Help would be really appreciate as I am new to Google App Engine and NDB

Not sure I understand. How could it fail after the last put and before the return?
Perhaps you want to run the code in a transaction:
https://cloud.google.com/appengine/docs/standard/python/ndb/transactions
You can run it in a transaction, and call ndb.Rollback if needed.

In Django, Why does my database manager fail?

I am learning managers for the first time.
I am trying to figure out how to set these managers correctly. It appears that managers are similar in philosophy as a SQL View.
Here is my manager and class:
class SubscriptionManager(models.Manager):
def active_officers(self):
officers = self.get_queryset().filter(Modified__gte=datetime.now()-timedelta(days=365)).filter(Subscription_Type='O')
return officers
def lifer(self):
return self.get_queryset().filter(Lifetime=True)
class Subscription(models.Model):
SType = (
('C', 'Civilian'),
('E', 'Enlisted'),
('O', 'Officer'),
)
Subscription_Type = models.CharField(max_length=1, choices=SType)
Lifetime = models.BooleanField(default=False)
Member = models.ForeignKey(Member)
Modified = models.DateField()
objects = SubscriptionManager()
def __str__(self):
return self.Member.first_name + " " + self.Member.last_name + ": " + self.Subscription_Type + "; last modified: " + self.Modified.strftime('%Y-%m-%d')
Once set I can still execute Subscription.objects.all() with no problems, bu if I fire off Subscription.objects.lifer(), I receive the following error: AttributeError: 'Manager' object has no attribute 'lifer'
I'm in python 3.4. What am I missing?
Thanks

Your code that you pasted above is correct and it works for me after copy and pasting.
Somewhere else in your code you have it listed as an attribute I believe Subscription.objects.lifer instead of the method Subscription.objects.lifer() Give that a search and I believe that should clear your error.

Django database: sorting by field in latest object in relative database

I have two db tables in django: "Device" table to store actual devices and "DeviceTest" table to store test checks for each device (Many Tests to one Device).
class Device(models.Model):
created_at = models.DateTimeField(auto_now_add=True)
name = models.CharField(max_length=20)
serial_number = models.CharField(max_length=20)
def __unicode__(self):
return self.name + " - " + self.serial_number
class DeviceTest(models.Model):
device = models.ForeignKey(Device)
created_at = models.DateTimeField(auto_now_add=True)
TEST_OK = '+'
TEST_ERROR = '-'
TEST_RESULT_CHOICES = (
(TEST_OK, 'Test Success'),
(TEST_ERROR, 'Test Error'),
)
status = models.CharField(max_length=1, choices=TEST_RESULT_CHOICES, default=TEST_ERROR)
comment = models.TextField(blank=True, default="")
tester = models.CharField(max_length=255)
action = models.CharField(max_length=255)
def __unicode__(self):
return self.device.name + " - " + \
self.device.serial_number + " - " + \
unicode(self.created_at) + " - " + \
u"Result (" + self.status + ")"
I've made HTML page with table of all devices in format:
name | serial number | created_at | last_test_time | last_test_status | hyperlink to test table page
Now I want to give user all sorting possibilities. I've done that through GET parameters for "name", "serial number" and "created_at" fields. And now I'm stuck with "last_test_time" and "last_test_status" fields.
Can this problem be solved with some standard Django functions? Something like Device.objects.all().filter(???).order_by(???)

You can annotate the queryset with the latest created_at time.
from django.db.models import Max
queryset = Device.objects.annotate(last_test_time=Max('devicetest__created_at'))
You can access last_test_time for each object.
for obj in queryset
print(obj.last_test_time)
And you can order the queryset by this field
queryset = queryset.order_by('last_test_time')
However, this is only a partial answer, because it does not help you get the status of that test.
You could do a lookup for each device individually.
for device in queryset:
device.latest_status = device.device_test.latest('created_at').status
This is ok if the number of devices is small, but it means one query for each device, which isn't ideal as the number of devices increases.
You can use prefetch_related to fetch all the related device tests in one query. (the code below is untested, I might have the syntax slightly wrong).
queryset = queryset.prefetch_related(Prefetch('devicetest', DeviceTest.objects.order_by('-created_at'))
You can then get the status via the first related device test.
for obj in queryset:
print obj.devicetest.all()[0].status
Note this is inefficient, because you have loaded all related items, not just the most recent one.

Django creating model instances without keyword arguments

In Django, if you want to create a model you can do it as follows:
class SomeModel(models.Model):
a = models.CharField(max_length=10)
b = models.CharField(max_length=20)
c = models.CharField(max_length=30)
d = models.CharField(max_length=40)
e = models.CharField(max_length=50)
f = models.CharField(max_length=60)
Now if you want to create an instance in the shell, you have to do:
> abcdef = SomeModel(a="stuff", b="stuff", c="stuff", d="stuff", e="stuff", f="stuff")
This gets really annoying if you have to keep creating model instances with long property names. Is there a way to simply send the arguments like you would with a normal Python object (as in without having to name the variables and just send the variables in order) ?
Like this:
> abcdef = SomeModel("stuff", "stuff", "stuff", "stuff", "stuff", "stuff")
As you would with a standard Python class in the init function.
Thanks :)
Here is my actual Python model:
class School(models.Model):
urn = models.CharField(max_length=6)
name = models.CharField(max_length=100)
house = models.CharField(max_length=50)
street = models.CharField(max_length=50)
county = models.CharField(max_length=50)
postcode = models.CharField(max_length=8)
def __unicode__(self):
return str(
'URN: ' + self.urn + ', ' +
'Name: ' + self.name + ', ' +
'House: ' + self.house + ', ' +
'Street: ' + self.street + ', ' +
'County: ' + self.county + ', ' +
'Postcode: ' + self.postcode + ', '
)

Have you tried the following code:
obj = SomeModel(None, "a_stuff", "b_stuff", "c_stuff", "d_stuff", "e_stuff", "f_stuff")
obj.save()
Hope these helps.

you can create instance by doing
abcdef = SomeModel()
Since instantiating doesnot touch the database, you can assign values later and then do .save().

Django - Getting/Saving large objects takes a lot of time

I'm trying to get a few million of items from a model, and parsing them. However, somehow it spends a lot of time trying to get the data saved.
These are the current models that I have:
class mapt(models.Model):
s = models.IntegerField(primary_key=True)
name = models.CharField(max_length=2000)
def __unicode__(self):
return str(self.s)
class datt(models.Model):
s = models.IntegerField(primary_key=True)
setid = models.IntegerField()
var = models.IntegerField()
val = models.IntegerField()
def __unicode(self):
return str(self.s)
class sett(models.Model):
setid = models.IntegerField(primary_key=True)
block = models.IntegerField()
username = models.IntegerField()
ts = models.IntegerField()
def __unicode__(self):
return str(self.setid)
class data_parsed(models.Model):
setid = models.IntegerField(max_length=2000, primary_key=True)
block = models.CharField(max_length=2000)
username = models.CharField(max_length=2000)
data = models.CharField(max_length=200000)
time = models.IntegerField()
def __unicode__(self):
return str(self.setid)
The s parameter for the datt model should actually act as a foreign key to mapt's s parameter. Furthermore, sett's setid field should act as a foreign key to setid's setid.
Lastly, data_parsed's setid is a foreign key to sett's models.
The algorithm is currently written this way:
def database_rebuild(start_data_parsed):
listSetID = []
#Part 1
for items in sett.objects.filter(setid__gte=start_data_parsed):
listSetID.append(items.setid)
uniqueSetID = listSetID
#Part 2
for items in uniqueSetID:
try:
SetID = items
settObject = sett.objects.get(setid=SetID)
UserName = mapt.objects.get(pk=settObject.username).name
TS = pk=settObject.ts
BlockName = mapt.objects.get(pk=settObject.block).name
DataPairs_1 = []
DataPairs_2 = []
DataPairs_1_Data = []
DataPairs_2_Data = []
for data in datt.objects.filter(setid__exact=SetID):
DataPairs_1.append(data.var)
DataPairs_2.append(data.val)
for Data in DataPairs_1:
DataPairs_1_Data.append(mapt.objects.get(pk=Data).name)
for Data in DataPairs_2:
DataPairs_2_Data.append(mapt.objects.get(pk=Data).name)
assert (len(DataPairs_1) == len(DataPairs_2)), "Length not equal"
#Part 3
Serialize = []
for idx, val in enumerate(DataPairs_1_Data):
Serialize.append(str(DataPairs_1_Data[idx]) + ":PARSEABLE:" + str(DataPairs_2_Data[idx]) + ":PARSEABLENEXT:")
Serialize_Text = ""
for Data in Serialize:
Serialize_Text += Data
Data = Serialize_Text
p = data_parsed(SetID, BlockName, UserName, Data, TS)
p.save()
except AssertionError, e:
print "Error:" + str(e.args)
print "Possibly DataPairs does not have equal length"
except Exception as e:
print "Error:" + str(sys.exc_info()[0])
print "Stack:" + str(e.args)
Basically, what it does is that:
Finds all sett objects that is greater than a number
Gets the UserName, TS, and BlockName, then get all the fields in datt field that correspond to a var and val field maps to the mapt 's' field. Var and Val is basically NAME_OF_FIELD:VALUE type of relationship.
Serialize all the var and val parameters so that I could get all the parameters from var and val that is spread across the mapt table in a row in data_parsed.
The current solution does everything I would like to, however, on a Intel Core i5-4300U CPU # 1.90Ghz, it parses around 15000 rows of data daily on a celery periodic worker. I have around 3355566 rows of data at my sett table, and it will take around ~23 days to parse them all.
Is there a way to speed up the process?
============================Updated============================
New Models:
class mapt(models.Model):
s = models.IntegerField(primary_key=True)
name = models.CharField(max_length=2000)
def __unicode__(self):
return str(self.s)
class sett(models.Model):
setid = models.IntegerField(primary_key=True)
block = models.ForeignKey(mapt, related_name='sett_block')
username = models.ForeignKey(mapt, related_name='sett_username')
ts = models.IntegerField()
def __unicode__(self):
return str(self.setid)
# class sett(models.Model):
# setid = models.IntegerField(primary_key=True)
# block = models.IntegerField()
# username = models.IntegerField()
# ts = models.IntegerField()
# def __unicode__(self):
# return str(self.setid)
class datt(models.Model):
s = models.IntegerField(primary_key=True)
setid = models.ForeignKey(sett, related_name='datt_setid')
var = models.ForeignKey(mapt, related_name='datt_var')
val = models.ForeignKey(mapt, related_name='datt_val')
def __unicode(self):
return str(self.s)
# class datt(models.Model):
# s = models.IntegerField(primary_key=True)
# setid = models.IntegerField()
# var = models.IntegerField()
# val = models.IntegerField()
# def __unicode(self):
# return str(self.s)
class data_parsed(models.Model):
setid = models.ForeignKey(sett, related_name='data_parsed_setid', primary_key=True)
block = models.CharField(max_length=2000)
username = models.CharField(max_length=2000)
data = models.CharField(max_length=2000000)
time = models.IntegerField()
def __unicode__(self):
return str(self.setid)
New Parsing:
def database_rebuild(start_data_parsed, end_data_parsed):
for items in sett.objects.filter(setid__gte=start_data_parsed, setid__lte=end_data_parsed):
try:
UserName = mapt.objects.get(pk=items.username_id).name
TS = pk=items.ts
BlockName = mapt.objects.get(pk=items.block_id).name
DataPairs_1 = []
DataPairs_2 = []
DataPairs_1_Data = []
DataPairs_2_Data = []
for data in datt.objects.filter(setid_id__exact=items.setid):
DataPairs_1.append(data.var_id)
DataPairs_2.append(data.val_id)
for Data in DataPairs_1:
DataPairs_1_Data.append(mapt.objects.get(pk=Data).name)
for Data in DataPairs_2:
DataPairs_2_Data.append(mapt.objects.get(pk=Data).name)
assert (len(DataPairs_1) == len(DataPairs_2)), "Length not equal"
Serialize = []
for idx, val in enumerate(DataPairs_1_Data):
Serialize.append(str(DataPairs_1_Data[idx]) + ":PARSEABLE:" + str(DataPairs_2_Data[idx]))
Data = ":PARSEABLENEXT:".join(Serialize)
p = data_parsed(items.setid, BlockName, UserName, Data, TS)
p.save()
except AssertionError, e:
print "Error:" + str(e.args)
print "Possibly DataPairs does not have equal length"
except Exception as e:
print "Error:" + str(sys.exc_info()[0])
print "Stack:" + str(e.args)

Defining lists by appending repeadedly is very slow. Use list comprehensions or even just the list() constructor.
In python you should not join a list of strings using for loops and +=, you should use join().
But that is not the primary bottleneck here. You have a lot of objects.get()s which each takes a database roundtrip. If you didn't have milions of rows in the mapt table, you should probably just make a dictionary mapping mapt primary keys to mapt objects.
Had you defined your foreign keys as foreign keys the django orm could help you do much of this in like five queries in total. That is, instead of SomeModel.objects.get(id=some_instance.some_fk_id) you can do some_instance.some_fk (which will only hit the databse the first time you do it for each instance). You can then even get rid of the foreign key query if some_instance had been initialized as some_instance = SomeOtherModel.objects.select_related('some_fk').get(id=id_of_some_instance).
Perhaps changing the models without changing the database will work.