Develop Reference

Items remain in the lists even after using del statement

I'm building an app using Python and Django and I'm facing a problem with some variables.
Below are simplified portions from my code to explain the problem (you can access the complete code in the repo):
# resources.py
class HREmployeeResource(ModelResource):
def after_import_row(self, row, row_result, row_number=None, **kwargs):
row_result.employee_code = row.get('Employee Code')
if not kwargs.get('dry_run'):
import_type = row_result.import_type
employee_code = row_result.employee_code
instance = kwargs.get('instance')
# we need the ones belong to these 2 companies in the report
if instance.company in ['Company 1', 'Company 2']:
self.report.setdefault(import_type, []).append(employee_code)
# views.py
import gc
from resources import HREmployeeResource
def import_hr_file(request):
if request.method == 'POST':
hr_file = request.FILES['hr-file']
data = get_import_data(hr_file)
hr_resource = HREmployeeResource()
result = hr_resource.import_data(data, dry_run=True)
if not result.has_errors():
result = hr_resource.import_data(data, dry_run=False)
ImportStatus.objects.create(
date=timezone.now(),
status='S1',
data=hr_resource.report
)
del result, hr_resource, data
gc.collect()
return redirect('core:import-report')
return render(request, 'core/upload-hr-file.html', {})
The ModelResource class is from a third-party library called django-import-export and the import_data method belongs to this class, you could find the code of the method here.
When I run the view import_hr_file for the first time everything is working fine, but when I execute it the second time I find that the old items still exist in the lists of the report dict of the instance of HREmployeeResource class.
Originally I'm using these two lines (below) to free up the memory after the import process.
del result, hr_resource, data
gc.collect()
But the memory is not freed, and now I found this problem (the old items of the previous run remain every time I run the view import_hr_file)
So, how can I fix this?

Why is "ObjectId('5efbe85b4aeb5d21e56fa81f')" not considered a valid ObjectId?

I am using PyMongo and I am trying to loop through an entire collection and display the ObjectId onto onto my Flask Web Page. However, when I write my method I keep getting the error "ObjectId('5efbe85b4aeb5d21e56fa81f')" is not a valid ObjectId.
The following is the code I am running
def get_class_names(self):
temp = list()
print("1")
for document_ in db.classes.find():
tempstr = document_.get("_id")
tempobjectid = ObjectId(tempstr)
temp.append(repr(tempobjectid))
print("2")
classes = list()
for class_ in temp:
classes.append(class_, Classes.get_by_id(class_).name)
return classes
How do I fix this?
Note: get_by_id, just takes in an ObjectId and finds it in the database.

The line
tempstr = document_.get("_id")
retrieves an ObjectId already. You then wrap it again in another ObjectId before calling repr on that. If you print(type(tempstr)), you'll see that it's an ObjectId.
Just do temp.append(tempstr).
BTW, you should rename the variable tempstr to tempId or something more appropriate.

web2py: How to execute instructions before delete using SQLFORM.smartgrid

I use SQLFORM.smartgrid to show a list of records from a table (service_types). In each row of the smartgrid there is a delete link/button to delete the record. I want to executive some code before smartgrid/web2py actually deletes the record, for example I want to know if there are child records (services table) referencing this record, and if any, flash a message telling user that record cannot be deleted. How is this done?
db.py
db.define_table('service_types',
Field('type_name', requires=[IS_NOT_EMPTY(), IS_ALPHANUMERIC()]),
format='%(type_name)s',
)
db.define_table('services',
Field('service_name',requires=[IS_NOT_EMPTY(),IS_NOT_IN_DB(db,'services.service_name')]),
Field('service_type','reference service_types',requires=IS_IN_DB(db,db.service_types.id,
'%(type_name)s',
error_message='not in table',
zero=None),
ondelete='RESTRICT',
),
Field('interest_rate','decimal(15,2)',requires=IS_DECIMAL_IN_RANGE(0,100)),
Field('max_term','integer'),
auth.signature,
format='%(service_name)s',
)
db.services._plural='Services'
db.services._singular='Service'
if db(db.service_types).count() < 1:
db.service_types.insert(type_name='Loan')
db.service_types.insert(type_name='Contribution')
db.service_types.insert(type_name='Other')
controller
def list_services():
grid = SQLFORM.smartgrid(db.services
, fields = [db.services.service_name,db.services.service_type]
)
return locals()
view
{{extend 'layout.html'}}
{{=grid}}

There are two options. First, the deletable argument can be a function that takes the Row object of a given record and returns True or False to indicate whether the record is deletable. If it returns False, the "Delete" button will not be shown for that record, nor the delete operation be allowed on the server.
def can_delete(row):
return True if [some condition involving row] else False
grid = SQLFORM.smartgrid(..., deletable=can_delete)
Second, there is an ondelete argument that takes the db Table object and the record ID. It is called right before the delete operation, so to prevent the delete, you can do a redirect within that function:
def ondelete(table, record_id):
record = table(record_id)
if [some condition]:
session.flash = 'Cannot delete this record'
redirect(URL())
grid = SQLFORM.smartgrid(..., ondelete=ondelete)
Note, if the grid is loaded via an Ajax component and its actions are therefore performed via Ajax, using redirect within the ondelete method as shown above will not work well, as the redirect will have no effect and the table row will still be deleted from the grid in the browser (even though the database record was not deleted). In that case, an alternative approach is to return a non-200 HTTP response to the browser, which will prevent the client-side Javascript from deleting the row from the table (the delete happens only on success of the Ajax request). We should also set response.flash instead of session.flash (because we are not redirecting/reloading the whole page):
def ondelete(table, record_id):
record = table(record_id)
if [some condition]:
response.flash = 'Cannot delete this record'
raise HTTP(403)
Note, both the deletable and ondelete arguments can be dictionaries with table names as keys, so you can specify different values for different tables that might be linked from the smartgrid.
Finally, notice the delete URLs look like /appname/list_services/services/delete/services/[record ID]. So, in the controller, you can determine if a delete is being requested by checking if 'delete' in request.args. In that case, request.args[-2:] represents the table name and record ID, which you can use to do any checks.

From Anthony's answer I chose the second option and came up with the following:
def ondelete_service_type(service_type_table, service_type_id):
count = db(db.services.service_type == service_type_id).count()
if count > 0:
session.flash = T("Cant delete")
#redirect(URL('default','list_service_types#'))
else:
pass
return locals()
def list_service_types():
grid = SQLFORM.smartgrid(db.service_types
, fields = [db.service_types.type_name, db.services.service_name]
, ondelete = ondelete_service_type
)
return locals()
But, if I do this...
if count > 0:
session.flash = T("Cant delete")
else:
pass
return locals()
I get this error:
And if I do this:
if count > 0:
session.flash = T("Cant delete")
redirect(URL('default','list_service_types#')) <== please take note
else:
pass
return locals()
I get the flash error message Cant delete but the record appears deleted from the list, and reappears after a page refresh with F5 (apparently because the delete was not allowed in the database, which is intended).
Which one should I fix and how?
Note
If any of these issue is resolved I can accept Anthony's answer.

get a list of comma separated values in python and use that list to perform a command as many times as necessary with each value

I'm sure this is something easy to do for someone with programming skills (unlike me). I am playing around with the Google Sites API. Basically, I want to be able to batch-create a bunch of pages, instead of having to do them one by one using the slow web form, which is a pain.
I have installed all the necessary files, read the documentation, and successfully ran this sample file. As a matter of fact, this sample file already has the python code for creating a page in Google Sites:
elif choice == 4:
print "\nFetching content feed of '%s'...\n" % self.client.site
feed = self.client.GetContentFeed()
try:
selection = self.GetChoiceSelection(
feed, 'Select a parent to upload to (or hit ENTER for none): ')
except ValueError:
selection = None
page_title = raw_input('Enter a page title: ')
parent = None
if selection is not None:
parent = feed.entry[selection - 1]
new_entry = self.client.CreatePage(
'webpage', page_title, '<b>Your html content</b>',
parent=parent)
if new_entry.GetAlternateLink():
print 'Created. View it at: %s' % new_entry.GetAlternateLink().href
I understand the creation of a page revolves around page_title and new_entry and CreatePage. However, instead of creating one page at a time, I want to create many.
I've done some research, and I gather I need something like
page_titles = input("Enter a list of page titles separated by commas: ").split(",")
to gather a list of page titles (like page1, page2, page3, etc. -- I plan to use a text editor or spreadsheet to generate a long list of comma separated names).
Now I am trying to figure out how to get that string and "feed" it to new_entry so that it creates a separate page for each value in the string. I can't figure out how to do that. Can anyone help, please?
In case it helps, this is what the Google API needs to create a page:
entry = client.CreatePage('webpage', 'New WebPage Title', html='<b>HTML content</b>')
print 'Created. View it at: %s' % entry.GetAlternateLink().href
Thanks.

Whenever you want to "use that list to perform a command as many times as necessary with each value", that's a for loop. (It may be an implicit for loop, e.g., in a map call or a list comprehension, but it's still a loop.)
So, after this:
page_titles = raw_input("Enter a list of page titles separated by commas: ").split(",")
You do this:
for page_title in page_titles:
# All the stuff that has to be done for each single title goes here.
# I'm not entirely clear on what you're doing, but I think that's this part:
parent = None
if selection is not None:
parent = feed.entry[selection - 1]
new_entry = self.client.CreatePage(
'webpage', page_title, '<b>Your html content</b>',
parent=parent)
if new_entry.GetAlternateLink():
print 'Created. View it at: %s' % new_entry.GetAlternateLink().href
And that's usually all there is to it.

You can use a for loop to loop over a dict object and create multiple pages. Here is a little snippet to get you started.
import gdata.sites.client
client = gdata.sites.client.SitesClient(
source=SOURCE_APP_NAME, site=site_name, domain=site_domain)
pages = {"page_1":'<b>Your html content</b>',
"page_2":'<b>Your other html content</b>'}
for title, content in pages.items():
feed = client.GetContentFeed()
parent = None
if selection is not None:
parent = feed.entry[selection - 1]
client.CreatePage('webpage', page_title, content, parent=parent)
as for feeding your script from an external source I would recommend something like a csv file
##if you have a csv file called pages.csv with the following lines
##page_1,<b>Your html content</b>
##page_2,<b>Your other html content</b>
import csv
with open("pages.csv", 'r') as f:
reader = csv.reader(f)
for row in reader:
title, content = row
client.CreatePage('webpage', title, content, parent=parent)

Mysterious behavior in try-except-finally

Excuse the vague title, I didn't know how else to state this.
I have a task worker request handler that fetches data from a URL and writes it to blobstore and saves the data's blob_key to a ListProperty in datastore. I've tried to simplifly the code for clarity here:
class Fetch(webapp2.RequestHandler):
def get(self):
url = self.request.get('url')
itemKey = self.request.get('itemKey')
item = MyModel.get(itemKey)
try:
result = urlfetch.fetch(url=url)
if result.status_code == 200:
saveDataResult = save_data(result.content, itemKey)
if saveDataResult is False:
raise Exception('error saving data')
else:
raise Exception('error fetching data: %s' % result.status_code)
item.status = 'success'
except Exception:
item.status = 'failed'
finally:
item.put()
def save_data(data, itemKey)
try:
#write data to blobstore and get its blob_key...
blob_key = files.blobstore.get_blob_key(file_name)
item = MyModel.get(itemKey)
item.blobKey.append(blob_key)
item.put()
return True
except:
return False
Now the problem I'm having is, when saveDataResult returns True, its status is set to 'success' but its blobKey property contains no value, even though a blob_key was generated and the data successfully written. I can't see what's causing this to save my life, please help.

Without much more information it is very difficult to determine what's happening. Here's my educated guess:
MyModel.get(itemKey) is called both in get() and save_data(). I surmise that it's returning two different objects representing the item. When the blobKey gets updated in save_data, the update is occurring only in the object fetched in save_data. When you later examine it outside that scope, you're looking at a different object.
Whether this is correct or not will depend on the implementation of MyModel.get().
Also, you do realize that you're calling item.put() twice, right?

The problem is here
finally:
item.put()
this single call overrides the data saved by save_data() because it references an older object of item.
My suggestion would be you do the status updates from save_data() i.e item.status = 'success'
or move item = MyModel.get(itemKey) to come after save_data() so you can fetch the updated object.

The problem is that when you call save_data() with item = MyModel.get(itemKey)
which is again called from the class Fetch you end up having two different objects and therefore overwriting the one in save_data() and hence when you go to your model datastore no data for blobkey is stored as its overwritten.
Try doing everything in the class or you do not use item = MyModel.get(itemKey) twice.