Develop Reference

Python Heroku Scheduler DB commit doesnt change values - DetachedInstanceError

I am using following code in the free heroku scheduler add-on to send emails to certain users. After the email was sent a value in the DB must be changed, to be more precise:
setattr(user, "stats_email_sent", True)
Somehow the db_session.commit() is executed but doesnt save the new value. Here is the code:
all_users = User.query.all()
for user in all_users:
if user.stats_email_sent is False and user.number_of_rooms > 0:
if date.today() <= user.end_offer_date and date.today() >= user.end_offer_date - relativedelta(days=10):
print user.id, user.email, user.number_of_rooms, user.bezahlt
if user.bezahlt is True:
with app.app_context():
print "app context true", user.id, user.email, user.number_of_rooms, user.bezahlt
html = render_template('stats_email_once.html', usersname=user.username)
subject = u"Update"
setattr(user, "stats_email_sent", True)
#send_email(user.email, subject, html, None)
else:
with app.app_context():
print "app context false", user.id, user.email, user.number_of_rooms, user.bezahlt
html = render_template('stats_email_once.html', usersname=user.username)
subject = u"Update"
setattr(user, "stats_email_sent", True)
#send_email(user.email, subject, html, None)
print "executing commit"
db_session.commit()
I tryed moving db_session.commit() right after setattr then it will work, but only for one user (for the first user).
setattr(user, "stats_email_sent", True)
db_session.commit()
And give me this in the logs:
sqlalchemy.orm.exc.DetachedInstanceError: Instance <User at 0x7f5c04d462d0> is not bound to a Session; attribute refresh operation cannot proceed
I also found some topics on detaching an instance, but I don't need to detach here anything, or?
EDIT:
I tryed now also adding db_session.expire_on_commit = False. This sadly had no effect.
I also looked on the bulk updates, which also didn't worked.
I even tryed to ignore and pass the DetachedInstanceError.
I cant believe that updating multiple rows at once is such an issue. I am running out of ideas here. Any help is appriciated. It either has no effect or is run into DetachedInstanceError.
EDIT
I solved the issue, had yesterday a similar one. I assume the query and its variables were consumed, thats why it didnt worked.
To solve this I created a list client_id_list = [] and appended all the ids of users which got the email and where the value needs to be changed.
Then I created a completely new query and all in all run the same code with the same logic, but the variables here were not consumed I guess? I wont answer the question again, because I am not sure whether this is true or not, here is the code which I appended to the code above which changes the values:
all_users_again = User.query.all()
for user in all_users_again:
if user.id in client_id_list:
setattr(user, "stats_email_sent", True)
db_session.commit()

Raw sql to json in Django, with Datetime and Decimal MySql columns

I am using Ajax to make some requests from client to server, I am using DJango and I have used some Raw Sql queries before, but all of my fields was Int, varchar and a Decimal, for the last one I had an enconding problem, but I overrided the "default" property of Json and everything worked.
But that was before, now I have a query wich gives me Decimal and DateTime fields, both of them gave me enconding errors, the overrided "default" doesn't work now, thats why with this new one I used DjangoJSONEncoder, but now I have another problem, and its not an encoding one, I am using dictfetchall(cursor) method, recomended on Django docs, to return a dictionary from the Sql query, because cursor.fetchall() gives me this error: 'tuple' object has no attribute '_meta'.
Before I just sended that dictionary to json.dumps(response_data,default=default) and everything was fine, but now for the encoding I have to use the following: json.dumps(response_data,cls=DjangoJSONEncoder) and if I send the dictionary in that way, I get this error:
SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data
And if I try to use the serializers, like this:
response_data2= serializers.serialize('json', list(response_data))
And later send response_data2 to dumps, I get this error:
'dict' object has no attribute '_meta'
This is the code for the MySql query:
def consulta_sql_personalizada(nombres,apellidos,puesto):
from django.db import connection, transaction
cursor = connection.cursor()
cursor.execute("""select E.idEmpleado as id,CONCAT(Per.nombres_persona,' ',Per.apellidos_persona) as nombre,P.nombre_puesto as puesto,E.motivo_baja_empleado as motivo_baja,E.fecha_contratacion_empleado AS fecha_contratacion,E.fecha_baja_empleado as fecha_baja, SUM(V.total_venta) AS ventas_mes,E.fotografia_empleado as ruta_fotografia from Empleado as E
inner join Puesto as P on E.Puesto_idPuesto=P.idPuesto
inner join Venta as V on V.vendedor_venta=E.idEmpleado
inner join Persona as Per on E.Persona_idPersona=Per.idPersona
where (Per.nombres_persona like %s OR Per.apellidos_persona like %s OR E.Puesto_idPuesto=%s)
AND E.estado_empleado=1 AND V.estado_venta=1
AND
(YEAR(V.fecha_venta) = YEAR(Now())
AND MONTH(V.fecha_venta) = MONTH(Now()))""",[nombres,apellidos,puesto])
row = dictfetchall(cursor)
return row
And this is the last part of the view that makes the query and send it to ajax using json:
response_data=consulta_sql_personalizada(rec_nombres,rec_apellidos,rec_puesto)
return HttpResponse(
json.dumps(response_data,cls=DjangoJSONEncoder),
content_type="application/json"
)
else:
return HttpResponse(
json.dumps({"nothing to see": "this isn't happening"}),
content_type="application/json"
)
What I want to know is, how can I parse the raw sql result to Json using that enconding?

Sorry, was my bad, i'm using JQuery ajax method, and in the "success" part I forgot to stop using json.parse to print the data in the console, the data was json already, that's why I had that line 1 column 1 error. My code worked exactly like it was posted here. If someone want to know how to make asynchronous requests, I followed this tutorial: Django form submissions using ajax

Session data not being stored during testing in Django

I am currently writing tests for our project, and I ran into an issue. We have this section of a view, which will redirect the user back to the page where they came from including an error message (that's being stored in the session):
if request.GET.get('error_code'):
"""
Something went wrong or the call was cancelled
"""
errorCode = request.GET.get('error_code')
if errorCode == 4201:
request.session['errormessage'] = _('Action cancelled by the user')
return HttpResponseRedirect('/socialMedia/manageAccessToken')
Once the HttpResponseRedirect kicks in, the first thing that the new view does is scan the session, to see if any error messages are stored in the session. If there are, we place them in a dictionary and then delete it from the session:
def manageAccessToken(request):
"""
View that handles all things related to the access tokens for Facebook,
Twitter and Linkedin.
"""
contextDict = {}
try:
contextDict['errormessage'] = request.session['errormessage']
contextDict['successmessage'] = request.session['successmessage']
del request.session['errormessage']
del request.session['successmessage']
except KeyError:
pass
We should now have the error message in a dictionary, but after printing the dictionary the error message is not there. I also printed the session just before the HttpResponseRedirect, but the session is an empty dictionary there as well.
This is the test:
class oauthCallbacks(TestCase):
"""
Class to test the different oauth callbacks
"""
def setUp(self):
self.user = User.objects.create(
email='test#django.com'
)
self.c = Client()
def test_oauthCallbackFacebookErrorCode(self):
"""
Tests the Facebook oauth callback view
This call contains an error code, so we will be redirected to the
manage accesstoken page. We check if we get the error message
"""
self.c.force_login(self.user)
response = self.c.get('/socialMedia/oauthCallbackFacebook/',
data={'error_code': 4201},
follow=True,
)
self.assertEqual('Action cancelled by the user', response.context['errormessage'])
It looks like the session can not be accessed or written to directly from the views during testing. I can, however, access a value in the session by manually setting it in the test by using the following bit of code:
session = self.c.session
session['errormessage'] = 'This is an error message'
session.save()
This is however not what I want, because I need the session to be set by the view as there are many different error messages in the entire view. Does anyone know how to solve this? Thanks in advance!

After taking a closer look I found the issue, it is in the view itself:
errorCode = request.GET.get('error_code')
if errorCode == 4201:
request.session['errormessage'] = _('Action cancelled by the user')
The errorCode variable is a string, and I was comparing it to an integer. I fixed it by changing the second line to:
if int(errorCode) == 4201:

Django Test Client does not create database entries

I'm creating unit tests for my views using Django's built-in Test Client to create mock requests.
The view I'm calling should create an object in the database. However, when I query the database from within the test method the object isn't there - it either hasn't been created or has been discarded on returning from the view.
Here's the view:
def apply_to_cmp(request, campaign_id):
""" Creates a new Application to 'campaign_id' for request.user """
campaign = Campaign.objects.get(pk = campaign_id)
if not Application.objects\
.filter(campaign = campaign, user = request.user)\
.exists():
application = Application(**{'campaign' : campaign,
'user' : request.user})
application.save()
return HttpResponseRedirect(request.META.get('HTTP_REFERER'))
This is the test that calls it:
def test_create_campaign_app(self):
""" Calls the method apply_to_cmp in .views """
c = Client()
c.login(username = self.username, password = self.password)
url = '/campaign/' + self.campaign.id + '/apply/'
response = c.get(url)
# Check whether request was successful (should return 302: redirect)
self.assertEqual(response.status_code, 302)
# Verify that an Application object was created
app_count = Application.objects\
.filter(user = self.user, campaign = self.campaign)\
.count()
self.assertEqual(app_count, 1)
This is the output from the running the test:
Traceback (most recent call last):
File "/test_views.py", line 40, in test_create_campaign_app
self.assertEqual(app_count, 1)
AssertionError: 0 != 1
The method apply_to_cmp is definitely being called, since response.status_code == 302, but still the Application object is not created. What am I doing wrong?
Edit: Solution
Client.login failed because the login system was not properly initialised in the setUp method. I fixed this by calling call_command('loaddata', 'initial_data.json') with initial_data.json containing the setup for the login system. Also, HttpResponseRedirect(request.META.get('HTTP_REFERER')) didn't work for obvious reasons. I changed that bit to
if request.META.get('HTTP_REFERER'):
return HttpResponseRedirect(request.META.get('HTTP_REFERER'))
return HttpResponse()
And therefore the test to
self.assertEqual(response.status_code, 200)
Thanks for your help!

Nothing stands out as particularly wrong with your code - but clearly either your test case or the code your are testing is not working the way you think. It is now time to question your assumptions.
The method apply_to_cmp is definitely being called, since response.status_code == 302
This is your first assumption, and it may not be correct. You might get a better picture of what is happening if you examine other details in the response object. For example, check the response.redirect_chain and confirm that it actually redirects where you expect it to:
response = c.get(url, follow=True)
self.assertEqual(response.redirect_chain, [<expected output here>])
What about other details? I can't see where self.username and self.password are defined from the code you provided. Are you 100% sure that your test code to login worked? c.login() returns 'True' or 'False' to indicate if the login was successful. In my test cases, I like to confirm that the login succeeds.
login_success = c.login(username = self.username, password = self.password)
self.assertTrue(login_success)
You can also be a bit more general. You find nothing if you check Application.objects.filter(user=self.user, campaign=self.campaign), but what about checking Application.objects.all()? You know that a specific item isn't in your database, but do you know what is stored in the database (if anything at all) in the test code at that time? Do you expect other items in there? Check to confirm that what you expect is true.
I think you can solve this one, but you'll need to be a bit more aggressive in your analysis of your test case, rather than just seeing that your app_count variable doesn't equal 1. Examine your response object, put in some debug statements, and question every assumption.

First of all, if you are subclassing from django.test.TestCase, please take in consideration the fact that each test is wrapped into transactions (official docs).
Then, you can add db logging to your project to see whether there was a hit to the database or not (official docs).
And finally be sure that you're using correct lookups at this line: filter(user = self.user, campaign = self.campaign)

Why am I getting a broken pipe error when adding data to the database?

Not sure if there's a better way to do this but I have a sign up page on my site and after a user signs up I add their initial data(stuff in the __init__ data model) then I start adding some other info in the same section which is giving me a broken pipe error. Oddly the code seems to work since the entries I'm expecting are in the database. I have tried moving around the .flush() command to see if it helps but it doesn't seem to be.
Traceback (most recent call last):
File "/Users/me/Dropbox/code/eclipseWorkSpace/website/pyramidwiki/lib/python2.7/site-packages/waitress-0.8.1-py2.7.egg/waitress/channel.py", line 134, in handle_write
flush()
File "/Users/me/Dropbox/code/eclipseWorkSpace/website/pyramidwiki/lib/python2.7/site-packages/waitress-0.8.1-py2.7.egg/waitress/channel.py", line 249, in _flush_some
num_sent = self.send(chunk)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/asyncore.py", line 365, in send
result = self.socket.send(data)
error: [Errno 32] Broken pipe
Here's my code:
if 'form.submitted' in request.params:
firstname = request.params['firstname']
lastname = request.params['lastname']
email = request.params['email']
password = request.params['password']
try:
new_user = Users(email, firstname, lastname, password)
DBSession.add(new_user)
#DBSession.flush() #commit so we get error if any
#add some other info
user_data = DBSession.query(Users).filter(Users.email==email).first()
user_data.join_date = datetime.datetime.now()
#create random number for verification url
user_data.vertified = id_generator(50)
DBSession.flush() #doesn't seem to make a difference where the flush is
return HTTPFound(location = request.route_url('new'))
Any ideas?

This may not answer you question directly, but "you're doing it all wrong"(tm) :)
You don't need to re-query User object after you added it to the session - what's more, trying to query it from the database without doing session.flush() first will result in an error because there's no record in the database yet. I'd do something like this:
if 'form.submitted' in request.params:
firstname = request.params['firstname']
lastname = request.params['lastname']
email = request.params['email']
password = request.params['password']
try:
new_user = Users(email, firstname, lastname, password)
new_user.join_date = datetime.datetime.now()
new_user.verified = id_generator(50)
DBSession.add(new_user)
DBSession.flush() # should fail if user email is in the database
return HTTPFound(location = request.route_url('new'))
Also, you need to check that all branches of execution (i.e. the except: clause, the else: clause of "if 'form.submitted' in request.params" return something meaningful - you may be getting the exception because your view function returns None in some conditions. Actually, that's probably what was happening - the "user_data = DBSession.query(Users)" line was raising an exception, and the except: part was not returning anything

I too struggled with the same problem on my Pyramid project and contrary to comments on github, it wasn't with waitress.
In my case, the problem of Error: Broken Pipe only occured whenever I used redirects (HTTPFound, HTTPMovedPermanently etc.). Since your function also uses HTTPFound, I think the problem is the same.
Atleast for me, this error was due to pyramid_debugtoolbar extension. The reason is probably that whenever our view does soemthing like
return HTTPFound(foo)
it sends out a 302 in the header and a Connection:Close. The pyramid_debugtoolbar extension adds a lengthy body to the response. The client on seeing the header, closes the connection and does not accept the lengthy debug body. Hence the Broken Pipe message (Atleast that is what Wireshark shows).
Try disabling the pyramid_debugtoolbar in your app's .ini, it might help.