Develop Reference

python json.loads() works but requests.json does not work, on an odata endpoint that produces 32MB of json

Have been happily using requests.json interchangably with json.loads().... but today I was using an odata endpoint that returns ~ 32MB of json... json.loads() worked fine but when I tried requests.json it raised an error. So there must be differences between the two? Any insights on what these might be... Using python 3.6, requests 2.18.4
# get json from odata endpoint and check for top-level "value" element...
try:
http_response = requests.get( url )
except Exception as errmsg:
sys.exit( "Error on http request ... Msg: " + str(errmsg) )
try:
# jsondata = json.loads( http_response.text ) # works fine
jsondata = http_response.json # doesn't work
except Exception as errmsg:
sys.exit( "Error on json conversion: " + str(errmsg) )
try:
rowset = jsondata["value"]
except Exception as errmsg:
sys.exit( "ERROR .... can't find 'value'. Msg: " + str(errmsg) )
The odata endpoint I need to call returns json that is lengthy (32,113,432 chars) and begins like this:
{"#odata.context":"http://xxx.yyy.org/JAXLIMS/Odata /$metadata#MPDInputs","#odata.count":27775,"value":[{"C_InputInstance_key":19203327,"InputValue":"129","InputName":"Experimenter", ....etc....
The error raised when I use requests.json is:
ERROR .... can't find 'value'. Msg: 'method' object is not subscriptable

'module' object is not subscriptable

here is very simplified version of my code , so pleas ignore syntax errors
i have a helper function basically reading a row from database using django orm and doing some validation finally return it using a dictionary
modVerify.py
def verify(request):
try :
req = Request.objects.get(id=request.POST.get('id'))
except :
return({'stat':'er' , 'error':-12})
return({'stat':'ok' , 'req':req})
here is where i get the error when im trying to use this above app
import modVerify.view
def verify(request):
result = modVerify.views.verify(request )
if(result['status'] == 'ok'):
req = modeVerify['req']
else :
print('ERROR !')
here is my error
TypeError at /api/verify
'module' object is not subscriptable
Request Method: POST
Request URL: site.com/api/verify
Django Version: 1.9.7
Exception Type: TypeError
Exception Value:
'module' object is not subscriptable
Exception Location: /home/somedomain/project/api/views.py in verify, line 98
Python Executable: /usr/local/bin/python3
Python Version: 3.4.4
which points to this line
req = modeVerify['req']
so why im getting this and is there a way around it or should i return row id back instead and read it again from database in the caller function ?

It seems like you should be doing
req = result['req']
instead of
req = modeVerify['req']

Raising exceptions with django_rest_framework

I have written a view that decrypts a GPG encrypted file and returns it as plain text. This works fine in general. The problem is, if the file is empty or otherwise contains invalid GPG data, gnupg returns an empty result rather than throw an exception.
I need to be able to do something like this inside decrypt_file to check to see if the decryption failed and raise an error:
if data.ok:
return str(data)
else:
raise APIException(data.status)
If I do this, I see the APIException raised in the Django debug output, but it's not translating to a 500 response to the client. Instead the client gets a 200 response with an empty body. I can raise the APIException in my get method and it sends a 500 response, but there I don't have access to the gnupg error message.
Here is a very simplified version of my view:
from rest_framework.views import APIView
from django.http import FileResponse
from django.core import files
from gnupg import GPG
class FileDownload(APIView):
def decrypt_file(self, file):
gpg = GPG()
data = gpg.decrypt(file.read())
return data
def get(self, request, id, format=None):
f = open('/tmp/foo', 'rb')
file = files.File(f)
return FileResponse(self.decrypt_file(file))
I have read the docs on DRF exception handling here, but it doesn't seem to provide a solution to this problem. I am fairly new to Django and python in general, so it's entirely possible I'm missing something obvious. Helpful advice would be appreciated. Thanks.

If you raise an error in python, every function in the trace re-raise it until someone catch it. You can first declare a exception :
from rest_framework.exceptions import APIException
class ServiceUnavailable(APIException):
status_code = 503
default_detail = 'Service temporarily unavailable, try again later.'
Then in your decrypt_file function, raise this exception if decryption is not successful. You can pass an argument to modify the message. Then, in the get method, you should call decrypt_file function and pass your file as an argument. If any things goes wrong, your function raise that exception and then, get method re-raise it until Django Rest Framework exception handler catch it.
EDIT:
In your decrypt function, do something like this:
from rest_framework.response import Response
def decrypt_file(self, file):
... # your codes
if data.ok:
return str(data)
else:
raise ServiceUnavailable
def get(self, request, id, format=None):
f = open('/tmp/foo', 'rb')
result = decrypt_file(f)
f.close()
return Response({'data': result})

Callback from External OAUTH in Django

Alrighty. Setup information - Django 1.8.3, Python 2.7.3, using Goodreads 0.2.4 api handler for Python (slightly modified). I'm at the point where Goodreads does a callback to the app, but the clickjacker middleware is catching it as an error and tossing out a 404. If I disable the clickjacker prevention (which I'd prefer not to do), I get an "argument of type 'type' is not iterable" error.
Relevant Code:
from goodreads import client
grClient = client.GoodreadsClient(<key>,<other_key>)
def goodReadsOAUTH_stage1(request,user):
try:
return HttpResponseRedirect(grClient.authenticate())
# function modified to return the URL rather than open a browser
except keyError:
return Http404
def goodReadsOAUTH_stage2(request):
if request.method == "GET":
if request.GET['authorize'] == 1:
grRecord = goodreadsOAUTHKeys(request.user,request.GET['oauth_token']
grRecord.save()
grClient.session.oauth_finalize()
return HttpResponseRedirect(reverse(request,'workroom:profile',kwargs={'user':request.user.username}))
else:
return Http404
else:
return Http404
And the URLconf for the two:
url(r'^social/(?P<user>[-\w\d]+)/goodreads/OAUTH/$', views.goodReadsOAUTH_stage1, name='goodreads-oauth-one'),
url(r'^social/goodreads/OAUTH/validation/$', views.goodReadsOAUTH_stage2, name='goodreads-oauth-two'),
And the ensuing error message!
type object 'Http404' has no attribute 'get'
Exception Location: /home/.../public/env/local/lib/python2.7/site-packages/django/middleware/clickjacking.py in process_response, line 31
And that line of code from the clickjacking protection:
if response.get('X-Frame-Options', None) is not None:
I'm essentially at a loss here as to how to get the callback to function correctly.

You are using Http404 wrong. Http404 is an Exception, use it like
raise Http404("Poll does not exist")
For the reference see section "Returning Errors"

Python GAE: How to return JSON-formated response even when an error occurs?

In my GAE application, I have several request handlers that return JSON-formated response. When one of these is called, if an error occurs (exception, or programming error), the output is not JSON: it is the stack trace.
What I need is:
Output without error:
{
"foo" : 1
"bar" : 2
"status" : "OK"
}
Output when an error occurs:
{
"status" : "ERR"
"errorMessage" : "An error occurred!"
}
My question is: What is the best practice to make sure that, in any case, the output will be a JSON-formated response? Of course, a common solution for every request handlers would be great.
Any help would be appreciated.

Sure - use the ereporter class (described here: https://stackoverflow.com/a/4296664/336505), but create a custom BaseHandler that formats your uncaught exceptions as JSON output:
class BaseHandler(webapp.RequestHandler):
def handle_exception(self, exception, debug_mode):
self.response.headers['Content-Type'] = 'application/json'
self.response.out.write(etc, etc) # format the exception

If an error occurs, to avoid getting a stack trace or other ugly output, you will need to employ a try ... except: http://docs.python.org/tutorial/errors.html
e.g.
try:
# ... your code ...
except TypeError as e:
# ... do something with this error type
except:
# ... generic exception catchall
# output that JSON response