Develop Reference

How to handle API responses(JSON) containing \x00 (or \u0000) in its data, and store the data in Postgres using django models?

I'm making an api call, and trying to store its response in Postgres using Django models.
Here's what I have been doing:
response = requests.post(url='some.url.com', data=json.dumps(data), headers={'some': 'header'})
response_data = json.loads(response.content.decode('utf-8'))
#handler is a object of a model
handler.api_response = response_data
handler.save()
But this used to fail, when my json had fields like 'field_name': '\x00\x00\x00\x00\x00'. It used to give following error :
DataError at /api/booking/reprice/
unsupported Unicode escape sequence
LINE 1: ... NULL, "api_status" = 0, "api_response" = '{"errorRe...
^
DETAIL: \u0000 cannot be converted to text.
CONTEXT: JSON data, line 1: ..."hoursConfirmed": 0, "field_name":...
How i tried to resolve this is by using the following:
response = requests.post(url='some.url.com', data=json.dumps(data), headers={'some': 'header'})
response_data = json.loads(response.content.decode('utf-8'))
#handler is a object of a model
handler.api_response = json.loads(json.dumps(response_data).encode("unicode-escape").decode())
handler.save()
The initial issue was solved then. But recently, when i got a field with value 'field2_name': 'Hey "Whats up"'. This thing failed by giving error:
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 143 (char 142)
Probably because json.loads() got confused with " inside the value as an enclosing " and not an escaped ".
Now, If i print the initial response just after json.loads(response.content.decode('utf-8')) statement, it shows the field as \x00\x00\x00\x00\x00.
But output of following code:
response = requests.post(url='some.url.com', data=json.dumps(data), headers={'some': 'header'})
response_data = json.loads(response.content.decode('utf-8'))
print(json.dumps(response_data))
This shows the field as \\u0000\\u0000\\u0000\\u0000\\u0000.
How does \x00 change to \\u0000
And how do i save this field into postgres tables ?
This is what i could think of.
json.loads(json.dumps(response_data).replace('\\u0000',''))
To add this statement before saving to postgres.
Is there a better way ?
Is the code response_data = json.loads(response.content.decode('utf-8')) wrong ? Or causing not to escape that particular character ?

FWIW, I recently ran into this and your proposed solution also worked for me. I think it's probably the simplest way to deal with it. Apparently this is the only character that can't go into a Postgres JSON column.
json.loads(json.dumps(response_data).replace('\\u0000',''))

Raw sql to json in Django, with Datetime and Decimal MySql columns

I am using Ajax to make some requests from client to server, I am using DJango and I have used some Raw Sql queries before, but all of my fields was Int, varchar and a Decimal, for the last one I had an enconding problem, but I overrided the "default" property of Json and everything worked.
But that was before, now I have a query wich gives me Decimal and DateTime fields, both of them gave me enconding errors, the overrided "default" doesn't work now, thats why with this new one I used DjangoJSONEncoder, but now I have another problem, and its not an encoding one, I am using dictfetchall(cursor) method, recomended on Django docs, to return a dictionary from the Sql query, because cursor.fetchall() gives me this error: 'tuple' object has no attribute '_meta'.
Before I just sended that dictionary to json.dumps(response_data,default=default) and everything was fine, but now for the encoding I have to use the following: json.dumps(response_data,cls=DjangoJSONEncoder) and if I send the dictionary in that way, I get this error:
SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data
And if I try to use the serializers, like this:
response_data2= serializers.serialize('json', list(response_data))
And later send response_data2 to dumps, I get this error:
'dict' object has no attribute '_meta'
This is the code for the MySql query:
def consulta_sql_personalizada(nombres,apellidos,puesto):
from django.db import connection, transaction
cursor = connection.cursor()
cursor.execute("""select E.idEmpleado as id,CONCAT(Per.nombres_persona,' ',Per.apellidos_persona) as nombre,P.nombre_puesto as puesto,E.motivo_baja_empleado as motivo_baja,E.fecha_contratacion_empleado AS fecha_contratacion,E.fecha_baja_empleado as fecha_baja, SUM(V.total_venta) AS ventas_mes,E.fotografia_empleado as ruta_fotografia from Empleado as E
inner join Puesto as P on E.Puesto_idPuesto=P.idPuesto
inner join Venta as V on V.vendedor_venta=E.idEmpleado
inner join Persona as Per on E.Persona_idPersona=Per.idPersona
where (Per.nombres_persona like %s OR Per.apellidos_persona like %s OR E.Puesto_idPuesto=%s)
AND E.estado_empleado=1 AND V.estado_venta=1
AND
(YEAR(V.fecha_venta) = YEAR(Now())
AND MONTH(V.fecha_venta) = MONTH(Now()))""",[nombres,apellidos,puesto])
row = dictfetchall(cursor)
return row
And this is the last part of the view that makes the query and send it to ajax using json:
response_data=consulta_sql_personalizada(rec_nombres,rec_apellidos,rec_puesto)
return HttpResponse(
json.dumps(response_data,cls=DjangoJSONEncoder),
content_type="application/json"
)
else:
return HttpResponse(
json.dumps({"nothing to see": "this isn't happening"}),
content_type="application/json"
)
What I want to know is, how can I parse the raw sql result to Json using that enconding?

Sorry, was my bad, i'm using JQuery ajax method, and in the "success" part I forgot to stop using json.parse to print the data in the console, the data was json already, that's why I had that line 1 column 1 error. My code worked exactly like it was posted here. If someone want to know how to make asynchronous requests, I followed this tutorial: Django form submissions using ajax

How to POST multiple FILES using Flask test client?

In order to test a Flask application, I got a flask test client POSTing request with files as attachment
def make_tst_client_service_call1(service_path, method, **kwargs):
_content_type = kwargs.get('content-type','multipart/form-data')
with app.test_client() as client:
return client.open(service_path, method=method,
content_type=_content_type, buffered=True,
follow_redirects=True,**kwargs)
def _publish_a_model(model_name, pom_env):
service_url = u'/publish/'
scc.data['modelname'] = model_name
scc.data['username'] = "BDD Script"
scc.data['instance'] = "BDD Stub Simulation"
scc.data['timestamp'] = datetime.now().strftime('%d-%m-%YT%H:%M')
scc.data['file'] = (open(file_path, 'rb'),file_name)
scc.response = make_tst_client_service_call1(service_url, method, data=scc.data)
Flask Server end point code which handles the above POST request is something like this
#app.route("/publish/", methods=['GET', 'POST'])
def publish():
if request.method == 'POST':
LOG.debug("Publish POST Service is called...")
upload_files = request.files.getlist("file[]")
print "Files :\n",request.files
print "Upload Files:\n",upload_files
return render_response_template()
I get this Output
Files:
ImmutableMultiDict([('file', <FileStorage: u'Single_XML.xml' ('application/xml')>)])
Upload Files:
[]
If I change
scc.data['file'] = (open(file_path, 'rb'),file_name)
into (thinking that it would handle multiple files)
scc.data['file'] = [(open(file_path, 'rb'),file_name),(open(file_path, 'rb'),file_name1)]
I still get similar Output:
Files:
ImmutableMultiDict([('file', <FileStorage: u'Single_XML.xml' ('application/xml')>), ('file', <FileStorage: u'Second_XML.xml' ('application/xml')>)])
Upload Files:
[]
Question:
Why request.files.getlist("file[]") is returning an empty list?
How can I post multiple files using flask test client, so that it can be retrieved using request.files.getlist("file[]") at flask server side ?
Note:
I would like to have flask client I dont want curl or any other client based solutions.
I dont want to post single file in multiple requests
Thanks
Referred these links already:
Flask and Werkzeug: Testing a post request with custom headers
Python - What type is flask.request.files.stream supposed to be?

You send the files as the parameter named file, so you can't look them up with the name file[]. If you want to get all the files named file as a list, you should use this:
upload_files = request.files.getlist("file")
On the other hand, if you really want to read them from file[], then you need to send them like that:
scc.data['file[]'] = # ...
(The file[] syntax is from PHP and it's used only on the client side. When you send the parameters named like that to the server, you still access them using $_FILES['file'].)

Lukas already addressed this,just providing these info as it may help someone
Werkzeug client is doing some clever stuff by storing requests data in MultiDict
#native_itermethods(['keys', 'values', 'items', 'lists', 'listvalues'])
class MultiDict(TypeConversionDict):
"""A :class:`MultiDict` is a dictionary subclass customized to deal with
multiple values for the same key which is for example used by the parsing
functions in the wrappers. This is necessary because some HTML form
elements pass multiple values for the same key.
:class:`MultiDict` implements all standard dictionary methods.
Internally, it saves all values for a key as a list, but the standard dict
access methods will only return the first value for a key. If you want to
gain access to the other values, too, you have to use the `list` methods as
explained below.
getList call looks for a given key in the "requests" dictionary. If the key doesn't exist, it returns empty list.
def getlist(self, key, type=None):
"""Return the list of items for a given key. If that key is not in the
`MultiDict`, the return value will be an empty list. Just as `get`
`getlist` accepts a `type` parameter. All items will be converted
with the callable defined there.
:param key: The key to be looked up.
:param type: A callable that is used to cast the value in the
:class:`MultiDict`. If a :exc:`ValueError` is raised
by this callable the value will be removed from the list.
:return: a :class:`list` of all the values for the key.
"""
try:
rv = dict.__getitem__(self, key)
except KeyError:
return []
if type is None:
return list(rv)
result = []
for item in rv:
try:
result.append(type(item))
except ValueError:
pass
return result

Picture property format using run with friends fb app

I am editing the Runwithfriends FB sample app to build one of my own. It was working fine and I was making a few changes here and there. I took a break from it for about a fortnight but now when I try to access the app, I get a strange python error:
C:\Program Files\Apache Software Foundation\Tomcat 7.0\webapps\ROOT\app\main.py in init_facebook(self=<main.RecentRunsHandler object>)
316 user_id=facebook.user_id, friends=friends,
317 access_token=facebook.access_token, name=me[u'name'],
=> 318 email=me.get(u'email'), picture=me[u'picture'])
319 user.put()
320 except KeyError, ex:
<..some portion clipped..>
class 'google.appengine.api.datastore_errors.BadValueError'>: Property picture must be a str or unicode instance, not a dict
args = ('Property picture must be a str or unicode instance, not a dict',)
message = 'Property picture must be a str or unicode instance, not a dict'"
I know this is a very generic error but its pointing to the following code. This code has always been there and I have never touched it. I really do not know where else to look now - I have searched exhaustively but couldnt find a clue. Sorry, if this is still too broad but I would be glad if you can tell me what other info can I provide to debug this :-(
def init_facebook(self):
"""Sets up the request specific Facebook and User instance"""
facebook = Facebook()
user = None
# initial facebook request comes in as a POST with a signed_request
if u'signed_request' in self.request.POST:
facebook.load_signed_request(self.request.get('signed_request'))
# we reset the method to GET because a request from facebook with a
# signed_request uses POST for security reasons, despite it
# actually being a GET. in webapp causes loss of request.POST data.
self.request.method = u'GET'
self.set_cookie(
'u', facebook.user_cookie, datetime.timedelta(minutes=1440))
elif 'u' in self.request.cookies:
facebook.load_signed_request(self.request.cookies.get('u'))
# try to load or create a user object
if facebook.user_id:
user = User.get_by_key_name(facebook.user_id)
if user:
# update stored access_token
if facebook.access_token and \
facebook.access_token != user.access_token:
user.access_token = facebook.access_token
user.put()
# refresh data if we failed in doing so after a realtime ping
if user.dirty:
user.refresh_data()
# restore stored access_token if necessary
if not facebook.access_token:
facebook.access_token = user.access_token
if not user and facebook.access_token:
me = facebook.api(u'/me', {u'fields': _USER_FIELDS})
try:
friends = [user[u'id'] for user in me[u'friends'][u'data']]
user = User(key_name=facebook.user_id,
user_id=facebook.user_id, friends=friends,
access_token=facebook.access_token, name=me[u'name'],
email=me.get(u'email'), picture=me[u'picture'])
user.put()
except KeyError, ex:
pass # ignore if can't get the minimum fields
self.facebook = facebook
self.user = user

Might have to do with the October 2012 Breaking Changes, quote:
/picture connection will return a dictionary when a callback is specified
We will start returning a dictionary containing the fields url, height, width, and is_silhouette when accessing the /picture connection for an object and specifying a callback property. Currently we just return the picture URL as a string.
So at this point in your code, where you are currently using picture=me[u'picture'], try accessing the url property of the picture dictionary instead. (If it has one; I can’t tell you for sure if this is applicable, since I don’t know if your code would be considered as having specified a callback property.)
If my assumption is correct, you could also enable the migration as described in the roadmap; but that will only make your app work in the old way until Oct. 3rd, so probably better to try and fix it right away.

This is the way to get the picture:
picture=me[u'picture'][u'data'][u'url']

Pyramid on App Engine gets "InvalidResponseError: header values must be str, got 'unicode'

I am using Pyramid 1.3 with the AppEngine 1.6.4 SDK on OS X 10.7.3. I am using Python 2.7 and have threadsafe true in app.yaml.
#view_config(route_name='manager_swms', permission='manager', renderer='manager/swms.jinja2')
def manager_swms(request):
"""Generates blobstore url and passes users swms in swms table"""
# generate url for any form upload that may occur
upload_url = blobstore.create_upload_url('/upload_swm')
user = get_current_user(request)
swms = DBSession.query(SWMS).filter_by(owner_id=int(user.id)).all()
return {
"analytics_id": analytics_id,
"user": get_current_user(request),
"upload_url": upload_url,
"swms": [(x.filename, x.blob_key) for x in swms]
}
class BlobstoreUploadHandler(object):
"""Base class for creation blob upload handlers."""
def __init__(self, *args, **kwargs):
self.__uploads = None
def get_uploads(self, field_name=None):
"""Get uploads sent to this handler.
Args:
field_name: Only select uploads that were sent as a specific field.
Returns:
A list of BlobInfo records corresponding to each upload.
Empty list if there are no blob-info records for field_name.
"""
if self.__uploads is None:
self.__uploads = {}
for key, value in self.request.params.items():
if isinstance(value, cgi.FieldStorage):
if 'blob-key' in value.type_options:
self.__uploads.setdefault(key, []).append(
blobstore.parse_blob_info(value))
if field_name:
try:
return list(self.__uploads[field_name])
except KeyError:
return []
else:
results = []
for uploads in self.__uploads.itervalues():
results += uploads
return results
#view_config(route_name='upload_swm', permission='manager')
class UploadHandler(BlobstoreUploadHandler):
''' Handles redirects from Blobstore uploads. '''
def __init__(self, request):
self.request = request
super(UploadHandler, self).__init__()
def __call__(self):
user = get_current_user(self.request)
for blob_info in self.get_uploads('file'):
new_swm = SWMS(
owner_id = int(user.id),
blob_key = str(blob_info.key()),
filename = blob_info.filename,
size = blob_info.size,
)
DBSession.add(new_swm)
DBSession.flush()
# redirect to swms page
return HTTPFound(location='/manager/swms')
In the above code, manager_swms() generates a page which includes a form for uploading a file into the Blobstore. The form works OK and I can see the blob appear in the Blobstore when the form is used. The redirect from the blobstore POST is then to /upload_swm where I successfully take BlobInfo details and place them into a SQL table. All of this is good and the final thing I want to do is redirect to the first page so another file can be uploaded if need be and I can show the list of files uploaded.
As per Pyramid documentation, I am using HTTPFound(location='/manager/swms') [the original page URL] to try and redirect however I get:
ERROR 2012-03-29 22:56:38,170 wsgi.py:208]
Traceback (most recent call last):
File "/Users/tim/work/OHSPro/var/parts/google_appengine/google/appengine/runtime/wsgi.py", line 196, in Handle
result = handler(self._environ, self._StartResponse)
File "lib/dist/pyramid/router.py", line 195, in __call__
foo = response(request.environ, start_response)
File "lib/dist/pyramid/httpexceptions.py", line 291, in __call__
foo = Response.__call__(self, environ, start_response)
File "lib/dist/webob/response.py", line 922, in __call__
start_response(self.status, headerlist)
File "/Users/tim/work/OHSPro/var/parts/google_appengine/google/appengine/runtime/wsgi.py", line 150, in _StartResponse
_GetTypeName(header[1]))
InvalidResponseError: header values must be str, got 'unicode'
INFO 2012-03-29 22:56:38,174 dev_appserver_blobstore.py:408] Upload handler returned 500
INFO 2012-03-29 22:56:38,193 dev_appserver.py:2884] "POST /_ah/upload/ahJkZXZ-cHJvdG8tc2NvaHNwcm9yGwsSFV9fQmxvYlVwbG9hZFNlc3Npb25fXxgTDA HTTP/1.1" 500 -
AppEngine is clearly objecting to unicode in the HTTP header but I'm not doing anything unusual AFAIK. If I drop into pdb and take a look at the HTTPFound object, the headers are:
ResponseHeaders([('Content-Type', 'text/html; charset=UTF-8'), ('Content-Length', '0'), ('Location', '/manager/swms')])
Why would I get a unicode problem from these?

so it looks like you are overriding appengine's supported webob which is 1.1.1 on the 2.7 runtime. And pyramid 1.3 depends on webob>=1.2. This is most likely the problem because it was the Blobstore handler stuff that was keeping the sdk held back at webob==0.9 until SDK 1.6.4 was released.
FWIW, this problem is expected to be resolved by SDK 1.6.5 (late April). The only reason I know this is because I was trying to get all this crap resolved when the 2.7 runtime was deemed ready for general use, yet the SDK didn't support it. see this issue for more details.
If possible, I would suggest running it with pyramid 1.2, I know that works fine on appengine. And Just hold off on moving to 1.3 for a few weeks. :)

Yet another answer... adding a str() fixes the problem, but not the root cause. I spent hours trying to figure out why one particular redirect raised this error while others didn't, before noticing that the URL for the bad redirect was missing the initial "/".
I have no idea why this is the case - possibly an incomplete path is processed differently from a complete one. But if you hit this error try changing:
self.redirect('home.view')
to:
self.redirect('/home.view')