I've been trying to figure out how to load JSON objects in Python.
def do_POST(self):
length = int(self.headers['Content-Length'])
decData = str(self.rfile.read(length))
print decData, type(decData)
"{'name' : 'journal2'}" <type 'str'>
postData = json.loads(decData)
print postData, type(postData)
#{'name' : 'journal2'} <type 'unicode'>
postData = json.loads(postData)
print postData, type(postData)
# Error: Expecting property name enclosed in double quotes
Where am I going wrong?
Error Code (JScript):
var data = "{'name':'journal2'}";
var http_request = new XMLHttpRequest();
http_request.open( "post", url, true );
http_request.setRequestHeader('Content-Type', 'application/json');
http_request.send(data);
True Code (JScript):
var data = '{"name":"journal2"}';
var http_request = new XMLHttpRequest();
http_request.open( "post", url, true );
http_request.setRequestHeader('Content-Type', 'application/json');
http_request.send(JSON.stringify(data));
True Code (Python):
def do_POST(self):
length = int(self.headers['Content-Length'])
decData = self.rfile.read(length)
postData = json.loads(decData)
postData = json.loads(postData)
Your JSON data is enclosed in extra quotes making it a JSON string, and the data contained within that string is not JSON.
Print repr(decData) instead, you'll get:
'"{\'name\' : \'journal2\'}"'
and the JSON library is correctly interpreting that as one string with the literal contents {'name' : 'journal2'}. If you stripped the outer quotes, the contained characters are not valid JSON, because JSON strings must always be enclosed in double quotes.
For all the json module is concerned, decData could just as well have contained "This is not JSON" and postData would have been set to u'This is not JSON'.
>>> import json
>>> decData = '''"{'name' : 'journal2'}"'''
>>> json.loads(decData)
u"{'name' : 'journal2'}"
>>> json.loads(json.loads(decData))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 326, in loads
return _default_decoder.decode(s)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting property name: line 1 column 1 (char 1)
Fix whatever is producing this string, your view is fine, it's the input that is broken.
To workaround the error 'Expecting property name enclosed in double quotes' you can do:
import json
data = "{'name' : 'journal2'}"
json.loads(json.dumps(data))
You can also go with eval:
data = "{'name' : 'test'}"
eval(data)
It works and returns you a dict.
Related
I want to work with RestAPI from kaufland.de website. I followed the instruction to sign a request from : this website .
Here is the code, they gave me as an example to run to sign a request by a SHA-256 HMAC in base64 encoding:
import hmac
import hashlib
import time
def sign_request(method, uri, body, timestamp, secret_key):
plain_text = "\n".join([method, uri, body, str(timestamp)])
digest_maker = hmac.new(secret_key, '', hashlib.sha256)
digest_maker.update(plain_text)
return digest_maker.hexdigest()
method = "POST"
uri = "https://www.kaufland.de/api/v1/units/"
body = ""
timestamp = time.time()
secret_key = "83cfe0909f4e57f05dd403"
print(sign_request(method, uri, body, timestamp, secret_key))
But the code above threw an error :
TypeError: key: expected bytes or bytearray, but got 'str'
I found a solution from SO1, added b in front of secret_key :
secret_key = b'83cfe0909f4e57f05dd403'
However, it still threw an error when I run :
TypeError: Unicode-objects must be encoded before hashing
So I followed the solution from SO2 by importing base64 package :
import hmac
import hashlib
import time
import base64
def sign_request(method, uri, body, timestamp, secret_key):
plain_text = "\n".join([method, uri, body, str(timestamp)])
digest_maker = hmac.new(secret_key, '', hashlib.sha256)
digest_maker.update(plain_text)
return base64.b64encode(digest_maker.hexdigest())
method = "POST"
uri = "https://www.kaufland.de/api/v1/units/"
body = ""
timestamp = time.time()
secret_key = b"a7d0cb1da1ddbc86c96ee5fedd341b7d8ebfbb2f5c83cfe0909f4e57f05dd403"
print(sign_request(method, uri, body, timestamp, secret_key))
But it still threw an error :
Traceback (most recent call last):
File "<ipython-input-54-83e727ea1edf>", line 22, in <module>
print(sign_request(method, uri, body, timestamp, secret_key))
File "<ipython-input-54-83e727ea1edf>", line 10, in sign_request
digest_maker = hmac.new(secret_key, '', hashlib.sha256)
File "C:\ProgramData\Anaconda3\lib\hmac.py", line 153, in new
return HMAC(key, msg, digestmod)
File "C:\ProgramData\Anaconda3\lib\hmac.py", line 88, in __init__
self.update(msg)
File "C:\ProgramData\Anaconda3\lib\hmac.py", line 96, in update
self.inner.update(msg)
TypeError: Unicode-objects must be encoded before hashing
Anyone can help me in this case ?
Try to encode your secret_key parameter on assign or in hmac.new.
Example:
secret_key = "83cfe0909f4e57f05dd403".encode('utf-8')
Or like I wrote above:
digest_maker = hmac.new(secret_key.encode('utf-8'), '', hashlib.sha256)
Upd:
import hmac
import hashlib
import time
def sign_request(method, uri, body, timestamp, secret_key):
plain_text = "\n".join([method, uri, body, str(timestamp)])
digest_maker = hmac.new(secret_key.encode('utf-8'), msg=''.encode('utf-8'), digestmod=digestmod)
digest_maker.update(plain_text.encode('utf-8'))
return digest_maker.hexdigest()
method = "POST"
uri = "https://www.kaufland.de/api/v1/units/"
body = ""
timestamp = time.time()
secret_key = "a7d0cb1da1ddbc86c96ee5fedd341b7d8ebfbb2f5c83cfe0909f4e57f05dd403"
digestmod = hashlib.sha256
print(sign_request(method, uri, body, timestamp, secret_key))
I would think there is not problem with that, but I am really having troubles with this piece of code and cannot seem to come up with a solution.
I have a dictionary whose keys are proper name, e.g. John Green, and I am using the API of the Sunlight Foundation to retrieve information about congress members (check here). Now I need to request using name and lastname so my code looks something like this:
for key in my_dict:
query_params2 = { 'apikey': 'xxxxxxxxxxx',
'firstname' : key.split()[0],
'lastname' : key.split()[-1]
}
endpoint2 = "http://services.sunlightlabs.com/api/legislators.get.json"
resp2 = requests.get(endpoint2, params = query_params2)
data2 = resp2.json().decode('utf-8')
print data2['response']['legislator']['bioguide_id']
Which gives some error I can not really interpret:
Traceback (most recent call last):
File "my_program.py", line 102, in <module>
data = resp.json()
File "//anaconda/lib/python2.7/site-packages/requests/models.py", line 741, in json
return json.loads(self.text, **kwargs)
File "//anaconda/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "//anaconda/lib/python2.7/json/decoder.py", line 365, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "//anaconda/lib/python2.7/json/decoder.py", line 383, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
I would guess it has something to do with the encoding, but I am not sure what and how to solve it.
Needless to say, if I put by hand a name and lastname, the request works perfectly.
Anybody can help with this? Thanks a lot!
This has nothing to do with the encoding. The answer simply isn't JSON. When I try your code with 'John' and 'Green' I get a 400 Bad Request and the content of the response is 'No Such Object Exists'.
Trying John Green in the web interface also comes up with an empty answer. Also the URL in the API documentation differs from the URL in your example.
The following works for me (again no John Green):
import requests
LEGISLATORS_URL = 'https://congress.api.sunlightfoundation.com/legislators'
API_KEY = 'xxxx'
def main():
names = [('John', 'Green'), ('John', 'Kerry')]
for first_name, last_name in names:
print 'Checking', first_name, last_name
response = requests.get(
LEGISLATORS_URL,
params={
'apikey': API_KEY,
'first_name': first_name,
'last_name': last_name,
'all_legislators': 'true'
}
).json()
print response['count']
if response['count'] > 0:
print response['results'][0]['bioguide_id']
if __name__ == '__main__':
main()
Output:
Checking John Green
0
Checking John Kerry
1
K000148
My code is:
class GameSession(JSONMessageHandler, UserHandlingMixin):
#allow_all
def post(self):
s = self.s = Session()
payload = self.message.body
# payload = json.loads(payload)
print 'payload start'
print payload
print 'payload end'
current_gamesession = self.check_gamesession(payload.prize_id,
payload.publisher_id)
What I see is:
payload start
prize_id=2&publisher_id=32&foreign_user_id=1234
payload end
ERROR:root:'str' object has no attribute 'prize_id'
Traceback (most recent call last):
File "/home/vagrant/src/brubeck/brubeck/request_handling.py", line 338, in __call__
rendered = fun(*self._url_args)
File "/vagrant/mysite/api/views.py", line 31, in wrapper
return func(self, *args, **kwargs)
File "/vagrant/mysite/api/views.py", line 629, in post
How do I get the form data?
** EDIT **
As an aside, that's if I send data with x-www-form-urlencoded. If I send as form-data, I get:
payload start
------WebKitFormBoundaryFX1GuivvAA42T3uk
Content-Disposition: form-data; name="prize_id"
2
------WebKitFormBoundaryFX1GuivvAA42T3uk
Content-Disposition: form-data; name="publisher_id"
1
------WebKitFormBoundaryFX1GuivvAA42T3uk
Content-Disposition: form-data; name="foreign_user_id"
2321
------WebKitFormBoundaryFX1GuivvAA42T3uk--
payload end
Traditionally you would create a cgi.FieldStorage object, which reads stdin (usually - there are CGI standards about what it does and when). That's a bit passé nowadays. Urlparse.parse_qs is designed to convert from form data to dict:
>>> import urlparse
>>> urlparse.parse_qs("prize_id=2&publisher_id=32&foreign_user_id=1234")
{'prize_id': ['2'], 'foreign_user_id': ['1234'], 'publisher_id': ['32']}
>>>
I am trying to write a python web app that will take some sql and a bunch of other things and return a Json file, the latter part is not the issue and I have not even put it in the script yet, the issue is that the url being passed is being utf-8 encoded and then url encoded
turning our example
query :SELECT + ;
test: 2
into
test=2&query=SELECT+%2B+%3B
This seems to be ok
but the receiving get seems to think that it can expand the codes back into chars
and it receives
test=2&query=SELECT+++;
then this is url decoded and it chops off the semicolon, and i want to keep the semicolon!
it also turns the +'s which are rightly spaces into spaces but the previous bug made the real plus code into a literal plus which turns it into a space!
{'test': '2', 'query': 'SELECT '}
code is as follows:
#!/usr/bin/python
import web
import psycopg2
import re
import urllib
import urlparse
urls = (
'/query', 'query',
'/data/(.*)', 'data'
)
app = web.application(urls, globals())
render = web.template.render('templates/')
class query:
def GET(self):
return render.query()
def POST(self):
i = web.input()
data = {}
data['query'] = i.sql.encode('utf-8')
data['test'] = '2'
murl = urllib.urlencode(data)
return "go!"
class data:
def GET(self, urlEncodedDict):
print "raw type:", type(urlEncodedDict)
print "raw:", urlEncodedDict
urlEncodedDict = urlEncodedDict.encode('ascii', 'ignore')
print "ascii type:", type(urlEncodedDict)
print "ascii:", urlEncodedDict
data = dict(urlparse.parse_qsl(urlEncodedDict, 1)) #bad bit
print "dict:", data
print "element:", data['query']
if ( re.match('SELECT [^;]+ ;', data['query'])):
return 'good::'+data['query']
else:
return 'Bad::'+data['query']
if __name__ == "__main__":
app.run()
Url generated from my test form is:
http://localhost:8080/data/test=2&query=SELECT+%2B+%3B
Output is as follows:
raw type: <type 'unicode'>
raw: test=2&query=SELECT+++;
ascii type: <type 'str'>
ascii: test=2&query=SELECT+++;
dict: {'test': '2', 'query': 'SELECT '}
element: SELECT
127.0.0.1:53272 - - [16/Nov/2012 11:05:44] "HTTP/1.1 GET /data/test=2&query=SELECT+++;" - 200 OK
127.0.0.1:53272 - - [16/Nov/2012 11:05:44] "HTTP/1.1 GET /favicon.ico" - 404 Not Found
I wish to get the same dict out of the get that i encode in the first place.
If you want to pass data into a GET request, you need to use the query string syntax using the question mark character [?] as a delimiter.
The URL should be:
http://localhost:8080/data/?test=2&query=SELECT+%2B+%3B
After that, you just have to use web.input() to get a dictionary with all arguments already decoded.
urls = (
'/query', 'query',
'/data/', 'data'
)
[...]
class data:
def GET(self):
data = web.input()
print "dict:", data
print "element:", data['query']
if ( re.match('SELECT [^;]+ ;', data['query'])):
return 'good::'+data['query']
else:
return 'Bad::'+data['query']
Result:
dict: <Storage {'test': u'2', 'query': u'SELECT + ;'}>
element: SELECT + ;
127.0.0.1:44761 - - [16/Nov/2012 15:06:06] "HTTP/1.1 GET /data/" - 200 OK
I'm trying to post a JSON object through a POST. I'm trying to do it as follows:
import json, urllib, urllib2
filename = 'test.json'
race_id = 2530
f = open(filename, 'r')
fdata = json.loads(f.read())
f.close()
prefix = 'localhost:8000'
count = 0
for points in fdata['positions'].iteritems():
print '--' + str(count) + '--------'
url = 'http://'+prefix+'/api/points'
parameters = {'point_data': json.dumps(points), 'race_id': race_id}
data = urllib.urlencode(parameters)
print data
request = urllib2.Request(url, data)
response = urllib2.urlopen(request)
count += 1
break;
print 'Finished adding points'
The data is then received on the other end (I'm using Google App Engine) with:
point_data = json.load(self.request.get('point_data'))
But I get the following error:
ERROR 2010-06-30 15:08:05,367
__init__.py:391] 'unicode' object has no attribute 'read' Traceback (most
recent call last): File
"/home/ian/workspace/google_appengine/google/appengine/ext/webapp/__init__.py",
line 513, in __call__
handler.post(*groups) File "/home/ian/workspace/codebase/track_builder/geo-api.py",
line 276, in post
point_data = json.load(self.request.get('point_data'))
File
"/home/ian/workspace/google_appengine/lib/django/django/utils/simplejson/__init__.py",
line 208, in load
return cls(encoding=encoding, **kw).decode(fp.read()) AttributeError: 'unicode' object has
no attribute 'read' INFO
2010-06-30 15:08:05,376
dev_appserver.py:3266] "POST
/api/points HTTP/1.1" 500 -
Any ideas on how to fix this?
EDIT: As requested here is an example of the points:
(u'1276859700',
{
u'24': {
u'tempc': u'-22.7',
u'gpsq': u'1',
u'altm': u'65527',
u'hd': u'112',
u'hdop': u'0.93',
u'bton': u'0',
u'maxv': u'20.15',
u'idit': u'1',
u'satc': u'10',
u'minv': u'20.15',
u'lat': u'35.271993',
u'btusr': u'0',
u'lng': u'-121.845353',
u'knots': u'7'
},
u'11': {
u'tempc': u'13.0',
u'gpsq': u'1',
u'altm': u'65535',
u'hd': u'130',
u'hdop': u'0.84',
u'bton': u'0',
u'maxv': u'15.96',
u'idit': u'1',
u'satc': u'12',
u'minv': u'15.88',
u'lat': u'34.877815',
u'btusr': u'0',
u'lng': u'-121.386116',
u'knots': u'8'
}
}
EDIT 2:
Thanks to Daniel Roseman and Nick Johnson who both caught my error. I've changed
point_data = json.loads(self.request.get('point_data'))
This has solved the error but, now I'm getting:
ERROR 2010-06-30 16:07:29,807 __init__.py:391] 'list' object has no attribute 'iteritems'
Traceback (most recent call last):
File "/home/ian/workspace/google_appengine/google/appengine/ext/webapp/__init__.py", line 513, in __call__
handler.post(*groups)
File "/home/ian/workspace/codebase/track_builder/geo-api.py", line 255, in post
for time, units in point_data.iteritems():
AttributeError: 'list' object has no attribute 'iteritems'
INFO 2010-06-30 16:07:29,816 dev_appserver.py:3266] "POST /api/points HTTP/1.1" 500 -
which relates to the following code:
class TrackPoint(webapp.RequestHandler):
def post(self):
point_data = json.loads(self.request.get('point_data'))
race_id = self.request.get('race_id')
added = []
failed = []
for time, units in point_data.iteritems():
for unit, data in units.iteritems():
...
Any ideas on this one?
It looks like self.request.get() is returning a unicode object rather than a file-like object. You could try using json.loads() instead of json.load().
json.load() expects a file object, but self.request.get returns the value of the parameter as a string.
The solution is easy: use json.loads.
Also, free tip: I presume from the name that you're bundling your own copy of the json library. App Engine actually includes a copy of simplejson that you can use - just do:
from django.utils import simplejson
first in js I recognize that json object must in STRING format (javascript file)
// using jquery, json2
var js = {"name":"nguyen","age":"1"};
$.post("/", {'data': JSON.stringify(js)}, function(ret){
alert(ret);
});
then in gae
from django.utils import simplejson as json
class PesonReq(webapp.RequestHandler):
def post(self):
t = json.loads(self.request.POST['data'])
self.response.out.write(t['name'])