I have been working with a couple of web APIs but this one has me perplexed. I can't work out what im going wrong.
this code works on one api, but not this one.
response = urllib.request.urlopen(self.query_base)
reader = codecs.getreader("utf-8")
obj = json.load(reader(response))
return obj
this gives me the following errror
UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f602' in position 4096: character maps to <undefined>
I have tried:
response = urllib.request.urlopen(self.query_base)
obj = json.load(response.decode("utf-8"))
return obj
which gives:
AttributeError: 'HTTPResponse' object has no attribute 'decode'
and,
response = urllib.request.urlopen(self.query_base).read()
obj = json.load(response)
return obj
which gives
AttributeError: 'bytes' object has no attribute 'read'
and,
response = urllib.request.urlopen(self.query_base)
obj = json.load(response)
which gives
TypeError: the JSON object must be str, not 'bytes'
along with maany other combinations of things I have found in other similar threads on here
I don't remember ever having this problem before, im sure ive missed something but I can't see what.
Your initial attempt is already correct. If it wasn't, you'd get decoding errors.
You have, instead, got encoding errors, moving from Unicode back to bytes. This is invariably caused by you using print() or trying to write the data to a file object. When printing, this is usually caused by your console not being able to handle the output. See python3 print unicode to windows xp console encode cp437 for example.
Your second attempt failed because you didn't read the response, then use json.loads() (since you are passing in a string now):
response = urllib.request.urlopen(self.query_base)
obj = json.loads(response.read().decode("utf-8"))
# ^ ^^^^^^^
return obj
Your 3rd attempt did use .read() but you forgot to decode that time, and again did not use json.loads():
response = urllib.request.urlopen(self.query_base).read()
# you didn't forget this ^^^^^^^
obj = json.loads(response.decode('utf-8'))
# ^ ^^^^^^^^^^^^^^^^
return obj
The last attempt passed in the raw response, without decoding the data.
response = urllib.request.urlopen(self.query_base).read()
obj = json.loads(response)
return obj
This code should work. json.load is looking to read from a file stream. json.loads is what you want when reading JSON from a string.
ok, just incase anyone has this issue and reads this thread later
response = urllib.request.urlopen(self.query_base)
reader = codecs.getreader("utf-8")
obj = json.load(reader(response))
return ascii(obj)
worked fine.
thanks to Martijn Pieters
Related
I have a JSON file hosted locally in my Django directory. It is fetched from that file to a view in views.py, where it is read in like so:
def Stops(request):
json_data = open(finders.find('JSON/myjson.json'))
data1 = json.load(json_data) # deserialises it
data2 = json.dumps(data1) # json formatted string
json_data.close()
return JsonResponse(data2, safe=False)
Using JsonResponse without (safe=False) returns the following error:
TypeError: In order to allow non-dict objects to be serialized set the safe parameter to False.
Similarly, using json.loads(json_data.read()) instead of json.load gives this error:
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
This is confusing to me - I have validated the JSON using an online validator. When the JSON is sent to the frontend with safe=False, the resulting object that arrives is a string, even after calling .json() on it in javascript like so:
fetch("/json").then(response => {
return response.json();
}).then(data => {
console.log("data ", data); <---- This logs a string to console
...
However going another step and calling JSON.parse() on the string converts the object to a JSON object that I can use as intended
data = JSON.parse(data);
console.log("jsonData", data); <---- This logs a JSON object to console
But this solution doesn't strike me as a complete one.
At this point I believe the most likely thing is that there is something wrong with the source JSON - (in the file character encoding?) Either that or json.dumps() is not doing what I think it should, or I am not understanding the Django API's JSONresponse function in a way I'm not aware of...
I've reached the limit of my knowledge on this subject. If you have any wisdom to impart, I would really appreciate it.
EDIT: As in the answer below by Abdul, I was reformatting the JSON into a string with the json.dumps(data1) line
Working code looks like:
def Stops(request):
json_data = open(finders.find('JSON/myjson.json'))
data = json.load(json_data) # deserialises it
json_data.close()
return JsonResponse(data, safe=False) # pass the python object here
Let's see the following lines of your code:
json_data = open(finders.find('JSON/myjson.json'))
data1 = json.load(json_data) # deserialises it
data2 = json.dumps(data1) # json formatted string
You open a file and get a file pointer in json_data, parse it's content and get a python object in data1 and then turn it back into a JSON string and store it into data2. Somewhat redundant right? Next you pass this JSON string to JsonResponse which will further try to serialize it into JSON!! Meaning you then get a string inside a string in JSON.
Try the following code instead:
def Stops(request):
json_data = open(finders.find('JSON/myjson.json'))
data = json.load(json_data) # deserialises it
json_data.close()
return JsonResponse(data, safe=False) # pass the python object here
Note: function names in python should ideally be in snake_case not PascalCase, hence instead of Stops you should use stops. See
PEP 8 -- Style Guide for Python
Code
This question already has answers here:
How can I parse (read) and use JSON?
(5 answers)
Closed 25 days ago.
In Python I'm getting an error:
Exception: (<type 'exceptions.AttributeError'>,
AttributeError("'str' object has no attribute 'read'",), <traceback object at 0x1543ab8>)
Given python code:
def getEntries (self, sub):
url = 'http://www.reddit.com/'
if (sub != ''):
url += 'r/' + sub
request = urllib2.Request (url +
'.json', None, {'User-Agent' : 'Reddit desktop client by /user/RobinJ1995/'})
response = urllib2.urlopen (request)
jsonStr = response.read()
return json.load(jsonStr)['data']['children']
What does this error mean and what did I do to cause it?
The problem is that for json.load you should pass a file like object with a read function defined. So either you use json.load(response) or json.loads(response.read()).
Ok, this is an old thread but.
I had a same issue, my problem was I used json.load instead of json.loads
This way, json has no problem with loading any kind of dictionary.
Official documentation
json.load - Deserialize fp (a .read()-supporting text file or binary file containing a JSON document) to a Python object using this conversion table.
json.loads - Deserialize s (a str, bytes or bytearray instance containing a JSON document) to a Python object using this conversion table.
You need to open the file first. This doesn't work:
json_file = json.load('test.json')
But this works:
f = open('test.json')
json_file = json.load(f)
If you get a python error like this:
AttributeError: 'str' object has no attribute 'some_method'
You probably poisoned your object accidentally by overwriting your object with a string.
How to reproduce this error in python with a few lines of code:
#!/usr/bin/env python
import json
def foobar(json):
msg = json.loads(json)
foobar('{"batman": "yes"}')
Run it, which prints:
AttributeError: 'str' object has no attribute 'loads'
But change the name of the variablename, and it works fine:
#!/usr/bin/env python
import json
def foobar(jsonstring):
msg = json.loads(jsonstring)
foobar('{"batman": "yes"}')
This error is caused when you tried to run a method within a string. String has a few methods, but not the one you are invoking. So stop trying to invoke a method which String does not define and start looking for where you poisoned your object.
AttributeError("'str' object has no attribute 'read'",)
This means exactly what it says: something tried to find a .read attribute on the object that you gave it, and you gave it an object of type str (i.e., you gave it a string).
The error occurred here:
json.load(jsonStr)['data']['children']
Well, you aren't looking for read anywhere, so it must happen in the json.load function that you called (as indicated by the full traceback). That is because json.load is trying to .read the thing that you gave it, but you gave it jsonStr, which currently names a string (which you created by calling .read on the response).
Solution: don't call .read yourself; the function will do this, and is expecting you to give it the response directly so that it can do so.
You could also have figured this out by reading the built-in Python documentation for the function (try help(json.load), or for the entire module (try help(json)), or by checking the documentation for those functions on http://docs.python.org .
Instead of json.load() use json.loads() and it would work:
ex:
import json
from json import dumps
strinjJson = '{"event_type": "affected_element_added"}'
data = json.loads(strinjJson)
print(data)
So, don't use json.load(data.read()) use json.loads(data.read()):
def findMailOfDev(fileName):
file=open(fileName,'r')
data=file.read();
data=json.loads(data)
return data['mail']
use json.loads() function , put the s after that ... just a mistake btw i just realized after i searched error
def getEntries (self, sub):
url = 'http://www.reddit.com/'
if (sub != ''):
url += 'r/' + sub
request = urllib2.Request (url +
'.json', None, {'User-Agent' : 'Reddit desktop client by /user/RobinJ1995/'})
response = urllib2.urlopen (request)
jsonStr = response.read()
return json.loads(jsonStr)['data']['children']
try this
Open the file as a text file first
json_data = open("data.json", "r")
Now load it to dict
dict_data = json.load(json_data)
If you need to convert string to json. Then use loads() method instead of load(). load() function uses to load data from a file so used loads() to convert string to json object.
j_obj = json.loads('["label" : "data"]')
I have this String that I need to pass into a REST request
{"notification":{"tag":"MyTag"}}
I'm trying to turn into an object using the JSON module in python.
This is my attempt so far
import json
obj = json.dumps([{'notification': ('{tag : MyTag}')}])
But it isn't parsed correctly so the REST request won't work. Anyone have any ideas?
Just dump your dictionary as is, replace:
obj = json.dumps([{'notification': ('{tag : MyTag}')}])
with:
obj = json.dumps({"notification": {"tag": "MyTag"}})
I have a webservice that reads a JSON object and it was giving me unicodeEncodeError exception. After googling a little bit, I saw How can I convert a dict to a unicode JSON string?
(I followed other questions that were related to unicodeEncodeError but I was still getting AttributeError: 'dict' object has no attribute 'content')
I did what was mentioned in that particular question and now I am getting ..... is not JSON serializable
Can anyone tell me what do I have to do now?
Following is my code:
def someMethod()
some_data = request.data
json_string1 = json.dumps(some_data) #GETTING ERROR ON THIS LINE
json_string2 = get_string(json_string1)
archive = call.send_json(json_string2)
def get_string(value):
find_unicode = re.compile('\\\\u([\da-f]{4})')
def get_parsed_unicode(x):
return chr(int(x.group(1), 16))
return find_unicode.sub(get_parsed_unicode, str(value))
Thanks for the help !!
When you're passing the object to the method, use foo.serealize(true). Include JQUERY to use this.
I have a function accepting requests from the network. Most of the time, the string passed in is not unicode, but sometimes it is.
I have code to convert everything to unicode, but it reports this error:
message.create(username, unicode(body, "utf-8"), self.get_room_name(),\
TypeError: decoding Unicode is not supported
I think the reason is the 'body' parameter is already unicode, so unicode() raises an exception.
Is there any way to avoid this exception, e.g. judge the type before the conversion?
You do not decode to UTF-8, you encode to UTF-8 or decode from.
You can safely decode from UTF8 even if it's just ASCII. ASCII is a subset of UTF8.
The easiest way to detect if it needs decoding or not is
if not isinstance(data, unicode):
# It's not Unicode!
data = data.decode('UTF8')
You can use either this:
try:
body = unicode(body)
except UnicodeDecodeError:
body = body.decode('utf8')
Or this:
try:
body = unicode(body, 'utf8')
except TypeError:
body = unicode(body)
Mark Pilgrim wrote a Python library to guess text encodings:
http://chardet.feedparser.org/
On Unicode and UTF-8, the first two sections of chapter 4 of his book ‘Dive into Python 3’ are pretty great:
http://diveintopython3.org/strings.html
This is what I use:
def to_unicode_or_bust(obj, encoding='utf-8'):
if isinstance(obj, basestring):
if not isinstance(obj, unicode):
obj = unicode(obj, encoding)
return obj
It's taken from this presentation: http://farmdev.com/talks/unicode/
And this is a sample code that uses it:
def hash_it_safe(s):
try:
s = to_unicode_or_bust(s)
return hash_it_basic(s)
except UnicodeDecodeError:
return hash_it_basic(s)
except UnicodeEncodeError:
assert type(s) is unicode
return hash_it_basic(s.encode('utf-8'))
Anyone have some thoughts on how to improve this code? ;)