Python - Having trouble creating a dictionary from a JSON - python

I'm trying to take the JSON from a twitter get_user query and turn it into a Python object that I can extract data from (twitter handle, location, screen name, etc.)
Here is what I created. I am not sure why it doesn't work.
api = tweepy.API(auth,parser=tweepy.parsers.JSONParser())
user = api.search_users('google.com')
t_dict = json.loads(user)
pprint(t_dict)
Error:
Traceback (most recent call last):
File "Get_User_By_URL.py", line 23, in <module>
t_dict = json.loads(user)
File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer


api.search_users is already returning a python object. It isn't a json string that needs to be parsed. According to tweetpy documentation search_users actually returns a list of users. So the following is possible:
for user in api.search_users('google.com'):
print user.screen_name

Related

python load json data error

I want to load this JSON data in python :
JSON data:
{"status":"success","result":{"users":{"5543295":{"status":{"payment_verified":false,"identity_verified":false,"email_verified":true,"deposit_made":false,"phone_verified":false,"facebook_connected":false,"profile_complete":true},"avatar_large":"/img/unknown.png","avatar_cdn":"//cdn6.f-cdn.com/img/unknown.png","spam_profile":null,"search_languages":null,"corporate_users":null,"support_status":null,"last_name":null,"suspended":null,"primary_language":"en","timezone":{"country":"US","offset":-7.0,"id":105,"timezone":"America/Los_Angeles"},"membership_package":null,"qualifications":[{"description":"Foundation vWorker Member","level":null,"icon_url":"/img/insignia/vworker.png","icon_name":"vworker","score_percentage":null,"user_percentile":null,"type":null,"id":null,"name":null}],"id":5543295,"badges":null,"hourly_rate":null,"responsiveness":null,"first_name":null,"display_name":"vw693015vw","tagline":null,"account_balances":null,"public_name":"vw693015vw","role":"employer","location":{"administrative_area":null,"city":null,"country":{"highres_flag_url":"/img/flags/highres_png/unknown.png","code":null,"name":"","flag_url_cdn":"//cdn5.f-cdn.com/img/flags/png/unknown.png","highres_flag_url_cdn":"//cdn5.f-cdn.com/img/flags/highres_png/unknown.png","flag_url":"/img/flags/png/unknown.png"},"vicinity":null,"longitude":null,"full_address":null,"latitude":null},"closed":false,"email":null,"username":"vw693015vw","is_local":null,"endorsements":null,"jobs":[],"employer_reputation":{"user_id":5543295,"last3months":{"completion_rate":null,"all":null,"incomplete_reviews":null,"complete":null,"on_time":null,"on_budget":null,"positive":0.0,"overall":0.0,"reviews":null,"category_ratings":{"communication":0.0,"clarity_spec":0.0,"payment_prom":0.0,"work_for_again":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":null},"entire_history":{"completion_rate":null,"all":null,"incomplete_reviews":null,"complete":null,"on_time":null,"on_budget":null,"positive":0.0,"overall":0.0,"reviews":null,"category_ratings":{"communication":0.0,"clarity_spec":0.0,"payment_prom":0.0,"work_for_again":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":null},"earnings_score":0.0,"role":"employer","job_history":{"job_counts":[],"count_other":0},"last12months":{"completion_rate":null,"all":null,"incomplete_reviews":null,"complete":null,"on_time":null,"on_budget":null,"positive":0.0,"overall":0.0,"reviews":null,"category_ratings":{"communication":0.0,"clarity_spec":0.0,"payment_prom":0.0,"work_for_again":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":null},"project_stats":{"draft":0,"open":0,"complete":0,"pending":0,"work_in_progress":0}},"company":"","registration_date":1041968572,"is_active":null,"avatar_large_cdn":"//cdn6.f-cdn.com/img/unknown.png","profile_description":"","address":null,"limited_account":false,"portfolio_count":0,"preferred_freelancer":false,"true_location":null,"primary_currency":{"code":"USD","name":"US Dollar","country":"US","sign":"$","exchange_rate":1.0,"id":1},"mobile_tracking":null,"test_user":false,"chosen_role":"both","reputation":{"user_id":5543295,"last3months":{"completion_rate":0.0,"all":0,"incomplete_reviews":0,"complete":0,"on_time":0.0,"on_budget":0.0,"positive":0.0,"overall":0.0,"reviews":0,"category_ratings":{"communication":0.0,"expertise":0.0,"hire_again":0.0,"quality":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":0},"entire_history":{"completion_rate":0.0,"all":0,"incomplete_reviews":0,"complete":0,"on_time":0.0,"on_budget":0.0,"positive":0.0,"overall":0.0,"reviews":0,"category_ratings":{"communication":0.0,"expertise":0.0,"hire_again":0.0,"quality":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":0},"earnings_score":0.0,"role":"freelancer","job_history":{"job_counts":[],"count_other":0},"last12months":{"completion_rate":0.0,"all":0,"incomplete_reviews":0,"complete":0,"on_time":0.0,"on_budget":0.0,"positive":0.0,"overall":0.0,"reviews":0,"category_ratings":{"communication":0.0,"expertise":0.0,"hire_{"status":"success","result":{"users":{"5543296":{"status":{"payment_verified":false,"identity_verified":false,"email_verified":true,"deposit_made":false,"phone_verified":false,"facebook_connected":false,"profile_complete":true},"avatar_large":"/img/unknown.png","avatar_cdn":"//cdn6.f-cdn.com/img/unknown.png","spam_profile":null,"search_languages":null,"corporate_users":null,"support_status":null,"last_name":null,"suspended":null,"primary_language":"en","timezone":{"country":"IN","offset":5.5,"id":164,"timezone":"Asia/Calcutta"},"membership_package":null,"qualifications":[{"description":"Foundation vWorker Member","level":null,"icon_url":"/img/insignia/vworker.png","icon_name":"vworker","score_percentage":null,"user_percentile":null,"type":null,"id":null,"name":null}],"id":5543296,"badges":null,"hourly_rate":null,"responsiveness":null,"first_name":null,"display_name":"vw1619364vw","tagline":null,"account_balances":null,"public_name":"vw1619364vw","role":"employer","location":{"administrative_area":null,"city":null,"country":{"highres_flag_url":"/img/flags/highres_png/unknown.png","code":null,"name":"","flag_url_cdn":"//cdn5.f-cdn.com/img/flags/png/unknown.png","highres_flag_url_cdn":"//cdn5.f-cdn.com/img/flags/highres_png/unknown.png","flag_url":"/img/flags/png/unknown.png"},"vicinity":null,"longitude":null,"full_address":null,"latitude":null},"closed":false,"email":null,"username":"vw1619364vw","is_local":null,"endorsements":null,"jobs":[],"employer_reputation":{"user_id":5543296,"last3months":{"completion_rate":null,"all":null,"incomplete_reviews":null,"complete":null,"on_time":null,"on_budget":null,"positive":0.0,"overall":0.0,"reviews":null,"category_ratings":{"communication":0.0,"clarity_spec":0.0,"payment_prom":0.0,"work_for_again":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":null},"entire_history":{"completion_rate":null,"all":null,"incomplete_reviews":null,"complete":null,"on_time":null,"on_budget":null,"positive":0.0,"overall":0.0,"reviews":null,"category_ratings":{"communication":0.0,"clarity_spec":0.0,"payment_prom":0.0,"work_for_again":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":null},"earnings_score":0.0,"role":"employer","job_history":{"job_counts":[],"count_other":0},"last12months":{"completion_rate":null,"all":null,"incomplete_reviews":null,"complete":null,"on_time":null,"on_budget":null,"positive":0.0,"overall":0.0,"reviews":null,"category_ratings":{"communication":0.0,"clarity_spec":0.0,"payment_prom":0.0,"work_for_again":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":null},"project_stats":{"draft":0,"open":0,"complete":0,"pending":0,"work_in_progress":0}},"company":"","registration_date":1124680669,"is_active":null,"avatar_large_cdn":"//cdn6.f-cdn.com/img/unknown.png","profile_description":"","address":null,"limited_account":false,"portfolio_count":0,"preferred_freelancer":false,"true_location":null,"primary_currency":{"code":"USD","name":"US Dollar","country":"US","sign":"$","exchange_rate":1.0,"id":1},"mobile_tracking":null,"test_user":false,"chosen_role":"both","reputation":{"user_id":5543296,"last3months":{"completion_rate":0.0,"all":0,"incomplete_reviews":0,"complete":0,"on_time":0.0,"on_budget":0.0,"positive":0.0,"overall":0.0,"reviews":0,"category_ratings":{"communication":0.0,"expertise":0.0,"hire_again":0.0,"quality":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":0},"entire_history":{"completion_rate":0.0,"all":0,"incomplete_reviews":0,"complete":0,"on_time":0.0,"on_budget":0.0,"positive":0.0,"overall":0.0,"reviews":0,"category_ratings":{"communication":0.0,"expertise":0.0,"hire_again":0.0,"quality":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":0},"earnings_score":0.0,"role":"freelancer","job_history":{"job_counts":[],"count_other":0},"last12months":{"completion_rate":0.0,"all":0,"incomplete_reviews":0,"complete":0,"on_time":0.0,"on_budget":0.0,"positive":0.0,"overall":0.0,"reviews":0,"category_ratings":{"communication":0.0,"expertise":0.0,"hire_agaiagain":0.0,"quality":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":0},"project_stats":null},"avatar":"/img/unknown.png","cover_image":{"current_image":{"url":"//cdn2.f-cdn.com/static/img/profiles/cover-product.jpg","width":1920,"height":550,"id":null,"description":""},"past_images":null},"corporate":null,"force_verify":null}}},"request_id":"2520481ca4d6b8130bdc20f6114d8037"}
My Python code :
import json
json_data = open('test.txt', 'r').read()
data = json.loads(json_data)
I'm sure JSON data is valid, because is result of one website, I need to load this in python and get username value.
Actual error :
MacBook-Pro-di-admin:freelancer11 admin$ python load.py
Traceback (most recent call last):
File "load.py", line 6, in <module>
data = json.loads(json_data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting : delimiter: line 1 column 4099 (char 4098)

So if you head over to your favorite JSON validator, like JSONlint, you will see the JSON is not valid, and that is precisely what the error in python means.

Invalid Control Character Error in Python 2.7.9

I am receiving the following error when running the below python script.
ValueError: Invalid control character at: line 7591 column 220620 (char 385678)
I did some research on this and it appeared that it would be resolved by passing 'strict=false' within json.dumps(), but I'm still receiving the same error. This is the only REST service that I have attempted to query that returns this error.
import arcgis
import json
from arcgis import ArcGIS
service = ArcGIS("http://mapping.dekalbcountyga.gov/arcgis/rest/services/LandUse/MapServer")
query = service.get(0, count_only=False)
json_query = json.dumps(query, strict=False)
f = open("dekalb_parcels.geojson", "w")
f.write(json_query)
f.close()
Any help that can be provided would be very appreciated. Thank you.
UPDATE - This is the full error that I am receiving.
Traceback (most recent call last):
File "G:\Python\Scripts\dekalb_parcel_query.py", line 8, in <module>
query = service.get(0, count_only=False)
File "C:\Python27\lib\site-packages\arcgis\arcgis.py", line 146, in get
jsobj = self.get_json(layer, where, fields, count_only, srid)
File "C:\Python27\lib\site-packages\arcgis\arcgis.py", line 90, in get_json
return response.json()
File "C:\Python27\lib\site-packages\requests\models.py", line 802, in json
return json.loads(self.text, **kwargs)
File "C:\Python27\lib\json\__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "C:\Python27\lib\json\decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Python27\lib\json\decoder.py", line 382, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Invalid control character at: line 7591 column 220620 (char 385678)

I was able to fix this issue by passing strict=False within return response.json()

printing json in readable format inside python

I am trying to prints json in readable form. I already checked previous threads and tried out.
using
JSON.stringify(response)
gives error:
NameError: name 'JSON' is not defined
Using
response = json.loads(urllib.urlopen(url).read())
parsed = json.loads(response)
print json.dumps(parsed, indent=4, sort_keys=True)
gives error:
Traceback (most recent call last):
File "p6.py", line 15, in <module>
parsed = json.loads(response)
File "/usr/lib/python2.7/json/__init__.py", line 328, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 365, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer

You get
NameError: name 'JSON' is not defined
because the first snippet is in JavaScript, not in Python.
As to the second snippet, you are calling json.loads() twice:
response = json.loads(urllib.urlopen(url).read()) # calling once
parsed = json.loads(response) # calling twice
Just call it once (and ensure that what you get from the HTTP server is actually JSON).

How to load unicode json in Python?

I've got some simple json formatted in unicode which I want to load using the usual python json.loads():
>>> er.rates
u"{u'sell': u'1.3477', u'buy': u'1.3588'}"
>>> json.loads(er.rates)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 365, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 381, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
So I tried using ensure_ascii=False:
>>> json.loads(er.rates, ensure_ascii=False)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 351, in loads
return cls(encoding=encoding, **kw).decode(s)
TypeError: __init__() got an unexpected keyword argument 'ensure_ascii'
Does anybody know how I can load this unicode json?

That is not json. It is a string representation of a python dict, which is something quite different.
You can use ast.literal_eval to load it.

Many API data providers return unicode string which is easily rendered in a browser. Unicode string (even when it looks like json) and json are not the same thing from a 'computer' perspective.
If you have a unicode (json-like) string you should be able to use json.loads(<your unicode json like string>)

Decoding JSON with Python

Why do I get
ValueError: No JSON object could be decoded
from this code:
import urllib.request,json
n = urllib.request.urlopen("http://graph.facebook.com/55")
d = json.loads(str(n.readall()))
The full error:
Traceback (most recent call last):
File "<pyshell#41>", line 1, in <module>
d= json.loads(str(n.readall()))
File "C:\Python33\lib\json\__init__.py", line 309, in loads
return _default_decoder.decode(s)
File "C:\Python33\lib\json\decoder.py", line 352, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Python33\lib\json\decoder.py", line 370, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
The output of str(n.readall()):
'b\'{"id":"55","name":"Jillian Copeland","first_name":"Jillian","last_name":"Copeland","username":"JCoMD","gender":"female","locale":"en_US"}\''
Maybe the b is throwing it off?
If that is the issue, how do I convert the binary stream from the readall to a string and not have that b?
I am trying to learn a little python so please keep that in mind.
I am using Python 3.3 in Windows.

I believe that this is an exact duplicate of this question, but sadly there's no accepted answer.
On my end, this works:
import urllib.request,json
n = urllib.request.urlopen("http://graph.facebook.com/55")
d= json.loads(n.readall().decode('utf-8'))

Categories

Resources