Develop Reference

reading text file back into a dictionary using json.loads - python

I piped the output of my Python script which accesses live twitter tweets to a file output.txt using:
$python scriptTweet.py > output.txt
Originally, the output returned by the script was a dictionary which got written to a text file.
Now i want to use the output.txt file to access tweets stored in it. But when i use this code to parse the text in output.txt into a python dictionary using json.loads():
tweetfile = open("output.txt")
pyresponse = json.loads('tweetfile.read()')
print type(pyresponse)
This error pops up:
pyresponse = json.loads('tweetfile.read()')
File "C:\Python27\lib\json\__init__.py", line 326, in loads
return _default_decoder.decode(s)
File "C:\Python27\lib\json\decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Python27\lib\json\decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
How should i convert the contents of file output.txt again into a dictionary?

'tweetfile.read()' is a string as you see it. You want to call this function:
with open("output.txt") as tweetfile:
pyresponse = json.loads(tweetfile.read())
or read it directly using json.load and let json read on the tweetfile itself:
with open("output.txt") as tweetfile:
pyresponse = json.load(tweetfile)

Related

I have a question concerning an issue I ran into while using the json lib in Python.
I'm tying to read a json file using the json.load(file) command using the following code:
import json
filename= '../Data/exampleFile.json'
histFile= open(filename, 'w+')
print(json.load(histFile))
The JSON file I am trying to read is valid according to some website I found: a screenshot of that validation, because I'm new and still lack the reputation...
The error message I'm getting is the following:
File ".\testLoad.py", line 5, in <module>
print(json.load(histFile))
File "C:\Users\...\Python\Python37\lib\json\__init__.py", line 296, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "C:\Users\...\Python\Python37\lib\json\__init__.py", line 348, in loads
return _default_decoder.decode(s)
File "C:\Users\...\Python\Python37\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "C:\Users\...\Python\Python37\lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Alright, so I believe it is not the file that is the issue, but the json.load(file) works for me in other cases.
Sadly I was not able to figure this error-message out on my own, so it would be amazing if someone with some more experience dealing with Python-JSON interaction could maybe help me out.

You opened the file for writing:
histFile= open(filename, 'w+')
# ^^^^
The w mode first truncates the file, so the file is empty (it doesn't matter here that the file can also be read from, the + sees to that but the file is truncated nonetheless). See the open() function documentation:
'w': open for writing, truncating the file first)
There is no JSON data in it to parse. This is why the exception tells you that parsing failed at the very start of the file:
Expecting value: line 1 column 1 (char 0)
There is no data in line one, column one.
If you wanted to open a file for both reading and writing without truncating it first, use 'r+' as the file mode.

I want to load this JSON data in python :
JSON data:
{"status":"success","result":{"users":{"5543295":{"status":{"payment_verified":false,"identity_verified":false,"email_verified":true,"deposit_made":false,"phone_verified":false,"facebook_connected":false,"profile_complete":true},"avatar_large":"/img/unknown.png","avatar_cdn":"//cdn6.f-cdn.com/img/unknown.png","spam_profile":null,"search_languages":null,"corporate_users":null,"support_status":null,"last_name":null,"suspended":null,"primary_language":"en","timezone":{"country":"US","offset":-7.0,"id":105,"timezone":"America/Los_Angeles"},"membership_package":null,"qualifications":[{"description":"Foundation vWorker Member","level":null,"icon_url":"/img/insignia/vworker.png","icon_name":"vworker","score_percentage":null,"user_percentile":null,"type":null,"id":null,"name":null}],"id":5543295,"badges":null,"hourly_rate":null,"responsiveness":null,"first_name":null,"display_name":"vw693015vw","tagline":null,"account_balances":null,"public_name":"vw693015vw","role":"employer","location":{"administrative_area":null,"city":null,"country":{"highres_flag_url":"/img/flags/highres_png/unknown.png","code":null,"name":"","flag_url_cdn":"//cdn5.f-cdn.com/img/flags/png/unknown.png","highres_flag_url_cdn":"//cdn5.f-cdn.com/img/flags/highres_png/unknown.png","flag_url":"/img/flags/png/unknown.png"},"vicinity":null,"longitude":null,"full_address":null,"latitude":null},"closed":false,"email":null,"username":"vw693015vw","is_local":null,"endorsements":null,"jobs":[],"employer_reputation":{"user_id":5543295,"last3months":{"completion_rate":null,"all":null,"incomplete_reviews":null,"complete":null,"on_time":null,"on_budget":null,"positive":0.0,"overall":0.0,"reviews":null,"category_ratings":{"communication":0.0,"clarity_spec":0.0,"payment_prom":0.0,"work_for_again":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":null},"entire_history":{"completion_rate":null,"all":null,"incomplete_reviews":null,"complete":null,"on_time":null,"on_budget":null,"positive":0.0,"overall":0.0,"reviews":null,"category_ratings":{"communication":0.0,"clarity_spec":0.0,"payment_prom":0.0,"work_for_again":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":null},"earnings_score":0.0,"role":"employer","job_history":{"job_counts":[],"count_other":0},"last12months":{"completion_rate":null,"all":null,"incomplete_reviews":null,"complete":null,"on_time":null,"on_budget":null,"positive":0.0,"overall":0.0,"reviews":null,"category_ratings":{"communication":0.0,"clarity_spec":0.0,"payment_prom":0.0,"work_for_again":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":null},"project_stats":{"draft":0,"open":0,"complete":0,"pending":0,"work_in_progress":0}},"company":"","registration_date":1041968572,"is_active":null,"avatar_large_cdn":"//cdn6.f-cdn.com/img/unknown.png","profile_description":"","address":null,"limited_account":false,"portfolio_count":0,"preferred_freelancer":false,"true_location":null,"primary_currency":{"code":"USD","name":"US Dollar","country":"US","sign":"$","exchange_rate":1.0,"id":1},"mobile_tracking":null,"test_user":false,"chosen_role":"both","reputation":{"user_id":5543295,"last3months":{"completion_rate":0.0,"all":0,"incomplete_reviews":0,"complete":0,"on_time":0.0,"on_budget":0.0,"positive":0.0,"overall":0.0,"reviews":0,"category_ratings":{"communication":0.0,"expertise":0.0,"hire_again":0.0,"quality":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":0},"entire_history":{"completion_rate":0.0,"all":0,"incomplete_reviews":0,"complete":0,"on_time":0.0,"on_budget":0.0,"positive":0.0,"overall":0.0,"reviews":0,"category_ratings":{"communication":0.0,"expertise":0.0,"hire_again":0.0,"quality":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":0},"earnings_score":0.0,"role":"freelancer","job_history":{"job_counts":[],"count_other":0},"last12months":{"completion_rate":0.0,"all":0,"incomplete_reviews":0,"complete":0,"on_time":0.0,"on_budget":0.0,"positive":0.0,"overall":0.0,"reviews":0,"category_ratings":{"communication":0.0,"expertise":0.0,"hire_{"status":"success","result":{"users":{"5543296":{"status":{"payment_verified":false,"identity_verified":false,"email_verified":true,"deposit_made":false,"phone_verified":false,"facebook_connected":false,"profile_complete":true},"avatar_large":"/img/unknown.png","avatar_cdn":"//cdn6.f-cdn.com/img/unknown.png","spam_profile":null,"search_languages":null,"corporate_users":null,"support_status":null,"last_name":null,"suspended":null,"primary_language":"en","timezone":{"country":"IN","offset":5.5,"id":164,"timezone":"Asia/Calcutta"},"membership_package":null,"qualifications":[{"description":"Foundation vWorker Member","level":null,"icon_url":"/img/insignia/vworker.png","icon_name":"vworker","score_percentage":null,"user_percentile":null,"type":null,"id":null,"name":null}],"id":5543296,"badges":null,"hourly_rate":null,"responsiveness":null,"first_name":null,"display_name":"vw1619364vw","tagline":null,"account_balances":null,"public_name":"vw1619364vw","role":"employer","location":{"administrative_area":null,"city":null,"country":{"highres_flag_url":"/img/flags/highres_png/unknown.png","code":null,"name":"","flag_url_cdn":"//cdn5.f-cdn.com/img/flags/png/unknown.png","highres_flag_url_cdn":"//cdn5.f-cdn.com/img/flags/highres_png/unknown.png","flag_url":"/img/flags/png/unknown.png"},"vicinity":null,"longitude":null,"full_address":null,"latitude":null},"closed":false,"email":null,"username":"vw1619364vw","is_local":null,"endorsements":null,"jobs":[],"employer_reputation":{"user_id":5543296,"last3months":{"completion_rate":null,"all":null,"incomplete_reviews":null,"complete":null,"on_time":null,"on_budget":null,"positive":0.0,"overall":0.0,"reviews":null,"category_ratings":{"communication":0.0,"clarity_spec":0.0,"payment_prom":0.0,"work_for_again":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":null},"entire_history":{"completion_rate":null,"all":null,"incomplete_reviews":null,"complete":null,"on_time":null,"on_budget":null,"positive":0.0,"overall":0.0,"reviews":null,"category_ratings":{"communication":0.0,"clarity_spec":0.0,"payment_prom":0.0,"work_for_again":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":null},"earnings_score":0.0,"role":"employer","job_history":{"job_counts":[],"count_other":0},"last12months":{"completion_rate":null,"all":null,"incomplete_reviews":null,"complete":null,"on_time":null,"on_budget":null,"positive":0.0,"overall":0.0,"reviews":null,"category_ratings":{"communication":0.0,"clarity_spec":0.0,"payment_prom":0.0,"work_for_again":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":null},"project_stats":{"draft":0,"open":0,"complete":0,"pending":0,"work_in_progress":0}},"company":"","registration_date":1124680669,"is_active":null,"avatar_large_cdn":"//cdn6.f-cdn.com/img/unknown.png","profile_description":"","address":null,"limited_account":false,"portfolio_count":0,"preferred_freelancer":false,"true_location":null,"primary_currency":{"code":"USD","name":"US Dollar","country":"US","sign":"$","exchange_rate":1.0,"id":1},"mobile_tracking":null,"test_user":false,"chosen_role":"both","reputation":{"user_id":5543296,"last3months":{"completion_rate":0.0,"all":0,"incomplete_reviews":0,"complete":0,"on_time":0.0,"on_budget":0.0,"positive":0.0,"overall":0.0,"reviews":0,"category_ratings":{"communication":0.0,"expertise":0.0,"hire_again":0.0,"quality":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":0},"entire_history":{"completion_rate":0.0,"all":0,"incomplete_reviews":0,"complete":0,"on_time":0.0,"on_budget":0.0,"positive":0.0,"overall":0.0,"reviews":0,"category_ratings":{"communication":0.0,"expertise":0.0,"hire_again":0.0,"quality":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":0},"earnings_score":0.0,"role":"freelancer","job_history":{"job_counts":[],"count_other":0},"last12months":{"completion_rate":0.0,"all":0,"incomplete_reviews":0,"complete":0,"on_time":0.0,"on_budget":0.0,"positive":0.0,"overall":0.0,"reviews":0,"category_ratings":{"communication":0.0,"expertise":0.0,"hire_agaiagain":0.0,"quality":0.0,"professionalism":0.0},"earnings":null,"rehire_rate":null,"incomplete":0},"project_stats":null},"avatar":"/img/unknown.png","cover_image":{"current_image":{"url":"//cdn2.f-cdn.com/static/img/profiles/cover-product.jpg","width":1920,"height":550,"id":null,"description":""},"past_images":null},"corporate":null,"force_verify":null}}},"request_id":"2520481ca4d6b8130bdc20f6114d8037"}
My Python code :
import json
json_data = open('test.txt', 'r').read()
data = json.loads(json_data)
I'm sure JSON data is valid, because is result of one website, I need to load this in python and get username value.
Actual error :
MacBook-Pro-di-admin:freelancer11 admin$ python load.py
Traceback (most recent call last):
File "load.py", line 6, in <module>
data = json.loads(json_data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting : delimiter: line 1 column 4099 (char 4098)

So if you head over to your favorite JSON validator, like JSONlint, you will see the JSON is not valid, and that is precisely what the error in python means.

I'm trying to take the JSON from a twitter get_user query and turn it into a Python object that I can extract data from (twitter handle, location, screen name, etc.)
Here is what I created. I am not sure why it doesn't work.
api = tweepy.API(auth,parser=tweepy.parsers.JSONParser())
user = api.search_users('google.com')
t_dict = json.loads(user)
pprint(t_dict)
Error:
Traceback (most recent call last):
File "Get_User_By_URL.py", line 23, in <module>
t_dict = json.loads(user)
File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer

api.search_users is already returning a python object. It isn't a json string that needs to be parsed. According to tweetpy documentation search_users actually returns a list of users. So the following is possible:
for user in api.search_users('google.com'):
print user.screen_name

I am doing a simple wrapper program around a software tool, and this wrapper will, among other things, parse a JSON configuration file that will be coded by the user as a way for the user to specify certain configutations to the program.
Within the program, I have the following lines of code:
with open(os.path.join(args.config_bin_dir, 'basic_config.json'), 'r') as jsondata :
print "JSON File contents:", jsondata.read()
basic_config.update(json.load(jsondata))
print "The basic_config after updating with JSON:", basic_config
jsondata.close()
The basic_config object is a dictionary object that gets constructed somewhere on top earlier up in the program, and over here I want to add the new dictionary key-value pairs obtained from parsing the user's JSON configuration file into the basic_config dictionary object.
When the program is run, I get the following error, a section of which is as follows:
File "path/to/python/script.py", line 42, in main
basic_config.update(json.load(jsondata))
File "/usr/lib/python2.7/json/__init__.py", line 290, in load
**kw)
File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
I'm quite puzzled by this error, but I noticed that it goes away so long as I remove the line print "JSON File contents:", jsondata.read() and run the code as either:
with open(os.path.join(args.config_bin_dir, 'basic_config.json'), 'r') as jsondata :
# print "JSON File contents:", jsondata.read()
basic_config.update(json.load(jsondata))
print "The basic_config after updating with JSON:", basic_config
jsondata.close()
or
with open(os.path.join(args.config_bin_dir, 'basic_config.json'), 'r') as jsondata :
# print "JSON File contents:", jsondata.read()
basic_config.update(json.load(jsondata))
# print "The basic_config after updating with JSON:", basic_config
jsondata.close()
I'll still get the same error if I comment out the line after the json.load method:
with open(os.path.join(args.config_bin_dir, 'basic_config.json'), 'r') as jsondata :
print "JSON File contents:", jsondata.read()
basic_config.update(json.load(jsondata))
# print "The basic_config after updating with JSON:", basic_config
jsondata.close()
So I'm guessing that it has to do with calling the read method on the JSON file object before parsing that same file. Could anyone please explain to me why this is so, and whether I am mistaken about the cause of the error?
Thank you very much!

When you call read() on a file handle, its current read position is advanced to the end of the file. This mirrors the behavior of the underlying file descriptors. You can reset to the start of a file via jsondata.seek(0, 0).

This question already has answers here:
Single vs double quotes in JSON
(12 answers)
Closed 8 years ago.
I have a json file that looks like this exactly:
{'data':[-99,19,1212,121,2131,323321,123]}
saved as list.json
and I have a python script that needs the list [-99,19,1212,121,2131,323321,123]
import json
with open ("list.json", "r") as myfile:
data=myfile.read().replace('\n', '')
spots1 = json.loads(data) #<-- Error
spots = spots1['data']
Error:
File "pythonscript.py", line 479, in <module>
spots1 = json.loads(data)
File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
obj, end = self.scan_once(s, idx)
I can't figure out what is wrong with my formatting.
Any help would be greatly appreciated!

Python wants you to use "(double quote) for properties names in a json dictionary and not '(Single Quote)
Try this String:
{"data":[-99,19,1212,121,2131,323321,123]}
See this question for more details: python: single vs double quotes in JSON