I am trying to write out a csv file from data in JSON format. I can get the fieldnames to write to the csv file but not the item value I need. This is my first time coding in python so any help would be appreciated. The json file can be found below for reference:
https://data.ny.gov/api/views/nqur-w4p7/rows.json?accessType=DOWNLOAD
Here is my error:
Traceback (most recent call last):
File "ChangeDataType.py", line 5, in <module>
data = json.dumps(inputFile)
File "/usr/lib64/python3.4/json/__init__.py", line 230, in dumps
return _default_encoder.encode(obj)
File "/usr/lib64/python3.4/json/encoder.py", line 192, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib64/python3.4/json/encoder.py", line 250, in iterencode
return _iterencode(o, 0)
File "/usr/lib64/python3.4/json/encoder.py", line 173, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <_io.TextIOWrapper name='rows.json?accessType=DOWNLOAD' mode='r' encoding='UTF-8'> is not JSON serializable
Here is my code:
import json
import csv
inputFile = open("rows.json?accessType=DOWNLOAD", "r")
data = json.dumps(inputFile)
with open("Data.csv","w") as csvfile:
writer = csv.DictWriter(csvfile, extrasaction='ignore', fieldnames=["date", "new_york_state_average_gal", "albany_average_gal", "binghamton_average_gal",\
"buffalo_average_gal", "nassau_average_gal", "new_york_city_average_gal", "rochester_average_gal", "syracuse_average_gal","utica_average_gal"])
writer.writeheader()
for row in data:
writer.writerow([row["date"], row["new_york_state_average_gal"], row["albany_average_gal"], row["binghamton_average_gal"],\
row["buffalo_average_gal"], row["nassau_average_gal"], row["new_york_city_average_gal"], row["rochester_average_gal"], row["syracuse\
_average_gal"],row["utica_average_gal"]])
If you want to read a JSON file you should use json.load instead of json.dumps:
data = json.load(inputFile)
Seems you're still having problems even opening the file.
Python json to CSV
You were told to use json.load
dumps takes an object to a string. You want to read JSON to a dictionary.
You therefore need to load the JSON file, and you can open two files at once
with open("Data.csv","w") as csvfile, open("rows.json?accessType=DOWNLOAD") as inputfile:
data = json.load(inputfile)
writer = csv.DictWriter(csvfile,...
Also, for example, considering the JSON data looks like "fieldName" : "syracuse_average_gal", and that is the only occurrence of the Syracuse average value, row["syracuse_average_gal"] is not correct.
Carefully inspect your JSON and figure out to parse it from the very top bracket
Related
I am reading a json file with python using below code:
import json
Ums = json.load(open('commerceProduct.json'))
for um in Ums :
des = um['description']
if des == None:
um['description'] = "Null"
with open("sample.json", "w") as outfile:
json.dump(um, outfile)
break
It is giving me the following error:
Traceback (most recent call last):
File "test.py", line 2, in <module>
Ums = json.load(open('commerceProduct.json'))
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/json/__init__.py", line 293, in load
return loads(fp.read(),
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/json/decoder.py", line 340, in decode
raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 5528 (char 5527)
while I am checking the json file, it looks fine.
The thing is it has one object on one line with deliminator being '\n'.
It is not corrupted since i have imported the same file in mongo.
Can someone please suggest what can be wrong in it ?
Thanks.
your JSON data is not in a valid format, one miss will mess up the python parser. Try to test your JSON data here, make sure it is in a correct format.
this return _default_decoder.decode(s) is returned when the python parser find somthing wrong in your json.
The code is valid and will work with a valid json doc.
You have one json object per line? That's not a valid json file. You have newline-delimited json, so consider using the ndjson package to read it. It has the same API as the json package you are familiar with.
import ndjson
Ums = ndjson.load(open('commerceProduct.json'))
...
I'm following this guide to secure the pickle files correctly but I'm not getting the same output. Granted I had to do some changes to run it the first time:
import hashlib
import hmac
import pickle
class Dummy:
pass
obj = Dummy()
data = pickle.dumps(obj)
digest = hmac.new(b'unique-key-here', data, hashlib.blake2b).hexdigest()
with open('temp.txt', 'wb') as output:
output.write(str(digest) + ' ' + data)
with open('temp.txt', 'r') as f:
data = f.read()
digest, data = data.split(' ')
expected_digest = hmac.new(b'unique-key-here', data, hashlib.blake2b).hexdigest()
if not secrets.compare_digest(digest, expected_digest):
print('Invalid signature')
exit(1)
obj = pickle.loads(data)
When I run this I get the following stacktrace:
File "test.py", line 21, in <module>
expected_digest = hmac.new(b'unique-key-here', data, hashlib.blake2b).hexdigest()
File "/usr/lib/python3.8/hmac.py", line 153, in new
return HMAC(key, msg, digestmod)
File "/usr/lib/python3.8/hmac.py", line 88, in __init__
self.update(msg)
File "/usr/lib/python3.8/hmac.py", line 96, in update
self.inner.update(msg)
TypeError: Unicode-objects must be encoded before hashing
Your problem is data = f.read(). .read() returns a string and hmac.new() wants bytes. Change the problem line to data = f.read().encode('utf-8') OR read the file in binary mode ('b' flag).
References:
7.2. Reading and Writing Files
open()
hmac.new()
.encode()
I ended up having to use the following methods for it to work:
pickle.loads(codecs.decode(pickle_data.encode(), 'base64'))
# and
codecs.encode(pickle.dumps(pickle_obj), "base64").decode()
Not sure why using .encode() and .decode() was still not working for me.
I am making a call to the AWS API using boto3 and Python and I am writing the JSON response to a JSONfile. I am then trying to convert the JSON file to a CSV file. When I attempt to do this with the csv writer() method, I get the above error and I am not sure why.
Code:
def ResponseConvert():
dynamo = boto3.client('dynamodb')
response = dynamo.scan(
TableName='XXXX'
)
with open('vuln_data.json', 'w') as outfile:
json.dump(response, outfile, indent=4)
f = open('vuln_data.json')
data = json.load(f)
f.close()
f = csv.writer(open('vuln_data.csv', 'wb+'))
f.writerow(data.keys())
for row in data:
f.writerow(row.values())
ResponseConvert()
Traceback:
Traceback (most recent call last):
File "response_convert.py", line 21, in <module>
ResponseConvert()
File "response_convert.py", line 19, in ResponseConvert
f.writerow(row.values())
AttributeError: 'unicode' object has no attribute 'values'
CSV writers expect a file handle, not a filename.
with open('filename.csv', 'w') as f:
writer = csv.writer(f)
...
You probably want a DictWriter instead, by the way. Don't rely on the order of keys and values matching up.
If I use this script to convert from Json to Csv in Python:
import json
import csv
with open("data.json") as file:
data = json.loads(file)
with open("data.csv", "w") as file:
csv_file = csv.writer(file)
for item in data:
csv_file.writerow([item['studio'], item['title']] + item['release_dates'].values())
It throws an error message:
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
File "C:\Python27\lib\json\__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "C:\Python27\lib\json\decoder.py", line 365, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer
You're using json.loads() when you should be using json.load().
json.loads is for reading from a string, whereas json.load is for reading from a file.
Visit here for more information: https://docs.python.org/2/library/json.html.
Also, unrelated, but you can chain with statements.
with open("data.json") as json_file, open("data.csv", "w") as csv_file:
csv_file = csv.writer(csv_file)
for item in json.load(json_file):
csv_file.writerow(...)
i am trying to extract from json data some information. On the following code, I first extract the part of json data that contains the information i want and i store it in a file. Then i am trying to open this file and i get the error that follows my code. Can you help me find where i am wrong?
import json
import re
input_file = 'path'
text = open(input_file).read()
experience = re.findall(r'Experience":{"positionsMpr":{"showSection":true," (.+?),"visible":true,"find_title":"Find others',text)
output_file = open ('/home/evi.nastou/Documenten/LinkedIn_data/Alewijnse/temp', 'w')
output_file.write('{'+experience[0]+'}')
output_file.close()
text = open('path/temp')
input_text = text.read()
data = json.load(input_text)
positions = json.dumps([s['companyName'] for s in data['positions']])
print positions
Error:
Traceback (most recent call last):
File "test.py", line 13, in <module>
data = json.load(input_text)
File "/home/evi.nastou/.pythonbrew/pythons/Python-2.7.2/lib/python2.7/json/__init__.py", line 274, in load
return loads(fp.read(),
AttributeError: 'str' object has no attribute 'read'
You want to use json.loads() (note the s), or pass in the file object instead of the result of .read():
text = open('path/temp')
data = json.load(text)
json.load() takes an open file object, but you were passing it a string; json.loads() takes a string.