I am making a call to the AWS API using boto3 and Python and I am writing the JSON response to a JSONfile. I am then trying to convert the JSON file to a CSV file. When I attempt to do this with the csv writer() method, I get the above error and I am not sure why.
Code:
def ResponseConvert():
dynamo = boto3.client('dynamodb')
response = dynamo.scan(
TableName='XXXX'
)
with open('vuln_data.json', 'w') as outfile:
json.dump(response, outfile, indent=4)
f = open('vuln_data.json')
data = json.load(f)
f.close()
f = csv.writer(open('vuln_data.csv', 'wb+'))
f.writerow(data.keys())
for row in data:
f.writerow(row.values())
ResponseConvert()
Traceback:
Traceback (most recent call last):
File "response_convert.py", line 21, in <module>
ResponseConvert()
File "response_convert.py", line 19, in ResponseConvert
f.writerow(row.values())
AttributeError: 'unicode' object has no attribute 'values'
CSV writers expect a file handle, not a filename.
with open('filename.csv', 'w') as f:
writer = csv.writer(f)
...
You probably want a DictWriter instead, by the way. Don't rely on the order of keys and values matching up.
Related
I am trying to write out a csv file from data in JSON format. I can get the fieldnames to write to the csv file but not the item value I need. This is my first time coding in python so any help would be appreciated. The json file can be found below for reference:
https://data.ny.gov/api/views/nqur-w4p7/rows.json?accessType=DOWNLOAD
Here is my error:
Traceback (most recent call last):
File "ChangeDataType.py", line 5, in <module>
data = json.dumps(inputFile)
File "/usr/lib64/python3.4/json/__init__.py", line 230, in dumps
return _default_encoder.encode(obj)
File "/usr/lib64/python3.4/json/encoder.py", line 192, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib64/python3.4/json/encoder.py", line 250, in iterencode
return _iterencode(o, 0)
File "/usr/lib64/python3.4/json/encoder.py", line 173, in default
raise TypeError(repr(o) + " is not JSON serializable")
TypeError: <_io.TextIOWrapper name='rows.json?accessType=DOWNLOAD' mode='r' encoding='UTF-8'> is not JSON serializable
Here is my code:
import json
import csv
inputFile = open("rows.json?accessType=DOWNLOAD", "r")
data = json.dumps(inputFile)
with open("Data.csv","w") as csvfile:
writer = csv.DictWriter(csvfile, extrasaction='ignore', fieldnames=["date", "new_york_state_average_gal", "albany_average_gal", "binghamton_average_gal",\
"buffalo_average_gal", "nassau_average_gal", "new_york_city_average_gal", "rochester_average_gal", "syracuse_average_gal","utica_average_gal"])
writer.writeheader()
for row in data:
writer.writerow([row["date"], row["new_york_state_average_gal"], row["albany_average_gal"], row["binghamton_average_gal"],\
row["buffalo_average_gal"], row["nassau_average_gal"], row["new_york_city_average_gal"], row["rochester_average_gal"], row["syracuse\
_average_gal"],row["utica_average_gal"]])
If you want to read a JSON file you should use json.load instead of json.dumps:
data = json.load(inputFile)
Seems you're still having problems even opening the file.
Python json to CSV
You were told to use json.load
dumps takes an object to a string. You want to read JSON to a dictionary.
You therefore need to load the JSON file, and you can open two files at once
with open("Data.csv","w") as csvfile, open("rows.json?accessType=DOWNLOAD") as inputfile:
data = json.load(inputfile)
writer = csv.DictWriter(csvfile,...
Also, for example, considering the JSON data looks like "fieldName" : "syracuse_average_gal", and that is the only occurrence of the Syracuse average value, row["syracuse_average_gal"] is not correct.
Carefully inspect your JSON and figure out to parse it from the very top bracket
With the following code I want to > open a file > read the contents and strip the non-required lines > then write the data to the file and also read the file for downstream analyses.
with open("chr2_head25.gtf", 'r') as f,\
open('test_output.txt', 'w+') as f2:
for lines in f:
if not lines.startswith('#'):
f2.write(lines)
f2.close()
Now, I want to read the f2 data and do further processing in pandas or other modules but I am running into a problem while reading the data(f2).
data = f2 # doesn't work
print(data) #gives
<_io.TextIOWrapper name='test_output.txt' mode='w+' encoding='UTF-8'>
data = io.StringIO(f2) # doesn't work
# Error message
Traceback (most recent call last):
File "/home/everestial007/PycharmProjects/stitcher/pHASE-Stitcher-Markov/markov_final_test/phase_to_vcf.py", line 64, in <module>
data = io.StringIO(f2)
TypeError: initial_value must be str or None, not _io.TextIOWrapper
The file is already closed (when the previous with block finishes), so you cannot do anything more to the file. To reopen the file, create another with statement and use the read attribute to read the file.
with open('test_output.txt', 'r') as f2:
data = f2.read()
print(data)
I'm having trouble designing a script that will delete designated rows and columns from a CSV file. Here's my script so far:
import csv
with open ("E:\\CSULB Grad\\2016 Spring\\Geog 588\\Project\\ACS_14_5YR_B02001_Copy.csv","rb") as csvfile:
reader=csv.reader(csvfile)
with open ("E:\\CSULB Grad\\2016 Spring\\Geog 588\\Project\\ACS_14_5YR_B02001_Copy.csv","ab+") as csvfile2:
writer=csv.writer(csvfile2)
for row in csvfile2:
del row['HD02_VD01']
writer.writerow(row)
Here's the traceback error I've been getting:
Traceback (most recent call last):
File "C:\Python27\ArcGIS10.3\Lib\site-packages\pythonwin\pywin\framework\scriptutils.py", line 326, in RunScript
exec codeObject in __main__.__dict__
File "E:\CSULB Grad\2016 Spring\Geog 588\Project\Script2.py", line 7, in <module>
del row['HD02_VD01']
TypeError: 'str' object does not support item deletion
You can't modify a CSV file in-place as you iterate over it. Consider creating a new CSV file that writes only the columns that you require for each row in the original file.
For instance:
with open(path,"wb") as csvfile:
writer = csv.DictWriter(csvfile, fieldnames = fieldNames)
for row in csvfile2:
writer.writerow({"Col1":row['...'],"Col2":row['...']})
I have passed .csv file to post request,
input_file = data.get('file', None)
with input_file as datasheet:
header = datasheet.readline()
Always I am getting error on second line. Also my file type is Unicode thats why it again giving error on third line for readline()
>>> with "test1.html" as fp:
... header = fp.readline()
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: __exit__
>>>
How to Read file with with stament:
code:
>>> with open("test1.html") as fp:
... header = fp.readline()
...
Check file is exits or not before doing any process.
Use os module
Demo:
>>> os.path.isfile("test1.html")
True
>>> os.path.isfile("nofile.html")
False
>>>
File Upload to server via post request in API testing using tastypie
fp = open("C:\sample_datasheet.csv", 'rb')
content = fp.read()
fp.close()
fd ={'file': "C:\sample_datasheet.csv", "content": content}
self.assertHttpOK(self.api_client.post('api of upload', format='json',\
org_id=2, content_type="multipart/form-data",\
data=fd))
and Save content from the data to server location in the view.
Considering that {u'file': u'C:\\sample_datasheet.csv'} is returned by the data.get() function, you have to obtain the file name and open it:
data = data.get('file', None)
fname = data["file"]
with open(fname, "r") as datasheet:
header = datasheet.readline()
I have the following json file named json.txt with the following data,
{"id":99903727,"nickname":"TEST_MLA_OFF","registration_date":"2010-12-03T14:19:33.000-04:00","country_id":"AR","user_type":"normal","logo":null,"points":0,"site_id":"MLA","permalink":"http://perfil.mercadolibre.com.ar/TEST_MLA_OFF","seller_reputation":{"level_id":null,"power_seller_status":null,"transactions":{"period":"12 months","total":25,"completed":25,"canceled":0,"ratings":{"positive":0,"negative":0,"neutral":1}}},"status":{"site_status":"deactive"}}
I obtained it using wget. I tried to load that json data with python using the following python code,
json_data = json.load('json.txt')
data = json.load(json_data)
json_data.close()
print data
but that throws the following error,
Traceback (most recent call last):
File "json-example.py", line 28, in <module>
main()
File "json-example.py", line 21, in main
json_data = json.load('json.txt')
File "/opt/sage-4.6.2-linux-64bit-ubuntu_8.04.4_lts-x86_64-Linux/local/lib/python/json/__init__.py", line 264, in load
return loads(fp.read(),
AttributeError: 'str' object has no attribute 'read'
I couldn't find googling what is the reason of the error.
Best regards.
Even better practice is to use the with statement.
with open('json.txt', 'r') as json_file:
data = json.load(json_file)
This makes sure the file gets closed properly without
you worrying about it.
You need to give json.load a file stream object:
json_file = open('json.txt')
data = json.load(json_file)
json_file.close()
print data