Develop Reference

Python csv.DictWriter not writing Dictory to writerow method

Python - 3.8
I am trying to write a csv file from a list of dictionary. I followed the official website example. writer is able to write headers. But it is not writing the rows from dictionary that i loops from the list.
csv_path = "/home/tmp/file.csv"
with open(csv_path, 'w') as csvfile:
writer = csv.DictWriter(csvfile, fieldnames=["RequestTimestamp", "MerchantId", "MerchantName", "ChannelName",
"MerchantUSN", "BatchNumber", "UniqueId", "TransactionId", "InvoiceId",
"ReferenceTimestamp", "CustomerName", "AccountHolder", "PaymentType",
"Currency", "Debit", "Credit", "SettledAmount", "Result", "bankResult", "ReconStatus"])
writer.writeheader()
for data in recons:
result_dict = utils.data_formatter_handler(data)
writer.writerow(data)

Append a file in Python 3 I/O operation on closed file

I'm able to open and write some headers to a csv file at the start of my code:
## Create the output file
with open(output_file, mode='w+') as csv_file:
fieldnames = ['Name', 'Instance ID', 'Type', 'State', 'Private IP', 'Public IP', 'Region', 'Availability Zone', 'Launch Time' ]
writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
writer.writeheader()
But in the same program I build a list of EC2 instances, and try to print that list to the same file later on in the code, and I get this error:
File "C:\Users\tdunphy\AppData\Local\Programs\Python\Python37-32\lib\csv.py", line 155, in writerow
return self.writer.writerow(self._dict_to_list(rowdict))
ValueError: I/O operation on closed file.
The is the code where I try to append to the file, that produces the above error:
with open(output_file, mode='a+') as csv_file:
writer.writerow({'Name': name, 'Instance ID': instance_id, 'Type': instance_type, 'State': instance_state, 'Private IP': private_ips, 'Public IP': public_ips, 'Region': aws_region, 'Availability Zone': availability_zone, 'Launch Time': launch_time_friendly})
Why can't I write to this file? What can I do to correct that?

You might need to add back this line before trying to writerow, as the old writer may still be referring to a "closed" file in memory:
writer = csv.DictWriter(csv_file, fieldnames=fieldnames)

with open(path, 'w+') as f1:
s = store(f1)
with open(path, 'a+') as f2:
s.operate_on()
if the same as
try:
f1 = open(path, 'w+') as f:
s = store(f1)
finally:
f1.close()
try:
f2 = open(path, 'a+')
s.operate_on()
finally:
f2.close()
Perhaps this is more obvious what you're problem is. You're storing an open file object, then closing it, then opening some other file object f2 which you never use and trying to operate on the closed file.
Your solution is to either put writer.writerow(...) in the first with block or just do
csv_file = open(output_file, mode='w+')
and be careful to close it later.
Alternatively, create a new csv.DictWriter with the new file object.

Getting error Nonetype in python 2.7

I am getting the error:
Traceback (most recent call last):
File "<stdin>", line 9, in <module>
AttributeError: 'NoneType' object has no attribute 'get'
Is there something wrong with the following script? I am using get() in the script because of some phone field names missing in Mongodb collections.
The sample data:
{
"_id": ObjectID("57e99f88b948da6c3be04366"),
"email": "jstahl2020#gmail.com",
"phone": [
{
"type": "Mobile",
"number": "250-851-1041"
}
]}
the python script :
import codecs
import csv
cursor = db.users.find ({}, {'_id':1, 'email': 1, 'phone.type':1, 'phone.number':1})
with codecs.open('applicatonmethod3.csv', 'w', encoding='utf-8') as outfile:
fields = ['_id', 'email', 'phone.type', 'phone.number']
write = csv.DictWriter(outfile, fieldnames=fields)
write.writeheader()
for x in cursor:
x_id = x.get('_id')
x_email = x['email']
y = x.get('phone')
z = {'_id': x_id,
'email': x_email,
'phone.type': y.get('type'),
'phone.number': y.get('number')}
write.writerow(z)
please someone can help me.

Are you connected to the correct database, the one that contains the users collection and not just the default database?

I worked out the problem. I just share it with you. here is the solution. thanks you for help.
import sys
reload(sys)
sys.setdefaultencoding('utf-8')
import codecs
import csv
cursor = db.users.find ({}, {'_id':1, 'phone.type':1, 'phone.number':1, 'phone.extension':1})
with codecs.open('phone.csv', 'w', encoding='utf-8') as outfile:
fields =['_id', 'phone.type', 'phone.number', 'phone.extension']
write = csv.DictWriter(outfile, fieldnames=fields)
write.writeheader()
for x in cursor:
x_id = x['_id']
for y in x.get('phone', {}):
z = {'_id': x_id,
'phone.type': y.get('type'),
'phone.number': y.get('number'),
'phone.extension': y.get('extension')}
print z
write.writerow(z)
cursor.close()

Python: csv to json converter value to key pair

I had a Python beginners course last year. Now I am trying to get a csv to json converter. I have searched quite some time and adapted and changed some of the code I found, until the output looked similar to what I want. I am using Python 3.4.2.
#kvorobiev this is an excerpt of my CSV, but it will do for the case. The first time Converting will work. After the second time you will see that the order of the headings will change within the json file.
The csv file looks like this
Document;Item;Category
4;10;C
What I am getting in the output file as of now (after applying the changes from kvorobiev):
[
{
"Item": "10",
"Category": "C",
"Document": "4"
};
]
The json string I want to get in the output file should look like:
[
{
"Document": "4",
"Item": "10",
"Category": "C"
},
]
You will notice the headings are in the wrong order.
Here is the code:
import json
import csv
csvfile = open('file1.csv', 'r')
jsonfile = open('file1.csv'.replace('.csv','.json'), 'w')
jsonfile.write('[' + '\n' + ' ')
fieldnames = csvfile.readline().replace('\n','').split(';')
num_lines = sum(1 for line in open('file.csv')) -1
reader = csv.DictReader(csvfile, fieldnames)
i = 0
for row in reader:
i += 1
json.dump(row, jsonfile, indent=4,sort_keys=False)
if i < num_lines:
jsonfile.write(',')
jsonfile.write('\n')
jsonfile.write(' ' + ']')
print('Done')
Thanks for helping.

Replace line
reader = csv.DictReader(csvfile, fieldnames)
with
reader = csv.DictReader(csvfile, fieldnames, delimiter=';')
Also, you open file1.csv and later get lines number from file.csv
num_lines = sum(1 for line in open('file.csv')) -2
Your solution could be reduced to
import json
import csv
csvfile = open('file1.csv', 'r')
jsonfile = open('file1.csv'.replace('.csv','.json'), 'w')
jsonfile.write('{\n[\n')
fieldnames = csvfile.readline().replace('\n','').split(';')
reader = csv.DictReader(csvfile, fieldnames, delimiter=';')
for row in reader:
json.dump(row, jsonfile, indent=4)
jsonfile.write(';\n')
jsonfile.write(']\n}')
If you want to save order of columns from csv you could use
from collections import OrderedDict
...
for row in reader:
json.dump(OrderedDict([(f, row[f]) for f in fieldnames]), jsonfile, indent=4)
jsonfile.write(';\n')
jsonfile.write(']\n}')

Best practice to store column-based data before to write to CSV in Python

I have the current code in Python 3:
import csv
if __name__ == '__main__':
sp500_data = [
{
'company': 'GOOGLE',
'headquarters': 'GOOGLEPLEX',
'industry': 'ADS',
'sector': 'TECH',
'symbol': 'GOOG'
},
{
'company': 'HEWLPA',
'headquarters': 'WHATEVER',
'industry': 'HARDWARE',
'sector': 'TECH',
'symbol': 'HP'
}
]
myfile = open("D:/test.csv", 'w', newline='')
wr = csv.DictWriter(myfile, delimiter='\t', quoting=csv.QUOTE_ALL, fieldnames=sp500_data[0].keys)
for sp500_company in sp500_data:
wr.writerow(sp500_company)
However this gives the following error:
Traceback (most recent call last):
File "D:\DEV\BlueTS\src\tsRetriever\dataRetriever\test.py", line 24, in <module>
wr.writerow(sp500_company)
File "C:\Python33\lib\csv.py", line 153, in writerow
return self.writer.writerow(self._dict_to_list(rowdict))
File "C:\Python33\lib\csv.py", line 146, in _dict_to_list
wrong_fields = [k for k in rowdict if k not in self.fieldnames]
File "C:\Python33\lib\csv.py", line 146, in <listcomp>
wrong_fields = [k for k in rowdict if k not in self.fieldnames]
TypeError: argument of type 'builtin_function_or_method' is not iterable
I would like to understand what I am doing wrong, and in addition to this, I would like to know what is the best way in Python to store column-based data which was originally organised in tables.

You forgot to call the .keys() method:
wr = csv.DictWriter(myfile, delimiter='\t', quoting=csv.QUOTE_ALL,
fieldnames=sp500_data[0].keys())
Note the () after sp500_data[0].keys; .keys is not an attribute, it is a method.
Using a csv.DictWriter() is an excellent method to turn data already in dictionary format into CSV data.