Develop Reference

Writing Dictionaries into a CSV

I am working on a web scraping project, I am trying to save the results into a CSV file,
data = {
'address_area': area,
'address_district': district,
'website': website,
'branch_rating': rating,
'facilities_delivery': delivery,
'facilities_banquet': banquet,
'facilites_shisha': shisha,
'faciliteis_minumum': minumium,
'facilites_reservation': reservation,
'facilities_free_wifi': wifi,
'facilities_smoking_permited': smoking,
'facilities_eat_out': eat_out,
'facilities_private_parking': parking,
'facilities_price_range': price_range,
'facilities_kids_ares': kids_ares,
'branch_no': branch_no}
mainlist.append(data)
with open('filetest.csv', 'w') as f:
writer = csv.writer(f)
for value in mainlist:
writer.writerow([value])
I want to save the key in the dictionary as columns and the values as a row,
(keep in mind that the value pair in the dictionary refers to a variable that extracts data from a web site)

Here is a solution that can work for you. I added a second data item to your code (data2) and renamed the initial data element to data1.
mainlist = []
#initial data item
data1 = {
'address_area': 'area1',
'address_district': 'district1',
'website': 'website1',
'branch_rating': 'rating1',
'facilities_delivery': 'delivery1',
'facilities_banquet': 'banquet1',
'facilites_shisha': 'shisha1',
'faciliteis_minumum': 'minumium1',
'facilites_reservation': 'reservation1',
'facilities_free_wifi': 'wifi1',
'facilities_smoking_permited': 'smoking1',
'facilities_eat_out': 'eat_out1',
'facilities_private_parking': 'parking1',
'facilities_price_range': 'price_range1',
'facilities_kids_ares': 'kids_aresa1',
'branch_no': 'branch_no1'}
mainlist.append(data1)
#second data item
data2 = {
'address_area': 'area2',
'address_district': 'district2',
'website': 'website2',
'branch_rating': 'rating2',
'facilities_delivery': 'delivery2',
'facilities_banquet': 'banquet2',
'facilites_shisha': 'shisha2',
'faciliteis_minumum': 'minumium2',
'facilites_reservation': 'reservation2',
'facilities_free_wifi': 'wifi2',
'facilities_smoking_permited': 'smoking2',
'facilities_eat_out': 'eat_out2',
'facilities_private_parking': 'parking2',
'facilities_price_range': 'price_range2',
'facilities_kids_ares': 'kids_aresa2',
'branch_no': 'branch_no2'}
mainlist.append(data2)
filename = 'filetest.csv'
headers = ",".join(data1.keys())
with open(filename, 'w') as f:
f.write(headers + '\n')
for item in mainlist:
f.write(','.join(str(item[key]) for key in item) + '\n')
print("All done. Check this file for results:",filename)

Saving a nested JSON response of what I'm able to call a print function on?

I'm having a difficult time figuring out how to save the results of an API call to a CSV. When I run the print function below, it prints exactly what I want saved. I've tried dictwriter, I've tried jsondumps, there's always something wrong with the output (putting everything on one row + doing columns too, or adding writing like t,h,i,s to the end of each entry).
fList = sys.argv[1]
with open(fList, "r", encoding='utf-8-sig') as ins:
for line in ins:
targetDomain = line
url1 = "https://websitehere={0}".format(
targetDomain)
response = requests.get(url1).json()
for subs in response["result"]["records"]:
with open('output.csv', 'a', newline="") as f:
print(subs)`
The code above prints:
{'domain': '***redacted out of caution***', 'firstSeen': 1629178483, 'lastSeen': 1629178507}
{'domain': '***redacted out of caution***', 'firstSeen': 1631174847, 'lastSeen': 1631174878}
{'domain': '***redacted out of caution***', 'firstSeen': 1630654337, 'lastSeen': 1630654337}
{'domain': '***redacted out of caution***', 'firstSeen': 1630465072, 'lastSeen': 1630465072}
Which is exactly what I want saved in a CSV but cannot for the life of me figure it out. Thank you.

I would use the csv module for this. Example with DictWriter (taken from the docs):
import csv
rows = [
{'domain': 'xx', 'firstSeen': 1629178483, 'lastSeen': 1629178507},
{'domain': 'xx', 'firstSeen': 1631174847, 'lastSeen': 1631174878}
]
with open('output.csv', 'w') as out_file:
writer = csv.DictWriter(out_file, fieldnames=list(rows[0].keys()))
writer.writeheader()
for row in rows:
writer.writerow(row)

I figured it out - Thank you.
with open('output.csv', 'a') as f:
print(json.dumps(subs), file=f)

Python csv.DictWriter not writing Dictory to writerow method

Python - 3.8
I am trying to write a csv file from a list of dictionary. I followed the official website example. writer is able to write headers. But it is not writing the rows from dictionary that i loops from the list.
csv_path = "/home/tmp/file.csv"
with open(csv_path, 'w') as csvfile:
writer = csv.DictWriter(csvfile, fieldnames=["RequestTimestamp", "MerchantId", "MerchantName", "ChannelName",
"MerchantUSN", "BatchNumber", "UniqueId", "TransactionId", "InvoiceId",
"ReferenceTimestamp", "CustomerName", "AccountHolder", "PaymentType",
"Currency", "Debit", "Credit", "SettledAmount", "Result", "bankResult", "ReconStatus"])
writer.writeheader()
for data in recons:
result_dict = utils.data_formatter_handler(data)
writer.writerow(data)

Append a file in Python 3 I/O operation on closed file

I'm able to open and write some headers to a csv file at the start of my code:
## Create the output file
with open(output_file, mode='w+') as csv_file:
fieldnames = ['Name', 'Instance ID', 'Type', 'State', 'Private IP', 'Public IP', 'Region', 'Availability Zone', 'Launch Time' ]
writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
writer.writeheader()
But in the same program I build a list of EC2 instances, and try to print that list to the same file later on in the code, and I get this error:
File "C:\Users\tdunphy\AppData\Local\Programs\Python\Python37-32\lib\csv.py", line 155, in writerow
return self.writer.writerow(self._dict_to_list(rowdict))
ValueError: I/O operation on closed file.
The is the code where I try to append to the file, that produces the above error:
with open(output_file, mode='a+') as csv_file:
writer.writerow({'Name': name, 'Instance ID': instance_id, 'Type': instance_type, 'State': instance_state, 'Private IP': private_ips, 'Public IP': public_ips, 'Region': aws_region, 'Availability Zone': availability_zone, 'Launch Time': launch_time_friendly})
Why can't I write to this file? What can I do to correct that?

You might need to add back this line before trying to writerow, as the old writer may still be referring to a "closed" file in memory:
writer = csv.DictWriter(csv_file, fieldnames=fieldnames)

with open(path, 'w+') as f1:
s = store(f1)
with open(path, 'a+') as f2:
s.operate_on()
if the same as
try:
f1 = open(path, 'w+') as f:
s = store(f1)
finally:
f1.close()
try:
f2 = open(path, 'a+')
s.operate_on()
finally:
f2.close()
Perhaps this is more obvious what you're problem is. You're storing an open file object, then closing it, then opening some other file object f2 which you never use and trying to operate on the closed file.
Your solution is to either put writer.writerow(...) in the first with block or just do
csv_file = open(output_file, mode='w+')
and be careful to close it later.
Alternatively, create a new csv.DictWriter with the new file object.

How to format a csv file to put all information on a line and then move down?

I have a file that takes information and writes it to a csv file. I am having trouble getting it to format the way I want. It is looping 10 times and the information is there I can confirm. I am including the code to show you the exact setup of the csv writing part.
Here is my code:
outfile = open('Accounts_Details.csv', 'a')
for i in range(0, 11):
#Calling all the above functions
soc_auth_requests()
create_account()
config_admin_create()
account_user_create()
account_activate()
account_config_DNS_create()
#Creating the dictionary for the CSV file with the data fields made and modified from before
#It is necessary this be done after the above method calls to ensure the data field values are correct
data = {
'Account_Name': acc_name,
'Account_Id': acc_id,
'User_Email': user_email,
'User_id': user_id
}
#Creating a csv file and writing the dictionary titled "data" to it
for key, value in sorted(data.items()):
outfile.write('\t' + str(value))
outfile.write('\n')
So I have four bits of data in the dict and I want the format to be laid out in the csv file so that the four bits of info are put on one line and when it loops through the for loop it moves to the next line and does the same there.
Ex.
name1, id1, email1, uId1
name2, id2, email2, uId2
name3, id3, email3, uId3
I assume it has to do with how I open the file, but I am not sure and can't figure it out.
Thanks for the help!
Here is the current output I am getting. I want all the 1's to be on one line and then move down.
name1
id1
email1
uID1
name2
id2
email2
uID2

Try deleting arguments from last statement:
for key, value in sorted(data.items()):
outfile.write('\t' + str(value))
# outfile.write('\n')
# modified for
outfile.close()
please let me know how it was and if this worked!
:)

I don't think your file opening arguments are the issue -- I wasn't able to replicate the issue. However, you could probably streamline the code and remove any possibility of an issue by using list comprehensions for your data:
for i in range(12):
data = {
'Account_Name': 'AccountName',
'Account_Id': '#12345',
'User_Email': 'a#b',
'User_id': 'LOGGER'
}
with open("test.txt", "a") as f:
f.write('{}\n'.format(', '.join(data.values())))

Just use the csv module which will automagically format the line for you:
outfile = open('Accounts_Details.csv', 'a')
writer = csv.DictWriter(outfile, [ 'Account_Name', 'Account_Id', 'User_Email', 'User_id' ])
# optionaly if you want a header line:
writer.writeheader()
for i in range(0, 11):
...
data = {
'Account_Name': acc_name,
'Account_Id': acc_id,
'User_Email': user_email,
'User_id': user_id
}
writer.writerow(data)
outfile.close()