Hello guys im learning how to code doing this small code creating a method to read a csv file I have object has no attribute mistake, can you help me please?
import csv
class LargeList:
def readCsv(self, filename):
with open(filename) as fp:
reader = csv.reader(fp, delimiter = ",", quotechar = '""')
for row in reader:
print(row)
if __name__ == '__main__':
csv = LargeList()
filename = 'N.csv'
csv.readCsv(filename)
I use this for working with csv files:
import csv
def open_csv(path):
'''return a list of dictionaries
'''
with open(path, 'r') as file:
reader = csv.DictReader(file)
return [dict(row) for row in reader]
print(open_csv('./test.csv'))
Input csv:
first_name,last_name,email
John,Doe,john.doe#example.com
Jane,Doe,jane.doe#example.com
Foo,Bar,foo.bar#example.com
Output:
[{'first_name': 'John', 'last_name': 'Doe', 'email': 'john.doe#example.com'}, {'first_name': 'Jane', 'last_name': 'Doe', 'email': 'jane.doe#example.com'}, {'first_name': 'Foo', 'last_name': 'Bar', 'email': 'foo.bar#example.com'}]
I've learned to let the csv package handle format parsing on its own, its pretty awesome at that!
Related
Each line in my csv file is data on a pet. Ex"Fish, Nemo, April 2nd, Goldfish, Orange." I would like to import that file and create a new object for that pet depending on its type(the first string in each line). For example data about the fish would be stored in a fish object. I then want to put each object into a list.
I've tried:
pets = []
with open('desktop/cs110/pets.csv', 'r') as file:
csvReader = csv.reader(file, delimiter=',')
for row_pets in csvReader:
pets.append(row_pets)
columnNames = ['firstCol', 'secondCol', 'thirdColomn']
lstPets = []
for row_pets in pets:
lstPets.append({key: value for key, value in zip(columnNames, row_pets)})
return lstPets
With csv.DictReader you can accomplish what your current code attempts by specifying fieldnames and assuming your "object" desired is a dictionary:
pets.csv
Fish,Nemo,April 2nd,Goldfish,Orange
Cat,Garfield,June 1st,Tabby,Orange
test.py
import csv
from pprint import pprint
with open('pets.csv', newline='') as file:
reader = csv.DictReader(file, fieldnames='type name bday species color'.split())
data = list(reader)
pprint(data)
Output
[{'bday': 'April 2nd',
'color': 'Orange',
'name': 'Nemo',
'species': 'Goldfish',
'type': 'Fish'},
{'bday': 'June 1st',
'color': 'Orange',
'name': 'Garfield',
'species': 'Tabby',
'type': 'Cat'}]
I am a fairly new dev and trying to parse "id" values from this file. Running into the issue below.
My python code:
import ast
from pathlib import Path
file = Path.home() /'AppData'/'Roaming'/'user-preferences-prod'
with open(file, 'r') as f:
contents = f.read()
ids = ast.literal_eval(contents)
profileids = []
for data in ids:
test= data.get('id')
profileids.append(test)
print(profileids))
This returns the error: ValueError: malformed node or string: <_ast.Name object at 0x0000023D8DA4D2E8> at ids = ast.literal_eval(contents)
A snippet of the content in my file of interest:
{"settings":{"defaults":{"value1":,"value2":,"value3":null,"value4":null,"proxyid":null,"sites":{},"sizes":[],"value5":false},"value6":true,"value11":,"user":{"value9":"","value8": ,"value7":"","value10":""},"webhook":"},'profiles':[{'billing': {'address1': '', 'address2': '', 'city': '', 'country': 'United States', 'firstName': '', 'lastName': '', 'phone': '', 'postalCode': '', 'province': '', 'usesBillingInformation': False}, 'createdAt': 123231231213212, 'id': '23123123123213, 'name': ''
I need this code to be looped as there are multiple id values that I am interested in and need them all to be entered into a list.Hopefully I explained it all. the file type is "file" according to windows, I just view its contents with notepad.
It appears to me that you have a file with a string representation of a dict (dictionary). So, what you need to do is:
string_of_dict →ast.literal_eval()→ dict
Open file and read in the text into a string variable. Currently I think this string is going into ids.
Then convert the string representation of dict into a dict using ast library as shown below. Reference
import ast
string_of_dict = "{'muffin' : 'lolz', 'foo' : 'kitty'}"
ast.literal_eval(string_of_dict)
Output:
{'muffin': 'lolz', 'foo': 'kitty'}
Solution
Something like this should most likely work. You may have to tweak it a little bit.
import ast
with open(file, 'r') as f:
contents = f.read()
ids = ast.literal_eval(contents)
profileids = []
for data in ids:
test= data.get('id')
profileids.append(test)
print(profileids)
I'm able to open and write some headers to a csv file at the start of my code:
## Create the output file
with open(output_file, mode='w+') as csv_file:
fieldnames = ['Name', 'Instance ID', 'Type', 'State', 'Private IP', 'Public IP', 'Region', 'Availability Zone', 'Launch Time' ]
writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
writer.writeheader()
But in the same program I build a list of EC2 instances, and try to print that list to the same file later on in the code, and I get this error:
File "C:\Users\tdunphy\AppData\Local\Programs\Python\Python37-32\lib\csv.py", line 155, in writerow
return self.writer.writerow(self._dict_to_list(rowdict))
ValueError: I/O operation on closed file.
The is the code where I try to append to the file, that produces the above error:
with open(output_file, mode='a+') as csv_file:
writer.writerow({'Name': name, 'Instance ID': instance_id, 'Type': instance_type, 'State': instance_state, 'Private IP': private_ips, 'Public IP': public_ips, 'Region': aws_region, 'Availability Zone': availability_zone, 'Launch Time': launch_time_friendly})
Why can't I write to this file? What can I do to correct that?
You might need to add back this line before trying to writerow, as the old writer may still be referring to a "closed" file in memory:
writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
with open(path, 'w+') as f1:
s = store(f1)
with open(path, 'a+') as f2:
s.operate_on()
if the same as
try:
f1 = open(path, 'w+') as f:
s = store(f1)
finally:
f1.close()
try:
f2 = open(path, 'a+')
s.operate_on()
finally:
f2.close()
Perhaps this is more obvious what you're problem is. You're storing an open file object, then closing it, then opening some other file object f2 which you never use and trying to operate on the closed file.
Your solution is to either put writer.writerow(...) in the first with block or just do
csv_file = open(output_file, mode='w+')
and be careful to close it later.
Alternatively, create a new csv.DictWriter with the new file object.
I am trying to write a code that will take in json values from Kafka and output them to a .csv file. The issue is that, for grades, the values have either science and math OR just english as nested objects.
This is what the data looks like:
{'id': 0, 'name': 'Susan', 'lastName': 'Johnsan', 'grades': {'science':
78, 'math': 89}}
{'id': 1, 'name': 'Mary', 'lastName': 'Davids', 'grades': {'english':
85}}
However when I run my code I keep getting the error TypeError: string indices must be integers.
from kafka import KafkaConsumer
import json
import csv
import sys
from datetime import datetime
import os
# connect to kafka topic
kaf = KafkaConsumer('students.all.events')
outputfile = 'C:\\Users\\Documents\\students_output.csv'
outfile = open(outputfile, mode='w', newline='')
master_key = ['id', 'name', 'lastName', 'science', 'math', 'english']
writer = csv.DictWriter(outfile, master_key, delimiter="|")
writer.writeheader()
'''
writer = csv.writer(outfile)
writer.writerow(['JSON_Data'])
'''
i = 1
for row in kaf:
if i < 5000:
json_row = json.loads(row.value)
print('Row: ', i)
print(json_row)
dict = {'id': json_row['id'], 'name': json_row['name'], 'lastName': json_row['lastName']}
for value in json_row['grades']:
if value['science'] is not None:
dict['science'] = value['science']
dict['math'] = value['math']
elif value['english'] is not None:
dict['english'] = value['english']
writer.writerow(dict)
i += 1
else:
break
outfile.close()
Please check if the value variable is actually of type dict, because the error you get, in general, means that you are trying to access a string object in a dict[key] way.
It looks like you have a typo - at least in the code that you pasted here. There is an extra double quote after the lastName key.
Based off of the help #TenorFlyy gave me, I changed my code to fix the issue:
from kafka import KafkaConsumer
import json
import csv
import sys
from datetime import datetime
import os
# connect to kafka topic
kaf = KafkaConsumer('students.all.events')
outputfile = 'C:\\Users\\Documents\\students_output.csv'
outfile = open(outputfile, mode='w', newline='')
master_key = ['id', 'name', 'lastName', 'science', 'math', 'english']
writer = csv.DictWriter(outfile, master_key, delimiter="|")
writer.writeheader()
'''
writer = csv.writer(outfile)
writer.writerow(['JSON_Data'])
'''
i = 1
for row in kaf:
if i < 5000:
json_row = json.loads(row.value)
print('Row: ', i)
print(json_row)
dict = {'id': json_row['id'], 'name': json_row['name'], 'lastName': json_row['lastName']}
for key, value in json_row['grades'].items():
dict[key] = value
writer.writerow(dict)
i += 1
else:
break
outfile.close()
A csv file names.csv has content:
first_name last_name
Baked Beans
Lovely Spam
Wonderful Spam
I would like to read it into a list of dictionaries, with the first row containing the keys:
>>> import csv
>>> with open('names.csv') as csvfile:
... reader = csv.DictReader(csvfile)
... for row in reader:
... print(row['first_name'], row['last_name'])
...
Baked Beans
Lovely Spam
Wonderful Spam
But is the type of reader csv.DictReader?
How can I convert reader into a list of dictionaries?
Thanks.
import csv
with open("in.csv") as csvfile:
reader = csv.DictReader(csvfile,delimiter=" ")
print(list(reader))
[{'first_name': 'Baked', 'last_name': 'Beans'}, {'first_name': 'Lovely', 'last_name': 'Spam'}, {'first_name': 'Wonderful', 'last_name': 'Spam'}]
If the delimiter is not actually a , you need to specify " " or whatever it is.
Just to clear any confusion, the code works fine for python3.6 also, the only difference is that using DictReader gives Orderdicts by default:
In [1]: import csv
...: with open("in.csv") as csvfile:
...: reader = csv.DictReader(csvfile, delimiter=" ")
...: print(list(reader))
...:
[OrderedDict([('first_name', 'Baked'), ('last_name', 'Beans')]), OrderedDict([('first_name', 'Lovely'), ('last_name', 'Spam')]), OrderedDict([('first_name', 'Wonderful'), ('last_name', 'Spam')])]
You can access keys exactly the same, an OrderedDict just keeps key insertion order:
In [2]: import csv
...: with open("in.csv") as csvfile:
...: reader = csv.DictReader(csvfile, delimiter=" ")
...: for dct in reader:
...: print(f"{dct['first_name']} {dct['last_name']}")
...:
...:
Baked Beans
Lovely Spam
Wonderful Spam
Which py3.6 actually does too, so if for some reason you really want a dict:
In [5]: import csv
...: with open("in.csv") as csvfile:
...: reader = csv.DictReader(csvfile, delimiter=" ")
...: for dct in map(dict, reader):
...: print(dct)
...: print(f"{dct['first_name']} {dct['last_name']}")
...:
...:
{'first_name': 'Baked', 'last_name': 'Beans'}
Baked Beans
{'first_name': 'Lovely', 'last_name': 'Spam'}
Lovely Spam
{'first_name': 'Wonderful', 'last_name': 'Spam'}
Wonderful Spam
The ordering retention on insertion in py3.6 is an implementation detail and may change, but if enough of us use it, it may just have to stay :)
Use list():
print(list(reader))
Demo:
>>> with open('names.csv') as csvfile:
... reader = csv.DictReader(csvfile, delimiter=" ")
... print(list(reader))
...
[{'first_name': 'Baked', 'last_name': 'Beans'}, {'first_name': 'Lovely', 'last_name': 'Spam'}, {'first_name': 'Wonderful', 'last_name': 'Spam'}]
Create function
import csv
def csv_to_dict(filename):
result_list=[]
with open(filename) as file_obj:
reader = csv.DictReader(file_obj, delimiter=delimiter)
for row in reader:
result_list.append(dict(row))
return result_list
and use it:
list_of_dicts = csv_to_dict('names.csv')
The csv.DictReader creates a dictreader object as noted in the title. This object is only useable while the file is open and cant be sliced.
To convert the object to a list as per the title....
list_of_dicts = list(your_dictreader_object)