A csv file names.csv has content:
first_name last_name
Baked Beans
Lovely Spam
Wonderful Spam
I would like to read it into a list of dictionaries, with the first row containing the keys:
>>> import csv
>>> with open('names.csv') as csvfile:
... reader = csv.DictReader(csvfile)
... for row in reader:
... print(row['first_name'], row['last_name'])
...
Baked Beans
Lovely Spam
Wonderful Spam
But is the type of reader csv.DictReader?
How can I convert reader into a list of dictionaries?
Thanks.
import csv
with open("in.csv") as csvfile:
reader = csv.DictReader(csvfile,delimiter=" ")
print(list(reader))
[{'first_name': 'Baked', 'last_name': 'Beans'}, {'first_name': 'Lovely', 'last_name': 'Spam'}, {'first_name': 'Wonderful', 'last_name': 'Spam'}]
If the delimiter is not actually a , you need to specify " " or whatever it is.
Just to clear any confusion, the code works fine for python3.6 also, the only difference is that using DictReader gives Orderdicts by default:
In [1]: import csv
...: with open("in.csv") as csvfile:
...: reader = csv.DictReader(csvfile, delimiter=" ")
...: print(list(reader))
...:
[OrderedDict([('first_name', 'Baked'), ('last_name', 'Beans')]), OrderedDict([('first_name', 'Lovely'), ('last_name', 'Spam')]), OrderedDict([('first_name', 'Wonderful'), ('last_name', 'Spam')])]
You can access keys exactly the same, an OrderedDict just keeps key insertion order:
In [2]: import csv
...: with open("in.csv") as csvfile:
...: reader = csv.DictReader(csvfile, delimiter=" ")
...: for dct in reader:
...: print(f"{dct['first_name']} {dct['last_name']}")
...:
...:
Baked Beans
Lovely Spam
Wonderful Spam
Which py3.6 actually does too, so if for some reason you really want a dict:
In [5]: import csv
...: with open("in.csv") as csvfile:
...: reader = csv.DictReader(csvfile, delimiter=" ")
...: for dct in map(dict, reader):
...: print(dct)
...: print(f"{dct['first_name']} {dct['last_name']}")
...:
...:
{'first_name': 'Baked', 'last_name': 'Beans'}
Baked Beans
{'first_name': 'Lovely', 'last_name': 'Spam'}
Lovely Spam
{'first_name': 'Wonderful', 'last_name': 'Spam'}
Wonderful Spam
The ordering retention on insertion in py3.6 is an implementation detail and may change, but if enough of us use it, it may just have to stay :)
Use list():
print(list(reader))
Demo:
>>> with open('names.csv') as csvfile:
... reader = csv.DictReader(csvfile, delimiter=" ")
... print(list(reader))
...
[{'first_name': 'Baked', 'last_name': 'Beans'}, {'first_name': 'Lovely', 'last_name': 'Spam'}, {'first_name': 'Wonderful', 'last_name': 'Spam'}]
Create function
import csv
def csv_to_dict(filename):
result_list=[]
with open(filename) as file_obj:
reader = csv.DictReader(file_obj, delimiter=delimiter)
for row in reader:
result_list.append(dict(row))
return result_list
and use it:
list_of_dicts = csv_to_dict('names.csv')
The csv.DictReader creates a dictreader object as noted in the title. This object is only useable while the file is open and cant be sliced.
To convert the object to a list as per the title....
list_of_dicts = list(your_dictreader_object)
Related
Each line in my csv file is data on a pet. Ex"Fish, Nemo, April 2nd, Goldfish, Orange." I would like to import that file and create a new object for that pet depending on its type(the first string in each line). For example data about the fish would be stored in a fish object. I then want to put each object into a list.
I've tried:
pets = []
with open('desktop/cs110/pets.csv', 'r') as file:
csvReader = csv.reader(file, delimiter=',')
for row_pets in csvReader:
pets.append(row_pets)
columnNames = ['firstCol', 'secondCol', 'thirdColomn']
lstPets = []
for row_pets in pets:
lstPets.append({key: value for key, value in zip(columnNames, row_pets)})
return lstPets
With csv.DictReader you can accomplish what your current code attempts by specifying fieldnames and assuming your "object" desired is a dictionary:
pets.csv
Fish,Nemo,April 2nd,Goldfish,Orange
Cat,Garfield,June 1st,Tabby,Orange
test.py
import csv
from pprint import pprint
with open('pets.csv', newline='') as file:
reader = csv.DictReader(file, fieldnames='type name bday species color'.split())
data = list(reader)
pprint(data)
Output
[{'bday': 'April 2nd',
'color': 'Orange',
'name': 'Nemo',
'species': 'Goldfish',
'type': 'Fish'},
{'bday': 'June 1st',
'color': 'Orange',
'name': 'Garfield',
'species': 'Tabby',
'type': 'Cat'}]
Hello guys im learning how to code doing this small code creating a method to read a csv file I have object has no attribute mistake, can you help me please?
import csv
class LargeList:
def readCsv(self, filename):
with open(filename) as fp:
reader = csv.reader(fp, delimiter = ",", quotechar = '""')
for row in reader:
print(row)
if __name__ == '__main__':
csv = LargeList()
filename = 'N.csv'
csv.readCsv(filename)
I use this for working with csv files:
import csv
def open_csv(path):
'''return a list of dictionaries
'''
with open(path, 'r') as file:
reader = csv.DictReader(file)
return [dict(row) for row in reader]
print(open_csv('./test.csv'))
Input csv:
first_name,last_name,email
John,Doe,john.doe#example.com
Jane,Doe,jane.doe#example.com
Foo,Bar,foo.bar#example.com
Output:
[{'first_name': 'John', 'last_name': 'Doe', 'email': 'john.doe#example.com'}, {'first_name': 'Jane', 'last_name': 'Doe', 'email': 'jane.doe#example.com'}, {'first_name': 'Foo', 'last_name': 'Bar', 'email': 'foo.bar#example.com'}]
I've learned to let the csv package handle format parsing on its own, its pretty awesome at that!
I am trying to write a code that will take in json values from Kafka and output them to a .csv file. The issue is that, for grades, the values have either science and math OR just english as nested objects.
This is what the data looks like:
{'id': 0, 'name': 'Susan', 'lastName': 'Johnsan', 'grades': {'science':
78, 'math': 89}}
{'id': 1, 'name': 'Mary', 'lastName': 'Davids', 'grades': {'english':
85}}
However when I run my code I keep getting the error TypeError: string indices must be integers.
from kafka import KafkaConsumer
import json
import csv
import sys
from datetime import datetime
import os
# connect to kafka topic
kaf = KafkaConsumer('students.all.events')
outputfile = 'C:\\Users\\Documents\\students_output.csv'
outfile = open(outputfile, mode='w', newline='')
master_key = ['id', 'name', 'lastName', 'science', 'math', 'english']
writer = csv.DictWriter(outfile, master_key, delimiter="|")
writer.writeheader()
'''
writer = csv.writer(outfile)
writer.writerow(['JSON_Data'])
'''
i = 1
for row in kaf:
if i < 5000:
json_row = json.loads(row.value)
print('Row: ', i)
print(json_row)
dict = {'id': json_row['id'], 'name': json_row['name'], 'lastName': json_row['lastName']}
for value in json_row['grades']:
if value['science'] is not None:
dict['science'] = value['science']
dict['math'] = value['math']
elif value['english'] is not None:
dict['english'] = value['english']
writer.writerow(dict)
i += 1
else:
break
outfile.close()
Please check if the value variable is actually of type dict, because the error you get, in general, means that you are trying to access a string object in a dict[key] way.
It looks like you have a typo - at least in the code that you pasted here. There is an extra double quote after the lastName key.
Based off of the help #TenorFlyy gave me, I changed my code to fix the issue:
from kafka import KafkaConsumer
import json
import csv
import sys
from datetime import datetime
import os
# connect to kafka topic
kaf = KafkaConsumer('students.all.events')
outputfile = 'C:\\Users\\Documents\\students_output.csv'
outfile = open(outputfile, mode='w', newline='')
master_key = ['id', 'name', 'lastName', 'science', 'math', 'english']
writer = csv.DictWriter(outfile, master_key, delimiter="|")
writer.writeheader()
'''
writer = csv.writer(outfile)
writer.writerow(['JSON_Data'])
'''
i = 1
for row in kaf:
if i < 5000:
json_row = json.loads(row.value)
print('Row: ', i)
print(json_row)
dict = {'id': json_row['id'], 'name': json_row['name'], 'lastName': json_row['lastName']}
for key, value in json_row['grades'].items():
dict[key] = value
writer.writerow(dict)
i += 1
else:
break
outfile.close()
I'm using poloniex trader api to get realtime market ticker info. It works perfect on console. When ever i request for market ticker in returns i get this {'last': '0.07269671', 'high24hr': '0.07379970', 'quoteVolume': '71582.56540639', 'isFrozen': '0', 'lowestAsk': '0.07277290', 'percentChange': '-0.00551274', 'id': 148, 'low24hr': '0.07124645', 'highestBid': '0.07269671', 'baseVolume': '5172.41552885'}
Problem is it's only storing item name/list name such as - low24hr, lowestAsk, highestBid etc. Not their value like -low24hr : 0.07124645
polo = Poloniex()
ticker_data = (polo('returnTicker')['BTC_ETH'])
out = csv.writer(open("myfile.csv","w"), delimiter=',',quoting=csv.QUOTE_ALL,)
out.writerow(ticker_data)
print(ticker_data)
Here is what my csv file looks like-
Your problem is that out.writerow(ticker_data) takes only the keys of the dictionary and writes them to the file. Try to use a csv.DictWriter:
with open('myfile.csv', 'w', newline='') as csv_file:
# Pass the keys of the `ticker_data` as the fieldnames.
writer = csv.DictWriter(csv_file, fieldnames=ticker_data, quoting=csv.QUOTE_ALL)
# Write the fieldnames.
writer.writeheader()
# Write the values.
writer.writerow(ticker_data)
with open('my_file.csv', 'w') as f:
for key, value in ticker_data.items(): f.write('{0},{1}\n'.format(key, value))
From here.
I want to import a CSV into multiple dictionaries in python
queryInclude,yahoo,value1
queryInclude,yahoo,value2
queryInclude,yahoo,value3
queryExclude,yahoo,value4
queryExclude,yahoo,value5
queryInclude,google,value6
queryExclude,google,value7
My ideal result would have row[0]=dictionary, row[1]=key, and row[2]=value or list of values
queryInclude = {
"yahoo": ["value1", "value2", "value3"],
"google": ["value6"] }
queryExclude = {
"yahoo": ["value4", "value5"],
"google": ["value7"] }
Here's my code:
import csv
queryList=[]
queryDict={}
with open('dictionary.csv') as csvfile:
reader = csv.reader(csvfile, delimiter=',', quotechar='|')
for row in reader:
queryDict[row[1]] = queryList.append(row[2])
print queryDict
{'yahoo': None}
{'yahoo': None}
{'yahoo': None}
{'yahoo': None}
{'yahoo': None}
{'google': None, 'yahoo': None}
{'google': None, 'yahoo': None}
I have the flexibility to change the CSV format if needed. My ideal result posted above is what I already had hard-coded into my app. I'm trying to make it easier to add more values down the road. I've spent many hours researching this and will continue to update if I make any more progress. My thought process looks like this... not sure how close I am to understanding how to structure my loops and combine like values while iterating through the CSV rows...
for row in reader:
where row[0] = queryInclude:
create a dictionary combining keys into a list of values
where row[0] = queryExclude:
create a dictionary combining keys into a list of values
Using defaultdict prevents having to account for the first element added to a dictionary. It declares the default type when the key is not present and must be a callable that creates the default object:
#! python3
import csv
from io import StringIO
from collections import defaultdict
from pprint import pprint
data = StringIO('''\
queryInclude,yahoo,value1
queryInclude,yahoo,value2
queryInclude,yahoo,value3
queryExclude,yahoo,value4
queryExclude,yahoo,value5
queryInclude,google,value6
queryExclude,google,value7
''')
D = defaultdict(lambda: defaultdict(list))
for d,k,v in csv.reader(data):
D[d][k].append(v)
pprint(D)
Output:
{'queryExclude': {'google': ['value7'],
'yahoo': ['value4', 'value5']},
'queryInclude': {'google': ['value6'],
'yahoo': ['value1', 'value2', 'value3']}}
Is this helping?
import StringIO
import csv
csvfile = StringIO.StringIO("""queryInclude,yahoo,value1
queryInclude,yahoo,value2
queryInclude,yahoo,value3
queryExclude,yahoo,value4
queryExclude,yahoo,value5
queryInclude,google,value6
queryExclude,google,value7""")
reader = csv.reader(csvfile, delimiter=',', quotechar='|')
dict1={}
for row in reader:
key1, provider, value1 = row
if not dict1.has_key(key1):
dict1[key1] = {}
if not dict1[key1].has_key(provider):
dict1[key1][provider] = []
dict1[key1][provider].append(value1)