Related
I have this code:
type(i[0]) == 'str'. How can I get it as a list? and later get print(i[0][0]) == 'a'
import csv
i = [['a', 'b'], 'c', 'd']
with open('a.csv', 'a', newline='', encoding='utf-8') as file:
writer = csv.writer(file, delimiter=';')
writer.writerow(i)
file.close()
with open('a.csv', 'r', newline='', encoding='utf-8') as file:
for i in csv.reader(file, delimiter=';'):
print(type(i[0]))
The reason it is returning str instead of list because the csv writer will always do the equivalent of str(['a','b']), which is "['a', 'b']",c,d, so you can use ast.literal_eval to get the list from the string list like this for the first element-
import csv
import ast
i = [['a', 'b'], 'c', 'd']
with open('a.csv', 'a', newline='', encoding='utf-8') as file:
writer = csv.writer(file, delimiter=';')
writer.writerow(i)
file.close()
with open('a.csv', 'r', newline='', encoding='utf-8') as file:
for i in csv.reader(file, delimiter=';'):
i[0] = ast.literal_eval(i[0])
print(type(i),type(i[0]), i[0][0])
You can use a for-loop. Like this:
example a.csv
hello, aloha, hola
bye, aloha, adios
python code:
import csv
col1 = []
with open('a.csv') as file:
r = csv.reader(file)
for row in r:
col1.append(row[0])
print(col1) # => ['hello', 'bye']
Alternatively, you could also use a list comprehension:
import csv
col1 = []
with open('a.csv') as file:
r = csv.reader(file)
col1 = [row[0] for row in r]
print(col1) # => ['hello', 'bye']
I have a csv file like following :
A, B, C, D
2,3,4,5
4,3,5,2
5,8,3,9
7,4,2,6
8,6,3,7
I want to fetch the B values from 3 rows at a time(for first iteration values would be 3,3,8) and save in some variable(value1=3,value2=3,value3=8) and pass it on to a function. Once those values are processed. I want to fetch the values from next 3 rows (value1=3,value2=8,value3=4) and so on.
The csv file is large.
I am a JAVA developer, if possible suggest the simplest possible code.
An easy solution would be the following:
import pandas as pd
data = pd.read_csv("path.csv")
for i in range(len(data)-2):
value1 = data.loc[i,"B"]
value2 = data.loc[i+1,"B"]
value3 = data.loc[i+2,"B"]
function(value1, value2, value3)
This is a possible solution (I have used the function proposed in this answer):
import csv
import itertools
# Function to iterate the csv file by chunks (of any size)
def grouper(n, iterable):
it = iter(iterable)
while True:
chunk = tuple(itertools.islice(it, n))
if not chunk:
return
yield chunk
# Open the csv file
with open('myfile.csv') as f:
csvreader = csv.reader(f)
# Read the headers: ['A', 'B', 'C', 'D']
headers = next(csvreader, None)
# Read the rest of the file by chunks of 3 rows
for chunk in grouper(3, csvreader):
# do something with your chunk of rows
print(chunk)
Printed result:
(['2', '3', '4', '5'], ['4', '3', '5', '2'], ['5', '8', '3', '9'])
(['7', '4', '2', '6'], ['8', '6', '3', '7'])
You can use pandas to read your csv with chunksize argument as described here (How can I partially read a huge CSV file?)
import pandas as pd
#Function that you want to apply to you arguments
def fn(A, B, C, D):
print(sum(A), sum(B), sum(C), sum(D))
#Iterate through the chunks
for chunk in pd.read_csv('test.csv', chunksize=3):
#Convert dataframe to dict
chunk_dict = chunk.to_dict(orient = 'list')
#Pass arguments to your functions
fn(**chunk_dict)
You can use csv module
import csv
with open('data.txt') as fp:
reader = csv.reader(fp)
next(reader) #skips the header
res = [int(row[1]) for row in reader]
groups = (res[idx: idx + 3] for idx in range(0, len(res) - 2))
for a, b, c in groups:
print(a, b, c)
Output:
3 3 8
3 8 4
8 4 6
I have a list of {n} dictionaries in a txt file. Each dictionary per line as illustrated below which i want exported in csv format with each key presented per column.
{'a':'1','b':'2','c':'3'}
{'a':'4','b':'5','c':'6'}
{'a':'7','b':'8','c':'9'}
{'a':'10','b':'11','c':'12'}
...
{'a':'x','b':'y','c':'z'}
i want csv output for {n} rows as below with index
a b c
0 1 2 3
1 4 5 6
2 7 8 9
... ... ... ...
n x y z
You can use ast.literal_eval (doc) to load your data from the text file.
With contents of input file file.txt:
{'a':'1','b':'2','c':'3'}
{'a':'4','b':'5','c':'6'}
{'a':'7','b':'8','c':'9'}
{'a':'10','b':'11','c':'12'}
{'a':'x','b':'y','c':'z'}
You could use this script to load the data and input file.csv:
import csv
from ast import literal_eval
with open('file.txt', 'r') as f_in:
lst = [literal_eval(line) for line in f_in if line.strip()]
with open('file.csv', 'w', newline='') as csvfile:
fieldnames = ['a', 'b', 'c']
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
writer.writeheader()
writer.writerows(lst)
file.csv will become:
a,b,c
1,2,3
4,5,6
7,8,9
10,11,12
x,y,z
Importing the file to LibreOffice:
x =[{'a':'1','b':'2','c':'3'},
{'a':'4','b':'5','c':'6'},
{'a':'7','b':'8','c':'9'},
{'a':'10','b':'11','c':'12'}]
n = len(x)
keys = list(x[0].keys())
newdict=dict()
for m in keys:
newdict[m]=[]
for i in range(n):
newdict[m].append(x[i][m])
newdict
Output is
{'a': ['1', '4', '7', '10'],
'b': ['2', '5', '8', '11'],
'c': ['3', '6', '9', '12']}
Or you can use pandas.concat which is used to combine DataFrames with the same columns.
import pandas as pd
x =[{'a':'1','b':'2','c':'3'},
{'a':'4','b':'5','c':'6'},
{'a':'7','b':'8','c':'9'},
{'a':'10','b':'11','c':'12'}]
xpd=[]
for i in x:
df=pd.DataFrame(i, index=[0])
xpd.append(df)
pd.concat(xpd, ignore_index=True)
How to open each column of a text file as separate list?
import csv
infile = open ('data.txt', 'r')
reader = csv.reader(infile, delimiter=' ')
col1 = [line[0] for line in reader]
print (col1)
col2 = [line[1] for line in reader]
print (col2)
But column2 is empty. Any idea???
The data.txt look as follows:
a d 1
b e 2
c f 3
In your code col2 is empty because after the first iteration(List comprehension for col1) the iterator is already exhausted.
You can zip with *:
import csv
with open('abc.csv') as f:
reader = csv.reader(f, delimiter=' ')
c1, c2, c3 = zip(*reader)
print c1
print c2
print c3
Output:
('a', 'b', 'c')
('d', 'e', 'f')
('1', '2', '3')
with open ('data.txt','r') as infile:
col1= [i.strip().split(' ')[0] for i in infile]
with open ('data.txt','r') as infile:
col2= [i.strip().split(' ')[1] for i in infile]
print (col1)
print (col2)
Here is my current code
a_reader = None
a_reader = open('data.csv', 'rU')
a_csv_reader = csv.reader(a_reader)
for row in a_csv_reader:
print row
a_reader.close()
count = 0
sum = 0.0
a_reader = open('data.csv', 'rU')
a_csv_reader = csv.reader(a_reader)
a_csv_reader.next()
for row in a_csv_reader:
if count != 0 and row[0] != '':
sum = sum + float(row[0])
count = count + 1
a_reader.close()
print 'Number of lines is:',count
print 'Sum is:',sum
return listStation
This produces the results below
['1', '476050', '7709929']
['2', '473971', '7707713']
['3', '465676', '7691097']
['4', '515612', '7702192']
['5', '516655', '7704405']
['6', '519788', '7713255']
['7', '538466', '7683341']
Number of lines is: 8
Sum is: 28.0
Ok now what I want to do is to split out the value of the ID, Easting and Northing and append them to a list to create one 2d list. Is it possible to do this? If so can you provide me the code?
rows = []
for row in a_csv_reader:
rows.append(row)
Will yield in rows:
[['1', '476050', '7709929']
['2', '473971', '7707713']
['3', '465676', '7691097']
['4', '515612', '7702192']
['5', '516655', '7704405']
['6', '519788', '7713255']
['7', '538466', '7683341']]
Try this:
import csv
def run():
count = 0
sum = 0.0
listStation = []
with open('data.csv', 'rU') as a_reader:
a_csv_reader = csv.reader(a_reader)
for row in a_csv_reader:
if count != 0:
if row[0] != '':
sum = sum + float(row[0])
listStation.append(map(int, row))
print 'row =', row
count = count + 1
print 'Number of lines is:',count
print 'Sum is:', sum
print listStation
if __name__ == '__main__':
run()
I don't have your data.csv file, to test with, but here's how I'd rewrite your code and make it produce the 2D list you want:
import csv
with open('test_data.csv', 'rU') as a_reader:
a_csv_reader = csv.reader(a_reader)
for row in a_csv_reader:
print row
with open('test_data.csv', 'rU') as a_reader:
a_csv_reader = csv.reader(a_reader)
a_csv_reader.next()
listStation = []
count = 0
total = 0.0
for row in a_csv_reader:
if count != 0 and row[0] != '':
total += float(row[0])
count += 1
listStation.append(map(int, row))
print 'Number of lines is:', count
print 'Sum is:', total
print 'listStation:', listStation
Output:
['ID', 'Easting', 'Northing']
['1', '476050', '7709929']
['2', '473971', '7707713']
['3', '465676', '7691097']
['4', '515612', '7702192']
['5', '516655', '7704405']
['6', '519788', '7713255']
['7', '538466', '7683341']
Number of lines is: 7
Sum is: 27.0
listStation: [[1, 476050, 7709929], [2, 473971, 7707713], [3, 465676, 7691097],
[4, 515612, 7702192], [5, 516655, 7704405], [6, 519788, 7713255],
[7, 538466, 7683341]]
Note, I changed the variable you named sum to total to prevent a conflict with the built-in sum() function.
The following might work (depends on the data though - do blank first columns/otherwise invalid numbers exist in the columns etc...):
from itertools import islice
import csv
with open('data.csv') as fin:
csvin = islice(csv.reader(fin), 1, None) # skip header
rows = [map(int, row) for row in csvin]
print 'Rows are:'
print rows
print 'Number of lines is:', len(stuff)
print 'Sum is:', sum(row[0] for row in stuff)
I suppose I would write your code with a DictReader and default dict:
import csv
data={}
with open('/tmp/sta.txt','r') as fin:
reader=csv.DictReader(fin)
for row in reader:
for k,v in row.items():
data.setdefault(k,[]).append(float(v))
print data
print 'Sum is:',sum(data['Station ID'])
print 'Number of lines is:',len(data['Station ID'])+1
Prints:
{'Station ID': [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0],
'Easting': [476050.0, 473971.0, 465676.0, 515612.0, 516655.0, 519788.0, 538466.0],
'Northing': [7709929.0, 7707713.0, 7691097.0, 7702192.0, 7704405.0, 7713255.0, 7683341.0]}
Sum is: 28.0
Number of lines is: 8