I have a CSV file that I need to read and process in python.
The CSV file contains tabular values as follows:
*aa
1 foo1 foo_bar1
2 foo2 foo_bar2
*bb
1.22 bla1 blabla1 blablabla22
1.33 bla2 ' ' blablabla33
Here aa and bb are the names of each table. Wherever table names occur, the name is preceded by a * and the rows below it are the rows of that table.
Note that each table can have:
different number of columns as well as rows.
There can also be empty columns representing missing values. I would like to keep them as ' ' after reading in.
However, we know exactly which tables are present in the csv file (i.e. the table names)
I need to read in the csv file and assign a table's entire content to one variable. I can think of a brute force way of doing this. However, since python has a csv module with read write operations, is there any built in functionality that could make this easier or more efficient for me?
Note: One of the major problem I've faced so far is that after reading in the csv file using csv.reader(), I see that aa's rows have additional empty columns. I believe this is because of the mismatch in the number of aa's and bb's columns. I also want to get rid of these additional empty columns without deleting the empty columns that represent actual missing values.
The cleanest way is to separate the tables before feeding each group to the csv reader. Here is a rough cut to get you started:
from itertools import takewhile
import csv
# Instead of *s*, you can use an open file object here
s = '''\
*aa
1,foo1,foo_bar1
2,foo2,foo_bar2
*bb
1.22,bla1,blabla1,blablabla22
1.33,bla2, ,blablabla33
'''.splitlines()
it = iter(s)
next(it)
for table in ['aa', 'bb']:
print(f'\nTable: {table}')
for row in csv.reader(takewhile(lambda r: not r.startswith('*'), it)):
print(row)
This produces:
Table: aa
['1', 'foo1', 'foo_bar1 ']
['2', 'foo2', 'foo_bar2']
Table: bb
['1.22', 'bla1', 'blabla1', 'blablabla22']
['1.33', 'bla2', ' ', 'blablabla33']
You could parse your csv file like so checking if the first value startswith a '*' and build a dict from it.
import csv
from collections import defaultdict
import pprint
csv_data = defaultdict(list)
with open('data.csv', 'r') as csv_file:
# filter empty lines
csv_reader = csv.reader(filter(lambda l: l.strip(',\n'), csv_file))
header = None
for row in csv_reader:
if row[0].startswith('*'):
header = row[0]
else:
# additional row processing if needed
csv_data[header].append(row)
pprint.pprint(csv_data)
# Output
defaultdict(<class 'list'>,
{'*aa': [['1', ' foo1', 'foo_bar1', ''],
['2', ' foo2', 'foo_bar2', '']],
'*bb': [['1.22', ' bla1', 'blabla1', 'blablabla22'],
['1.333', ' bla2', '', 'blablabla3']]})
If you want to remove the excess elements from a table due to another being larger, one option is
csv_data[header].append(row[:col_nums[header]])
Where as you mentioned you know how many columns your table should have
col_nums = {'*aa' : 3, '*bb' : 4}
defaultdict(<class 'list'>,
{'*aa': [['1', ' foo1', 'foo_bar1'],
['2', ' foo2', 'foo_bar2']],
'*bb': [['1.22', ' bla1', 'blabla1', 'blablabla22'],
['1.333', ' bla2', '', 'blablabla3']]})
If I misread it and you only know the max number of columns and not the number of columns for each table, then you could instead do.
def trim_row(row):
for i, item in enumerate(reversed(row)):
if not item:
break
return row[:len(row) - i]
# use it like so
csv_data[header].append(trim_row(row))
Have you considered using pandas?
import pandas as pd
df = pd.read_csv('foo.csv', sep=r'/s+', header=None) #if there is table headings, remove header = None
You do not need to add any line to the top of the file.
This reads files with different number of rows and columns into a dataframe. You can perform all sorts of actions in it now. For ex:
Empty elements are represented by NaN, which means Not a Number. You can replace it with ' ' just by writing
df.fillna(' ')
To fit your use case, from what i understand, you have multiple tables in the same csv file, try this:
df = pd.read_csv("foo.csv", header=None, names=range(3))
table_names = ["*aa", "*bb", "*cc"..]
groups = df[0].isin(table_names).cumsum()
tables = {g.iloc[0,0]: g.iloc[1:] for k,g in df.groupby(groups)}
This will create a list of tables with key as table name and value as the table itself.
for k,v in tables.items():
print("table:", k)
print(v)
print()
You can find more details in the documentation.
Related
I've two CSV file named as file1.csv and file2.csv in file2.csv there is only one column which contain only five records and in file1.csv I've three column which contain more than thousand records I want to get those records which contain in file2.csv
for example this is my file1.csv
'A J1, Jhon1',jhon1#jhon.com, A/B-201 Test1
'A J2, Jhon2',jhon2#jhon.com, A/B-202 Test2
'A J3, Jhon3',jhon3#jhon.com, A/B-203 Test3
'A J4, Jhon4',jhon4#jhon.com, A/B-204 Test4
.......and more records
and inside my file2.csv I've only five records right now but in future it can be many
A/B-201 Test1
A/B-2012 Test12
A/B-203 Test3
A/B-2022 Test22
so I've to find records from my file1.csv at index[2] or index[-1]
this is what I did but it not giving me any output it just returning empty list
import csv
file1 = open('file1.csv','r')
file2 = open('file2.csv','r')
f1 = list(csv.reader(file1))
f2 = list(csv.reader(file2))
new_list = []
for i in f1:
if i[-1] in f2:
new_list.append(i)
print('New List : ',new_list)
it gives me output like this
New List : []
Please help if I did any thing wrong correct me.
Method 1: pandas
This task can be done with relative ease using pandas. DataFrame documentation here.
Example:
In the example below, the two CSV files are read into two DataFrames. The DataFrames are merged using an inner join on the matching columns.
The output shows the merged result.
import pandas as pd
df1 = pd.read_csv('file1.csv', names=['col1', 'col2', 'col3'], quotechar="'", skipinitialspace=True)
df2 = pd.read_csv('file2.csv', names=['match'])
df = pd.merge(df1, df2, left_on=df1['col3'], right_on=df2['match'], how='inner')
The quotechar and skipinitialspace parameters are used as the first column in file1 is quoted and contains a comma, and there is leading whitespace after the comma before the last field.
Output:
col1 col2 col3
0 A J1, Jhon1 jhon1#jhon.com A/B-201 Test1
1 A J3, Jhon3 jhon3#jhon.com A/B-203 Test3
If you choose, the output can easily be written back to a CSV file as:
df.to_csv('path/to/output.csv')
For other DataFrame operations, refer to the documentation linked above.
Method 2: Core Python
The method below does not use any libraries, only core Python.
Read the matches from file2 into a list.
Iterate over file1 and search each line to determine if the last value is a match for an item in file2.
Report the output.
Any subsequent data cleaning (if required) will be up to your personal requirements or use-case.
Example:
output = []
# Read the matching values into a list.
with open('file2.csv') as f:
matches = [i.strip() for i in f]
# Iterate over file1 and place any matches into the output.
with open('file1.csv') as f:
for i in f:
match = i.split(',')[-1].strip()
if any(match == j for j in matches):
output.append(i)
Output:
["'A J1, Jhon1',jhon1#jhon.com, A/B-201 Test1\n",
"'A J3, Jhon3',jhon3#jhon.com, A/B-203 Test3\n"]
Use sets or dicts for in checks (complexity is O(1) for them, instead of O(N) for lists and tuples).
Have a look at convtools library (github): it has Table helper for working with table data as with streams (table docs)
from convtools import conversion as c
from convtools.contrib.tables import Table
# creating a set of allowed values
allowed_values = {
item[0] for item in Table.from_csv("input2.csv").into_iter_rows(tuple)
}
result = list(
# reading a file with custom quotechar
Table.from_csv("input.csv", dialect=Table.csv_dialect(quotechar="'"))
# stripping last column values
.update(COLUMN_2=c.col("COLUMN_2").call_method("strip"))
# filtering based on allowed values
.filter(c.col("COLUMN_2").in_(c.naive(allowed_values)))
# returning iterable of tuples
.into_iter_rows(tuple)
# # OR outputting csv if needed
# .into_csv("result.csv")
)
"""
>>> In [36]: result
>>> Out[36]:
>>> [('A J1, Jhon1', 'jhon1#jhon.com', 'A/B-201 Test1'),
>>> ('A J3, Jhon3', 'jhon3#jhon.com', 'A/B-203 Test3')]
"""
I am a beginner user of Python and would like to clean the csv file for analysis purpose. However, I am facing the problem with the code.
def open_dataset(file_name):
opened_file = open(file_name)
read_file = reader(opened_file, delimiter=",")
data = list(read_file)
return data
def column(filename):
filename = open_dataset(filename)
for row in filename:
print(row)
with the code above, the output is like
['Name;Product;Sales;Country;Website']
[';Milk;30;Germany;something.com']
[';;;USA;']
['Chris;Milk;40;;']
I would like to have the output following:
['Name','Product','Sales','Country','Website']
[NaN,'Milk','30','Germany','something.com']
[NaN,NaN,NaN,'USA',NaN]
['Chris','Milk',40,NaN,NaN]
I defined a delimiter as "," but still ";" used. I don't know why it is happening. Also Even if I tried to replace the space with "NaN, but still every space is replaced by "NaN".
Would be really appreciated if someone could give me tips
After all, I would like to analyse each column such as percentage of "NaN" etc.
Thank you!
You can get the result that you want by:
specifying ';' as the delimiter when constructing a reader object
passing each row through a function that converts empty cells to 'NaN' (or some other value of your choice)
Here is some sample code:
import csv
def row_factory(row):
return [x if x != '' else 'NaN' for x in row]
with open(filename, 'r', newline='') as f:
reader = csv.reader(f, delimiter=';')
for row in reader:
print(row_factory(row))
Output:
['Name', 'Product', 'Sales', 'Country', 'Website']
['NaN', 'Milk', '30', 'Germany', 'something.com']
['NaN', 'NaN', 'NaN', 'USA', 'NaN']
['Chris', 'Milk', '40', 'NaN', 'NaN']
You need to specify the correct delimiter:
read_file = reader(opened_file, delimiter=";")
Your CSV file appears to be using a semicolon rather than a comma, so you need to tell reader() what to use.
Tip:
filename = open_dataset(filename)
Don't reassign a variable to mean something else. Before this line executes, filename is a string with the name of the file to open. After this assignment, filename is now a list of rows from the file. Use a different variable name instead:
rows = open_dataset(filename)
Now the two variables are distinct and their meaning is clear from the names. Of course, feel free to use something other than rows other than filename.
You might want to look into using pandas. It can make data processing a lot easier, up to and including reading multi file formats.
if you want to read a csv:
import pandas as pd:
my_file = '/pat/to/my_csv.csv'
pd.read_csv(my_file)
That's because your lists contain only one element, and that element is a single string, in order to parse a string into a list you can split it.
This should do what you need:
for row in filename:
parsed_row = row[0].split(';')
for i in range(0, len(parsed_row)):
if parsed_row[i] == '':
parsed_row[i] = None
print(parsed_row)
I made you a Repl.it example
I am preparing for a test and one of the topics is to parse tabular data without using csv/panda packages.
The ask is to take data with an arbitrary number of columns and convert it into a dictionary. The delimiter can be a space, colon or comma. For instance, here is some data with comma as the delimiter -
person,age,nationality,language, education
Jack,18,Canadian,English, bs
Rahul,25,Indian,Hindi, ms
Mark,50,American,English, phd
Kyou, 21, Japanese, English, bs
This should be converted into a dictionary format like this -
{'person': ['Jack', 'Rahul', 'Mark', 'Kyou'], 'age': ['18', '25', '50', '21'], 'education': ['doc', 'eng', 'llb', 'ca'], 'language': ['English', 'Hindi', 'English', 'English'
], 'nationality': ['Canadian', 'Indian', 'American', 'Japanese']}
Columns can vary among different files. My program should be flexible to handle this variety. For instance, in the next file there might be another column titled "gender".
I was able to get this working but feel my code is very "clunky". It works but I would like to do something more "pythonic".
from collections import OrderedDict
def parse_data(myfile):
# initialize myd as an ordered dictionary
myd = OrderedDict()
# open file with data
with open (myfile, "r") as f:
# use readlines to store tabular data in list format
data = f.readlines()
# use the first row to initialize the ordered dictionary keys
for item in data[0].split(','):
myd[item.strip()] = [] # initializing dict keys with column names
# variable use to access different column values in remaining rows
i = 0
# access each key in the ordered dict
for key in myd:
'''Tabular data starting from line # 1 is accessed and
split on the "," delimiter. The variable "i" is used to access
each column incrementally. Ordered dict format of myd ensures
columns are paired appropriately'''
myd[key] = [ item.split(',')[i].strip() for item in data[1:]]
i += 1
print dict(myd)
# my-input.txt
parse_data("my-input.txt")
Can you please suggest how can I make my code "cleaner"?
Here is a more pythonic way to approach this.
def parse(file):
with open(file, 'r') as f:
headings = f.readline().strip().split(',')
values = [l.strip().split(',') for l in f]
output_dict = {h: v for h, v in zip(headings, [*zip(*values)])}
return output_dict
print(parse('test.csv'))
First, take the first line in the file as the headings to use for the keys in the dictionary (this will break with duplicate headings)
Then, all the remaining values are read into a list of lists of strings using a list comprehension.
Finally the dictionary is compiled by zipping the list of headings with a transpose (thats what the [*zip(*values))] represents - if you are willing to use numpy you can replace this with numpy.array(values).T for example)
Slightly better version
def parse_data(myfile):
# read lines and strip out extra whitespaces and newline characters
lines = [line.strip() for line in open(myfile,"r").readlines()]
dict = {} # initialize our dict variable
# start loop from second line
for x in range(1,len(lines)):
# for each line split values and store them in dict[col]
for y in range(len(lines[0].split(","))):
# if col is not present in dict create new column and initialize it with a list
if lines[0].split(",")[y] not in dict:
dict[lines[0].split(",")[y]] = []
# store the corresponding column value to the dict
dict[lines[0].split(",")[y]].append(lines[x].split(",")[y])
parse_data("my-input.txt")
See it in action here.
Hope it helps!
I'm trying to read in a .csv file and extract specific columns so that I can output a single table that essentially performs a 'GROUP BY' on a particular column and aggregates certain other columns of interest (similar to how you would in SQL) but I'm not too familiar how to do this easily in Python.
The csv file is in the following form:
age,education,balance,approved
30,primary,1850,yes
54,secondary,800,no
24,tertiary,240,yes
I've tried to import and read in the csv files to parse the three columns I care about and iterate through them to put them into three separate array lists. I'm not too familiar with packages and how to get these into a data frame or matrix with 3 columns so that I can then iterate through them mutate or perform all of the aggregated output field (see below expected results).
with open('loans.csv') as csvfile:
readCSV = csv.reader(csvfile, delimiter = ',')
next(readCSV) ##skips header row
education = []
balance = []
loan_approved = []
for row in readCSV:
educat = row[1]
bal = row[2]
approve = row[3]
education.append(educat)
balance.append(bal)
loan_approved.append(approve)
print(education)
print(balance)
print(loan_approved)
The output would be a 4x7 table of four rows (grouped by education level) and the following headers:
Education|#Applicants|Min Bal|Max Bal|#Approved|#Rejected|%Apps Approved
Primary ...
Secondary ...
Terciary ...
It seems to be much simpler by using Pandas instead. For instance, you can read only the columns that you care for instead of all of them:
import Pandas as pd
df = pd.read_csv(usecols=['education', 'balance', 'loan_approved'])
Now, to group by education level, you can find all the unique entries for that column and group them:
groupby_education = {}
for level in list(set(df['education'])):
groupby_education[level] = df.loc[df['education'] == level]
print(groupby_education)
I hope this helped. Let me know if you still need help.
Cheers!
Input:
I have two csv files (file1.csv and file2.csv).
file1 looks like:
ID,Name,Gender
1,Smith,M
2,John,M
file2 looks like:
name,gender,city,id
Problem:
I want to compare the header of file1 with file2 and copy the data of the matching columns. The header in file1 need to be in lowercase prior to finding the matching columns in file2.
Output:
the output should be like this:
name,gender,city,id # name,gender,and id are the only matching columns btw file1 and file2
Smith,M, ,1 # the data copied for name, gender, and id columns
John,M, ,2
I have tried the following code so far:
import csv
file1 = csv.DictReader(open("file1.csv")) #reading file1.csv
file1_Dict = {} # the dictionary of lists that will store the keys and values as list
for row in file1:
for column, value in row.iteritems():
file1_Dict.setdefault(column,[]).append(value)
for key in file1_Dict: # converting the keys of the dictionary to lowercase
file1_Dict[key.lower()] = file1_Dict.pop(key)
file2 = open("file2.csv") #reading file2.csv
file2_Dict ={} # store the keys into a dictionary with empty values
for row2 in file2:
row2 = row2.split(",")
for i in row2:
file2_Dict[i] = ""
Any idea how to solve this problem?
I had a crack on this problem using python without taking performance into consideration. Took me quite a while, phew!
This is my solution.
import csv
csv_data1_filepath = './file1.csv'
csv_data2_filepath = './file2.csv'
def main():
# import nem config and data into memory
data1 = list(csv.reader(file(csv_data1_filepath, 'r')))
data2 = list(csv.reader(file(csv_data2_filepath, 'r')))
file1_header = data1[0][:] # Get f1 header
file2_header = data2[0][:] # Get f1 header
lowered_file1_header = [item.lower() for item in file1_header] # lowercase it
lowered_file2_header = [item.lower() for item in file2_header] # do it for header 2 anyway
col_index_dict = {}
for column in lowered_file1_header:
if column in file2_header:
col_index_dict[column] = lowered_file1_header.index(column)
else:
col_index_dict[column] = -1 # mark as column that will not be worked on later
for column in lowered_file2_header:
if not column in lowered_file1_header:
col_index_dict[column] = -1 # mark as column that will not be worked on later
# build header
output = [col_index_dict.keys()]
is_header = True
for row in data1:
if is_header is False:
rowData = []
for column in col_index_dict:
column_index = col_index_dict[column]
if column_index != -1:
rowData.append(row[column_index])
else:
rowData.append('')
output.append(rowData)
else:
is_header = False
print(output)
if __name__ == '__main__':
main()
This will give you the output of:
[
['gender', 'city', 'id', 'name'],
['M', '', '1', 'Smith'],
['M', '', '2', 'John']
]
Note that the output kind of lost its ordering but this should be fixable by using the ordered dictionary instead.
Hope this helps.
You don't need Python for this. This is a task for SQL.
SQLite Browser supports CSV Import. Take the below steps to get the desired output:
Download and install SQLite Browser
Create a new Database
Import both CSV's as tables (let's say the table names are file1 and file2, respectively)
Now, you can decide how you want to match the data sets. If you only want to match the files on ID, then you can do something like:
select *
from file1 f1
inner join file2 f2
on f1.id = f2.id
If you want to match on every column, you can do something like:
select *
from file1 f1
inner join file2 f2
on f1.id = f2.id and f1.name = f2.name and f1.gender = f2.gender
Finally, simply export the query results back to a CSV.
I spent a lot of time myself trying to perform tasks like this with scripting languages. The benefit of using SQL is that you simply tell what you want to match on, and then let the database do the optimization for you. Generally, it ends up doing the matching faster than any code I could write.
In case you're interested, python also has a sqlite module that comes out-of-the-box. I've gravitated towards using this as my source for data in python scripts for the above reason, and I simply import the CSV's required in SQLite browser before running the python script.