For a project I have devices who send payloads and I should store them on a localfile, but I have memory limitation and I dont want to store more than 2000 data rows. again for the memory limitation I cannot have a database so I chose to store data in csv file.
I tried to use open('output.csv', 'r+') as f: ; I'm appending the rows to the end of my csv and I have to check each time the lenght with sum(1 for line in f) to be sure its not more than 2000.
The big problem starts when I reach 2000 rows and I want to ideally delete the first row and add another row to the end or start to write rows from the beginning of the file and overwrite the old rows without deleting evrything, but I dont know how to do it. I tried to use open('output.csv', 'w+') or open('output.csv', 'a+') but it will delete all the contents with w+ while writing only one row and by a+ it just continues to append to the end. I on the otherhand I cannot count the number of rows anymore with both. can you pleas help me which command should I use to start to rewrite each line from the beginning or delete one line from the beginning and append one to the end? I will also appriciate if you can tell me if there is a better chioce than csv files for storing many data or I can use a better way to count the number of rows.
This should help. See comments inline
import pandas as pd
allowed_length = 2 # Set it to the required value
df = pd.read_csv('output.csv') #Read your csv file to df
row_count = df.shape[0] #Get row count
df.loc[row_count] = ['Fridge', 15] #Insert row at end of df. In my case it has only 2 values
#if count of dataframe is greater or equal to allowed_length, the delete first row
if row_count >= allowed_length:
df = df.drop(df.head(1).index)
df.to_csv('output.csv', index=False)
Related
I want to create a program in which reads a CSV file and writes in another file. My problem is, the file I'm ready is kinda big and I don't want to go through every column by doing this:
columns = defaultdict(list)
reader = csv.DictReader(csvfile)
for row in reader:
for (k,v) in row.items():
columns[k].append(v)
print(columns['name'])
print(columns['id'])
...
I wanted to, instead, do columns[0] to find 'name', and so on. Is there any way I can do this?
You are now reading the CSV with a DictReader this creates the columns based on names, in your case you could just use the reader:
columns = defaultdict(list)
reader = csv.reader(csvfile)
next(reader) # to skip the header row
for row in reader:
for i, v in enumerate(row):
columns[i].append(v)
print(columns[0])
print(columns[1])
I'm not sure that I understand your question. If you are asking, "can I read only the first column?", then the short answer is no. CSV is specifically designed to read a fixed number of columns from variable length records. More specifically, the data is organized as a list of rows, not a list of columns. You can't just seek past what you don't want to read. It sounds like what you are trying to do is reorganized your data into columns.
If you want to minimize the processing of what you do read, it sounds like all you need to do is use csv.reader and skip the first row containing the header. Each row from the reader will return a list of strings and the construction of this list should be less expensive than a map.
If you collect the list of rows you can then put it in a numpy array. A numpy array will allow you to access columns (e.g., x[:, 0]) or rows (e.g., x[0, :]).
Given that I am not entirely sure what you are asking, my answers may not not be what you are looking for; however, whatever your problem is, I am certain you cannot avoid reading the entire file.
I've got a lot of csv files which contain strings. I would like to import the strings in python 3 from the multiple csvs to a master csv but making sure that no duplicates which are already contained in the master csv are added.
I've written some code but I'm unsure of how to get the print to be written to the master csv and how to check for duplicates.
My current code is:
output = [ ]
f = open( 'example.csv' , 'r' )
for line in f:
cells = line.split( "," )
output.append( ( cells[ 3 ]))
f.close( )
print (output)
Any help would be appreciated.
Thanks in advance.
The answer really depends on how big those CSV files are i.e. how many words do you expect to end up in master CSV. Based on that you can have more or less optimized Python code.
First things first, you should provide some sort of example since from what is shown, you take strings from third column and put them in output list.
One solution could be this:
from csv import reader
words = set()
# open master CSV file in case it already exists and load all words
# now, this is the part where you didn't give an example of how master CSV should look like
# I'll assume its just a word per line text file
with open(MASTER_CSV_FILE, 'r') as f:
for line in f:
words.append(line)
with open(NEW_CSV_FILE, 'r') as f:
for columns in reader(f):
words.append(columns[3])
# here again, I'll just write word per line in MASTER_CSV_FILE
with open(MASTER_CSV_FILE, 'w') as f:
for word in words:
f.write(word + '\n')
I have based my answer on next assumptions:
master CSV file is actually word per line text file (due to a lack of examples),
new CSV file always have at least 3 comma separated values in each row,
you just want to dedupe words and do not want to count number duplicates.
Here's another way that may work for you.
import pandas as pd
# Create a DataFrame that will be used to load all the data.
# The duplicates will be removed once all the csv's have been
# loaded
df = pd.DataFrame()
# Read the contents of the csv files into the DataFrame.
# I'm assuming all the csv's have the same data format.
for f in os.listdir():
if f.endswith(".csv"):
df = df.append(pd.read_csv(f))
# Eliminate the duplicates. This will use the values in
# all the columns of the DataFrame to determine whether
# a particular row is a duplicate.
df.drop_duplicates(inplace=True)
You can then convert the DataFrame back to a csv file by using df.to_csv() if needed.
Hope that helps.
I have a very large csv file with millions of rows and a list of the row numbers that I need.like
rownumberList = [1,2,5,6,8,9,20,22]
I know there is something called skiprows that helps to skip several rows when reading csv file like that
df = pd.read_csv('myfile.csv',skiprows = skiplist)
#skiplist would contain the total row list deducts rownumberList
However, since the csv file is very large, directly selecting the rows that I need could be more efficient. So I was wondering are there any methods to select rows when using read_csv? Not try to select rows using dataframe afterwards, since I try to minimize the time of reading file.Thanks.
There is a parameter called nrows : int, default None
Number of rows of file to read. Useful for reading pieces of large files (Docs)
pd.read_csv(file_name,nrows=int)
In case you need some part in the middle. Use both skiprows as well as nrows in read_csv.if skiprows indicate the beginning rows and nrows will indicate the next number of rows after skipping eg.
Example:
pd.read_csv('../input/sample_submission.csv',skiprows=5,nrows=10)
This will select data from the 6th row to 16 row
Edit based on comment:
Since there is a list this one might help i.e
li = [1,2,3,5,9]
r = [i for i in range(max(li)) if i not in li]
df = pd.read_csv('../input/sample_submission.csv',skiprows=r,nrows= max(li))
# This will skip the rows you dont want as well as limit the number of rows to maximum of the list.
import pandas as pd
rownumberList = [1,2,5,6,8,9,20,22]
df = pd.read_csv('myfile.csv',skiprows=lambda x: x not in rownumberList)
for pandas 0.25.1, pandas read_csv, you can pass callable function to skiprows
I am not sure about read_csv() from Pandas (there is though a way to use an iterator for reading a large file in chunks), but you can read the file line by line (lazy-loading, not reading the whole file in memory) with csv.reader (or csv.DictReader), leaving only the desired rows with the help of enumerate():
import csv
import pandas as pd
DESIRED_ROWS = {1, 17, 28}
with open("input.csv") as input_file:
reader = csv.reader(input_file)
desired_rows = [row for row_number, row in enumerate(reader)
if row_number in DESIRED_ROWS]
df = pd.DataFrame(desired_rows)
(assuming you would like to pick random/discontinuous rows and not a "continuous chunk" from somewhere in the middle - in that case #James's idea to have "start and "stop" would work generally better).
import pandas as pd
df = pd.read_csv('Data.csv')
df.iloc[3:6]
Returns rows 3 through 5 and all columns.
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.iloc.html
From de documentation you can see that skiprows can take an integer or a list as values to remove some lines.
So basicaly you can tell it to remove all but those you want. For this you first need to know the number in lines in the file (best if you know beforehand) by open it and counting as following:
with open('myfile.csv') as f:
row_count = sum(1 for row in f)
Now you need to create the complementary list (here are sets but also works, don't know why). First you create the one from 1 to the number of rows and then substract the numbers of the rows you want to read.
skiplist = set(range(1, row_count+1)) - set(rownumberList)
Finally you can read the csv as normal.
df = pd.read_csv('myfile.csv',skiprows = skiplist)
here is the full code:
import pandas as pd
with open('myfile.csv') as f:
row_count = sum(1 for row in f)
rownumberList = [1,2,5,6,8,9,20,22]
skiplist = set(range(1, row_count+1)) - set(rownumberList)
df = pd.read_csv('myfile.csv', skiprows=skiplist)
you could try this
import pandas as pd
#making data frame from a csv file
data = pd.read_csv("your_csv_flie.csv", index_col ="What_you_want")
# retrieving multiple rows by iloc method
rows = data.iloc [[1,2,5,6,8,9,20,22]]
You will not be able to circumvent the read time when accessing a large file. If you have a very large CSV file, any program will need to read through it at least up to the point where you want to begin extracting rows. Really, that is what databases are designed for.
However, if you want to extract rows 300,000 to 300,123 from a 10,000,000 row CSV file, you are better off reading just the data you need into Python before converting it to a data frame in Pandas. For this you can use the csv module.
import csv
import pandas
start = 300000
stop = start + 123
data = []
with open('/very/large.csv', 'r') as fp:
reader = csv.reader(fp)
for i, line in enumerate(reader):
if i >= start:
data.append(line)
if i > stop:
break
df = pd.DataFrame(data)
for i in range (1,20)
the first parameter is the first row and the last parameter is the last row...
I'm trying to append multiple columns of a csv to multiple lists. Column 1 will go in list 1, column 2 will go in list 2 etc...
However I want to be able to not hard code in the number of columns so it could work with multiple csv files. So I've used a column count to assign how many lists there should be.
I'm coming unstuck when trying to append values to these lists though. I've initiated a count that should be able to assign the right column to the right list however it seems like the loop just exits after the first loop and wont append the other columns to the list.
import csv
#open csv
f = open('attendees1.csv')
csv_f = csv.reader(f)
#count columns
first_row = next(csv_f)
num_cols = len(first_row)
#create multiple lists (within lists) based on column count
d = [[] for x in xrange(num_cols)]
#initiate count
count = 0
#im trying to state that whilst the count is less than the amount of columns, rows should be appended to lists, which list and which column will be defined by the [count] value.
while count < (num_cols):
for row in csv_f:
d[count].append(row[count])
count += 1
print count
print d
The iteration for row in csv_f: does not reset after each instance of the while loop, thus this loop exits immediately after the first time through.
You can read in everything as a list of rows, then transpose it to create a list of columns:
import csv
with open('attendees1.csv', 'r') as f:
csv_f = csv.reader(f)
first_row = next(csv_f) # Throw away the first row
d = [row for row in csv_f]
d = zip(*d)
See Transpose a matrix in Python.
If you want to keep re-reading the CSV file in the same manner as the OP, you can do that as well (but this is extremely inefficient):
while count < (num_cols):
for row in csv_f:
d[count].append(row[count])
count += 1
print count
f.seek(0) # rewind to the beginning of the file
next(csv_f) # throw away the first line again
See Python csv.reader: How do I return to the top of the file?.
Transposing the list of rows is a very elegant answer. There is another solution, not so elegant, but a little more transparent for a beginner.
Read rows, and append each element to the corresponding list, like so:
for row in csv_f:
for i in range(len(d)):
d[i].append(row[i])
I'm "pseudo" creating a .bib file by reading a csv file and then following this structure writing down every thing including newline characters. It's a tedious process but it's a raw form on converting csv to .bib in python.
I'm using Pandas to read csv and write row by row, (and since it has special characters I'm using latin1 encoder) but I'm getting a huge problem: it only reads the first row. From the official documentation I'm using their method on reading row by row, which only gives me the first row (example 1):
row = next(df.iterrows())[1]
But if I remove the next() and [1] it gives me the content of every column concentrated in one field (example 2).
Why is this happenning? Why using the method in the docs does not iterate through all rows nicely? How would be the solution for example 1 but for all rows?
My code:
import csv
import pandas
import bibtexparser
import codecs
colnames = ['AUTORES', 'TITULO', 'OUTROS', 'DATA','NOMEREVISTA','LOCAL','VOL','NUM','PAG','PAG2','ISBN','ISSN','ISSN2','ERC','IF','DOI','CODEN','WOS','SCOPUS','URL','CODIGO BIBLIOGRAFICO','INDEXAÇÕES',
'EXTRAINFO','TESTE']
data = pandas.read_csv('test1.csv', names=colnames, delimiter =r";", encoding='latin1')#, nrows=1
df = pandas.DataFrame(data=data)
with codecs.open('test1.txt', 'w', encoding='latin1') as fh:
fh.write('#Book{Arp, ')
fh.write('\n')
rl = data.iterrows()
for i in rl:
ix = str(i)
fh.write(' Title = {')
fh.write(ix)
fh.write('}')
fh.write('\n')
PS: I'm new to python and programming, I know this code has flaws and it's not the most effective way to convert csv to bib.
The example row = next(df.iterrows())[1] intentionally only returns the first row.
df.iterrows() returns a generator over tuples describing the rows. The tuple's first entry contains the row index and the second entry is a pandas series with your data of the row.
Hence, next(df.iterrows()) returns the next entry of the generator. If next has not been called before, this is the very first tuple.
Accordingly, next(df.iterrows())[1] returns the first row (i.e. the second tuple entry) as a pandas series.
What you are looking for is probably something like this:
for row_index, row in df.iterrows():
convert_to_bib(row)
Secondly, all your writing to your file handle fh must happen within the block with codecs.open('test1.txt', 'w', encoding='latin1') as fh:
because at the end of the block the file handle will be closed.
For example:
with codecs.open('test1.txt', 'w', encoding='latin1') as fh:
# iterate through all rows
for row_index, row in df.iterrows():
# iterate through all elements in the row
for colname in df.columns:
row_element = row[colname]
fh.write('%s = {%s},\n' % (colname, str(row_element)))
Still I am not sure if the names of the columns exactly match the bibtex fields you have in mind. Probably you have to convert these first. But I hope you get the principle behind the iterations :-)