Develop Reference

Skip file if value is not in data using python

With my current code, I am trying to skip a csv file if it does not contain a value within the actual data that I am looking for.
basically if it has "PROD_NAME" as a column, then it looks for that string and replaces it with the second string in that statement, but the first file in my folder does not have this column name and so the script fails. I've looked into ways to skip but have only seen ways to skip based on the filename itself and not the data within a file not having the correct information. Any help would be appreciated. Thanks!
def worker(files):
filenames = glob.glob(dest_dir + '\\*.csv')
for filename in filenames:
my_file = Path(os.path.join(dest_dir, filename))
#read header
with open(filename) as f:
read_data = f.read()
header = read_data[:read_data.find('!1')]
idx = header.find('\n')
# read data
df1 = pd.read_csv(filename, skiprows=1, encoding='ISO-8859-1', nrows=1) # read column header only - to get the list of columns
dtypes = {}
for col in df1.columns:# make all columns text, to avoid formatting errors
dtypes[col] = 'str'
df1 = pd.read_csv(filename, dtype=dtypes, skiprows=1, encoding='ISO-8859-1', quotechar="'", delimiter='\t')
df1.loc[df1['PROD_NAME'].str.contains('NA_NRF'), 'PROD_NAME'] = 'FA_GUAR'
file_count += 1 # count the fil
worker(files)

Could you just add an if statement before your transformation
if 'PROD_NAME' in df1.columns:
df1.loc[df1['PROD_NAME'].str.contains('NA_NRF'), 'PROD_NAME'] = 'FA_GUAR'
file_count += 1 # count the fil

How to write the filename and rowcount in a csv in python?

I've been trying to make a CSV from a big list of another CSVs and here's the deal: I want to get the names of these CSV files and put them in the CSV that I want to create, plus, I also need the row count from the CSV files that I'm getting the names of, here's what I've tried so far:
def getRegisters(file):
results = pd.read_csv(file, header = None, error_bad_lines= False, sep = '\t', low_memory = False)
print(len(results))
return len(results)
path = "C:/Users/gdldieca/Desktop/TESTSFORPANW/New folder"
dirs = os.listdir(path)
with open("C:/Users/gdldieca/Desktop/TESTSFORPANW/New folder/FilesNames.csv", 'w', newline='') as f:
writer = csv.writer(f, delimiter = '\t')
writer.writerow(("File", "Rows"))
for names in dirs:
sfile = getRegisters("C:/Users/gdldieca/Desktop/TESTSFORPANW/New folder/" + str(names))
writer.writerow((names, sfile))
However I can't seem to get the files row count even tho Pandas actually returns it. I'm getting this error:
_csv.Error: iterable expected, not int
The final result would be something like this written into the CSV
File1 90
File2 10

If you are using pandas , I think you can use also for make a csv file with all values that you need..Here an alternative
import os
import pandas as pd
directory='D:\\MY\\PATH\\ALLCSVFILE\\'
#create a list for add all
rows_list = []
for filename in os.listdir(directory):
if filename.endswith(".csv"):
file=os.path.join(directory, filename)
df=pd.read_csv(file)
#Count rows
rowcount=len(df.index)
new_row = {'namefile':filename, 'count':rowcount}
rows_list.append(new_row)
#pass list to dataframe
df1 = pd.DataFrame(rows_list)
print(df1)
df1.to_csv('test.csv', sep=',')
result :

Pandas Dataframe append one row at a time into CSV

Im trying to send a pandas dataframe into a csv file
import pandas as pd
import os
case_array = [['2017042724', '05/18/2017'], ['2017042723', '05/18/2017'], ['2017042722', '05/18/2017'], ['2017042721', '05/18/2017']]
filename = 'case_array.csv'
path = "C:\shares\folder"
fullpath = os.path.join(path, filename)
for case_row in case_array:
df = pd.DataFrame(case_row)
try:
with open(fullpath, 'w+') as f:
df.to_csv(f, header=False)
print('Success')
except:
print('Unable to Write CSV')
try:
df = pd.read_csv(fullpath)
print(df)
except:
print('Unable to Read CSV')
but its inserting each row as a column, inserting a header column (was set to False) and overwriting the previous insertion:
0 2017042721
1 05/18/2017
If I insert the entire array it will insert rows without the header row. (This is the correct result I want) The issue is the script I writing I need to insert each row at a time.
How do I get pandas dataframe to insert a row instead of a column?
edit1
like this:
0 1
2017042721 05/18/2017
2017042723 05/18/2017

You do not have to loop over the array to do it. You can make a dataframe out of the array and have it written to a csv using to_csv().
case_array = [['2017042724', '05/18/2017'], ['2017042723', '05/18/2017'], ['2017042722', '05/18/2017'], ['2017042721', '05/18/2017']]
df=pd.DataFrame(case_array)
df.to_csv(fullpath, header=False)
EDIT
If you must iterate over the array you below code:
for case_row in case_array:
df = pd.DataFrame(case_row).T
try:
with open(fullpath, 'a') as f:
df.to_csv(f, header=False, index=False)
print('Success')
except:
print('Unable to Write CSV')

Writing single CSV header with pandas

I'm parsing data into lists and using pandas to frame and write to an CSV file. First my data is taken into a set where inv, name, and date are all lists with numerous entries. Then I use concat to concatenate each iteration through the datasets I parse through to a CSV file like so:
counter = True
data = {'Invention': inv, 'Inventor': name, 'Date': date}
if counter is True:
df = pd.DataFrame(data)
df = df[['Invetion', 'Inventor', 'Date']]
else:
df = pd.concat([df, pd.DataFrame(data)])
df = df[['Invention', 'Inventor', 'Date']]
with open('./new.csv', 'a', encoding = utf-8) as f:
if counter is True:
df.to_csv(f, index = False, header = True)
else:
df.to_csv(f, index = False, header = False)
counter = False
The counter = True statement resides outside of my iteration loop for all the data I'm parsing so it's not overwriting every time.
So this means it only runs once through my data to grab the first df set then concats it thereafter. The problem is that even though counter is only True the first round and works for my first if-statement for df it does not work for my writing to file.
What happens is that the header is written over and over again - regardless to the fact that counter is only True once. When I swap the header = False for when counter is True then it never writes the header.
I think this is because of the concatenation of df holding onto the header somehow but other than that I cannot figure out the logic error.
Is there perhaps another way I could also write a header once and only once to the same CSV file?

It's hard to tell what might be going wrong without seeing the rest of the code. I've developed some test data and logic that works; you can adapt it to fit your needs.
Please try this:
import pandas as pd
early_inventions = ['wheel', 'fire', 'bronze']
later_inventions = ['automobile', 'computer', 'rocket']
early_names = ['a', 'b', 'c']
later_names = ['z', 'y', 'x']
early_dates = ['2000-01-01', '2001-10-01', '2002-03-10']
later_dates = ['2010-01-28', '2011-10-10', '2012-12-31']
early_data = {'Invention': early_inventions,
'Inventor': early_names,
'Date': early_dates}
later_data = {'Invention': later_inventions,
'Inventor': later_names,
'Date': later_dates}
datasets = [early_data, later_data]
columns = ['Invention', 'Inventor', 'Date']
header = True
for dataset in datasets:
df = pd.DataFrame(dataset)
df = df[columns]
mode = 'w' if header else 'a'
df.to_csv('./new.csv', encoding='utf-8', mode=mode, header=header, index=False)
header = False
Alternatively, you can concatenate all of the data in the loop and write out the dataframe at the end:
df = pd.DataFrame(columns=columns)
for dataset in datasets:
df = pd.concat([df, pd.DataFrame(dataset)])
df = df[columns]
df.to_csv('./new.csv', encoding='utf-8', index=False)
If your code cannot be made to conform to this API, you can forego writing the header in to_csv altogether. You can detect whether the output file exists and write the header to it first if it does not:
import os
fn = './new.csv'
if not os.path.exists(fn):
with open(fn, mode='w', encoding='utf-8') as f:
f.write(','.join(columns) + '\n')
# Now append the dataframe without a header
df.to_csv(fn, encoding='utf-8', mode='a', header=False, index=False)

I found the same problem. Pandas dataframe to csv works fine if the dataframe is finished and no need to do anything beyond any tutorial.
However if our program is making results and we are appending them, it seems that we find the repetitive header writing problem
In order to solve this consider the following function:
def write_data_frame_to_csv_2(dict, path, header_list):
df = pd.DataFrame.from_dict(data=dict, orient='index')
filename = os.path.join(path, 'results_with_header.csv')
if os.path.isfile(filename):
mode = 'a'
header = 0
else:
mode = 'w'
header = header_list
with open(filename, mode=mode) as f:
df.to_csv(f, header=header, index_label='model')
If the file does not exist we use write mode and header is equal to header list. When this is false, and the file exists we use append and header changed to 0.
The function receives a simple dictionary as parameter, In my case I used:
model = { 'model_name':{'acc':0.9,
'loss':0.3,
'tp':840,
'tn':450}
}
Using the function form ipython console several times produces expected result:
write_data_frame_to_csv_2(model, './', header_list)
Csv generated:
model,acc,loss,tp,tn
model_name,0.9,0.3,840,450
model_name,0.9,0.3,840,450
model_name,0.9,0.3,840,450
model_name,0.9,0.3,840,450
Let me know if it helps.
Happy coding!

just add this check before setting header property if you are using an index to iterate over API calls to add data in csv file.
if i > 0:
dataset.to_csv('file_name.csv',index=False, mode='a', header=False)
else:
dataset.to_csv('file_name.csv',index=False, mode='a', header=True)

Pandas: Continuously write from function to csv

I have a function set up for Pandas that runs through a large number of rows in input.csv and inputs the results into a Series. It then writes the Series to output.csv.
However, if the process is interrupted (for example by an unexpected event) the program will terminate and all data that would have gone into the csv is lost.
Is there a way to write the data continuously to the csv, regardless of whether the function finishes for all rows?
Prefarably, each time the program starts, a blank output.csv is created, that is appended to while the function is running.
import pandas as pd
df = pd.read_csv("read.csv")
def crawl(a):
#Create x, y
return pd.Series([x, y])
df[["Column X", "Column Y"]] = df["Column A"].apply(crawl)
df.to_csv("write.csv", index=False)

This is a possible solution that will append the data to a new file as it reads the csv in chunks. If the process is interrupted the new file will contain all the information up until the interruption.
import pandas as pd
#csv file to be read in
in_csv = '/path/to/read/file.csv'
#csv to write data to
out_csv = 'path/to/write/file.csv'
#get the number of lines of the csv file to be read
number_lines = sum(1 for row in (open(in_csv)))
#size of chunks of data to write to the csv
chunksize = 10
#start looping through data writing it to a new file for each chunk
for i in range(1,number_lines,chunksize):
df = pd.read_csv(in_csv,
header=None,
nrows = chunksize,#number of rows to read at each loop
skiprows = i)#skip rows that have been read
df.to_csv(out_csv,
index=False,
header=False,
mode='a',#append data to csv file
chunksize=chunksize)#size of data to append for each loop

In the end, this is what I came up with. Thanks for helping out!
import pandas as pd
df1 = pd.read_csv("read.csv")
run = 0
def crawl(a):
global run
run = run + 1
#Create x, y
df2 = pd.DataFrame([[x, y]], columns=["X", "Y"])
if run == 1:
df2.to_csv("output.csv")
if run != 1:
df2.to_csv("output.csv", header=None, mode="a")
df1["Column A"].apply(crawl)

I would suggest this:
with open("write.csv","a") as f:
df.to_csv(f,header=False,index=False)
The argument "a" will append the new df to an existing file and the file gets closed after the with block is finished, so you should keep all of your intermediary results.

I've found a solution to a similar problem by looping the dataframe with iterrows() and saving each row to the csv file, which in your case it could be something like this:
for ix, row in df.iterrows():
row['Column A'] = crawl(row['Column A'])
# if you wish to mantain the header
if ix == 0:
df.iloc[ix - 1: ix].to_csv('output.csv', mode='a', index=False, sep=',', encoding='utf-8')
else:
df.iloc[ix - 1: ix].to_csv('output.csv', mode='a', index=False, sep=',', encoding='utf-8', header=False)