Develop Reference

Combine csv files with identical columns and unescape html code

Dear all i often need to concat csv files with identical headers (i.e. but them into one big file). Usually i just use pandas, but i now need to operate in an enviroment were i am not at liberty to install any library. The csv and the html libs do exsit.
I also need to remove all remaining html tags like %amp; for the apercent symbol which are still within the data. I do know in which columns these can come up.
I thought aboug doing it like this, and the concat part of my code seems to work fine:
import CSV
import html
for file in files: # files is a list of csv files.
with open(file, "rt", encoding="utf-8-sig") as source, open(outfilePath, "at", newline='',
encoding='utf-8-sig') as result:
d_reader = csv.DictReader(source,delimiter=";")
# Set header based on first file in file_list:
if file == test_files[0]:
Common_header = d_reader.fieldnames
# Define DcitwriterObject
wtr = csv.DictWriter(result, fieldnames=Common_header, lineterminator='\n', delimiter=";")
# Write Header only once to emtpy file
if result.tell() == 0:
wtr.writeheader()
# If i remove this block i get my concateneated singe csv file as a result
# Howerver the html tags/encoded symbols are sill present.
for row in d_reader:
print(html.unescape(row['ColA'])) # This prints the unescaped Values in the column correctly
# If i kepp these two lines, i get an empty file with just the header as a result of the concatenation
row['ColA'] = html.unescape(row['ColA'])
row['ColB'] = html.unescape(row['ColB'])
wtr.writerows(d_reader)
I would have thought the simply suppling the encoding='utf-8-sig' part to the result file would be sufficient to get rid of the html symbols but that does not work. If you could give me a hint what i am doint wrong in the usage of the code containing the html.unescape function in my code that would be nice.
Thank you in advance

Python: How to create a new dataframe with first row when a specific value

I am reading csv files into python using:
df = pd.read_csv(r"C:\csvfile.csv")
But the file has some summary data, and the raw data start if a value "valx" is found. If "valx" is not found then the file is useless. I would like to create news dataframes that start when "valx" is found. I have been trying for a while with no success. Any help on how to achieve this is greatly appreciated.

Unfortunately, pandas only accepts skiprows for rows to skip in the beginning. You might want to parse the file before creating the dataframe.
As an example:
import csv
with open(r"C:\csvfile.csv","r") as f:
lines = csv.reader(f, newline = '')
if any('valx' in i for i in lines):
data = lines
Using the Standard Libary csv module, you can read file and check if valx is in the file, if it is found, the content will be returned in the data variable.
From there you can use the data variable to create your dataframe.

How write and append data in csv file and store it in list

I made csv file in my python code itself and going to append next data in ti it but the error is comming
io.UnsupportedOperation: not readable
I tried code is:
df.to_csv('timepass.csv', index=False)
with open(r'timepass.csv', 'a') as f:
writer = csv.reader(f)
your_list = list(writer)
print(your_list)
want to append next data and store in the same csv file. so that csv file having both previous and current data.
so please help me to find out..
Thanks in advance...

It is so simple just try this:
import pandas as pd
df = pd.read_excel("NSTT.xlsx","Sheet1") #reading Excel
print(df) #Printing data frame
df.to_excel("new.xlsx") #Writing Dataframe into New Excel file
Now here if you want to append data in the same file then use
df.to_excel("new.xlsx","a")
And no need to add in a list as you can directly access the data same as a list with data frame only you have to define the location .

Please check this.
You can use pandas in python to read csv and write csv:
import pandas as pd
df = pd.read_csv("csv file")
print(df)

Try:
with open(r'timepass.csv', 'r') as f:
reader = list(csv.reader(f))
print(reader)
Here you are opening your file as r, which means read-only and assigning the list contents to reader with list(csv.reader(f)). Your earlier code a opens the file for appending only where in the documentation is described as:
'a' opens the file for appending; any data written to the file is
automatically added to the end
and does not support the read().
And if you want to append data to the csv file from a different list, use the with open as a with the writer method.
with open('lake.csv','a') as f:
csv.writer(f,[1,2,3]) #dummy list [1,2,3]
Or directly from the pandas.DataFrame.to_csv method from your new dataframe, with header = False so as not to append headers:
df.to_csv('timepass.csv', index=False)
df_new.to_csv(r'timepass.csv', mode='a', header=False) #once you have updated your dataframe, you can directly append it to the same csv file

you can use pandas for appending two csv quickly
import pandas as pd
dataframe1=pd.read_csv("a.csv")
dataframe2=pd.read_csv("b.csv")
dataframe1=dataframe1.append(dataframe2)
dataframe1=dataframe1.reset_index(drop=True)
dataframe1.to_csv("a.csv")

Python write multiple numpy array to csv file

I am trying to add features of multiple images by converting them from raw data format to .csv.
I have read and displayed features of two images via print function but during addition of contents to csv, i am only able to add single numpy array. I want to add few thousand images in same csv.
Below is printed output, but csv only shows one array (having features of single image).
Image showing code and output

I have done this by using following code:
with open("output.csv", "a") as f:
writer = csv.writer(f)
writer.writerows(data_read[2])

How to export data (which is as result of Python program) from command line?

I am working on a Python program, and I have results on the command line.
Now I need to do analysis on the results, so I need all results as exported in any format like either SQL, or Excel or CSV format.
Can some tell me how can i do that ?
import csv
x1=1 x2=2
while True:
show = [ dict(x1=x1+1 , x2=x2+2)]
print('Received', show )
with open('large1.csv','w') as f1:
writer=csv.writer(f1, delimiter=' ',lineterminator='\n\n',)
writer.writerow(show)
x1=x1+1
x2=x2+1
Here this is infinite loop and I want to have a csv file containing 2 column of x1 and x2. and with regularly updated all values of x1 and x2 row wise (1 row for 1 iteration)
But by this code I'm getting a csv file which is named as 'large1.csv' and containing only one row (last updated values of x1 and x2).
So how can I get my all values of x1 and x2 as row was in python.

Just use the csv format it can easily imported into Excel and
the python standard library supports csv out of the box. #See python-csv

One way to handle this is to use the CSV module, and specifically a Writer object to write the output to a CSV file (perhaps even instead of writing to stdout). The documentation has several examples, including this one:
import csv
with open('some.csv', 'w', newline='') as f:
writer = csv.writer(f)
writer.writerows(someiterable)
You should then be able to import the CSV file easily in Excel if that is what you want.

Have a look at the open() documentation.
Mode w will truncate the file, which means it will replace the contents. Since you call that every loop iteration, you are continuously deleting the file and replacing it with a new one. Mode a appends to the file and is maybe what you want. You also might consider opening the file outside of the loop, in that case w might be the correct mode.