I have a program which will run on a cron job and write the output to a CSV file.
I get it to write correctly but I would like to make the program write headers on the first row when the file is created. Is there a way of the program checking if there are any rows in the CSV file and, if not, writing the headers.
Just use a flag:
headers_written = False
then when writing rows:
if not headers_written:
writer.writerow(headers)
headers_written = True
writer.writerow(somerow)
You'd actually postpone creating the writer until you are sure you have stuff to write:
writer = None
# ...
if not writer:
writer = csv.writer(open(filename, 'wb'))
writer.writerow(headers)
writer.writerow(somerow)
Related
I am trying to create an IOT weather station where in a Python file it would receive data from a temperature sensor and save it in a CSV along with the second(s) it was received.
Since this is going to be plotted in a graph in real time, I can't have it save new data without clearing the CSV file out first since if I did, the line in the graph will overlap, because of the eventual repetition of seconds in the CSV file.
This is a sample of what is saved in the CSV file:
Celcius,Time
56,20.50
57,20.50
58,20.50
59,20.50
00,20.50
I want the code to clear the CSV file once the seconds reach 60 (or 00), so it can be repopulated with new data.
This is what I have come up with so far:
with open('D:\\WORKSTUFF\\Coding\\ADET\\txt files\\testing.csv', 'a') as csv_file:
csv_writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
info = {
"Time": secondsNow[2],
"Celcius": secondsNow[2]
}
if (secondsNow[2] == "00"):
print("CLEAR CLEAR CLEAR")
csv_file.close()
else:
csv_writer.writerow(info)
You're opening the CSV file in append mode during the minute to add data to the end, what you could then do is close and re-open it in write mode at the end of the minute to start with a fresh file:
while True:
info = {
"Time": secondsNow[2],
"Celcius": secondsNow[2]
}
if secondsNow[2] == '00':
# open in write mode to start again with a blank file
with open('D:\\WORKSTUFF\\Coding\\ADET\\txt files\\testing.csv', 'w') as csv_file:
csv_writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
csv_writer.writerow(info)
else:
# open in append mode to add data to the end
with open('D:\\WORKSTUFF\\Coding\\ADET\\txt files\\testing.csv', 'a') as csv_file:
csv_writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
csv_writer.writerow(info)
When you open a csv file in python and write something into, it automatically overwrites what was already written in the file.
You can clear the file by passing the shell command via python os module:
import os
...
print(CLEAR CLEAR CLEAR)
exit_code = os.system(": > $FILEPATH")
...
I'am having trouble with python csv module I'am trying to write a newline in a csv file is there any reson why it would not work?
Code:
csv writing function
def write_response_csv(name,games,mins):
with open("sport_team.csv",'w',newline='',encoding='utf-8') as csv_file:
fieldnames=['Vardas','Žaidimai','Minutės']
writer = csv.DictWriter(csv_file,fieldnames=fieldnames)
writer.writeheader()
writer.writerow({'Vardas':name,'Žaidimai':games,"Minutės":mins})
with requests.get(url,headers=headers) as page:
content = soup(page.content,'html.parser')
content = content.findAll('table',class_='table01 tablesorter')
names = find_name(content)
times = 0
for name in names:
matches = find_matches(content,times)
min_in_matches = find_min(content,times)
times +=1
csv_file = write_response_csv(name,matches,min_in_matches)
try:
print(name,matches,min_in_matches)
except:
pass
When you call your write_response_csv function it is reopening the file and starting at line 1 again in the csv file and each new line of data you are passing to that function is overwriting the previous one written. What you could do try is creating the csv file outside of the scope of your writer function and setting your writer function to append mode instead of write mode. This will ensure that it will write the data on the next empty csv line, instead of starting at line 1.
#Outside of function scope
fieldnames=['Vardas','Žaidimai','Minutės']
#Create sport_team.csv file w/ headers
with open('sport_team.csv', 'w',encoding='utf-8') as csv_file:
writer = csv.DictWriter(csv_file, fieldnames)
writer.writeheader()
#Write response function
def write_response_csv(name,games,mins):
with open('sport_team.csv','a',encoding='utf-8') as csv_file:
writer = csv.DictWriter(csv_file, fieldnames)
writer.writerow({'Vardas':name,'Žaidimai':games,"Minutės":mins})
Note:
You will run into the same issue if you are reusing this script to continuously add new lines of data to the same file because each time you run it the code that creates the csv file will essentially recreate a blank sport_team.csv file with the headers. If you would like to reuse the code to continuously add new data, I would look into using os.path and utilizing it to confirm if sport_team.csv exists already and if so, to not run that code after the fieldnames.
Try using metabob, it find code errors for you. I've been using it as a Python beginner, and has been pretty successful with it.
I want to create a csv file and write data to it dynamically my script have to keep running 24/7 and csv files have to be created and written every 24 hours, right now all files are created when the program ends.
with open(file_name, 'r+') as f:
myDataList = f.readlines()
nameList = []
for line in myDataList:
entry = line.split(',')
nameList.append(entry[0])
if name not in nameList:
now = datetime.datetime.now()
dtString = now.strftime('%H:%M:%S')
writer = csv.writer(f)
writer.writerow(name, dtString)
Thanks in advance
Remove the file context. Use the earlier way of writing file. And keep doing flush() and fsync() on the file like shown below. That ensures that data is written to the file on disk.
f = open(FILENAME, MODE)
f.write(data)
f.write(data)
f.flush() #important part
os.fsync(f) # important part
For more info: see this link
I have 200 files with dates in the file name. I would like to add date from this file name into new column in each file.
I created macro in Python:
import pandas as pd
import os
import openpyxl
import csv
os.chdir(r'\\\\\\\')
for file_name in os.listdir(r'\\\\\\'):
with open(file_name,'r') as csvinput:
reader = csv.reader(csvinput)
all = []
row = next(reader)
row.append('FileName')
all.append(row)
for row in reader:
row.append(file_name)
all.append(row)
with open(file_name, 'w') as csvoutput:
writer = csv.writer(csvoutput, lineterminator='\n')
writer.writerows(all)
if file_name.endswith('.csv'):
workbook = openpyxl.load_workbook(file_name)
workbook.save(file_name)
csv_filename = pd.read_csv(r'\\\\\\')
csv_data= pd.read_csv(csv_filename, header = 0)
csv_data['filename'] = csv_filename`
Right now I see "InvalidFileException: File is not a zip file" and only first file has added column with the file name.
Can you please advise what am I doing wrong? BTW I,m using Python 3.4.
Many thanks,
Lukasz
First problem, this section:
with open(file_name, 'w') as csvoutput:
writer = csv.writer(csvoutput, lineterminator='\n')
writer.writerows(all)
should be indented, to be included in the for loop. Now it is only executed once after the loop. This is why you only get one output file.
Second problem, the exception is probably caused by openpyxl.load_workbook(file_name). Presumably openpyxl can only open actual Excel files (which are .zip files with other extension), no CSV files. Why do you want to open and save it after all? I think you can just remove those three lines.
I am using Blair's Python script which modifies a CSV file to add the filename as the last column (script appended below). However, instead of adding the file name alone, I also get the Path and File name in the last column.
I run the below script in windows 7 cmd with the following command:
python C:\data\set1\subseta\add_filename.py C:\data\set1\subseta\20100815.csv
The resulting ID field is populated by the following C:\data\set1\subseta\20100815.csv, although, all I need is 20100815.csv.
I'm new to python so any suggestion is appreciated!
import csv
import sys
def process_file(filename):
# Read the contents of the file into a list of lines.
f = open(filename, 'r')
contents = f.readlines()
f.close()
# Use a CSV reader to parse the contents.
reader = csv.reader(contents)
# Open the output and create a CSV writer for it.
f = open(filename, 'wb')
writer = csv.writer(f)
# Process the header.
header = reader.next()
header.append('ID')
writer.writerow(header)
# Process each row of the body.
for row in reader:
row.append(filename)
writer.writerow(row)
# Close the file and we're done.
f.close()
# Run the function on all command-line arguments. Note that this does no
# checking for things such as file existence or permissions.
map(process_file, sys.argv[1:])
Use os.path.basename(filename). See http://docs.python.org/library/os.path.html for more details.