i have written a python program which makes an api call to a webserver once every minute and then parse the json response and saves parsed values in to the csv files.
here is the code that is saving the values into the csv file :
with open('data.csv', 'a', newline='') as file:
writer = csv.writer(file)
writer.writerow([current_time,SHORTPERC, LONGPERC, SHORTvolume, longVolume, longPositions, shortPositions])
how can i make it so that it saves the header only once on the top most row and not on every row ?
UPDATE:
here is a bit of more code to make api call and write the data to file :
from apscheduler.schedulers.blocking import BlockingScheduler
from apscheduler.triggers.cron import CronTrigger
import requests
import json
import csv
from datetime import datetime
def fn():
print("Starting...")
session_id = "auZsJ4F2RsQNJxSPTMDt2238324"
Outlook='http://www.myfxbook.com/api/get-community-outlook.json?session=' + session_id
Outlook_response = requests.get(Outlook)
Outlook_data = Outlook_response.json()['symbols']
now = datetime.now()
current_time = now.strftime("%H:%M")
EURUSD=Outlook_data[0]
SHORTPERC=EURUSD['shortPercentage']
LONGPERC =EURUSD['longPercentage']
SHORTvolume=EURUSD['shortVolume']
longVolume=EURUSD['longVolume']
longPositions=EURUSD['longPositions']
shortPositions=EURUSD['shortPositions']
with open('data.csv', 'a', newline='') as file:
writer = csv.writer(file)
writer.writerow([current_time,SHORTPERC, LONGPERC, SHORTvolume, longVolume, longPositions, shortPositions])
with open('data1.csv', 'a', newline='') as file:
writer = csv.writer(file)
writer.writerow([SHORTvolume, longVolume])
with open('data2.csv', 'a', newline='') as file:
writer = csv.writer(file)
writer.writerow([SHORTPERC, LONGPERC])
i cant post the full code cuz it will be very ugly since its around 700 lines long , but the above mentioned code should work to create the csv file
this is how one of my csv files look :
07:11,31,69,555.55,1265.14,4750,2607
07:12,31,69,555.55,1265.16,4751,2607
07:13,31,69,555.55,1265.16,4751,2607
07:14,30,70,555.56,1267.36,4752,2608
07:15,30,70,555.56,1267.36,4752,2608
07:16,30,70,555.56,1267.36,4752,2608
07:17,30,70,555.46,1267.36,4752,2607
07:18,31,69,558.61,1267.36,4752,2610
07:19,31,69,558.61,1267.37,4753,2610
07:20,31,69,561.58,1267.37,4753,2611
07:21,31,69,561.61,1267.37,4753,2613
07:22,31,69,561.65,1267.37,4753,2614
07:23,31,69,561.65,1267.36,4752,2614
this is just part of the csv file , more rows keep adding as time passes
EDIT 2:
answer suggested by Sparkofska seems to work but somehow it ends up giving an empty row in between every line like this:
Time,ShortPer,LongPer,ShortVolume,LongVolume,ShortPosition,LongPosition
05:47,44,56,19528.8,24789.27,65223,48630
05:48,44,56,19529.04,24789.27,65223,48633
code :
EURUSD=Outlook_data[0]
SHORTPERC=EURUSD['shortPercentage']
LONGPERC =EURUSD['longPercentage']
SHORTvolume=EURUSD['shortVolume']
longVolume=EURUSD['longVolume']
longPositions=EURUSD['longPositions']
shortPositions=EURUSD['shortPositions']
filename='EURUSD.csv';
def write_row_header_aware(filename, row):
if not os.path.exists(filename) or os.stat(filename).st_size == 0:
with open(filename, 'a') as file:
writer = csv.writer(file)
writer.writerow(['Time', 'ShortPer', 'LongPer','ShortVolume','LongVolume','ShortPosition','LongPosition'])
with open(filename, 'a') as file:
writer = csv.writer(file)
writer.writerow([current_time,SHORTPERC, LONGPERC, SHORTvolume, longVolume, longPositions, shortPositions])
write_row_header_aware(filename, [current_time,SHORTPERC, LONGPERC, SHORTvolume, longVolume, longPositions, shortPositions])
print("done...")
You could wrap the writerow function to have it automatically add the header if needed.
If your output csv file is not empty, we can assert the header was already written and simply append the row. Otherwise (file not exist or empty) we write the header before appending the row.
import os
def write_row_header_aware(filename, row):
# in case file doesn't exist or is empty
if not os.path.exists(filename) or os.stat(filename).st_size == 0:
# write header
with open(filename, 'a', newline='') as file:
writer = csv.writer(file)
writer.writerow(['current_time', 'SHORTPERC', 'LONGPERC', ...])
# write line as usual
with open(filename, 'a', newline='') as file:
writer = csv.writer(file)
writer.writerow(row)
write_row_header_aware('data.csv', [current_time, SHORTPERC, LONGPERC, ...])
Please make a check to know if the file exists, if it already exists use append to write rows to file else write the headers. By this way you could avoid writing headers multiple times. Please refer to this link
The following code is intended to remove tags from a csv file. Tags are labelled as follows (the tag here is hot as an example): tag,hot
It iterates through the current csv file, writing every line to a temporary file except the tag that needs to be removed, then renames the temporary file on top of the original. Currently, it adds whitespace between every line in the original, eg:
tag,tag1
tag,tag2
tag,tag3
tag,tag4
tag,tag5
goes to the following if tag3 was passed as the tag to be removed:
tag,tag1
tag,tag2
tag,tag4
tag,tag5
As can be seen an extra line is where tag3 would have been, as well as there being blank lines between every other line. I have seen other threads that suggested adding newline='' to when the files are opened, but I have done this and it still does not solve the problem (although there was actually more whitespace before, with 2 lines between every value rather than 1). Not sure why this hasn't fixed it.
Code:
tempfile = NamedTemporaryFile(mode="w", delete=False, newline='')
CSVpart = imageNumber + ".csv"
lowercase = tag.lower()
filePath = os.path.join(os.curdir, "static", "csv", CSVpart)
if os.path.isfile(filePath):
with open(filePath, 'r', newline='') as csvfile, tempfile:
reader = csv.reader(csvfile, delimiter=',')
writer = csv.writer(tempfile)
for row in reader:
if row != ['tag', lowercase]:
writer.writerow(row)
else:
removed = True //ignore, this makes sense in context
shutil.move(tempfile.name, filePath)
I created the following, which works. Cannot figure out what is different about it:
import shutil, csv, os
from tempfile import NamedTemporaryFile
tempfile = NamedTemporaryFile(mode="w", delete=False, newline='')
tagToRemove = "hot"
filePath = os.path.join(os.curdir, "test.csv")
with open(filePath, 'r', newline='') as csvfile, tempfile:
reader = csv.reader(csvfile, delimiter=',')
writer = csv.writer(tempfile)
for row in reader:
if row != ['tag', tagToRemove]:
writer.writerow(row)
shutil.move(tempfile.name, filePath)
I have many CSV files, need to read all the files in loop and write file name and all the columns (header in row 1) in an output file.
Example
Input csv file 1 (test1.csv)
Id, Name, Age, Location
1, A, 25, India
Input csv file 2 (test2.csv)
Id, ProductName
1, ABC
Outputfile
test1.csv Id
test1.csv Name
test1.csv Age
test1.csv Location
test2.csv Id
test2.csv ProductName
Many thanks for your help.
Update:
This code works fine for this purpose:
import os
import csv
ofile = open('D:\Anuj\Personal\OutputFile/AHS_File_Columns_Info.csv', 'w')
directory = os.path.join('D:\Anuj\Personal\Python')
for root, dirs, files in os.walk(directory):
for file in files:
fullfilepath = directory + "/" + file
with open(fullfilepath,'r') as f:
output = file +','+ f.readline()
ofile.write(output)
clean solution using csv module for reading and writing
open output file and create a csv.writer instance on its handle
open each input file and create a csv.reader instance on their handle
get first row using next on the csv.reader iterator: gets titles as list (with a small post-processing to remove the spaces)
write titles alongside the current filename in a loop
code:
import csv
files=["test1.csv","test2.csv"]
with open("output.tsv","w",newline='') as fw:
cw = csv.writer(fw,delimiter="\t") # output is tab delimited
for filename in files:
with open(filename,'r') as f:
cr = csv.reader(f)
# get title
for column_name in (x.strip() for x in next(cr)):
cw.writerow([filename,column_name])
There are several advantages using csv module, the most important being that quoting & multi-line fields/titles are managed properly.
But I'm not sure I understand you correctly.
import csv
from typing import List
from typing import Tuple
TableType = List[List[str]]
def load_csv_table(file_name: str) -> Tuple[List[str], TableType]:
with open(file_name) as csv_file:
csv_reader = csv.reader(csv_file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)
headers = next(csv_reader)
data_table = list(csv_reader)
return headers, data_table
def save_csv_table(file_name: str, headers: List[str], data_table: TableType):
with open(file_name, 'w', newline='') as csv_file:
writer = csv.writer(csv_file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)
writer.writerow(headers)
for row in data_table:
writer.writerow(row)
input_files = ['file1.csv', 'file2.csv', 'file3.csv']
new_table = []
new_headers = []
for file_name in input_files:
headers, data_table = load_csv_table(file_name)
if not new_headers:
new_headers = ['Source'] + headers
new_table.extend(([file_name] + line for line in data_table))
save_csv_table('output.csv', new_headers, new_table)
A simple method is to use readline() on the file object:
files=["test1.csv","test2.csv"]
for my_file in files:
with open(my_file,'r') as f:
print my_file, f.readline()
I have data in a file and I need to write it to CSV file in specific column. The data in file is like this:
002100
002077
002147
My code is this:
import csv
f = open ("file.txt","r")
with open("watout.csv", "w") as output:
for line in f :
c.writerows(line)
It is always writes on the first column. How could I resolve this?
Thanks.
This is how I solved the problem
f1 = open ("inFile","r") # open input file for reading
with open('out.csv', 'w',newline="") as f:up # output csv file
writer = csv.writer(f)
with open('in.csv','r') as csvfile: # input csv file
reader = csv.reader(csvfile, delimiter=',')
for row in reader:
row[7] = f1.readline() # edit the 8th column
writer.writerow(row)
f1.close()
python 2 users replace
with open('out.csv', 'w',newline="") as f:
by
with open('out.csv', 'wb') as f:
EDIT: Thanks for the answers guys, got what I needed!!
Basically I am trying to take what I have stored in my textfile and I am trying to write that into a .csv file. In my file are tweets that I have stored and I am trying to have one tweet in each cell in my .csv file.
Right now it is only taking one tweet and creating a .csv file with it and I need it to take all of them. Any help is greatly appreciated. Here is what I have so far.
with open('reddit.txt', 'rb') as f:
reader = csv.reader(f, delimiter=':', quoting = csv.QUOTE_NONE)
for row in reader:
print row
cr = csv.writer(open('reddit.csv', 'wb'))
cr.writerow(row)
You'll need to create the writer outside of the loop:
with open('reddit.txt', 'rb') as input_file:
reader = csv.reader(input_file, delimiter=':', quoting = csv.QUOTE_NONE)
with open('reddit.csv', 'wb') as output_file:
writer = csv.writer(output_file)
for row in reader:
writer.writerow(row)
Although here it might be cleaner to open the files without with:
input_file = open('reddit.txt', 'rb')
output_file = open('reddit.csv', 'wb')
reader = csv.reader(input_file, delimiter=':', quoting=csv.QUOTE_NONE)
writer = csv.writer(output_file)
for row in reader:
writer.writerow(row)
input_file.close()
output_file.close()
Or you can still use with and just have a really long line:
with open('reddit.txt', 'rb') as input_file, open('reddit.csv', 'wb') as output_file:
reader = csv.reader(input_file, delimiter=':', quoting = csv.QUOTE_NONE)
writer = csv.writer(output_file)
for row in reader:
writer.writerow(row)
The line cr = csv.writer(open('reddit.csv', 'wb')) is inside the for loop. You need to open the file just once, place this line after
reader = csv.reader(f, delimiter=':', quoting = csv.QUOTE_NONE)
Then write to it as you did in each loop iteration.