Develop Reference

Python 3 dictwriter appending rows to csv blank line in Windows

I have a CSV file which is created once a day based on a webhook, and then saved to S3. Any future webhooks will open the file and create an entry before closing the file.
def create_new_file(self, row, file_name):
with open(f"/tmp/{file_name}", 'w', encoding='UTF8', newline='') as f:
writer = csv.DictWriter(f, fieldnames=self.fieldnames)
writer.writeheader()
writer.writerow(row)
self.s3.upload_file(f"/tmp/{file_name}", self.bucket_name, file_name)
def append_to_file(self, row, file_name):
self.s3.download_file(self.bucket_name, file_name, f"/tmp/{file_name}")
with open(f"/tmp/{file_name}", 'r+', encoding='UTF8', newline='') as f:
last_number = f.readlines()[-1].split(",")[0]
writer = csv.DictWriter(f, fieldnames=self.fieldnames)
writer.writerow(row)
I'm encountering the common blank line issue with Python3, but only when appending items. There is no blank line between the header and the first entry so I am thinking it's something to do with how I am opening the existing file - at the moment it is in r+ mode. Am I using the wrong mode type?

CSV file to list of lines?

I have a txt file and I want to save each line as a list in a new file with fname as the new file name. But the output is not being saved. What am I missing?
import csv
with open('file.txt', 'rU') as csvfile:
reader = csv.reader(csvfile, delimiter='\t')
i = 1
for line in reader:
fname = line[0] + line[1]
#print fname
with open(fname, 'w') as out:
out.write(line)
i +=1

To do what you want, you need to fix two things, one is to open the output files in "append" mode so their previous contents aren't wiped-out everytime something additional is written to them.
Secondly you need some way to know the raw data from the file for each csv row it reads. This can be difficult when you use an extension like the csv module and don't know the internals (which you shouldn't use anyway).
To work around that in this case, you can pass a custom csvfile argument to the csv.reader that will give you the information needed. Basically a small preprocessor of the data being read. Here's what I mean:
import csv
def pre_reader(file):
"""Generator that remembers last line read."""
for line in file:
pre_reader.lastline = line
yield line
with open('test_gen.csv', 'rU') as csvfile:
reader = csv.reader(pre_reader(csvfile), delimiter='\t')
i = 1
for line in reader:
fname = line[0] + line[1]
#print fname
with open(fname, 'a') as out:
out.write(pre_reader.lastline)
i +=1

Change:
with open(fname, 'w') as out:
out.write(line)
To:
with open(fname, 'a') as out:
out.write(line)
w Opens a file for writing only. Overwrites the file if the file exists. If the file does not exist, creates a new file for writing.
a Opens a file for appending. The file pointer is at the end of the file if the file exists. That is, the file is in the append mode. If the file does not exist, it creates a new file for writing.
Better way:
import csv
with open('file.txt', 'rU') as csvfile, open(fname, 'w') as out:
reader = csv.reader(csvfile, delimiter='\t')
i = 1
for line in reader:
fname = line[0] + line[1]
out.write(line)

You cannot write a list so change penultimate line to **out.write(str(line))**
import csv
with open('file.txt', 'rU') as csvfile:
reader = csv.reader(csvfile, delimiter='\t')
i = 1
for line in reader:
fname = line[0] + line[1]
#print fname
with open(fname, 'w') as out:
------> out.write(str(line))
i +=1

Python read CSV file columns and write file name and column name in a csv file

I have many CSV files, need to read all the files in loop and write file name and all the columns (header in row 1) in an output file.
Example
Input csv file 1 (test1.csv)
Id, Name, Age, Location
1, A, 25, India
Input csv file 2 (test2.csv)
Id, ProductName
1, ABC
Outputfile
test1.csv Id
test1.csv Name
test1.csv Age
test1.csv Location
test2.csv Id
test2.csv ProductName
Many thanks for your help.
Update:
This code works fine for this purpose:
import os
import csv
ofile = open('D:\Anuj\Personal\OutputFile/AHS_File_Columns_Info.csv', 'w')
directory = os.path.join('D:\Anuj\Personal\Python')
for root, dirs, files in os.walk(directory):
for file in files:
fullfilepath = directory + "/" + file
with open(fullfilepath,'r') as f:
output = file +','+ f.readline()
ofile.write(output)

clean solution using csv module for reading and writing
open output file and create a csv.writer instance on its handle
open each input file and create a csv.reader instance on their handle
get first row using next on the csv.reader iterator: gets titles as list (with a small post-processing to remove the spaces)
write titles alongside the current filename in a loop
code:
import csv
files=["test1.csv","test2.csv"]
with open("output.tsv","w",newline='') as fw:
cw = csv.writer(fw,delimiter="\t") # output is tab delimited
for filename in files:
with open(filename,'r') as f:
cr = csv.reader(f)
# get title
for column_name in (x.strip() for x in next(cr)):
cw.writerow([filename,column_name])
There are several advantages using csv module, the most important being that quoting & multi-line fields/titles are managed properly.

But I'm not sure I understand you correctly.
import csv
from typing import List
from typing import Tuple
TableType = List[List[str]]
def load_csv_table(file_name: str) -> Tuple[List[str], TableType]:
with open(file_name) as csv_file:
csv_reader = csv.reader(csv_file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)
headers = next(csv_reader)
data_table = list(csv_reader)
return headers, data_table
def save_csv_table(file_name: str, headers: List[str], data_table: TableType):
with open(file_name, 'w', newline='') as csv_file:
writer = csv.writer(csv_file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)
writer.writerow(headers)
for row in data_table:
writer.writerow(row)
input_files = ['file1.csv', 'file2.csv', 'file3.csv']
new_table = []
new_headers = []
for file_name in input_files:
headers, data_table = load_csv_table(file_name)
if not new_headers:
new_headers = ['Source'] + headers
new_table.extend(([file_name] + line for line in data_table))
save_csv_table('output.csv', new_headers, new_table)

A simple method is to use readline() on the file object:
files=["test1.csv","test2.csv"]
for my_file in files:
with open(my_file,'r') as f:
print my_file, f.readline()

JSON like data to CSV file in python - not showing headers correctly

I am transforming JSON like data to CSV and having a few issues.
The code is here:
import json
import csv
def parse_file(inputed_file):
with open(input_file, 'r') as inputed_file:
content = inputed_file.readlines()
split_file = open('test.csv', 'w')
for line in content:
lines = line.split('\t')
data = json.loads(lines[0])
writer = csv.DictWriter(split_file, fieldnames = ["title", "firstname"], delimiter = ',')
writer.writeheader()
The problem is this is adding a header on each row for the data, I want to only have the header displayed once. Then add this for the data to go below the headers:
writer.writerow(data)
I have looked at this and tried it but failed: How can I convert JSON to CSV?.

Create the DictWriter outside the loop, and just call writer.writeheader() there. Then call writer.writerow() inside the loop.
def parse_file(inputed_file):
with open(input_file, 'r') as inputed_file:
content = inputed_file.readlines()
split_file = open('test.csv', 'w')
writer = csv.DictWriter(split_file, fieldnames = ["title", "firstname"], delimiter = ',')
writer.writeheader()
for line in content:
lines = line.split('\t')
data = json.loads(lines[0])
writer.writerow(data)

Taking data from text file and writing it as a .csv file in python

EDIT: Thanks for the answers guys, got what I needed!!
Basically I am trying to take what I have stored in my textfile and I am trying to write that into a .csv file. In my file are tweets that I have stored and I am trying to have one tweet in each cell in my .csv file.
Right now it is only taking one tweet and creating a .csv file with it and I need it to take all of them. Any help is greatly appreciated. Here is what I have so far.
with open('reddit.txt', 'rb') as f:
reader = csv.reader(f, delimiter=':', quoting = csv.QUOTE_NONE)
for row in reader:
print row
cr = csv.writer(open('reddit.csv', 'wb'))
cr.writerow(row)

You'll need to create the writer outside of the loop:
with open('reddit.txt', 'rb') as input_file:
reader = csv.reader(input_file, delimiter=':', quoting = csv.QUOTE_NONE)
with open('reddit.csv', 'wb') as output_file:
writer = csv.writer(output_file)
for row in reader:
writer.writerow(row)
Although here it might be cleaner to open the files without with:
input_file = open('reddit.txt', 'rb')
output_file = open('reddit.csv', 'wb')
reader = csv.reader(input_file, delimiter=':', quoting=csv.QUOTE_NONE)
writer = csv.writer(output_file)
for row in reader:
writer.writerow(row)
input_file.close()
output_file.close()
Or you can still use with and just have a really long line:
with open('reddit.txt', 'rb') as input_file, open('reddit.csv', 'wb') as output_file:
reader = csv.reader(input_file, delimiter=':', quoting = csv.QUOTE_NONE)
writer = csv.writer(output_file)
for row in reader:
writer.writerow(row)

The line cr = csv.writer(open('reddit.csv', 'wb')) is inside the for loop. You need to open the file just once, place this line after
reader = csv.reader(f, delimiter=':', quoting = csv.QUOTE_NONE)
Then write to it as you did in each loop iteration.