I have two files: src.csv and dst.csv. The code below reads the second row from src.csv and appends it to dst.csv. The issue is the output in dst.csv is contained within double quotes ("").
Expected result:
10, 5, 5, 10, 1
Output:
"10, 5, 5, 10, 1"
I have tried using quoting=csv.QUOTE_NONE, escapechar=' ' in csv.writer and it does remove the quotes though the output now contains a blank space after each csv value.
Here is my code:
import csv
with open('src.csv', 'r') as src, open('dst.csv', 'a', newline='') as dst:
wr = csv.writer(dst, dialect='excel', delimiter=',', quoting=csv.QUOTE_NONE, escapechar=' ')
next(src)
for row in src:
wr.writerow([row.rstrip('\n')])
Any suggestions?
You don't split the source file rows into columns so you just ended up writing a 1 column csv. Use a reader instead:
import csv
with open('src.csv', 'r') as src, open('dst.csv', 'a', newline='') as dst:
wr = csv.writer(dst, dialect='excel', delimiter=',', quoting=csv.QUOTE_NONE, escapechar=' ')
next(src)
reader = csv.reader(src)
for row in reader:
wr.writerow(row)
I think you have to use csv.reader() to read row as list of number - now you read row as one string and csv.writer has to add "" because you have , in this string.
Related
I'm trying to write a code that gets a list of strings, and writes a csv file that containes all of these strings in one line.
here's what I got so far-
import csv
l = ['column1', 'column2',....]
with open('csvfile', 'w', newline='') as f:
writer = csv.writer(f, delimiter=' ', quotechar='|', quoting=csv.QUOTE_MINIMAL)
for x in range(1):
writer.writerow(l)
I want the file to look like this:
column1 column2,...
but for some reason, it ends up looking like this:
column1 column2,...<\n>
I'm using windows and python 3.6.7.
Thanks in advance!
You dont need to use csv.writter you can just do it with Python's file handlers. Also dont use list as a variable name as its a keyword in Python.
list_columns = ['column1', 'column2', 'column3']
f = open('csvfile.csv', 'w')
for item in list_columns:
f.write(item)
f.write(' ')
f.close()
Output: column1 column2 column3
If you want to use csv.writer, you will need to write to a string buffer and then do some editing:
import csv
from io import StringIO
l = ['column1', 'column2']
out = StringIO('')
writer = csv.writer(out, delimiter=' ', quotechar='|', quoting=csv.QUOTE_MINIMAL)
for x in range(4):
writer.writerow(l)
# replace newlines with single space and get rid of carriage returns:
rows = out.getvalue().replace('\n', ' ').replace('\r', '')
# final character is a space. Don't output:
with open('csvfile', 'w') as f:
f.write(rows[0:-1])
How to specify (when reading a file) and assign (in output) this 3-character string"," as .csv delimiter? For example: "col1","col2","col3","col4"
The code where it needs to be used for reading and output:
import csv
with open('a.csv', 'r') as infile, open('reordered.csv', 'a') as outfile:
fieldnames = ['B', 'C', 'A', 'D']
writer = csv.DictWriter(outfile, delimiter=',', fieldnames=fieldnames)
writer.writeheader()
for row in csv.DictReader(infile):
writer.writerow(row)
I tried to look it up (https://docs.python.org/2.0/ref/strings.html), etc. but it is not clear.
UPDATE: As pointed out by others, this may not be "," delimiter, but a , delimiter and "values". Regardless, my values contain commas ,, so the "," pattern between columns helps to maintain column structure.
You'll want to define the quote character, setting the delimiter will not get you the desired results.
import csv
with open('a.csv', 'r') as f:
reader = csv.reader(f, delimiter=',', quotechar='"')
for row in reader:
print row
Edit: As for changes to your code, it would look somewhat like this:
with open('a.csv', 'r') as infile, open('reordered.csv', 'a') as outfile:
fieldnamesout = ['B', 'C', 'A', 'D']
fieldnamesin = ['A', 'B', 'C', 'D']
reader = csv.DictReader(infile, delimiter=',', quotechar='"', fieldnames=fieldnamesin)
writer = csv.DictWriter(outfile, delimiter=',', quotechar='"', fieldnames=fieldnamesout, quoting=csv.QUOTE_ALL)
for row in reader:
writer.writerow(row)
Note the quoting=csv.QUOTE_ALL, which instructs the writer to quote all fields according to the quotechar, this may or may not be what you want, other options would be quoting=csv.QUOTE_NONNUMERIC.
Seems like your values are string literals. I don't know if the cvs module can handle this out of the box. It is not a parsing problem it is a data processing problem.
Your values are in the quote chars. So either you truncate them when processing your data field_value.replace('"', '') and add them back later on '"{}"'.format(field_value) or you straight operate on the data fields leaving the quote char in place.
I have data, example :
2017/06/07 10:42:35,THREAT,url,192.168.1.100,52.25.xxx.xxx,Rule-VWIRE-03,13423523,,web-browsing,80,tcp,block-url
2017/06/07 10:43:35,THREAT,url,192.168.1.101,52.25.xxx.xxx,Rule-VWIRE-03,13423047,,web-browsing,80,tcp,allow
2017/06/07 10:43:36,THREAT,end,192.168.1.100,52.25.xxx.xxx,Rule-VWIRE-03,13423047,,web-browsing,80,tcp,block-url
2017/06/07 10:44:09,TRAFFIC,end,192.168.1.101,52.25.xxx.xxx,Rule-VWIRE-03,13423111,,web-browsing,80,tcp,allow
2017/06/07 10:44:09,TRAFFIC,end,192.168.1.103,52.25.xxx.xxx,Rule-VWIRE-03,13423111,,web-browsing,80,tcp,block-url
How to parse that only get data columns 4,5,7, and 12 in all rows?
This is my code :
import csv
file=open('filename.log', 'r')
f=open('fileoutput', 'w')
lines = file.readlines()
for line in lines:
result.append(line.split(' ')[4,5,7,12])
f.write (line)
f.close()
file.close()
The right way with csv.reader and csv.writer objects:
import csv
with open('filename.log', 'r') as fr, open('filoutput.csv', 'w', newline='') as fw:
reader = csv.reader(fr)
writer = csv.writer(fw)
for l in reader:
writer.writerow(v for k,v in enumerate(l, 1) if k in (4,5,7,12))
filoutput.csv contents:
192.168.1.100,52.25.xxx.xxx,13423523,block-url
192.168.1.101,52.25.xxx.xxx,13423047,allow
192.168.1.100,52.25.xxx.xxx,13423047,block-url
192.168.1.101,52.25.xxx.xxx,13423111,allow
192.168.1.103,52.25.xxx.xxx,13423111,block-url
This is wrong:
line.split(' ')[4,5,7,12]
You want this:
fields = line.split(' ')
fields[4], fields[5], fields[7], fields[12]
a solution using pandas
import pandas as pd
df = pd.read_csv('filename.log', sep=',', header=None, index_col=False)
df[[3, 4, 6, 11]].to_csv('fileoutput.csv', header=False, index=False)
Note the use of [3, 4, 6, 11] instead of [4, 5, 7, 12] to account for 0-indexing in the dataframe's columns.
Content of fileoutput.csv:
192.168.1.100,52.25.xxx.xxx,13423523,block-url
192.168.1.101,52.25.xxx.xxx,13423047,allow
192.168.1.100,52.25.xxx.xxx,13423047,block-url
192.168.1.101,52.25.xxx.xxx,13423111,allow
192.168.1.103,52.25.xxx.xxx,13423111,block-url
You're on the right path, but your syntax is off. Here's an example using csv module:
import csv
log = open('filename.log')
# newline='\n' to prevent csv.writer to include additional newline when writing to file
log_write = open('fileoutput', 'w', newline='\n')
csv_log = csv.reader(log, delimiter=',')
csv_writer = csv.writer(log_write, delimiter=',')
for line in csv_log:
csv_writer.writerow([line[0], line[1], line[2], line[3]]) # output first 4 columns
log.close()
log_write.close()
Looking at the list compressions, you could have something like this without necessarily using csv module
file=open('filename.log','r')
f=open('fileoutput', 'w')
lines = file.readlines()
for line in lines:
f.write(','.join(line.split(',')[i] for i in [3,4,6,11]))
f.close()
file.close()
Notice the indices are 3,4,6,11 for our zero index based list
output
cat fileoutput
192.168.1.100,52.25.xxx.xxx,13423523,block-url
192.168.1.101,52.25.xxx.xxx,13423047,allow
192.168.1.100,52.25.xxx.xxx,13423047,block-url
192.168.1.101,52.25.xxx.xxx,13423111,allow
192.168.1.103,52.25.xxx.xxx,13423111,block-url
I am trying to add a column to a csv file that combines strings from two other columns. Whenever I try this I either get an output csv with only the new column or an output with all of the original data and not the new column.
This is what I have so far:
with open(filename) as csvin:
readfile = csv.reader(csvin, delimiter=',')
with open(output, 'w') as csvout:
writefile = csv.writer(csvout, delimiter=',', lineterminator='\n')
for row in readfile:
result = [str(row[10]) + ' ' + str(row[11])]
writefile.writerow(result)
Any help would be appreciated.
No input to test, but try this. Your current approach doesn't include the existing data for each row that already exists in your input data. extend will take the list that represents each row and then add another item to that list... equivalent to adding a column.
import csv
with open(filename) as csvin:
readfile = csv.reader(csvin, delimiter=',')
with open(output, 'w') as csvout:
writefile = csv.writer(csvout, delimiter=',', lineterminator='\n')
for row in readfile:
row.extend([str(row[10]) + ' ' + str(row[11])])
writefile.writerow(row)
I assume that glayne wants to combine column 10 and 11 into one.
In my approach, I concentrate on how to transform a single row first:
def transform_row(input_row):
output_row = input_row[:]
output_row[10:12] = [' '.join(output_row[10:12])]
return output_row
Once tested to make sure that it works, I can move on to replace all rows:
with open('data.csv') as inf, open('out.csv', 'wb') as outf:
reader = csv.reader(inf)
writer = csv.writer(outf)
writer.writerows(transform_row(row) for row in reader)
Note that I use the writerows() method to write multiple rows in one statement.
Below code snippet combines strings in column 10 and column 11 in each row and add that to the end of the each row
import csv
input = 'test.csv'
output= 'output.csv'
with open(input, 'rb') as csvin:
readfile = csv.reader(csvin, delimiter=',')
with open(output, 'wb') as csvout:
writefile = csv.writer(csvout, delimiter=',', lineterminator='\n')
for row in readfile:
result = row + [row[10]+row[11]]
writefile.writerow(result)
I have some data that needs to be written to a CSV file. The data is as follows
A ,B ,C
a1,a2 ,b1 ,c1
a2,a4 ,b3 ,ct
The first column has comma inside it. The entire data is in a list that I'd like to write to a CSV file, delimited by commas and without disturbing the data in column A. How can I do that? Mentioning delimiter = ',' splits it into four columns on the whole.
Just use the csv.writer from the csv module.
import csv
data = [['A','B','C']
['a1,a2','b1','c1']
['a2,a4','b3','ct']]
fname = "myfile.csv"
with open(fname,'wb') as f:
writer = csv.writer(f)
for row in data:
writer.writerow(row)
https://docs.python.org/library/csv.html#csv.writer
No need to use the csv module since the ',' in the first column is already part of your data, this will work:
with open('myfile.csv', 'w') as f:
for row in data:
f.write(', '.join(row))
f.write('\n')
You could try the below.
Code:
import csv
import re
with open('infile.csv', 'r') as f:
lst = []
for line in f:
lst.append(re.findall(r',?(\S+)', line))
with open('outfile.csv', 'w', newline='') as w:
writer = csv.writer(w)
for row in lst:
writer.writerow(row)
Output:
A,B,C
"a1,a2",b1,c1
"a2,a4",b3,ct