I have a CSV file which has three columns. The third column "row[02]" has a listing of IP's which contains a list of IP's that are comma delimited. I would like to turn the following:
"column01", "column02", "192.168.1.1, 192.168.2.1, 192.168.3.1"
into (minus the bullets):
column01, column02, 192.168.1.1
column01, column02, 192.168.2.1
column01, column02, 192.168.3.1
Provided is my code, but the output keeps displaying the following:
row01, row02, 192.168.1.1, 192.168.2.1, 192.168.3.1
Please let me know if any other information is needed. Thanks in advance.
Original Code:
#!/usr/bin/python
import csv, sys, time, logging
s = time.strftime('%Y%m%d%H%M%S')
# Create exception file from standard output
class Logger(object):
def __init__(self):
self.terminal = sys.stdout
self.log = open((s) + "_" + sys.argv[1], "a")
def write(self, message):
self.terminal.write(message)
self.log.write(message)
def main():
# Start screen capture to output into CSV file
sys.stdout = Logger()
# File input argument
with open(sys.argv[1], "rb") as f:
reader = csv.reader(f, delimiter=',')
for row in reader:
print row[0], ",", row[1], ",", row[2]
if __name__=='__main__':
main()
Updated Code:
#!/usr/bin/python
import csv, sys, time, logging
s = time.strftime('%Y%m%d%H%M%S')
# Create exception file from standard output
class Logger(object):
def __init__(self):
self.terminal = sys.stdout
self.log = open((s) + "_" + sys.argv[1], "a")
def write(self, message):
self.terminal.write(message)
self.log.write(message)
def main():
# Start screen capture to output file
sys.stdout = Logger()
# File input argument
with open(sys.argv[1], "rb") as f:
reader = csv.reader(f, delimiter=',', skipinitialspace=True)
for row in reader:
network = row[0].replace(" ","")
network_group = row[1].replace(" ","")
address = row[2].replace(',','\n').replace(" ","")
if "-" in address: #parse network ranges
try:
print IP(address), network, network_group
except:
netrange = address.replace('-'," ").replace(" ",",")
print netrange, ",", network, ",", network_group
else:
print address, ",", network, ",", network_group
if __name__=='__main__':
main()
The reason it's printing this:
row01, row02, 192.168.1.1, 192.168.2.1, 192.168.3.1
Is that you asked it to do exactly that for each row:
for row in reader:
print row[0], ",", row[1], ",", row[2]
If you want it to do something different, you have to tell it to do something different. It can't read your mind. So, if you want to, e.g., split row[2] on commas, you need to write some code that does that. Like this:
for row in reader:
addresses = row[2].split(", ")
for address in addresses:
print row[0], ",", row[1], ",", address
Related
why there is no content in the "file object?"
Tried changing to append and there's content since the second run. What am I missing here?
code below:
import random
from io import StringIO
class WriteMyStuff(object):
def __init__(self, writer):
self.writer = writer
def write(self):
write_text = "this is a silly message\n"
self.writer.write(write_text)
fh = open('test.txt', 'w')
w1 = WriteMyStuff(fh)
w1.write()
fh.close
sioh = StringIO()
w2 = WriteMyStuff(sioh)
w2.write()
print("file object: ", open('test.txt', 'r').read())
print("StringIO object:", sioh.getvalue())
Thanks!
This happened because you forgot to call the close method properly in line 17. You called it like an attribute where it needs to be called with the parenthesis .close(). Here's, the fixed version:
import random
from io import StringIO
class WriteMyStuff(object):
def __init__(self, writer):
self.writer = writer
def write(self):
write_text = "this is a silly message\n"
self.writer.write(write_text)
fh = open("test.txt", "w")
w1 = WriteMyStuff(fh)
w1.write()
fh.close()
sioh = StringIO()
w2 = WriteMyStuff(sioh)
w2.write()
print("file object: ", open("test.txt", "r").read())
print("StringIO object:", sioh.getvalue())
This will print out:
file object: this is a silly message
StringIO object: this is a silly message
This code still has one subtle bug, it hasn't closed the second file handler object in the line before the last line.
You can use contextmanages if you don't want to handle closing the file handlers manually. Consider this example:
import random
from io import StringIO
class WriteMyStuff(object):
def __init__(self, writer):
self.writer = writer
def write(self):
write_text = "this is a silly message\n"
self.writer.write(write_text)
with open("test.txt", "w") as fh:
w1 = WriteMyStuff(fh)
w1.write()
sioh = StringIO()
w2 = WriteMyStuff(sioh)
w2.write()
with open("test.txt", "r") as f:
print("file object: ", f.read())
print("StringIO object:", sioh.getvalue())
The with block takes care of closing the file handler after you go out of scope.
I have an application that works. But in the interest of attempting to understand functions and python better. I am trying to split it out into various functions.
I"m stuck on the file_IO function. I'm sure the reason it does not work is because the main part of the application does not understand reader or writer. To better explain. Here is a full copy of the application.
Also I'm curious about using csv.DictReader and csv.DictWriter. Do either provide any advantages/disadvantages to the current code?
I suppose another way of doing this is via classes which honestly I would like to know how to do it that way as well.
#!/usr/bin/python
""" Description This script will take a csv file and parse it looking for specific criteria.
A new file is then created based of the original file name containing only the desired parsed criteria.
"""
import csv
import re
import sys
searched = ['aircheck', 'linkrunner at', 'onetouch at']
def find_group(row):
"""Return the group index of a row
0 if the row contains searched[0]
1 if the row contains searched[1]
etc
-1 if not found
"""
for col in row:
col = col.lower()
for j, s in enumerate(searched):
if s in col:
return j
return -1
#Prompt for File Name
def file_IO():
print "Please Enter a File Name, (Without .csv extension): ",
base_Name = raw_input()
print "You entered: ",base_Name
in_Name = base_Name + ".csv"
out_Name = base_Name + ".parsed.csv"
print "Input File: ", in_Name
print "OutPut Files: ", out_Name
#Opens Input file for read and output file to write.
in_File = open(in_Name, "rU")
reader = csv.reader(in_File)
out_File = open(out_Name, "wb")
writer = csv.writer(out_File, delimiter=',', quotechar='"', quoting=csv.QUOTE_ALL)
return (reader, writer)
file_IO()
# Read header
header = reader.next()
stored = []
writer.writerow([header[0], header[3]])
for i, row in enumerate(reader):
g = find_group(row)
if g >= 0:
stored.append((g, i, row))
stored.sort()
for g, i, row in stored:
writer.writerow([row[0], row[3]])
# Closing Input and Output files.
in_File.close()
out_File.close()
If I were you, I'd only separate find_group.
import csv
def find_group(row):
GROUPS = ['aircheck', 'linkrunner at', 'onetouch at']
for idx, group in enumerate(GROUPS):
if group in map(str.lower, row):
return idx
return -1
def get_filenames():
# this might be the only other thing you'd want to factor
# into a function, and frankly I don't really like getting
# user input this way anyway....
basename = raw_input("Enter a base filename (no extension): ")
infilename = basename + ".csv"
outfilename = basename + ".parsed.csv"
return infilename, outfilename
# notice that I don't open the files yet -- let main handle that
infilename, outfilename = get_filenames()
with open(infilename, 'rU') as inf, open(outfilename, 'wb') as outf:
reader = csv.reader(inf)
writer = csv.writer(outf, delimiter=',',
quotechar='"', quoting=csv.QUOTE_ALL)
header = next(reader)
writer.writerow([[header[0], header[3]])
stored = sorted([(find_group(row),idx,row) for idx,row in
enumerate(reader)) if find_group(row) >= 0])
for _, _, row in stored:
writer.writerow([row[0], row[3]])
I am trying to write a code that takes in a csv, runs a ping on the value in the first column and then outputs the status to the second column. Everything in the code runs fine until it tries to write out to the csv at which time I get this error
line 35, in writer.writerows(columns)
TypeError: 'str' does not support the buffer interface
import os
import csv
from collections import defaultdict
i = 0
#read file
columns = defaultdict(list)
with open('hosts.csv') as f:
reader = csv.DictReader(f)
for row in reader:
for (k,v) in row.items():
columns[k].append(v)
f.close()
print('[DEBUG]', columns['host'])
print('[DEBUG] 1st host is', (columns['host'])[0])
print('[DEBUG]', columns['status'])
#ping hosts
hostname = (columns['host'])[i]
response = os.system("ping -n 1 " + hostname)
print ("[DEBUG]", response)
if response == 0:
print (hostname, 'is up')
(columns['status'])[i] = 'Up'
i = i+1
else:
print (hostname, 'is down')
(columns['status'])[i] = 'Down'
i = i+1
#write results
with open("hosts.csv", "wb") as f:
writer =csv.writer(f)
print("[DEBUG] just before write rows")
writer.writerows(columns)
print("[DEBUG] after write rows")
f.close()
The csv contains the following
host,status,name
8.8.8.8,down,google.com
and should return
host,status,name
8.8.8.8,Up,google.com
I am using Python 3.4
You are reading the CSV in one format and writing in another one, where columns is defaultdict with list of values inside a dict.
Here's a better way to solve this problem, maintaing the original file structure:
import os
import csv
with open('hosts.csv') as f:
reader = csv.DictReader(f)
rows = list(reader)
hosts = [row['host'] for row in rows]
statuses = [row['status'] for row in rows]
print('[DEBUG]', hosts)
print('[DEBUG] 1st host is', hosts[0])
print('[DEBUG]', statuses)
for row in rows:
#ping hosts
hostname = row['host']
response = os.system("ping -n 1 " + hostname)
print ("[DEBUG]", response)
if response == 0:
print (hostname, 'is up')
row['status'] = 'Up'
else:
print (hostname, 'is down')
row['status'] = 'Down'
#write results
with open("hosts.csv", "wb") as f:
writer = csv.DictWriter(f, reader.fieldnames)
# to maintain the same structure from the original file, rewrite header in original position
writer.writeheader()
print("[DEBUG] just before write rows")
writer.writerows(rows)
print("[DEBUG] after write rows")
Before instantiate csv.DictWriter, you can change the field names that you want to be in the new file:
newfieldnames = csvreader.fieldnames
lastfield = newfieldnames.pop() # remove last field
if 'field_name' in newfieldnames:
newfieldnames.remove('field_name') # remove by field name
writer = csv.DictWriter(f, newfieldnames)
trying to process text file and would like to see output in idle as well as redirected to text file. due to text formatting, i need to keep statement in print function (end = ''). i am looking for a way to use "end = ''" and "file=output_file" simultaneously with print function.
import re
input_file = open ('E:\input.txt','r')
output_file = open ('E:\output.txt','w')
for line in input_file:
if re.match('ab|cd', line):
print ('line with ab or cd: ', end = '',file=output_file )
print (line, end = '',file=output_file)
print (' --------', file=output_file)
print (' --------',file=output_file)
print ('!',file=output_file)
else:
print (line,file=output_file)
To write to several places using a single print(), you could define a file-like object:
class TeeFile(object):
def __init__(self, *files):
self.files = files
def write(self, data):
for f in self.files:
f.write(data)
def flush(self):
for f in self.files:
f.flush()
Example:
import sys
file = TeeFile(sys.stderr, output_file)
print("abc", end="", file=file)
file.flush()
Python 2.4
For my example I have a 2 column csv file
Eg:
HOST, FILE
server1, /path/to/file1
server2, /path/to/file2
server3, /path/to/file3
I would like to get the file size of the object at PATH for each row in the csv FILE, then add that value to the csv FILE on a new column.
Making it:
HOST, PATH, FILESIZE
server1, /path/to/file1, 6546542
server2, /path/to/file2, 46546343
server3, /path/to/file3, 87523
Ive tried a couple methods but havnt had much success.
The code below executes fileSizeCmd (du -b) on the PATH and outputs the filezie correctly, but I havnt figured out how to use the data to add to the csv FILE
import datetime
import csv
import os, time
from subprocess import Popen, PIPE, STDOUT
now = datetime.datetime.now()
fileSizeCmd = "du -b"
SP = " "
# Try to get disk size and append to another row after entry above
#st = os.stat(row[3])
#except IOError:
#print "failed to get information about", file
#else:
#print "file size:", st[ST_SIZE]
#print "file modified:", time.asctime(time.localtime(st[ST_MTIME]))
incsv = open('my_list.csv', 'rb')
try:
reader = csv.reader(incsv)
outcsv = open('results/results_' + now.strftime("%m-%d-%Y") + '.csv', 'wb')
try:
writer = csv.writer(outcsv)
for row in reader:
p = Popen(fileSizeCmd + SP + row[1], shell=True, stdin=PIPE, stdout=PIPE, stderr=PIPE)
stdout, empty = p.communicate()
print 'Command: %s\nOutput: %s\n' % (fileSizeCmd + SP + row[1], stdout)
# Results in bytes example
#
# Output:
# 8589935104 /path/to/file
#
# Write 8589935104 to new column of csv FILE
finally:
outcsv.close()
finally:
incsv.close()
Sketch w/o error handling:
#!/usr/bin/env python
import csv
import os
filename = "sample.csv"
# localhost, 01.html.bak
# localhost, 01.htmlbak
# ...
def filesize(filename):
# no need to shell out for filesize
return os.stat(filename).st_size
with open(filename, 'rb') as handle:
reader = csv.reader(handle)
# result is written to sample.csv.updated.csv
writer = csv.writer(open('%s.updated.csv' % filename, 'w'))
for row in reader:
# need to strip filename, just in case
writer.writerow(row + [ filesize(row[1].strip()) ])
# result
# localhost, 01.html.bak,10021
# localhost, 01.htmlbak,218982
# ...
You can
1) read the cvs content into a list of tuple of (server, filename)
2) collect the file size for each element of this list
3) package the result into another tuple (server, filename, filesize) into another list ('result')
4) write out the result to new file
First, getting file size is a lot easier than using subprocess (see os.stat):
>>> os.stat('/tmp/file').st_size
100
Second, you're on the right track with your writer object writing to a different file, but you just need to add a column to the row lists you're getting back from the reader and then feed them to writerow on the writer (see here). Something like this:
>>> writerfp = open('out.csv', 'w')
>>> writer = csv.writer(writerfp)
>>> for row in csv.reader(open('in.csv', 'r')):
... row.append('column')
... writer.writerow(row)
...
>>> writerfp.close()