Develop Reference

How to write to a CSV line by line?

I have data which is being accessed via http request and is sent back by the server in a comma separated format, I have the following code :
site= 'www.example.com'
hdr = {'User-Agent': 'Mozilla/5.0'}
req = urllib2.Request(site,headers=hdr)
page = urllib2.urlopen(req)
soup = BeautifulSoup(page)
soup = soup.get_text()
text=str(soup)
The content of text is as follows:
april,2,5,7
may,3,5,8
june,4,7,3
july,5,6,9
How can I save this data into a CSV file.
I know I can do something along the lines of the following to iterate line by line:
import StringIO
s = StringIO.StringIO(text)
for line in s:
But i'm unsure how to now properly write each line to CSV
EDIT---> Thanks for the feedback as suggested the solution was rather simple and can be seen below.
Solution:
import StringIO
s = StringIO.StringIO(text)
with open('fileName.csv', 'w') as f:
for line in s:
f.write(line)

General way:
##text=List of strings to be written to file
with open('csvfile.csv','wb') as file:
for line in text:
file.write(line)
file.write('\n')
OR
Using CSV writer :
import csv
with open(<path to output_csv>, "wb") as csv_file:
writer = csv.writer(csv_file, delimiter=',')
for line in data:
writer.writerow(line)
OR
Simplest way:
f = open('csvfile.csv','w')
f.write('hi there\n') #Give your csv text here.
## Python will convert \n to os.linesep
f.close()

You could just write to the file as you would write any normal file.
with open('csvfile.csv','wb') as file:
for l in text:
file.write(l)
file.write('\n')
If just in case, it is a list of lists, you could directly use built-in csv module
import csv
with open("csvfile.csv", "wb") as file:
writer = csv.writer(file)
writer.writerows(text)

I would simply write each line to a file, since it's already in a CSV format:
write_file = "output.csv"
with open(write_file, "wt", encoding="utf-8") as output:
for line in text:
output.write(line + '\n')
I can't recall how to write lines with line-breaks at the moment, though :p
Also, you might like to take a look at this answer about write(), writelines(), and '\n'.

To complement the previous answers, I whipped up a quick class to write to CSV files. It makes it easier to manage and close open files and achieve consistency and cleaner code if you have to deal with multiple files.
class CSVWriter():
filename = None
fp = None
writer = None
def __init__(self, filename):
self.filename = filename
self.fp = open(self.filename, 'w', encoding='utf8')
self.writer = csv.writer(self.fp, delimiter=';', quotechar='"', quoting=csv.QUOTE_ALL, lineterminator='\n')
def close(self):
self.fp.close()
def write(self, elems):
self.writer.writerow(elems)
def size(self):
return os.path.getsize(self.filename)
def fname(self):
return self.filename
Example usage:
mycsv = CSVWriter('/tmp/test.csv')
mycsv.write((12,'green','apples'))
mycsv.write((7,'yellow','bananas'))
mycsv.close()
print("Written %d bytes to %s" % (mycsv.size(), mycsv.fname()))
Have fun

What about this:
with open("your_csv_file.csv", "w") as f:
f.write("\n".join(text))
str.join() Return a string which is the concatenation of the strings in iterable.
The separator between elements is
the string providing this method.

In my situation...
with open('UPRN.csv', 'w', newline='') as out_file:
writer = csv.writer(out_file)
writer.writerow(('Name', 'UPRN','ADMIN_AREA','TOWN','STREET','NAME_NUMBER'))
writer.writerows(lines)
you need to include the newline option in the open attribute and it will work
https://www.programiz.com/python-programming/writing-csv-files

How to loop through a list of strings in Python

I'm a bit new to Python and I am trying to simplify my existing code.
Right now, I have the code repeated 5 times with different strings. I'd like to have the code one time and have it run through a list of strings.
Currently what I have:
def wiScanFormat():
File = open("/home/pi/gpsMaster/WiScan.txt", "r")
data = File.read()
File.close()
MAC = data.replace("Address:", "\nAddress, ")
File = open("/home/pi/gpsMaster/WiScan.txt", "w")
File.write(MAC)
File.close()
File = open("/home/pi/gpsMaster/WiScan.txt", "r")
data = File.read()
File.close()
SSID = data.replace("ESSID:", "\nESSID, ")
File = open("/home/pi/gpsMaster/WiScan.txt", "w")
File.write(SSID)
File.close()
File = open("/home/pi/gpsMaster/WiScan.txt", "r")
data = File.read()
File.close()
FREQ = data.replace("Frequency:", "\nFrequency, ")
File = open("/home/pi/gpsMaster/WiScan.txt", "w")
File.write(FREQ)
File.close()
File = open("/home/pi/gpsMaster/WiScan.txt", "r")
data = File.read()
File.close()
QUAL = data.replace("Quality", "\nQuality, ")
File = open("/home/pi/gpsMaster/WiScan.txt", "w")
File.write(QUAL)
File.close()
File = open("/home/pi/gpsMaster/WiScan.txt", "r")
data = File.read()
File.close()
SIG = data.replace("Signal level", "\nSignal Level, ")
File = open("/home/pi/gpsMaster/WiScan.txt", "w")
File.write(SIG)
File.close()
What I'd like to have:
ORG = ['Address:', 'ESSID:'...etc]
NEW = ['\nAddress, ' , '\nESSID, ' , ... etc]
and run that through:
File = open("/home/pi/gpsMaster/WiScan.txt", "r")
data = File.read()
File.close()
ID = data.replace("ORG", "NEW")
File = open("/home/pi/gpsMaster/WiScan.txt", "w")
File.write(ID)
File.close()
I've tried running exactly what I put up, but it does not seem to format it the way I need to.
The output from above looks like:
Cell 46 - Address: xx:xx:xx:xx:xx:xx ESSID:"MySSID" Frequency:2.412 GHz (Channel 1) Quality=47/100 Signal level=48/100 Quality=47/100 Signal level=48/100
But it is supposed to look like this (And it does when I run that same block over the strings separately):
xx:xx:xx:xx:xx:xx MySSID 5.18 GHz (Channel 36) 0.81 0.99
How should I go about looping this block of code through my list of strings?
There two strings that I would need for the find and replace, old and new, so they would have to work together. These lists will be the same size, obviously, and I need them to be in the correct order. Address with address, ESSID with ESSID, etc.
Thanks in advance!

Try something like this:
ORG = ['Address:', 'ESSID:'...etc]
NEW = ['\nAddress, ' , '\nESSID, ' , ... etc]
File = open("/home/pi/gpsMaster/WiScan.txt", "r")
data = File.read()
File.close()
for org, new in zip(ORG, NEW):
data = data.replace(org, new)
File = open("/home/pi/gpsMaster/WiScan.txt", "w")
File.write(data)
File.close()
(Note the way zip works: https://docs.python.org/2/library/functions.html#zip)

If I am reading your question right, you are opening the same file, making a small alteration, saving it, and then closing it again, five times. You could just open it once, make all the alterations, and then save it. For instance, like this:
filename = "/home/pi/gpsMaster/WiScan.txt"
with open(filename, 'r') as fin:
data = fin.read()
data = data.replace("Address:", "\nAddress, ")
data = data.replace("ESSID:", "\nESSID, ")
data = data.replace("Frequency:", "\nFrequency, ")
data = data.replace("Quality", "\nQuality, ")
data = data.replace("Signal level", "\nSignal Level, ")
with open(filename, 'w') as fout:
fout.write(data)
If you want to use lists (ORG and NEW) for your replacements, you could do this:
with open(filename, 'r') as fin:
data = fin.read()
for o,n in zip(ORG, NEW):
data = data.replace(o,n)
with open(filename, 'w') as fout:
fout.write(data)

Given your ORG and NEW, the simplest way to do this would be something like:
# Open once for both read and write; use with statement for guaranteed close at end of block
with open("/home/pi/gpsMaster/WiScan.txt", "r+") as f:
data = f.read() # Slurp file
f.seek(0) # Seek back to beginning of file
# Perform all replacements
for orig, repl in zip(ORG, NEW):
data = data.replace(orig, repl)
f.write(data) # Write new data over old
f.truncate() # If replacement shrunk file, truncate extra

You could just do this:
def wiScanFormat(path = "/home/pi/gpsMaster/WiScan.txt"):
# List of tuples with strings to find and strings to replace with
replacestr = [
("Address:", "\nAddress, "),
("ESSID:", "\nESSID, "),
("Frequency:", "\nFrequency, "),
("Quality", "\nQuality, "),
("Signal level", "\nSignal Level, ")
]
with open(path, "r") as file: # Open a file
data = file.read()
formated = data
for i in replacestr: # Loop over each element (tuple) in the list
formated = formated.replace(i[0], i[1]) # Replace the data
with open(path, "w") as file:
written = file.write(formated) # Write the data
return written

Python converting text file to all caps

I'm new to python, and I'm trying to figure out how to write a program that prompts the user for the name of a text file, converts the contents of the text file to all caps, and then saves it as a new file.

Using Python 2.7.6 this works for me:
filename = raw_input("File Name: ")
with open(filename, 'r+') as f:
text = f.read()
f.seek(0)
f.write(text.upper())
f.truncate()

import os
def main():
fp = raw_input('Filename: ')
if fp and os.path.isfile(fp):
with open(fp, 'r') as f:
txt = f.read()
newfp = '{0}_upper{1}'.format(*os.path.splitext(fp))
with open(newfp, 'w') as f:
f.write(txt.upper())
if __name__ == '__main__':
main()

How do I send the output of my first operation into the input of the second?

I am trying to find a way to send the output of crop_rows called at the very bottom into the input of delete_Apps called directly after, but I'm unsure as to what to feed it. Also, do I need to remove:
file_obj.close()
from the crop_rows function so that my script continues to run through both functions?
import os, csv, sys, Tkinter, tkFileDialog as fd
# stop tinker shell from opening as only needed for file dialog
root = Tkinter.Tk()
root.withdraw()
#crop_rows deletes the extra info automatically generated in the Waders report
def crop_rows(in_path):
# read file into memory
file_obj = open(in_path, 'rb')
reader = csv.reader(file_obj, delimiter='\t')
data = []
for row in reader:
if not row or not any(row):
break #stop at empty row
else:
data.append(row)
file_obj.close()
print 'Found', len(data), 'rows of data without empty lines.'
conf = raw_input('delete remaining lines? (Y|N): ').upper()[0]
if conf == 'Y':
# write data to file
file_obj = open(in_path, 'wb')
writer = csv.writer(file_obj)
writer.writerows(data)
file_obj.close
#delete_Apps deletes and leads that are currently Applicants in gHire as their Status
def delete_Apps(in_path):
# read file into memory
file_obj = open(in_path, 'rb')
reader = csv.reader(file_obj, delimiter='\t')
data = []
for row in reader:
if 'Applicant' not in row:
data.append(row)
file_obj.close()
print 'Found', len(data), 'Leads with Applicant in gHire as Status.'
conf = raw_input('delete these leads? (Y|N): ').upper()[0]
if conf == 'Y':
# write data to file
file_obj = open(in_path, 'wb')
writer = csv.writer(file_obj)
writer.writerows(data)
file_obj.close
def main():
in_path = None
prog_name = sys.argv[0]
# check if in_path are inlcuded as cmd line args...
if len(sys.argv) > 1:
in_path = sys.argv[1]
if not os.path.exists(in_path):
print 'Usage:', prog_name, '[file_path>]'
print 'cannot find the file provided for file_path:\n', in_path
sys.exit("Error - invalid excel_file_path arg")
else:
try:
# set current working directory to user's my documents folder
os.chdir(os.path.join(os.getenv('userprofile'),'documents'))
except:
pass
# ask user for path to file...
while not in_path:
print "Please select the file to read data from ..."
try:
in_path = fd.askopenfilename()
except:
print 'Error selecting file.'
if not in_path:
cont = raw_input('Do you want to continue? (Y|N): ').upper()[0]
if cont == 'N':
sys.exit("Error - unable to select input file")
crop_rows(in_path)
delete_Apps(in_path)
if __name__ == '__main__':
main()

Change crop_rows into a generator function (see https://wiki.python.org/moin/Generators) and use it in delete_Apps.

Read the .csv file separately
Change crop_rows so that it takes rows as input and returns the new data
Then change delete_Apps to take rows input and to return the modified data
Finally write the .csv file
You can then simply nest calls to the functions, like this:
def read_input_file(in_path):
# use with not to worry about closing the file
with open(in_path, 'rb') as file_obj:
reader = csv.reader(file_obj, delimiter='\t')
return list(reader)
def crop_rows(input_rows):
data = []
for row in input_rows:
if not row or not any(row):
break #stop at empty row
else:
data.append(row)
print 'Found', len(data), 'rows of data without empty lines.'
conf = raw_input('delete remaining lines? (Y|N): ').upper()[0]
if conf == 'Y':
return data
else:
# return unmodified data
return input_rows
def delete_Apps(input_rows):
data = []
for row in input_rows:
if 'Applicant' not in row:
data.append(row)
print 'Found', len(data), 'Leads with Applicant in gHire as Status.'
conf = raw_input('delete these leads? (Y|N): ').upper()[0]
if conf == 'Y':
return data
else:
return input_rows
def write_output_file(data, out_path):
wirg open(out_path, 'wb') as file_obj:
writer = csv.writer(file_obj)
writer.writerows(data)
file_obj.close()
The final call:
file_path='/tmp/whatever.csv'
write_output_file(delete_Apps(crop_rows(read_input_file(file_path)), file_path)

Want to read multiple csv file one by one and filepaths are stored in a text file using python

here is my code for readinng individual cell of one csv file. but want to read multiple csv file one by one from .txt file where csv file paths are located.
import csv
ifile = open ("C:\Users\BKA4ABT\Desktop\Test_Specification\RDBI.csv", "rb")
data = list(csv.reader(ifile, delimiter = ';'))
REQ = []
RES = []
n = len(data)
for i in range(n):
x = data[i][1]
y = data[i][2]
REQ.append (x)
RES.append (y)
i += 1
for j in range(2,n):
try:
if REQ[j] != '' and RES[j]!= '': # ignore blank cell
print REQ[j], ' ', RES[j]
except:
pass
j += 1
And csv file paths are stored in a .txt file like
C:\Desktop\Test_Specification\RDBI.csv
C:\Desktop\Test_Specification\ECUreset.csv
C:\Desktop\Test_Specification\RDTC.csv
and so on..

You can read stuff stored in files into variables. And you can use variables with strings in them anywhere you can use a literal string. So...
with open('mytxtfile.txt', 'r') as txt_file:
for line in txt_file:
file_name = line.strip() # or was it trim()? I keep mixing them up
ifile = open(file_name, 'rb')
# ... the rest of your code goes here
Maybe we can fix this up a little...
import csv
with open('mytxtfile.txt', 'r') as txt_file:
for line in txt_file:
file_name = line.strip()
csv_file = csv.reader(open(file_name, 'rb', delimiter=';'))
for record in csv_file[1:]: # skip header row
req = record[1]
res = record[2]
if len(req + res):
print req, ' ', res

you just need to add a while which will read your file containing your list of files & paths upon your first open statement, for example
from __future__ import with_statement
with open("myfile_which_contains_file_path.txt") as f:
for line in f:
ifile = open(line, 'rb')
# here the rest of your code

You need to use a raw string string your path contains \
import csv
file_list = r"C:\Users\BKA4ABT\Desktop\Test_Specification\RDBI.csv"
with open(file_list) as f:
for line in f:
with open(line.strip(), 'rb') as the_file:
reader = csv.reader(the_file, delimiter=';')
for row in reader:
req,res = row[1:3]
if req and res:
print('{0} {1}'.format(req, res))