I'am having trouble with python csv module I'am trying to write a newline in a csv file is there any reson why it would not work?
Code:
csv writing function
def write_response_csv(name,games,mins):
with open("sport_team.csv",'w',newline='',encoding='utf-8') as csv_file:
fieldnames=['Vardas','Žaidimai','Minutės']
writer = csv.DictWriter(csv_file,fieldnames=fieldnames)
writer.writeheader()
writer.writerow({'Vardas':name,'Žaidimai':games,"Minutės":mins})
with requests.get(url,headers=headers) as page:
content = soup(page.content,'html.parser')
content = content.findAll('table',class_='table01 tablesorter')
names = find_name(content)
times = 0
for name in names:
matches = find_matches(content,times)
min_in_matches = find_min(content,times)
times +=1
csv_file = write_response_csv(name,matches,min_in_matches)
try:
print(name,matches,min_in_matches)
except:
pass
When you call your write_response_csv function it is reopening the file and starting at line 1 again in the csv file and each new line of data you are passing to that function is overwriting the previous one written. What you could do try is creating the csv file outside of the scope of your writer function and setting your writer function to append mode instead of write mode. This will ensure that it will write the data on the next empty csv line, instead of starting at line 1.
#Outside of function scope
fieldnames=['Vardas','Žaidimai','Minutės']
#Create sport_team.csv file w/ headers
with open('sport_team.csv', 'w',encoding='utf-8') as csv_file:
writer = csv.DictWriter(csv_file, fieldnames)
writer.writeheader()
#Write response function
def write_response_csv(name,games,mins):
with open('sport_team.csv','a',encoding='utf-8') as csv_file:
writer = csv.DictWriter(csv_file, fieldnames)
writer.writerow({'Vardas':name,'Žaidimai':games,"Minutės":mins})
Note:
You will run into the same issue if you are reusing this script to continuously add new lines of data to the same file because each time you run it the code that creates the csv file will essentially recreate a blank sport_team.csv file with the headers. If you would like to reuse the code to continuously add new data, I would look into using os.path and utilizing it to confirm if sport_team.csv exists already and if so, to not run that code after the fieldnames.
Try using metabob, it find code errors for you. I've been using it as a Python beginner, and has been pretty successful with it.
Related
I've written a script in python which is able to fetch the title of different posts from a webpage and write them to a csv file. As the site updates it's content very frequently, I like to append the new result first in that csv file where there are already list of old titles available.
I've tried with:
import csv
import time
import requests
from bs4 import BeautifulSoup
url = "https://stackoverflow.com/questions/tagged/python"
def get_information(url):
response = requests.get(url)
soup = BeautifulSoup(response.content, 'lxml')
for title in soup.select(".summary .question-hyperlink"):
yield title.text
if __name__ == '__main__':
while True:
with open("output.csv","a",newline="") as f:
writer = csv.writer(f)
writer.writerow(['posts'])
for items in get_information(url):
writer.writerow([items])
print(items)
time.sleep(300)
The above script which when run twice can append the new results after the old results.
Old data are like:
A
F
G
T
New data are W,Q,U.
The csv file should look like below when I rerun the script:
W
Q
U
A
F
G
T
How can I append the new result first in an existing csv file having old data?
Inserting data anywhere in a file except at the end requires rewriting the whole thing. To do this without reading its entire contents into memory first, you could create a temporary csv file with the new data in it, append the data from the existing file to that, delete the old file and rename the new one.
Here's and example of what I mean (using a dummy get_information() function to simplify testing).
import csv
import os
from tempfile import NamedTemporaryFile
url = 'https://stackoverflow.com/questions/tagged/python'
csv_filepath = 'updated.csv'
# For testing, create a existing file.
if not os.path.exists(csv_filepath):
with open(csv_filepath, 'w', newline='') as f:
writer = csv.writer(f)
writer.writerows([item] for item in 'AFGT')
# Dummy for testing.
def get_information(url):
for item in 'WQU':
yield item
if __name__ == '__main__':
folder = os.path.abspath(os.path.dirname(csv_filepath)) # Get dir of existing file.
with NamedTemporaryFile(mode='w', newline='', suffix='.csv',
dir=folder, delete=False) as newf:
temp_filename = newf.name # Save filename.
# Put new data into the temporary file.
writer = csv.writer(newf)
for item in get_information(url):
writer.writerow([item])
print([item])
# Append contents of existing file to new one.
with open(csv_filepath, 'r', newline='') as oldf:
reader = csv.reader(oldf)
for row in reader:
writer.writerow(row)
print(row)
os.remove(csv_filepath) # Delete old file.
os.rename(temp_filename, csv_filepath) # Rename temporary file.
Since you intend to change the position of every element of the table, you need to read the table into memory and rewrite the entire file, starting with the new elements.
You may find it easier to (1) write the new element to a new file, (2) open the old file and append its contents to the new file, and (3) move the new file to the original (old) file name.
It's a strange issue, I have a method where I read a csv file so I created an unit test for it, is something as simple as this:
def test_csv(self):
with open(self.csv_file_path, 'rb') as csv_file:
response = csv_parser_method(csv_file)
assert response here
So if I add a pdb breakpoint there and check the content of self.csv_file_path file it's empty
(Pdb) import csv
(Pdb) reader = csv.reader(csv_file, delimiter=str(','))
(Pdb) [row for row in reader]
[]
That's strange, if I open a normal shell it has content and of course the file has content...
Your csv_parser_method already reads the entire CSV file and the csv_file file object therefore already has its pointer positioned at the end of the file, so when you use csv.reader to try to read it, it gets nothing since there is no more content after the position of the end of the file.
You can use the seek method to reset the file pointer back to the beginning of the file so that csv.reader can read the file:
csv_file.seek(0)
reader = csv.reader(csv_file, delimiter=str(','))
Please look at the pseudocode below:
def main():
queries = ['A','B','C']
for query in queries:
filename = query + '.csv'
writer = csv.writer(open(filename, 'wt', encoding = 'utf-8'))
...
FUNCTION (query)
def FUNCTION(query):
...
writer.writerow(XXX)
I'd like to write to multiple csv files, so I use for loop to generate different file names, followed by writing into the file in another def()
However, this is not working, the file will be empty.
If I try to get rid of using main() or stop for loop:
writer = csv.writer(open(filename, 'wt', encoding = 'utf-8'))
...
FUNCTION (query)
def FUNCTION(query):
...
writer.writerow(XXX)
It'll work.
I don't know why? Anything related to for loop or main()?
A simple fix is to pass the file handle and not the name to FUNCTION. Since the file has been opened in main, you don't need/want the name in the subroutine, just the file handle so change the call to FUNCTION(writer) and the definition to
def FUNCTION(writer):
and use writer.writerow(xxx) wherever you need to stream output in the subroutine.
Note: you changed the name of the file pointer from writer to write in your example.
I think the possible reason is you didn't close the file pointer. You can use the context manager like:
with open(filename, 'wt', encoding = 'utf-8') as f:
writer = csv.writer(f)
...
FUNCTION (query)
which will help you auto close file.
I have a program which will run on a cron job and write the output to a CSV file.
I get it to write correctly but I would like to make the program write headers on the first row when the file is created. Is there a way of the program checking if there are any rows in the CSV file and, if not, writing the headers.
Just use a flag:
headers_written = False
then when writing rows:
if not headers_written:
writer.writerow(headers)
headers_written = True
writer.writerow(somerow)
You'd actually postpone creating the writer until you are sure you have stuff to write:
writer = None
# ...
if not writer:
writer = csv.writer(open(filename, 'wb'))
writer.writerow(headers)
writer.writerow(somerow)
I am using Blair's Python script which modifies a CSV file to add the filename as the last column (script appended below). However, instead of adding the file name alone, I also get the Path and File name in the last column.
I run the below script in windows 7 cmd with the following command:
python C:\data\set1\subseta\add_filename.py C:\data\set1\subseta\20100815.csv
The resulting ID field is populated by the following C:\data\set1\subseta\20100815.csv, although, all I need is 20100815.csv.
I'm new to python so any suggestion is appreciated!
import csv
import sys
def process_file(filename):
# Read the contents of the file into a list of lines.
f = open(filename, 'r')
contents = f.readlines()
f.close()
# Use a CSV reader to parse the contents.
reader = csv.reader(contents)
# Open the output and create a CSV writer for it.
f = open(filename, 'wb')
writer = csv.writer(f)
# Process the header.
header = reader.next()
header.append('ID')
writer.writerow(header)
# Process each row of the body.
for row in reader:
row.append(filename)
writer.writerow(row)
# Close the file and we're done.
f.close()
# Run the function on all command-line arguments. Note that this does no
# checking for things such as file existence or permissions.
map(process_file, sys.argv[1:])
Use os.path.basename(filename). See http://docs.python.org/library/os.path.html for more details.