Develop Reference

Python iterating over excel files in a folder

I am interested in getting this script to open an excel file, and save it again as a .csv or .txt file. I'm pretty sure the problem with this is the iteration - I haven't coded it correctly to iterate properly over the contents of the folder. I am new to Python, and I managed to get this code to sucessfully print a copy of the contents of the items in the folder by the commented out part. Can someone please advise what needs to be fixed?
My error is: raise XLRDError('Unsupported format, or corrupt file: ' + msg)
from xlrd import open_workbook
import csv
import glob
import os
import openpyxl
cwd= os.getcwd()
print (cwd)
FileList = glob.glob('*.xlsx')
#print(FileList)
for i in FileList:
rb = open_workbook(i)
wb = copy(rb)
wb.save('new_document.csv')

I would just use:
import pandas as pd
import glob
import os
file_list = glob.glob('*.xlsx')
for file in file_list:
filename = os.path.split(file, )[1]
pd.read_excel(file).to_csv(filename.replace('xlsx', 'csv'), index=False)

It appears that your error is related to the excel files, not because of your code.
Check that your files aren't also open in Excel at the same time.
Check that your files aren't encrypted.
Check that your version of xlrd supports the files you are reading
In the above order. Any of the above could have caused your error.

How to merge multiple CSV files in a folder to a single file on Azure?

I have written this code and its showing no error. But I am not able to see that output file. Any help will be appreciated.
from os import chdir
from glob import glob
import pandas as pd
def produceOneCSV(list_of_files, file_out):
result_obj = pd.concat([pd.read_csv(file,encoding='utf-8') for file in list_of_files])
result_obj.to_csv(file_out, index=False, encoding='utf-8')
root = "FOLDER PATH"
chdir(root)
file_pattern = ".csv"
list_of_files = [file for file in glob(root+'*.csv')]
file_out = "ConsolidateOutput.csv"
produceOneCSV(list_of_files, file_out)

Check your folder path as well as your app path and permissions
I ran this code with no problems. I started with 3 CSVs and ended up with one. However, there were a few configuration issues which caused me to not see the CSV at first. The full code is below with real paths for reference.
Here are the things I had to fix:
As it stands, this code stores in the folder where this code is located. Is that intentional? It often isn't, so go check where your file.py is to see if your CSV is there.
Check that you have the proper permissions to write to that folder. It wasn't a problem here, but it has been an issue for projects in the past.
Check that your root folder is correct and there are actually CSVs there. While the code throws an error when it doesn't find any CSVs on my local, maybe your setup does it differently.
Here is my full working code:
from os import chdir
from glob import glob
import pandas as pd
def produceOneCSV(list_of_files, file_out):
result_obj = pd.concat([pd.read_csv(file, encoding='utf-8') for file in list_of_files])
result_obj.to_csv(file_out, index=False, encoding='utf-8')
root = "C:\\Users\\Matthew\\PycharmProjects\\stackoverflow\\"
chdir(root)
file_pattern = ".csv"
list_of_files = [file for file in glob(root + '*.csv')]
file_out = "ConsolidateOutput.csv"
produceOneCSV(list_of_files, file_out)

issues while reading a file in python

I'm struggling with reading a file in python, the py file and CSV file are in the same folder but the VSCode makes an error and can't find the file:
import csv
with open('file.csv','r') as f:
reader = reader(f)
...
how can I fix this??
and the error is:
Exception has occurred: FileNotFoundError
[Errno 2] No such file or directory: 'file.csv'

If you run:
import os
os.getcwd()
You'll find out your current working directory which I assume is not the one you were expecting. If you're running the python script through VS code it could be using it could be the directory which you have open on the left hand side.
So either run the python using the correct working directory or use an absolute path like this:
import csv
with open('pathname/file.csv','r') as f:
reader = reader(f)

There might be an issue with your relative path settings.
Try this:
import os
import csv
dir = os.path.dirname(__file__)
filename = os.path.join(dir, 'file.csv')
with open(filename,'r') as f:
reader = reader(f)

Are you using spyder?
If so, please check if the current working path is the path your py file locates.

import csv
with open('file.csv','r') as f:
reader = csv.reader(f)
in this case your file.csv should be in folder where is your python script (current working folder)
or, instead of 'file.csv' you can put absolute path

Django can't find a file stored in my application folder

I have a folder that stores a json file in my django application folder, ie, test_data/data.json.
In my tests.py, I am trying to read this file using the following code:
with open('/test_data/data.json', 'r') as f:
self.response_data = json.load(f)
However, I keep on getting the following error:
FileNotFoundError: [Errno 2] No such file or directory: '/test_data/data.json'
What am I doing wrong? Thanks.
Edit: I tried removing the leading slash, yet I still get the same error.

import os
try this
with open(os.getcwd() + '/test_data/data.json', 'r') as f:
self.response_data = json.load(f)

If you're opening files in directories close to where your code is, it is common to place
import os
DIRNAME = os.path.dirname(__file__) # the directory of this file
at the top of the file.
Then you can open files in a test_data subdirectory with
with open(os.path.join(DIRNAME, 'test_data', 'data.json'), 'rb') as fp:
self.response_data = json.load(fp)
you probably want to open json files, which should be utf-8 encoded, in 'rb' (read-binary) mode.

Pandas DataFrame Not Saving To File

I'm learning Python and can't seem to get pandas dataframes to save. I'm not getting any errors, the file just doesn't appear in the folder.
I'm using a windows10 machine, python3, jupyter notebook and saving to a local google drive folder.
Any ideas?
import feedparser
import pandas as pd
rawrss = [
'http://newsrss.bbc.co.uk/rss/newsonline_uk_edition/front_page/rss.xml',
'https://www.yahoo.com/news/rss/',
'http://www.huffingtonpost.co.uk/feeds/index.xml',
'http://feeds.feedburner.com/TechCrunch/',
]
posts = []
for url in rawrss:
feed = feedparser.parse(url)
for post in feed.entries:
posts.append((post.title, post.link, post.summary))
df = pd.DataFrame(posts, columns=['title', 'link', 'summary']) # pass data to init
df.to_csv('df.to_csv('c:\\Users\\username\\Documents\\myfilename.csv', index=False)', index=False)

The file should be saved in the current working directory.
import os
cwd = os.getcwd()
print(cwd)

In the last line of your code change this:
df.to_csv('C://myfilename.csv', index=False)
Now your file is saved in C drive.
You can change the path as per you wish.
for eg.
df.to_csv('C://Folder//myfilename.csv', index=False)
2.Alternatively ,If you want to locate where your file is stored.
import os
print(os.getcwd())
This gives you the directory where the files are stored.
you can also change your working directory as per your wish.
Just at the beginning of your code
import os
os.chdir("path_to_folder")
In this case then no need of specifying path at the time of saving it to CSV.

You can write a function that saves the file and returns a boolean value as the following:
import os
def save_data(path, file, df):
if (df.to_csv(saving_path + file + '.csv', index = False)):
return True
else:
return False
But you have to provide the right path though.

add this code to the bottom of your file
import os
print(os.getcwd())
That's where your file is
Try writing a simple file with a new script.
F = open(your_path_with_filename, 'w')
F.write("hello")
F.close()