I'm trying to store a pandas dataframe to a tempfile in csv format (in windows), but am being hit by:
[Errno 13] Permission denied: 'C:\Users\Username\AppData\Local\Temp\tmpweymbkye'
import tempfile
import pandas
with tempfile.NamedTemporaryFile() as temp:
df.to_csv(temp.name)
Where df is the dataframe. I've also tried changing the temp directory to one I am sure I have write permissions:
tempfile.tempdir='D:/Username/Temp/'
This gives me the same error message
Edit:
The tempfile appears to be locked for editing as when I change the loop to:
with tempfile.NamedTemporaryFile() as temp:
df.to_csv(temp.name + '.csv')
I can write the file in the temp directory, but then it is not automatically deleted at the end of the loop, as it is no longer a temp file.
However, if I change the code to:
with tempfile.NamedTemporaryFile(suffix='.csv') as temp:
training_data.to_csv(temp.name)
I get the same error message as before. The file is not open anywhere else.
I encountered the same error message and the issue was resolved after adding "/df.csv" to file_path.
df.to_csv('C:/Users/../df.csv', index = False)
Check your permissions and, according to this post, you can run your program as an administrator by right click and run as administrator.
We can use the to_csv command to do export a DataFrame in CSV format. Note that the code below will by default save the data into the current working directory. We can save it to a different folder by adding the foldername and a slash to the file
verticalStack.to_csv('foldername/out.csv').
Check out your working directory to make sure the CSV wrote out properly, and that you can open it! If you want, try to bring it back into python to make sure it imports properly.
newOutput = pd.read_csv('out.csv', keep_default_na=False, na_values=[""])
ref
Unlike TemporaryFile(), the user of mkstemp() is responsible for deleting the temporary file when done with it.
With the use of this function may introduce a security hole in your program. By the time you get around to doing anything with the file name it returns, someone else may have beaten you to the punch. mktemp() usage can be replaced easily with NamedTemporaryFile(), passing it the delete=False paramete.
Read more.
After export to CSV you can close your file with temp.close().
with tempfile.NamedTemporaryFile(delete=False) as temp:
df.to_csv(temp.name + '.csv')
temp.close()
Sometimes，you need check the file path that if you have right permission to read and write file. Especially when you use relative path.
xxx.to_csv('%s/file.csv'%(file_path), index = False)
Sometimes, it gives that error simply because there is another file with the same name and it has no permission to delete the earlier file and replace it with the new file.
So either name the file differently while saving it,
or
If you are working on Jupyter Notebook or a other similar environment, delete the file after executing the cell that reads it into memory. So that when you execute the cell which writes it to the machine, there is no other file that exists with that name.
I encountered the same error. I simply had not yet saved my entire python file. Once I saved my python file in VS code as "insertyourfilenamehere".py to documents(which is in my path), I ran my code again and I was able to save my data frame as a csv file.
As per my knowledge, this error pops up when one attempt to save the file that have been saved already and currently open in the background.
You may try closing those files first and then rerun the code.
Just give a valid path and a file name
e.g:
final_df.to_csv('D:\Study\Data Science\data sets\MNIST\sample.csv')
Related
Hi I have a beginner problem. So I wanted to access csv file with jupyter notebook and I am using python. I am opening the jupyter notebook on visual studio code. So here is my code
import pandas as pd
df3 = pd.read_csv("D:/medali.csv")
imax = df3["bronze"].idxmax()
df3[imax:imax+1]
The thing is I kept stuck with the error
FileNotFoundError: [Errno 2] No such file or directory: 'D:/medali.csv'
I assume it is due to pathway problem so I've put the .ipynb file with the .csv file in one folder but it does not work. How to solve the error?
The easiest thing to do, assuming you're on windows os, is to go to the file, right click, select "copy as file path", and then put that in the place of "D:/medali.csv". That should fix the issue, but you may find that you also have to set the file path string as a raw string to keep it from being messed up by the \ or / characters that windows uses. To do this, type a single "r" in front of the file path string, without the quotations. Just the character "r".
Another thought to try is that you may need to actually "Open" the file first, and then try to read from it. Given that you're in python, I would recommend the following syntax:
import pandas as pd
with open(r"filepath.csv", "r") as f:
df3 = pd.read_csv("D:/medali.csv")
imax = df3["bronze"].idxmax()
df3[imax:imax+1]
This is best practice because when you open the file with the "with" keyword, it will close after the block under it has executed automatically.
I tried creating a code where the file, when you run creates a replica of itself and deletes the original file.
Here is my code:
import shutil
import os
loc=os.getcwd()
shutil.move("./aa/test.py", loc, copy_function=shutil.copy2)
But the issue with this is that:
this code is only 1 time usable and to use it again, I need to change the name of the file or delete the newly created file and then run it again.
Also, If I run it inside a folder, It will always create the new file outside the folder (in a dir up from the exceuting program).
How Do I fix this?
Some Notes:
The copy should be made at the exact place where the original file was.
The folder was empty, just having this file. The file doesn't needs to be in a folder but I just used it as a test instance.
Yes, I understand that if I delete the original file it should stop working. I actually have a figure in my mind of how It should work:
First, a new file with the exact same content in it will be made > in the same path as the original file (with a different name probably).
Then, the original file will be deleted and the 2nd file (which is > the copy of the original file) will be renamed as the exact name and > extension as of the original file which got deleted.
This thing above should repeat every time I run the .py file (containing this code) thus making this code portable and suitable for multiple uses.
Maybe the code to be executed after the file deletion can be stored in memory cache (I guess?).
Easiest way (in pseudo code):
Get name of current script.
Read all contents in memory.
Delete current script.
Write memory contents into new file with the same name.
this code is only 1 time usable and to use it again, I need to change the name of the file or delete the newly created file and then run it again.
That is of course because the file is called differently. You could approach this by having no other files in that folder, or always prefixing the filename in the same way, so that you can find the file although it always is called differently.
Also, If I run it inside a folder, It will always create the new file outside the folder (in a dir up from the exceuting program).
That is because you move it from ./aa to ./. You can take the path of the file and reuse it, apart for the filename, and then it would be in the same folder.
Hey TheKaushikGoswami,
I believe your code does exactly what you told him to and as everybody before me stated, surely only works once. :)
I would like to throw in another idea:
First off I'd personally believe that shutil.move is more of a method for actually moving a file into another directory, as you did by accident.
https://docs.python.org/3/library/shutil.html#shutil.move
So why not firstly parameterize your folder (makes it easier for gui/cmd access) and then just copy to a temporary file and then copying from that temporary file. That way you wont get caught in errors raised if you try to create files already existing and have an easy-to-understand code!
Like so:
import shutil
import os
try:
os.mkdir('./aa/')
except:
print('Folder already exists!')
dest= './aa/'
file= 'test.py'
copypath = dest + 'tmp' + file
srcpath = dest + file
shutil.copy2(srcpath, copypath, follow_symlinks=True)
os.remove(srcpath)
shutil.copy2(copypath, srcpath, follow_symlinks=True)
os.remove(copypath)
But may I ask what your use-case is for that since it really doesn't change anything for me other than creating an exact same file?
I had a quick google of this but couldn't find anything. I'm using os to get a list of all the file names in the current working directory using the following code:
path = os.getcwd()
files = os.listdir(path)
The list of files returns fine, but the last element has an extra '~$' that isn't in the actual file name. For example:
files
['File1.xlsx', 'File2.xlsx', '~$File3.xlsx']
This is then causing an issue when I iterate through these files to try and import them, as I get the error of:
[Errno 2] No such file or directory: 'C:\\Users\\$File3.xlsx'
If anyone knows why this happens and how I can fix/prevent it, that would be great!
Just thought I'd answer in case anyone else has this issue.
It's nothing to do with os. It happened because I had File3 open in Excel while pulling the list of file names. I've found out that opening a microsoft document creates a temporary 'lock' file, which are denoted by '~$' (this is how it can re-open unsaved data if it crashes etc).
I found the below from here:
The files you are describing are so-called owner files (sometimes
referred to as "lock" files). An owner file is created when you work
with a document ... and it should be deleted when you save your
document and exit.
There's also a SO question about this within Microsoft files, which can be found here
I am learning Python through 'Automate the Boring Stuff With Python' First Edition. In chapter 12, pg 267, we are supposed to open a file called example.xlsx.
The author's code reads:
import openpyxl
wb = openpyxl.load_workbook('example.xlsx')
type(wb)
However, when I try to open this file, I get the following error (this is the last line of the error):
FileNotFoundError: [Errno 2] No such file or directory: 'example.xlsx'
I know this file exists, because I downloaded it myself and am looking at it right now.
I have tried moving it to the current location in which Python 3.8 is, I have tried saving it with my Automate the Boring Stuff files that I've been working on the desktop, and I have tried saving it in every conceivable location on my machine, but I continue getting this same message.
I have imported openpyxl without error, but when I enter the line
wb = openpyxl.load_workbook('example.xlsx')
I have entered the entire pathway for the example.xlsx in the parenthesis, and I continue to get the same error.
What am I doing wrong? How am I supposed to open an Excel workbook?
I still don't understand how I am doing wrong, but this one is incredibly infuriating, and I feel incredibly stupid, because it must be something simple.
Any insight/help is greatly appreciated.
Your error is unambigous — your file in a supposed directory don't exist. Believe me.
For Python is irrelevant, whether you see it. Python itself must see it.
Specify the full path, using forward slashes, for example:
wb = openpyxl.load_workbook('C:/users/John/example.xlsx')
Or find out your real current (working) directory — and not the one supposed by you — with commands
import os
print(os.getcwd())
then move your example.xlsx to it, and then use only the name of your file
wb = openpyxl.load_workbook('example.xlsx')
You may also verify its existence with commands — use copy/paste from your code to avoid typos in the file name / path
import os.path
print(os.path.exists('example.xlsx')) # True, if Python sees it
or
import os.path
print(os.path.exists('C:/users/John/example.xlsx')) # True, if exists
to be sure that I'm right, i.e. that the error is not in the function openpyxl.load_workbook() itself, but in its parameter (the path to the file) provided by you.
I notice that the extension of the example file is not the same as described in the book, is example.csv. I was facing the same frustration as you
I'm using jupyter notebook pandas to_csv doesn't output the dataframe to a file.
I tried to use to_csv to output a dataframe to a csv file by setting the working directory or specify the directory, and it didn't create any file. The code ran and didn't produce any error message, but when I opened the folder, there was no such file.
I tried a different IO, and it did show the result had been output.
from io import StringIO
output = StringIO()
a.to_csv(output)
print(output.getvalue())
I got the following output:
,a
0,1
1,2
2,3
but again to_csv('filepath/filename.csv') doesn't output any file.
PS: I can read any file in any directory using read_csv().
Update
If I save the file df.to_csv('testfile.csv') then do pd.read_csv('testfile.csv')
I can read the file but cannot see it in the directory.
Also, doing [x for x in os.listdir() if x == 'testfile.csv'] will list the file.
I think the issue is that you're running a Jupyter Notebook so the "current directory" for the notebook is probably somewhere in "C:\Users\user_name\AppData...".
Try running os.getcwd() on its own in your notebook. It probably won't be the same folder as where the *.ipynb file is saved. So as #Chris suggested in comments, this:
df.to_csv(os.getcwd()+'\\file.csv')
... will send your csv into the AppData folder.
You could either change the working directory for the Jupyter notebook, or you could use a fully specified filename like:
df.to_csv('C:\\Users\\<user_name>\\Desktop\\file.csv')
(Note: this also tripped me up in VS-Code while using the interactive iPython execution that happens when you press shift+enter. Interestingly, in VS-Code, if you use ctrl+shift+p and select "Python: Run selection..." it executes in your default terminal, which doesn't have this problem.)
you probably forgot to add the name of the file after your path, so it will named your file as the last character of your path, which you can see on the home page of jupyter.
should be:
df.to_csv('path/filename.csv', ....)
rather than
df.to_csv('path.csv'......)
I had the same problem using spyder. In my case it was caused by the internet security tool (COMODO) I used, which somehow executed spyder in a sandbox or so and not allowed it to write to the "normal" directories. Instead it saved the result to a folder named C:VTRoot\HarddiskVolume2\users\. You can try to save a file with a quite unique name a.to_csv('very_unique_filename.csv') and then search in the windows explorer for that filename to find the folder it is stored in. If the reason is some tool like this, changing it's settings may help.
Maybe you do not have access to your output folder.
First try the current dir, like to_csv('tmp.csv').
Then check the directory's ownership by using ls -l.
This will give you a normal document even though you have used to_csv:
df.to_csv(**'df',** index = False)
Make sure to use 'df.csv' this will ensure the output CSV file.
df.to_csv(**'df.csv'**, index = False)
in my case the files were saved to my apps root folder (as expected).
but i needed to restart Jupyter even though i had auto reload enabled:
%load_ext autoreload
%autoreload 2
meaning, reload works for most changes, but it did not work when i added df.to_csv until restarting jupyter notebook