I'm using jupyter notebook pandas to_csv doesn't output the dataframe to a file.
I tried to use to_csv to output a dataframe to a csv file by setting the working directory or specify the directory, and it didn't create any file. The code ran and didn't produce any error message, but when I opened the folder, there was no such file.
I tried a different IO, and it did show the result had been output.
from io import StringIO
output = StringIO()
a.to_csv(output)
print(output.getvalue())
I got the following output:
,a
0,1
1,2
2,3
but again to_csv('filepath/filename.csv') doesn't output any file.
PS: I can read any file in any directory using read_csv().
Update
If I save the file df.to_csv('testfile.csv') then do pd.read_csv('testfile.csv')
I can read the file but cannot see it in the directory.
Also, doing [x for x in os.listdir() if x == 'testfile.csv'] will list the file.
I think the issue is that you're running a Jupyter Notebook so the "current directory" for the notebook is probably somewhere in "C:\Users\user_name\AppData...".
Try running os.getcwd() on its own in your notebook. It probably won't be the same folder as where the *.ipynb file is saved. So as #Chris suggested in comments, this:
df.to_csv(os.getcwd()+'\\file.csv')
... will send your csv into the AppData folder.
You could either change the working directory for the Jupyter notebook, or you could use a fully specified filename like:
df.to_csv('C:\\Users\\<user_name>\\Desktop\\file.csv')
(Note: this also tripped me up in VS-Code while using the interactive iPython execution that happens when you press shift+enter. Interestingly, in VS-Code, if you use ctrl+shift+p and select "Python: Run selection..." it executes in your default terminal, which doesn't have this problem.)
you probably forgot to add the name of the file after your path, so it will named your file as the last character of your path, which you can see on the home page of jupyter.
should be:
df.to_csv('path/filename.csv', ....)
rather than
df.to_csv('path.csv'......)
I had the same problem using spyder. In my case it was caused by the internet security tool (COMODO) I used, which somehow executed spyder in a sandbox or so and not allowed it to write to the "normal" directories. Instead it saved the result to a folder named C:VTRoot\HarddiskVolume2\users\. You can try to save a file with a quite unique name a.to_csv('very_unique_filename.csv') and then search in the windows explorer for that filename to find the folder it is stored in. If the reason is some tool like this, changing it's settings may help.
Maybe you do not have access to your output folder.
First try the current dir, like to_csv('tmp.csv').
Then check the directory's ownership by using ls -l.
This will give you a normal document even though you have used to_csv:
df.to_csv(**'df',** index = False)
Make sure to use 'df.csv' this will ensure the output CSV file.
df.to_csv(**'df.csv'**, index = False)
in my case the files were saved to my apps root folder (as expected).
but i needed to restart Jupyter even though i had auto reload enabled:
%load_ext autoreload
%autoreload 2
meaning, reload works for most changes, but it did not work when i added df.to_csv until restarting jupyter notebook
Related
Hi I have a beginner problem. So I wanted to access csv file with jupyter notebook and I am using python. I am opening the jupyter notebook on visual studio code. So here is my code
import pandas as pd
df3 = pd.read_csv("D:/medali.csv")
imax = df3["bronze"].idxmax()
df3[imax:imax+1]
The thing is I kept stuck with the error
FileNotFoundError: [Errno 2] No such file or directory: 'D:/medali.csv'
I assume it is due to pathway problem so I've put the .ipynb file with the .csv file in one folder but it does not work. How to solve the error?
The easiest thing to do, assuming you're on windows os, is to go to the file, right click, select "copy as file path", and then put that in the place of "D:/medali.csv". That should fix the issue, but you may find that you also have to set the file path string as a raw string to keep it from being messed up by the \ or / characters that windows uses. To do this, type a single "r" in front of the file path string, without the quotations. Just the character "r".
Another thought to try is that you may need to actually "Open" the file first, and then try to read from it. Given that you're in python, I would recommend the following syntax:
import pandas as pd
with open(r"filepath.csv", "r") as f:
df3 = pd.read_csv("D:/medali.csv")
imax = df3["bronze"].idxmax()
df3[imax:imax+1]
This is best practice because when you open the file with the "with" keyword, it will close after the block under it has executed automatically.
I know of the %%writefile .py cell-magic command but it causes the cell to not run. What I want is for the cell to run and create the .py as well.
Edit 1: I only want the original code in the cell to be saved as a .py file. I don't need the results
I would try to:
Use the %%writefile magic in the next cell, referencing the last one
or
Use the %save line magic instead.
There are some ways to deal with it.
Follow two ideas:
%%writefile in the first cell followed by %run yourscript or !python 'yourscript.py'
Convert your code into a string, run it with exec and save it.
The first option is the most common, and you can pass arguments. The second option can be attractive because you don't depend on a Jupyter Notebook to use it.
I launched Jupyter Notebook, created a new notebook in python, imported the necessary libraries and tried to access a .xlsx file on the desktop with this code:
haber = pd.read_csv('filename.xlsx')
but error keeps popping up. Want a reliable way of accessing this file on my desktop without incurring any error response
This is an obvious path problem, because your notebook is not booted on the desktop path, you must indicate the absolute path to the desktop file, or the relative path relative to the jupyter boot directory.
You will need to enter the full path of your excel file.
First:
Open your excel file, right click on the file and click on "Copy path to clipboard".
Second:
Next paste your path in your script. Mine looks something like this:
#only using one backslash "\"
'C:\Users\...YourFileName.xlsx'
Third:
You will likely have to modify this path by adding two "\" instead of one "\" in each spot you only see one backslash.
For example, my new path would now look like this:
#using two backslashes now "\\"
'C:\\Users\\...YourFileName.xlsx'
An example of your final output will look like this:
haber = pd.read_csv('C:\\Users\\...YourFileName.xlsx')
If you are using linux
/home/(your user name)/Desktop/(your filename)
if you are on windows
C:\Users\(your user name)\Desktop\( your filename)
and if your python file is on same path where dataset file is then just give the file name with extension
I'm very new to programming, and to vscode.
I'm learning Python and currently I am learning about working with files.
The path looks like this: /home/anewuser/learning/chapter10.
The problem: completely basic "read file in python" lesson does not work in vscode because no such file or directory error raises when running my .py file, located in ~/learning/chapter10. But vscode wants that my .txt file I am supposed to open in python, to be in ~/learning directory, then it works. I don't like this behaviour.
All I want is to be able to read file placed in the directory where the .py file is. How to do this?
Because in your case ~/learning is the default cwd (current working directory), VSCode looks for pi_digits.txt in that location. If you put pi_digits.txt beside file_reader.py (which is located at ~/learning/chapter10), you'll have to specify the path (by prepending chapter10/ to the .txt file).
So you should do this:
with open('chapter10/pi_digits.txt') as file_object:
contents = file_object.read()
print(contents)
If you want to change the default current working directory (for example you want to change it to ~/learning/chapter10), you'll have to do the following:
~/learning/chapter10/file_reader.py
import os # first you need to import the module 'os'
# set the cwd to 'chapter10'
os.chdir('chapter10')
# now 'file_reader.py' and 'pi_digits.txt' are both in the cwd
with open('pi_digits.txt') as file_object:
contents = file_object.read()
print(contents)
With os.chdir('chapter10') you've set chapter10 as the default cwd, in which VSCode now will look for pi_digits.txt.
For detailed information about os.chdir() you can read through the official documentation or take a look at this post on stackoverflow.
In "User Settings", use the search bar to look for "python.terminal.executeInFileDir" and set (=) its value to "true" instead of "false".
I took this answer from here
How to run python interactive in current file's directory in Visual Studio Code?
this is my first time putting an answer on StackOverflow
so I apologize if I didn't do it the right way
I'm trying to store a pandas dataframe to a tempfile in csv format (in windows), but am being hit by:
[Errno 13] Permission denied: 'C:\Users\Username\AppData\Local\Temp\tmpweymbkye'
import tempfile
import pandas
with tempfile.NamedTemporaryFile() as temp:
df.to_csv(temp.name)
Where df is the dataframe. I've also tried changing the temp directory to one I am sure I have write permissions:
tempfile.tempdir='D:/Username/Temp/'
This gives me the same error message
Edit:
The tempfile appears to be locked for editing as when I change the loop to:
with tempfile.NamedTemporaryFile() as temp:
df.to_csv(temp.name + '.csv')
I can write the file in the temp directory, but then it is not automatically deleted at the end of the loop, as it is no longer a temp file.
However, if I change the code to:
with tempfile.NamedTemporaryFile(suffix='.csv') as temp:
training_data.to_csv(temp.name)
I get the same error message as before. The file is not open anywhere else.
I encountered the same error message and the issue was resolved after adding "/df.csv" to file_path.
df.to_csv('C:/Users/../df.csv', index = False)
Check your permissions and, according to this post, you can run your program as an administrator by right click and run as administrator.
We can use the to_csv command to do export a DataFrame in CSV format. Note that the code below will by default save the data into the current working directory. We can save it to a different folder by adding the foldername and a slash to the file
verticalStack.to_csv('foldername/out.csv').
Check out your working directory to make sure the CSV wrote out properly, and that you can open it! If you want, try to bring it back into python to make sure it imports properly.
newOutput = pd.read_csv('out.csv', keep_default_na=False, na_values=[""])
ref
Unlike TemporaryFile(), the user of mkstemp() is responsible for deleting the temporary file when done with it.
With the use of this function may introduce a security hole in your program. By the time you get around to doing anything with the file name it returns, someone else may have beaten you to the punch. mktemp() usage can be replaced easily with NamedTemporaryFile(), passing it the delete=False paramete.
Read more.
After export to CSV you can close your file with temp.close().
with tempfile.NamedTemporaryFile(delete=False) as temp:
df.to_csv(temp.name + '.csv')
temp.close()
Sometimes，you need check the file path that if you have right permission to read and write file. Especially when you use relative path.
xxx.to_csv('%s/file.csv'%(file_path), index = False)
Sometimes, it gives that error simply because there is another file with the same name and it has no permission to delete the earlier file and replace it with the new file.
So either name the file differently while saving it,
or
If you are working on Jupyter Notebook or a other similar environment, delete the file after executing the cell that reads it into memory. So that when you execute the cell which writes it to the machine, there is no other file that exists with that name.
I encountered the same error. I simply had not yet saved my entire python file. Once I saved my python file in VS code as "insertyourfilenamehere".py to documents(which is in my path), I ran my code again and I was able to save my data frame as a csv file.
As per my knowledge, this error pops up when one attempt to save the file that have been saved already and currently open in the background.
You may try closing those files first and then rerun the code.
Just give a valid path and a file name
e.g:
final_df.to_csv('D:\Study\Data Science\data sets\MNIST\sample.csv')