I need to create a file using NamedTemporaryFile and download some data into it:
f = NamedTemporaryFile(suffix='.csv', delete=False)
s3.download_fileobj(bucket, f"{files_dir}/metadata/{table_name}.csv.gz", f)
After that I need to load data into mysql table using LOAD DATA INFILE
But I get an error:
File '/tmp/tmp00f3funu.csv' not found (OS errno 13 - Permission denied)
I suppose mysql root user doesn't have enough permissions to read temp file.
This temp file has permissions to read only for owner of this file.
What should I do to resolve this issue?
here you either need to run as SQL user, which might not be the best idea
or just change file permissions after file is created using os.chmod
see detailed answer here: https://stackoverflow.com/a/16249655/7453765
Related
I am really new in Python.
I have made a python program for merging pdfs. The problem is when the program want to save the completed merged pdf on the server (at work), it doesn't work. I get an Error: Permission denied. But when I say to my program that the merged pdf file must be saved on a local folder (on my pc), it works perfectly. How can I fix this?
An example of pathpdflocation_compleet (local on my pc) :
C:\Users\myname\Documents\TF\Compleet\pdfcompleet.pdf
An example of pathpdflocation_compleet (at work) :
W:\xxx\xxx\xxx\xx\pdfcompleet.pdf
Can I give my python script permission for saving this pdf file on the server?
I have searched on the web, but I don't find anything about this.
I have already tried with double backslashes and / or \.
I have already tried to give the full name of the server.
'''
python
'''
from PyPDF2 import PdfFileMerger
from openpyxl import load_workbook
filepath=input ('Path of the Merge Excel: ')
wb=load_workbook(filepath,data_only = True)
fiche1="A1"
file1=sheet[fiche1].value
fiche2="A2"
file2=sheet[fiche2].value
pdf2merge = [file1,file2,...]
merger = PdfFileMerger()
for pdf in pdf2merge:
merger.append(pdf)
with open(pathpdflocation_compleet, 'wb') as fout:
merger.write(fout)
The error is this:
IOError: [Errno 13] Permission denied
I want the file to be saved on the server.
I am trying to use django-dbbackup module to backup my database. After running dbbackup it saves a db dump with a name something like this:
default-user-2018-03-22-220805.psql
Then I delete 1 last row in one tables of my db. Next I run dbrestore and get the follow:
> Finding latest backup Restoring backup for database 'default' and
> server 'None'
> Restoring: default-user-2018-03-22-220805.psql
> Restore tempfile created: 407.0 KiB
> Are you sure you want to continue?
> [Y/n] y
> Following files were affected
But after that - nothing happens. The deleted row is not restored in my db.
I have read some where (unfortunately already lost that page) that to restore my db the dump file must be in .tar format.
Also I have tried to use that .psql with pgAdmin 4 - to restore db via UI tool. But got an error that input file is not a valid archive.
And last I tried to use that file with Windows cmd running pd_restore and got:
pg_restore: [archiver] input file does not appear to be a valid archive
So the question is: how to use django-dbbackup dbrestore with its
generated file or how to change the extension format of its output
files if it is not possible to restore from .psql files?
P.S. I have also found in the source code a row extension = 'psql', tried to change that, and it worked - I got a .tar file on the output but next dbrestore said that:
Finding latest backup
CommandError: There's no backup file available.
You can change file backup extension with this variable
DBBACKUP_FILENAME_TEMPLATE = '{databasename}-{servername}-{datetime}.sql'
You define DBBACKUP_FILENAME_TEMPLATE variable in settings.py
This setting change extension backup file from dump to sql.
You can change sql extension file with another extension.
I'm trying to store a pandas dataframe to a tempfile in csv format (in windows), but am being hit by:
[Errno 13] Permission denied: 'C:\Users\Username\AppData\Local\Temp\tmpweymbkye'
import tempfile
import pandas
with tempfile.NamedTemporaryFile() as temp:
df.to_csv(temp.name)
Where df is the dataframe. I've also tried changing the temp directory to one I am sure I have write permissions:
tempfile.tempdir='D:/Username/Temp/'
This gives me the same error message
Edit:
The tempfile appears to be locked for editing as when I change the loop to:
with tempfile.NamedTemporaryFile() as temp:
df.to_csv(temp.name + '.csv')
I can write the file in the temp directory, but then it is not automatically deleted at the end of the loop, as it is no longer a temp file.
However, if I change the code to:
with tempfile.NamedTemporaryFile(suffix='.csv') as temp:
training_data.to_csv(temp.name)
I get the same error message as before. The file is not open anywhere else.
I encountered the same error message and the issue was resolved after adding "/df.csv" to file_path.
df.to_csv('C:/Users/../df.csv', index = False)
Check your permissions and, according to this post, you can run your program as an administrator by right click and run as administrator.
We can use the to_csv command to do export a DataFrame in CSV format. Note that the code below will by default save the data into the current working directory. We can save it to a different folder by adding the foldername and a slash to the file
verticalStack.to_csv('foldername/out.csv').
Check out your working directory to make sure the CSV wrote out properly, and that you can open it! If you want, try to bring it back into python to make sure it imports properly.
newOutput = pd.read_csv('out.csv', keep_default_na=False, na_values=[""])
ref
Unlike TemporaryFile(), the user of mkstemp() is responsible for deleting the temporary file when done with it.
With the use of this function may introduce a security hole in your program. By the time you get around to doing anything with the file name it returns, someone else may have beaten you to the punch. mktemp() usage can be replaced easily with NamedTemporaryFile(), passing it the delete=False paramete.
Read more.
After export to CSV you can close your file with temp.close().
with tempfile.NamedTemporaryFile(delete=False) as temp:
df.to_csv(temp.name + '.csv')
temp.close()
Sometimes,you need check the file path that if you have right permission to read and write file. Especially when you use relative path.
xxx.to_csv('%s/file.csv'%(file_path), index = False)
Sometimes, it gives that error simply because there is another file with the same name and it has no permission to delete the earlier file and replace it with the new file.
So either name the file differently while saving it,
or
If you are working on Jupyter Notebook or a other similar environment, delete the file after executing the cell that reads it into memory. So that when you execute the cell which writes it to the machine, there is no other file that exists with that name.
I encountered the same error. I simply had not yet saved my entire python file. Once I saved my python file in VS code as "insertyourfilenamehere".py to documents(which is in my path), I ran my code again and I was able to save my data frame as a csv file.
As per my knowledge, this error pops up when one attempt to save the file that have been saved already and currently open in the background.
You may try closing those files first and then rerun the code.
Just give a valid path and a file name
e.g:
final_df.to_csv('D:\Study\Data Science\data sets\MNIST\sample.csv')
I am trying to write a datafame to a csv file using the following:
fxRollPath = 'C:\Users\stacey\Documents\scripts\FXFiles'
fxData.to_csv(fxRollPath,fxRoll'+ str(index)+'.csv')
Where fxRoll is a dataframe
But I get the error:
PermissionError: [Errno 13] Permission denied: 'C:\Users\stacey\Documents\scripts\FXFiles'
What have I done wrong?
Just realized, you are actually trying to save to a target directory path instead of file path.
Docs of path_or_buf for DataFrame.to_csv : "string or file handle, default None. File path or object, if None is provided the result is returned as a string."
So change your code to:
fxData.to_csv('{0}\{1}{2}{3}'.format(fxRollPath, fxRoll, str(index), '.csv'))
Maybe you need close the first version your csv which now is open
I got this problem too. Not sure if you have the same situation as me.
I had the same filename in the same directory and I wanted to override the old csv file. Instead of overriding the old file, I deleted it and then save it to solve this problem.
os.remove('filename')
df.to_csv('filename.csv')
Edit: I deleted this but I'm going to undelete because I think it could be useful. And post what was actually happening which I didn't realize at the time.
Original Question: I'm trying to open a set of excel files where one is currently open by another user. All of the other files work, but this one I get 'permission denied' errors.
Windows gives you the option to "read only" open the file, but I can't seem to find an equivalent in python (xlrd), so I thought I would copy the file to a temp location and open it; but that doesn't work either. In either case, i get:
IOError: [Errno 13] Permission denied:
Is it possible to either:
open an excel file in read only mode (like windows) in xlrd
copy a file currently in use by another user
thanks !
As it turns out, I the files that were failing were also open by other users, so when I went to access them through windows explorer it seemed natural another user having the file open would cause the permission denied error; however the real issue is that as I was looping through the folder I was attempting to access the temp files created by Excel for the other user who was accessing the file.
So, for a given file "old_file.xlsx" that a user had open, their instance of excel created "~$old_file.xlsx"
How I got around this was:
files_to_check = [f for f in os.listdir(PATH) if os.path.isfile(f)]
files_to_check = [f for f in files_to_check if '~' not in f and 'xlsx' in f]
basically, just making sure that they're the non-temp, xlsx files that I planned to go through.