Shutil.copy IO Error2 when directory exists - python

I'm encountering a troublesome problem with my code and I've been unable to figure it out. Basically I am copying files from a local directory on my computer to a Dropbox folder that acts as a project repository for me and some other folks.
I keep hitting an IO Error when executing the shutil.copy line. Errno 2, N osuch file or directory. However the directory and file both exist. When I change the directory to a different/test location (test_dir in my code), the code runs perfectly fine. Any insights would be greatly appreciated.
import os, os.path
import re
import shutil
import sys
#File Location
directory_list = "path where files are located"
#Dropbox base directory:
dropbox = "path to dropbox directory"
test_dir = "path to test directory on my local machine"
sed_files = os.listdir(directory_list)
for i in sed_files:
#print i.split("BBB")[0]
#df
copy_dir = re.sub(r'XXX',r'_',i.split("BBB")[0])
copy_dir = re.sub(r'ZZZ',r'/',copy_dir)
copy_dir = dropbox + copy_dir + "/FIXED/"
if not os.path.exists(copy_dir):
os.makedirs(copy_dir)
shutil.copy(directory_list+i,copy_dir)
#print directory_list+i
#os.rename(copy_dir+i,copy_dir+i.split("BBB")[1])
Traceback error is:
Traceback (most recent call last):
File "copy_SE_files.py", line 25, in <module> shutil.copy(direcotry_list+i,copydir)
File "C:\Python27\ArcGIS10.1\lib\shutil.py", line 116, in copy copyfile(src,dst)
File "C:\Python27\ArcGIS10.1\lib\shutil.py", line 82, in copyfile with open(dst, 'wb') as fdst:
IOError: [Errno 2] No such file or directory: 'C:/Users/myusername/Dropbox/NASA_HyspIRI_Project/Field_Data/Spectra/CVARS/April2014/LemonTrees/04172014_SE_LemonTreeCanopy/SE_Files/SpectraZZZCVARSZZZApril2014ZZZLemonTreesZZZZ04172014XXXSEXXXLemonTreeCanopyZZZSEXXXFilesBBBCVARS_na_LemonTrees_Bareground1_4deg_Refl_00355.sed'
Problem solved thanks to the keen eyes of stack overflow. Modified the line to read:
shutil.copy(directory_list+i,'\\\\?\\'+os.path.abspath(copy_dir))

You're failing because the combined length of the path is greater than Window's MAX_PATH limit. C:/Users/myusername/Dropbox/NASA_HyspIRI_Project/Field_Data/Spectra/CVARS/April2014/LemonTrees/04172014_SE_LemonTreeCanopy/SE_Files/SpectraZZZCVARSZZZApril2014ZZZLemonTreesZZZZ04172014XXXSEXXXLemonTreeCanopyZZZSEXXXFilesBBBCVARS_na_LemonTrees_Bareground1_4deg_Refl_00355.sed is 274 characters long, and without going to some trouble, most Windows file manipulation APIs won't work properly with a path longer than MAX_PATH (which is 260, and one of them is reserved for the NUL terminator).
Assuming Python uses the correct APIs, you can make it work with the extended path prefix, \\?\ (and it may require you to use backslashes rather than forward slashes in your path; I'm not clear on that; read the docs).

The first thing that jumped out at me was this line:
shutil.copy(directory_list+i,copy_dir)
Consider changing it to
shutil.copy(os.path.join(directory_list,i),copy_dir)
IOW, use os.path.join() when concatenating file paths.

One work around is:
try:
shutil.copy(src, dest)
except:
try:
shutil.copy(src, "\\\\?\\" + dest)
#If Long Path as per Maximum Path limitation Windows
except:
self.failed_TC=True
print("Failed to move the script "+os.path.basename(src)+" to "+dest)

Related

FileNotFoundError: [Errno 2] - Possible work network issue?

I am trying to read content of a file on my work network from my work network. I copy and pasted a code snippet from a google search and modified it to the below. Why might I still be getting [Errno 2] (I have changed some of the path names for this question board)
The file path in my file explorer shows that "> This PC > word common common" and I don't have "This PC" in my path. I tried adding that into the place I would think it goes in the string. That didn't solve it.
I tried making sure I have matching capitalization. That didn't solve it.
I tried renaming the file to have a *.txt on the end. That didn't solve it.
I tried the different variations of // and / and \ with and without the r predecessor and while that did eliminate the first error I was getting. It didn't help this error.
(Looking at the code errors in the right gutter is says my line length is greater than the PEP8 standard. While I doubt that is the root of my problem, if you can throw in the 'right' wrap method for a file path that long that would be helpful.)
myfile = open("z:/abcdefg/abc123_proj2/word_general/word common common/Users/Mariee/Python/abc_abc_ab_Full_Report_12345-1_R9999_962019_9246", "rt") # open lorem.txt for reading text
contents = myfile.read() # read the entire file into a string
myfile.close() # close the file
print(contents) # print contents
Full Error Copy:
C:\Users\e087680\PycharmProjects\FailureCompiling\venv\Scripts\python.exe C:/Users/e087680/PycharmProjects/FailureCompiling/FirstScriptAttempt.py
Traceback (most recent call last):
File "C:/Users/e087680/PycharmProjects/FailureCompiling/FirstScriptAttempt.py", line 1, in
myfile = open("z:/abcdefg/abc123_proj2/word_general/word common common/Users/Mariee/Python/abc_abc_ab_Full_Report_12345-1_R9999_962019_9246", "rt") # open lorem.txt for reading text
FileNotFoundError: [Errno 2] No such file or directory: 'z:/abcdefg/abc123_proj2/word_general/word common common/Users/Mariee/Python/abc_abc_ab_Full_Report_12345-1_R9999_962019_9246'
EDIT
DEBUG EFFORTS
working to figure out how to change directory. Just in case that is the problem. Tested this code bit
import os
path = "z:/abcdefg/abc123_proj2/word_general/word common common/Users/Mariee/Python/abc_abc_ab_Full_Report_12345-1_R9999_962019_9246"
os.chdir(path)
isExist = os.path.exists(path)
print(isExist)
Received this error
C:\Users\e087680\PycharmProjects\FailureCompiling\venv\Scripts\python.exe C:/Users/e087680/PycharmProjects/FailureCompiling/ScriptDebugJunkFile.py
Traceback (most recent call last):
File "C:/Users/e087680/PycharmProjects/FailureCompiling/ScriptDebugJunkFile.py", line 5, in <module>
os.chdir(path)
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'z:/abcdefg/abc123_proj2/word_general/word common common/Users/Mariee/Python/abc_abc_ab_Full_Report_12345-1_R9999_962019_9246'
My intention for adding the picture below is to show how File Explorer displays the file path for my file
FileExplorerPathScreenShot
EDIT
I think this confirms that my 'OS' doesn't have my file.
from os import path
path.exists("PCC_ESS_FC_Full_Report_65000122-1_R0016_962019_9246")
def main():
print ("File exists:" + str(path.exists('PCC_ESS_FC_Full_Report_65000122-1_R0016_962019_9246')))
if __name__== "__main__":
main()
Output
File exists: False
I thought OS was a standard variable for Operating system. Now I'm not sure.
EDIT
Using Cmd in DOS, I confirmed that my path for the z: is correct
EDIT - Success
I ran
import os
print( os.listdir("z:/"))
Confirmed I don't need the monster string of folders.
Confirmed, although explorer doesn't show it, it is a *.txt file
Once I implemented these two items the first code worked fine.
Thank you #Furas
To open and read a file specify the filename in your path:
myfile = open("U:/matrix_neo/word common common/hello world.txt", "rt") # open file
contents = myfile.read() # read the entire file into a string
myfile.close() # close the file
print(contents) # print contents
The U: is a mapped drive in my network.
I did not find any issue with your change dir example. I used a path on my U: path again and it returned True.
import os
path = "U:/matrix_neo/word common common"
os.chdir(path)
isExist = os.path.exists(path)
print(isExist)
The check the attributes on the directory that you are trying to read from. Also try to copy the file to a local drive for a test and see if you can read the file and also check if it exists.
This is an alternative to the above and uses your path to make sure that the long file path works:
import os
mypath = "z:/abcdefg/abc123_proj2/word_general/word common common/Users/Mariee/Python/abc_abc_ab_Full_Report_12345-1_R9999_962019_9246"
myfile = 'whatever is your filename.txt'
if not os.path.isdir(mypath):
os.makedirs (mypath)
file_path = os.path.join(mypath, myfile)
print(file_path)
if os.path.exists(file_path) is True:
with open(file_path) as filein:
contents = filein.read()
print(contents)
I tested this code using a long csv file.,Replace the variable myfile with whatever is your file name.

How to deal with OSError: [Errno 20] Not a directory: '.DS_Store'?

I want to copy some pictures from on directory to another directory, and here is my code:
import os.path
import shutil
def copyFile(sourceDir,targetDir):
for files in os.listdir(sourceDir):
sourceFile=os.path.join(sourceDir,files)
if os.path.isfile(sourceFile) and sourceFile.find('.jpg')>0:
shutil.copy(sourceFile,targetDir)
for i in os.listdir('/Users/liuchong/Desktop/LFW/new'):
copyFile(i,'/Users/liuchong/Desktop/LFW/lfw')
But when I run it ,the terminal tells me thatOSError: [Errno 20] Not a directory: '.DS_Store'
I know 'DS_dstore' is a hidden file in Mac, but how can I solve this bug?
It is worth to mention what is the general meaning of the "Not a directory" error (Errno 20). It means that you are trying to operate with a path which sub-path is a real file and not a directory. That is it is a malformed, incorrect path.
Example: file.txt/test.txt where file.txt is an existing true file (not a directory).
Python will return this error for every I/O operation with such a file:
$ python -c 'import os; os.path.getsize("file.txt/test.txt");'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python2.7/genericpath.py", line 57, in getsize
return os.stat(filename).st_size
OSError: [Errno 20] Not a directory: 'file.txt/test.txt'
The OS will also return this error:
$ stat file.txt/test.txt
stat: cannot stat 'file.txt/test.txt': Not a directory
Every time you have this error it means that you concatenate file+file somewhere in your program.
This error is Unix OS specific. On Windows you should get "File not found" for such malformed path.
Your logic seems badly broken. You iterate through all the files in the directory, passing each one to copyFile. But inside that function, you again try to iterate through each file in the "directory" passed to the function: except that you're not passing only directories to the function, you're passing each file found in the original directory.
It's not clear what you are trying to do, but I think you need to remove one of those calls to listdir and the associated loop.
When you do os.listdir('/Users/liuchong/Desktop/LFW/lfw'), it returns both directories and files. You have to check the entry for directory before passing it as an argument.
def copyFile(sourceDir,targetDir):
for files in os.listdir(sourceDir):
sourceFile=os.path.join(sourceDir,files)
if os.path.isfile(sourceFile) and sourceFile.find('.jpg')>0:
shutil.copy(sourceFile,targetDir)
for i in os.listdir('/Users/liuchong/Desktop/LFW/lfw'):
if os.path.isdir(i):
i = os.path.join( '/Users/liuchong/Desktop/LFW/lfw', i)
copyFile(i,'/Users/liuchong/Desktop/LFW/lfw')
os.listdir('/Users/liuchong/Desktop/LFW/lfw')
returns a list which contains the names of entries in the directory that is given by the user. List is based on random choices.
To address this issue (I know I am late) here's a python 3 compatible code:-
import time
import os
from shutil import copyfile
def Copyfile(in_path, out_path, quantity = None):
filelist = os.listdir(in_path)
counter = 0
print("Copying to %r directory. Please wait... " % out_path)
start = time.time()
for i in range(quantity):
copyfile(filelist[i], out_path+filelist[i])
counter+=1
sys.stdout.write("\rTotal images copied : %r" % counter)
sys.stdout.flush()
end = time.time()
print("This move operation took %r seconds to run." %(end - start))

Renaming a single file in python

I'm trying to rename an audio file but I keep getting OSError: [Errno 2] No such file or directory.
In my program, each user has a directory that holds the users files. I obtain the path for each user by doing the following:
current_user_path = os.path.join(current_app.config['UPLOAD_FOLDER'], user.username)
/Users/johnsmith/Documents/pythonprojects/project/files/john
Now I want to obtain the path for the existing file I want to rename:
current_file_path = os.path.join(current_user_path,form.currentTitle.data)
/Users/johnsmith/Documents/pythonprojects/project/files/john/File1
The path for the rename file will be:
new_file_path = os.path.join(current_user_path, form.newTitle.data)
/Users/johnsmith/Documents/pythonprojects/project/files/john/One
Now I'm just running the rename command:
os.rename(current_file_path, new_file_path)
you can use os.rename for rename single file.
to avoid
OSError: [Errno 2] No such file or directory.
check if file exist or not.
here is the working example:
import os
src = "D:/test/Untitled003.wav"
dst = "D:/test/Audio001.wav"
if os.path.isfile(src):
os.rename(src, dst)
If the OS says there's no such file or directory, that's the gospel truth. You're making a lot of assumptions about where the file is, constructing a path to it, and renaming it. It's a safe bet there's no such file as the one named by current_file_path, or no directory to new_file_path.
Try os.stat(current_file_path), and similarly double-check the new file path with os.stat(os.posixpath.dirname(new_file_path)). Once you've got them right, os.rename will work if you've got permissions.
Try changing the current working directory to the one you want to work with. This code below should give you a simple walk through of how you should go about it:
import os
print (os.getcwd())
os.chdir(r"D:\Python\Example")
print (os.getcwd())
print ("start")
def rename_files():
file_list= os.listdir(r"D:\Python\Example")
print(file_list)
for file_name in file_list:
os.rename(file_name,file_name.translate(None,"0123456789")) rename_files()
print("stop")
print (os.getcwd())

Python. IOError: [Errno 13] Permission denied: when i'm copying file

I have two folders: In, Out - it is not system folder on disk D: - Windows 7. Out contain "myfile.txt" I run the following command in python:
>>> shutil.copyfile( r"d:\Out\myfile.txt", r"D:\In" )
Traceback (most recent call last):
File "<pyshell#39>", line 1, in <module>
shutil.copyfile( r"d:\Out\myfile.txt", r"D:\In" )
File "C:\Python27\lib\shutil.py", line 82, in copyfile
with open(dst, 'wb') as fdst:
IOError: [Errno 13] Permission denied: 'D:\\In'
What's the problem?
Read the docs:
shutil.copyfile(src, dst)
Copy the contents (no metadata) of the file named src to a file
named dst. dst must be the complete target file name; look at copy()
for a copy that accepts a target directory path.
use
shutil.copy instead of shutil.copyfile
example:
shutil.copy(PathOf_SourceFileName.extension,TargetFolderPath)
Use shutil.copy2 instead of shutil.copyfile
import shutil
shutil.copy2('/src/dir/file.ext','/dst/dir/newname.ext') # file copy to another file
shutil.copy2('/src/file.ext', '/dst/dir') # file copy to diff directory
I solved this problem, you should be the complete target file name for destination
destination = pathdirectory + filename.*
I use this code fir copy wav file with shutil :
# open file with QFileDialog
browse_file = QFileDialog.getOpenFileName(None, 'Open file', 'c:', "wav files (*.wav)")
# get file name
base = os.path.basename(browse_file[0])
os.path.splitext(base)
print(os.path.splitext(base)[1])
# make destination path with file name
destination= "test/" + os.path.splitext(base)[0] + os.path.splitext(base)[1]
shutil.copyfile(browse_file[0], destination)
First of all, make sure that your files aren't locked by Windows, some applications, like MS Office, locks the oppened files.
I got erro 13 when i was is trying to rename a long file list in a directory, but Python was trying to rename some folders that was at the same path of my files. So, if you are not using shutil library, check if it is a directory or file!
import os
path="abc.txt"
if os.path.isfile(path):
#do yor copy here
print("\nIt is a normal file")
Or
if os.path.isdir(path):
print("It is a directory!")
else:
#do yor copy here
print("It is a file!")
Visual Studio 2019
Solution : Administrator provided full Access to this folder "C:\ProgramData\Docker"
it is working.
ERROR: File IO error seen copying files to volume: edgehubdev. Errno: 13, Error Permission denied : [Errno 13] Permission denied: 'C:\ProgramData\Docker\volumes\edgehubdev\_data\edge-chain-ca.cert.pem'
[ERROR]: Failed to run 'iotedgehubdev start -d "C:\Users\radhe.sah\source\repos\testing\AzureIotEdgeApp1\config\deployment.windows-amd64.json" -v' with error: WARNING! Using --password via the CLI is insecure. Use --password-stdin.
ERROR: File IO error seen copying files to volume: edgehubdev. Errno: 13, Error Permission denied : [Errno 13] Permission denied: 'C:\ProgramData\Docker\volumes\edgehubdev\_data\edge-chain-ca.cert.pem'
use
> from shutil import copyfile
>
> copyfile(src, dst)
for src and dst use:
srcname = os.path.join(src, name)
dstname = os.path.join(dst, name)
This works for me:
import os
import shutil
import random
dir = r'E:/up/2000_img'
output_dir = r'E:/train_test_split/out_dir'
files = [file for file in os.listdir(dir) if os.path.isfile(os.path.join(dir, file))]
if len(files) < 200:
# for file in files:
# shutil.copyfile(os.path.join(dir, file), dst)
pass
else:
# Amount of random files you'd like to select
random_amount = 10
for x in range(random_amount):
if len(files) == 0:
break
else:
file = random.choice(files)
shutil.copyfile(os.path.join(dir, file), os.path.join(output_dir, file))
Make sure you aren't in (locked) any of the the files you're trying to use shutil.copy in.
This should assist in solving your problem
I avoid this error by doing this:
Import lib 'pdb' and insert 'pdb.set_trace()' before 'shutil.copyfile', it would just like this:
import pdb
...
print(dst)
pdb.set_trace()
shutil.copyfile(src,dst)
run the python file in a terminal, it will execute to the line 'pdb.set_trace()', and now the 'dst' file will print out.
copy the 'src' file by myself, and substitute and remove the 'dst' file which has been created by the above code.
Then input 'c' and click the 'Enter' key in the terminal to execute the following code.
well the questionis old, for new viewer of Python 3.6
use
shutil.copyfile( "D:\Out\myfile.txt", "D:\In" )
instead of
shutil.copyfile( r"d:\Out\myfile.txt", r"D:\In" )
r argument is passed for reading file not for copying

Python IOError exception when creating a long file

I get an IOError shown below when trying to open a new file using "open (fname, 'w+')". The complete error message is below.
The file does not exist, but I verified using "os.access(dir_name, os.W_OK)" and "os.path.exists (dir_name)" that the parent directory for the file does exist.
I am wondering if the file name is just too long for Windows, or if I am doing something wrong. Any tips would be appreciated. Thank you very much.
Error message:
IOError: [Errno 2] No such file or directory: 'C:\Documents and
Settings\Administrator\op_models\Corp_Network_Nov12\abcde_corporate_nov_12.project\abcde_corporate_nov_12-ctr.rptd.dir\ctr\Non
Business Hours for
Weeknights\hourly_data_for_2_weeks\1294897740\json.data\Link\0\Link
Utilization\analyzer393146160-data0.js'
In the Windows API the maximum path length is limited to 260 characters.
http://msdn.microsoft.com/en-us/library/aa365247%28v=vs.85%29.aspx
Update: prepend "\\?\" to the path.
Here is some related code which works for me (I have very long file names and paths):
for d in os.walk(os.getcwd()):
dirname = d[0]
files = d[2]
for f in files:
long_fname = u"\\\\?\\" + os.getcwd() + u"\\" + dirname + u"\\" + f
if op.isdir(long_fname):
continue
fin = open(long_fname, 'rb')
...
Note that for me it only worked with a combination of all of the following:
Prepend '\\?\' at the front.
Use full path, not relative path.
Use only backslashes.
In Python, the filename string must be a unicode string, for example u"abc", not "abc".
Also note, for some reason os.walk(..) returned some of the directories as files, so above I check for that.
You can monkey patch the tarfile module with this:
import tarfile
def monkey_patch_tarfile():
import os
import sys
if sys.platform not in ['cygwin', 'win32']:
return
def long_open(name, *args, **kwargs):
# http://msdn.microsoft.com/en-us/library/aa365247%28v=vs.85%29.aspx#maxpath
if len(name) >= 200:
if not os.path.isabs(name):
name = os.path.join(os.getcwd(), name)
name = "\\\\?\\" + os.path.normpath(name)
return long_open.bltn_open(name, *args, **kwargs)
long_open.bltn_open = tarfile.bltn_open
tarfile.bltn_open = long_open
monkey_patch_tarfile()
If it's not the length of the filename, it's the contents of the filename...
Python is treating '\12' as a control sequence.
>>> fn='C:\Documents and Settings\Administrator\op_models\Corp_Network_Nov12\abcde_corporate_nov_12.project\abcde_corporate_nov_12-ctr.rptd.dir\ctr\Non Business Hours for Weeknights\hourly_data_for_2_weeks\1294897740\json.data\Link\0\Link Utilization\analyzer393146160-data0.js'
>>> print fn
C:\Documents and Settings\Administrator\op_models\Corp_Network_Nov12bcde_corporate_nov_12.projectbcde_corporate_nov_12-ctr.rptd.dir\ctr\Non Business Hours for Weeknights\hourly_data_for_2_weeks
94897740\json.data\Link\Link Utilizationnalyzer393146160-data0.js
Using raw strings for Windows filenames will help:
>>> fn=r'C:\Documents and Settings\Administrator\op_models\Corp_Network_Nov12\abcde_corporate_nov_12.project\abcde_corporate_nov_12-ctr.rptd.dir\ctr\Non Business Hours for Weeknights\hourly_data_for_2_weeks\1294897740\json.data\Link\0\Link Utilization\analyzer393146160-data0.js'
>>> print fn
C:\Documents and Settings\Administrator\op_models\Corp_Network_Nov12\abcde_corporate_nov_12.project\abcde_corporate_nov_12-ctr.rptd.dir\ctr\Non Business Hours for Weeknights\hourly_data_for_2_weeks\1294897740\json.data\Link\0\Link Utilization\analyzer393146160-data0.js
Update
Alternatively, use forward slashes '/' instead of backslashes '\', since these will work on all operating systems and will save you hassles with backslashes right at the end of a pathname as in your comments.
See also os.path.join() .
Update 2
Simplified demonstration of problem:
>>> open('.\12\n\r\file.txt')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IOError: [Errno 2] No such file or directory: '.\n\n\r\x0cile.txt'
>>> open('./12/n/r/file.txt')
<open file './12/n/r/file.txt', mode 'r' at 0x7ff83f98>
C:\Users\johnysweb>copy .\12\n\r\file.txt con
Blah
1 file(s) copied.
Check the length of the entire path and then append the necessary Windows long path format. It should be noted that this doesn't work for accessing data from remote directories i.e. paths that begin with '\\some_remote_location\..' so you will need to map that directory locally in order to get "long path" to work.
if len(path_and_file) > 250: #I think the max is 260 but I left a buffer :)
path_and_file = '\\\\?\\'+path_and_file

Categories

Resources