Develop Reference

Python cannot find exiting path

While preparing my dataset for a machine learning model I stumbled upon a bug (at least from my perspective). I was trying to move files around in Windows 10 but my system always told me that my file does not exist (FileNotFoundError: [Errno 2] No such file or directory: '../path/to/file'), but I specifically probed for the file before which returned True.
What I was trying in my IPython console was the following (which can reproduce the bug):
import os
import shutil
path = '../path/to/file' # You can see I use relative paths
out_path = '../path/to/out/'
os.makedirs(out_path) # Make sure paths do exist
_, name = os.path.split(path)
out_path += name
# Produces almost the same error FileNotFoundError: [WinError 3] The system cannot find the given directory: path
# os.rename(path, out_path)
shutil.move(path, out_path)
You obviously need a file to move at '../path/to/file'. I tested whether I use the correct path with os.path.exist(path) which returned True.
Do I simply misread the documentation and the usage is different to what I am thinking? I understood it like the first argument is the path to the file to move and the second one is the path for the file after moving it.
Or is there really a bug in my Python environment?
I also tried moving the file with absolute paths returned by os.path.abspath(), but this did not work either.
FYI: I am using Windows 10 and Python 3.8.12.
Edit: I found the mistake I did... After 2.5 days of labeling image data it appears like I had an underscore mistaken for a hyphen which is why my output directory seems to be malformed... :/

You are not doing a relative path to the current directory, but to the parent directory of the current directory with a double dot '../path' and not single dot './path'. So if your files aren't in the parent directory of the current directory, they won't be found.
When you try to append the file name to the directory path "out_path" you do a string append. which just makes a new file with the combined names rather than move the file into the directory. Use the os.path.join() function for this.
Here is code I made that corrects these two points and works on my end.
import os
import shutil
path = './somefile.txt' # CORRECTED: One dot for current directory
out_path = './somedir'
try:
os.makedirs(out_path) # Make sure paths do exist
except Exception as e:
print(e)
_, name = os.path.split(path)
out_path = os.path.join(out_path, name) # CORRECTED: Use appropriate function
# Produces almost the same error FileNotFoundError: [WinError 3] The system cannot find the given directory: path
# os.rename(path, out_path)
shutil.move(path, out_path)

Rename Single .txt File In Multiple Subdirectories

I have ~60 subdirectories in a single directory. Each of these contain thousands of files, but they all contain a file named test_all_results.txt.
What I would like to do is to rename each test_all_results.txt file so that it now has the name:
foldername_all_results.txt
What is the best way to do this?

Easily accomplished using Python os interface.
Assuming you are currently in the main directory:
import os
#get a list of all sub directories
subdir = os.listdir()
for dir in subdir:
if os.path.isdir(dir): #check if directory
os.chdir(dir) #move to sub directory
os.rename('test_all_results.txt', 'foldername_all_results.txt')
os.chdir('..') #return to main directory

Using python in Linux, make this:
import os
os.system("mv old_name.txt new_name.txt")
You can automatize with a loop, renaming all filenames.

You can do:
(change your code accordingly)
import os
# current directory is the target
direct = "."
for path, dirs, files in os.walk(direct):
for f in files:
if os.path.splitext(f)[0] == "test_all_results.txt":
os.rename(os.path.join(path, f), os.path.join(path, "foldername_all_results.txt"))

There's an answer that tells you to use the os.system() method, if you do decide to call Linux commands from Python, I'd advise that you use the subprocess module instead.
Here's how you'd run the mv command with two arguments using subprocess.call:
import subprocess
subprocess.call(["mv", "filename.txt", "new-name.txt"])
INFO: here's an old (but relevant) article that explains why it's dangerous to use these methods.
Good luck.

How can I delete multiple files using a Python script?

I am playing around with some python scripts and I ran into a problem with the script I'm writing. It's supposed to find all the files in a folder that meets the criteria and then delete it. However, it finds the files, but at the time of deleting the file, it says that the file is not found.
This is my code:
import os
for filename in os.listdir('C:\\New folder\\'):
if filename.endswith(".rdp"):
os.unlink(filename)
And this is the error I get after running it:
FileNotFoundError: [WinError 2] The system cannot find the file specified:
Can somebody assist with this?

os.unlink takes the path to the file, not only its filename. Try pre-pending your filename with the dirname. Like this
import os
dirname = 'C:\\New folder\\'
for filename in os.listdir(dirname):
if filename.endswith(".rdp"):
# Add your "dirname" to the file path
os.unlink(dirname + filename)

You could alternatively use os.walk, however it might go deeper than you want:
import os
for root, sub, file in os.walk("/media/"):
if file.endswith(".rdp"):
os.unlink(f'{root}/{file}')

nested directory appear while creating zip file

I am quite new to python.Here i am trying to create zip file of "diveintomark-diveintopython3-793871b' directory.I changed the current working directory using os.chdir() function.The zip file is created but the problem is when i extract the zip file i get the the following directory
Users/laiba/Desktop/diveintomark-diveintopython3-793871b
but i only want diveintomark-diveintopython3-793871b folder inside my zip folder not the whole nested directory created .Why is this happening and how i can solve this?
import zipfile, os
os.chdir('c:\\Users\\laiba\\Desktop')
myzip=zipfile.ZipFile('diveZip.zip','w',zipfile.ZIP_DEFLATED)
for folder,subfolder,file in os.walk('diveintomark-diveintopython3-793871b'):
myzip.write(folder)
for each in subfolder:
myzip.write(os.path.abspath(os.path.join(folder,each)))
for each in file:
myzip.write(os.path.abspath(os.path.join(folder,each)))

you could use argument arcname: name of the item in the archive as opposed to the full path name. But here you don't need it because you already are in the correct directory. Just drop the abspath and you're done (and also the duplicate folder entry)
import zipfile, os
os.chdir('c:\\Users\\laiba\\Desktop')
myzip=zipfile.ZipFile('diveZip.zip','w',zipfile.ZIP_DEFLATED)
for folder,subfolder,file in os.walk('diveintomark-diveintopython3-793871b'):
for each in subfolder+file:
myzip.write(os.path.join(folder,each))
myzip.close()
This is possible to do without changing directories but more complex, also more elegant since you don't have to chdir
import zipfile, os
root_dir = r"c:\Users\laiba\Desktop"
myzip=zipfile.ZipFile(os.path.join(root_dir,'diveZip.zip'),'w',zipfile.ZIP_DEFLATED)
for folder,subfolder,file in os.walk(os.path.join(root_dir,'diveintomark-diveintopython3-793871b')):
for each in subfolder+file:
source = os.path.join(folder,each)
# remove the absolute path to compose arcname
# also handles the remaining leading path separator with lstrip
arcname = source[len(root_dir):].lstrip(os.sep)
# write the file under a different name in the archive
myzip.write(source,arcname=arcname)
myzip.close()

Error while using listdir in Python

I'm trying to get the list of files in a particular directory and count the number of files in the directory. I always get the following error:
WindowsError: [Error 3] The system cannot find the path specified: '/client_side/*.*'
My code is:
print len([name for name in os.listdir('/client_side/') if os.path.isfile(name)])
I followed the code example given here.
I am running the Python script on Pyscripter and the directory /client_side/ do exists. My python code is in the root folder and has a sub-folder called "client_side". Can someone help me out on this?

This error occurs when you use os.listdir on a path which does not refer to an existing path.
For example:
>>> os.listdir('Some directory does not exist')
Traceback (most recent call last):
File "<interactive input>", line 1, in <module>
WindowsError: [Error 3] : 'Some directory does not exist/*.*'
If you want to use os.listdir, you need to either guarantee the existence of the path that you would use, or use os.path.exists to check the existence first.
if os.path.exists('/client_side/'):
do something
else:
do something
Suppose your current working directory is c:\foobar, os.listdir('/client_side/') is equivalent to os.listdir('c:/client_side'), while os.listdir('client_side/') is equivalent to os.listdir('c:/foobar/client_side'). If your client_side directory is not in the root, such error will occur when using os.listdir.
For your 0 ouput problem, let us recall os.listdir(path)
Return a list containing the names of the entries in the directory given by path. The list is in arbitrary order. It does not include the special entries '.' and '..' even if they are present in the directory.
and os.path.isfile(path).
Return True if path is an existing regular file. This follows symbolic links, so both islink() and isfile() can be true for the same path.
listdir returns neither the absolute paths nor relative paths, but a list of the name of your files, while isfile requires path. Therefore, all of those names would yield False.
To obtain the path, we can either use os.path.join , concat two strings directly.
print ([name for name in os.listdir(path)
if os.path.isfile(os.path.join(path, name))])
Or
print ([name for name in os.listdir('client_side/')
if os.path.isfile('client_side/' + name)])

I decided to change the code into:
def numOfFiles(path):
return len(next(os.walk(path))[2])
and use the following the call the code:
print numOfFiles("client_side")
Many thanks to everyone who told me how to pass the windows directory correctly in Python and to nrao91 in here for providing the function code.
EDIT: Thank you eryksun for correcting my code!

Two things:
os.listdir() does not do a glob pattern matching, use the glob module for that
probably you do not have a directory called '/client_side/*.*', but maybe one
without the . in the name
The syntax you used works fine, if the directory you look for exists, but there is no directory called '/client_side/.'.
In addition, be careful if using Python 2.x and os.listdir, as the results on windows are different when you use u'/client_side/' and just '/client_side'.

You can do just
os.listdir('client_side')
without slashes.

As I can see a WindowsError, Just wondering if this has something to do with the '/' in windows ! Ideally, on windows, you should have something like os.path.join('C:','client_side')

You want:
print len([name for name in os.listdir('./client_side/') if os.path.isfile(name)])
with a "." before "/client_side/".
The dot means the current path where you are working (i.e. from where you are calling your code), so "./client_side/" represents the path you want, which is specified relatively to your current directory.
If you write only "/client_side/", in unix, the program would look for a folder in the root of the system, instead of the folder that you want.

If you just want to see all the files in the directory where your script is located, you can use os.path.dirname(sys.argv[0]). This will give the path of the directory where your script is.
Then, with fnmatch function you can obtain the list of files in that directory with a name and/or extension specified in the filenamevariable.
import os,sys
from fnmatch import fnmatch
directory = os.path.dirname(sys.argv[0]) #this determines the directory
file_name= "*" #if you want the python files only, change "*" to "*.py"
for path, subdirs, files in os.walk(directory):
for name in files:
if fnmatch(name, file_name):
print (os.path.join(path, name))
I hope this helps.

Checking for existence is subject to a race. Better to handle the error (beg forgiveness instead of ask permission). Plus, in Python 3 you can suppress errors. Use suppress from contextlib:
with suppress(FileNotFoundError):
for name in os.listdir('foo'):
print(name)