Develop Reference

How to suppress a folder or a file with os module [duplicate]

How can I delete a file or folder?

os.remove() removes a file.
os.rmdir() removes an empty directory.
shutil.rmtree() deletes a directory and all its contents.
Path objects from the Python 3.4+ pathlib module also expose these instance methods:
pathlib.Path.unlink() removes a file or symbolic link.
pathlib.Path.rmdir() removes an empty directory.

Python syntax to delete a file
import os
os.remove("/tmp/<file_name>.txt")
or
import os
os.unlink("/tmp/<file_name>.txt")
or
pathlib Library for Python version >= 3.4
file_to_rem = pathlib.Path("/tmp/<file_name>.txt")
file_to_rem.unlink()
Path.unlink(missing_ok=False)
Unlink method used to remove the file or the symbolik link.
If missing_ok is false (the default), FileNotFoundError is raised if the path does not exist.
If missing_ok is true, FileNotFoundError exceptions will be ignored (same behavior as the POSIX rm -f command).
Changed in version 3.8: The missing_ok parameter was added.
Best practice
First, check if the file or folder exists and then delete it. You can achieve this in two ways:
os.path.isfile("/path/to/file")
Use exception handling.
EXAMPLE for os.path.isfile
#!/usr/bin/python
import os
myfile = "/tmp/foo.txt"
# If file exists, delete it.
if os.path.isfile(myfile):
os.remove(myfile)
else:
# If it fails, inform the user.
print("Error: %s file not found" % myfile)
Exception Handling
#!/usr/bin/python
import os
# Get input.
myfile = raw_input("Enter file name to delete: ")
# Try to delete the file.
try:
os.remove(myfile)
except OSError as e:
# If it fails, inform the user.
print("Error: %s - %s." % (e.filename, e.strerror))
Respective output
Enter file name to delete : demo.txt
Error: demo.txt - No such file or directory.
Enter file name to delete : rrr.txt
Error: rrr.txt - Operation not permitted.
Enter file name to delete : foo.txt
Python syntax to delete a folder
shutil.rmtree()
Example for shutil.rmtree()
#!/usr/bin/python
import os
import sys
import shutil
# Get directory name
mydir = raw_input("Enter directory name: ")
# Try to remove the tree; if it fails, throw an error using try...except.
try:
shutil.rmtree(mydir)
except OSError as e:
print("Error: %s - %s." % (e.filename, e.strerror))

Use
shutil.rmtree(path[, ignore_errors[, onerror]])
(See complete documentation on shutil) and/or
os.remove
and
os.rmdir
(Complete documentation on os.)

Here is a robust function that uses both os.remove and shutil.rmtree:
def remove(path):
""" param <path> could either be relative or absolute. """
if os.path.isfile(path) or os.path.islink(path):
os.remove(path) # remove the file
elif os.path.isdir(path):
shutil.rmtree(path) # remove dir and all contains
else:
raise ValueError("file {} is not a file or dir.".format(path))

You can use the built-in pathlib module (requires Python 3.4+, but there are backports for older versions on PyPI: pathlib, pathlib2).
To remove a file there is the unlink method:
import pathlib
path = pathlib.Path(name_of_file)
path.unlink()
Or the rmdir method to remove an empty folder:
import pathlib
path = pathlib.Path(name_of_folder)
path.rmdir()

Deleting a file or folder in Python
There are multiple ways to Delete a File in Python but the best ways are the following:
os.remove() removes a file.
os.unlink() removes a file. it is a Unix name of remove() method.
shutil.rmtree() deletes a directory and all its contents.
pathlib.Path.unlink() deletes a single file The pathlib module is available in Python 3.4 and above.
os.remove()
Example 1: Basic Example to Remove a File Using os.remove() Method.
import os
os.remove("test_file.txt")
print("File removed successfully")
Example 2: Checking if File Exists using os.path.isfile and Deleting it With os.remove
import os
#checking if file exist or not
if(os.path.isfile("test.txt")):
#os.remove() function to remove the file
os.remove("test.txt")
#Printing the confirmation message of deletion
print("File Deleted successfully")
else:
print("File does not exist")
#Showing the message instead of throwig an error
Example 3: Python Program to Delete all files with a specific extension
import os
from os import listdir
my_path = 'C:\Python Pool\Test\'
for file_name in listdir(my_path):
if file_name.endswith('.txt'):
os.remove(my_path + file_name)
Example 4: Python Program to Delete All Files Inside a Folder
To delete all files inside a particular directory, you simply have to use the * symbol as the pattern string.
#Importing os and glob modules
import os, glob
#Loop Through the folder projects all files and deleting them one by one
for file in glob.glob("pythonpool/*"):
os.remove(file)
print("Deleted " + str(file))
os.unlink()
os.unlink() is an alias or another name of os.remove() . As in the Unix OS remove is also known as unlink.
Note: All the functionalities and syntax is the same of os.unlink() and os.remove(). Both of them are used to delete the Python file path.
Both are methods in the os module in Python’s standard libraries which performs the deletion function.
shutil.rmtree()
Example 1: Python Program to Delete a File Using shutil.rmtree()
import shutil
import os
# location
location = "E:/Projects/PythonPool/"
# directory
dir = "Test"
# path
path = os.path.join(location, dir)
# removing directory
shutil.rmtree(path)
Example 2: Python Program to Delete a File Using shutil.rmtree()
import shutil
import os
location = "E:/Projects/PythonPool/"
dir = "Test"
path = os.path.join(location, dir)
shutil.rmtree(path)
pathlib.Path.rmdir() to remove Empty Directory
Pathlib module provides different ways to interact with your files. Rmdir is one of the path functions which allows you to delete an empty folder. Firstly, you need to select the Path() for the directory, and then calling rmdir() method will check the folder size. If it’s empty, it’ll delete it.
This is a good way to deleting empty folders without any fear of losing actual data.
from pathlib import Path
q = Path('foldername')
q.rmdir()

How do I delete a file or folder in Python?
For Python 3, to remove the file and directory individually, use the unlink and rmdir Path object methods respectively:
from pathlib import Path
dir_path = Path.home() / 'directory'
file_path = dir_path / 'file'
file_path.unlink() # remove file
dir_path.rmdir() # remove directory
Note that you can also use relative paths with Path objects, and you can check your current working directory with Path.cwd.
For removing individual files and directories in Python 2, see the section so labeled below.
To remove a directory with contents, use shutil.rmtree, and note that this is available in Python 2 and 3:
from shutil import rmtree
rmtree(dir_path)
Demonstration
New in Python 3.4 is the Path object.
Let's use one to create a directory and file to demonstrate usage. Note that we use the / to join the parts of the path, this works around issues between operating systems and issues from using backslashes on Windows (where you'd need to either double up your backslashes like \\ or use raw strings, like r"foo\bar"):
from pathlib import Path
# .home() is new in 3.5, otherwise use os.path.expanduser('~')
directory_path = Path.home() / 'directory'
directory_path.mkdir()
file_path = directory_path / 'file'
file_path.touch()
and now:
>>> file_path.is_file()
True
Now let's delete them. First the file:
>>> file_path.unlink() # remove file
>>> file_path.is_file()
False
>>> file_path.exists()
False
We can use globbing to remove multiple files - first let's create a few files for this:
>>> (directory_path / 'foo.my').touch()
>>> (directory_path / 'bar.my').touch()
Then just iterate over the glob pattern:
>>> for each_file_path in directory_path.glob('*.my'):
... print(f'removing {each_file_path}')
... each_file_path.unlink()
...
removing ~/directory/foo.my
removing ~/directory/bar.my
Now, demonstrating removing the directory:
>>> directory_path.rmdir() # remove directory
>>> directory_path.is_dir()
False
>>> directory_path.exists()
False
What if we want to remove a directory and everything in it?
For this use-case, use shutil.rmtree
Let's recreate our directory and file:
file_path.parent.mkdir()
file_path.touch()
and note that rmdir fails unless it's empty, which is why rmtree is so convenient:
>>> directory_path.rmdir()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "~/anaconda3/lib/python3.6/pathlib.py", line 1270, in rmdir
self._accessor.rmdir(self)
File "~/anaconda3/lib/python3.6/pathlib.py", line 387, in wrapped
return strfunc(str(pathobj), *args)
OSError: [Errno 39] Directory not empty: '/home/username/directory'
Now, import rmtree and pass the directory to the funtion:
from shutil import rmtree
rmtree(directory_path) # remove everything
and we can see the whole thing has been removed:
>>> directory_path.exists()
False
Python 2
If you're on Python 2, there's a backport of the pathlib module called pathlib2, which can be installed with pip:
$ pip install pathlib2
And then you can alias the library to pathlib
import pathlib2 as pathlib
Or just directly import the Path object (as demonstrated here):
from pathlib2 import Path
If that's too much, you can remove files with os.remove or os.unlink
from os import unlink, remove
from os.path import join, expanduser
remove(join(expanduser('~'), 'directory/file'))
or
unlink(join(expanduser('~'), 'directory/file'))
and you can remove directories with os.rmdir:
from os import rmdir
rmdir(join(expanduser('~'), 'directory'))
Note that there is also a os.removedirs - it only removes empty directories recursively, but it may suit your use-case.

This is my function for deleting dirs. The "path" requires the full pathname.
import os
def rm_dir(path):
cwd = os.getcwd()
if not os.path.exists(os.path.join(cwd, path)):
return False
os.chdir(os.path.join(cwd, path))
for file in os.listdir():
print("file = " + file)
os.remove(file)
print(cwd)
os.chdir(cwd)
os.rmdir(os.path.join(cwd, path))

shutil.rmtree is the asynchronous function,
so if you want to check when it complete, you can use while...loop
import os
import shutil
shutil.rmtree(path)
while os.path.exists(path):
pass
print('done')

import os
folder = '/Path/to/yourDir/'
fileList = os.listdir(folder)
for f in fileList:
filePath = folder + '/'+f
if os.path.isfile(filePath):
os.remove(filePath)
elif os.path.isdir(filePath):
newFileList = os.listdir(filePath)
for f1 in newFileList:
insideFilePath = filePath + '/' + f1
if os.path.isfile(insideFilePath):
os.remove(insideFilePath)

For deleting files:
os.unlink(path, *, dir_fd=None)
or
os.remove(path, *, dir_fd=None)
Both functions are semantically same. This functions removes (deletes) the file path. If path is not a file and it is directory, then exception is raised.
For deleting folders:
shutil.rmtree(path, ignore_errors=False, onerror=None)
or
os.rmdir(path, *, dir_fd=None)
In order to remove whole directory trees, shutil.rmtree() can be used. os.rmdir only works when the directory is empty and exists.
For deleting folders recursively towards parent:
os.removedirs(name)
It remove every empty parent directory with self until parent which has some content
ex. os.removedirs('abc/xyz/pqr') will remove the directories by order 'abc/xyz/pqr', 'abc/xyz' and 'abc' if they are empty.
For more info check official doc: os.unlink , os.remove, os.rmdir , shutil.rmtree, os.removedirs

To remove all files in folder
import os
import glob
files = glob.glob(os.path.join('path/to/folder/*'))
files = glob.glob(os.path.join('path/to/folder/*.csv')) // It will give all csv files in folder
for file in files:
os.remove(file)
To remove all folders in a directory
from shutil import rmtree
import os
// os.path.join() # current working directory.
for dirct in os.listdir(os.path.join('path/to/folder')):
rmtree(os.path.join('path/to/folder',dirct))

To avoid the TOCTOU issue highlighted by Éric Araujo's comment, you can catch an exception to call the correct method:
def remove_file_or_dir(path: str) -> None:
""" Remove a file or directory """
try:
shutil.rmtree(path)
except NotADirectoryError:
os.remove(path)
Since shutil.rmtree() will only remove directories and os.remove() or os.unlink() will only remove files.

My personal preference is to work with pathlib objects - it offers a more pythonic and less error-prone way to interact with the filesystem, especially if You develop cross-platform code.
In that case, You might use pathlib3x - it offers a backport of the latest (at the date of writing this answer Python 3.10.a0) Python pathlib for Python 3.6 or newer, and a few additional functions like "copy", "copy2", "copytree", "rmtree" etc ...
It also wraps shutil.rmtree:
$> python -m pip install pathlib3x
$> python
>>> import pathlib3x as pathlib
# delete a directory tree
>>> my_dir_to_delete=pathlib.Path('c:/temp/some_dir')
>>> my_dir_to_delete.rmtree(ignore_errors=True)
# delete a file
>>> my_file_to_delete=pathlib.Path('c:/temp/some_file.txt')
>>> my_file_to_delete.unlink(missing_ok=True)
you can find it on github or PyPi
Disclaimer: I'm the author of the pathlib3x library.

I recommend using subprocess if writing a beautiful and readable code is your cup of tea:
import subprocess
subprocess.Popen("rm -r my_dir", shell=True)
And if you are not a software engineer, then maybe consider using Jupyter; you can simply type bash commands:
!rm -r my_dir
Traditionally, you use shutil:
import shutil
shutil.rmtree(my_dir)

python yaml path after deployment

So this is a question about how to handle settings files and relative paths in python (probably also something about best practice).
So I have coded a smaller project that i want to deploy to a docker image and everything is set up now except when I try to run the python task (Through cron) I get the error: settings/settings.yml not found.
tree .
├───settings
│ └───settings/settings.yml
└───main.py
And am referencing the yml file as
open('settings/settings.yml', 'r') as f:
config = yaml.load(f, Loader=yaml.FullLoader)
I can see this is what is causing the problem but am unsure about how to fix it. I wish to reference the main file basically by using the entry_points from setuptools in the future so my quick fix with cd'ing before python main.py will not be a lasting solution.

Instead of hardcoding a path as a string, you can find the directories and build the file path with os.path. For example:
import os
import yaml
current_dir = os.path.dirname(os.path.abspath(__file__))
settings_dir = os.path.join(current_dir, "settings")
filename = "settings.yml"
settings_path = os.path.join(settings_dir, filename)
with open(settings_path, "r") as infile:
settings_data = yaml.load(infile)
This way it can be run in any file system and the python file can be called from any directory.

python: robustely import module from working directory

I define parameters for my programs in a python module for simplicity. These parameters are then loaded using import. Therefore, I have to make sure to always load from the working directory, and nowhere else (independent form the location of the executed script or available modules in python path).
I found two solutions. First modifying the path:
import sys
from os import getcwd
import_path = sys.path
sys.path = [str(getcwd(), ]
import xxx
sys.path = import_path
or using importlib
from pathlib import Path
from importlib.util import module_from_spec, spec_from_file_location
spec = spec_from_file_location('xxx', str(Path('.').expanduser()/'xxx.py'))
xxx = module_from_spec(spec)
spec.loader.exec_module(xxx)
Of course this can be wrapped into a context manager or a function respectively.
What would the pythonic way be to do this? Do those two approaches have advantages and disadvantages?
I checked How can I import a Python library located in the current working directory?
as well as Import python package from local directory into interpreter, but they lack the focus on robustness.

Local imports can be achieved by modifying the path. A context manager is a suitable solution:
import sys
from pathlib import Path
from contextlib import contextmanager
#contextmanager
def local_import(dir_=None):
"""Only import modules within `dir_` (default: cwd)."""
if dir_ is None:
dir_ = Path.cwd()
else:
dir_ = Path(dir_).absolute().resolve(strict=True)
import_path0 = sys.path[0]
sys.path[0] = str(dir_)
try:
yield
finally:
sys.path[0] = import_path0
Then the local import can be done using the standard import syntax
with local_import():
import xxx
This solution relys on the order, in which the paths are scanned, thus we temporarily replace sys.path[0]. We replace it, instead of prepending to avoid import conflicts with the script directory.
Note
You have to be careful to avoid name conflicts, as the import statement is used, modules with identical names will be imported only once. Thus if different modules with the same name exist in the working directory and in the original sys.path[0], only one of them will be imported. Thus, local_import should only be used for scripts, that only use the standard library or installed third party libraries, but not for scripts that import other scripts from the directory. For the unlikely case that you want to import different files with the same name, the following function can be used:
import uuid
from importlib.util import module_from_spec, spec_from_file_location
def import_file(file, content=None):
"""Try importing `file` as module avoiding name clashes.
If `content` is given `content = import_file('file.py', 'content')`
roughly corresponds to `from file import content`
else `file = import_file('file.py')`
roughly corresponds to `import file`.
Parameters
----------
file : str or Path
The Python file corresponding to the module.
content : str, optional
What to import from the module (optional).
"""
file = Path(file).expanduser().resolve(strict=True)
print(file)
spec = spec_from_file_location(file.stem + str(uuid.uuid4()), str(file))
module = module_from_spec(spec)
spec.loader.exec_module(module)
if content:
print(module)
return getattr(module, content)
else:
return module

python import a module with source code in temporary file

import tempfile
tmp = tempfile.NamedTemporaryFile(delete=True)
try:
# do stuff with temp
tmp.write(b'def fun():\n\tprint("hello world!")\n')
if __name__ == '__main__':
func = __import__(tmp.name)
func.fun()
finally:
tmp.close() # deletes the file
So I want to create a temporary file, add some source code to it and then import the module and call the function, but I always run into this error:
ModuleNotFoundError: No module named '/var/folders/3w/yyp887lx4018h9s5sr0bwhkw0000gn/T/tmp84bk0bic'
It doesn't seem to find the module of the temporary file. How do I solve this?

There are a few problems with your code:
Your filename does not end with .py, but Python modules are expected to. You can fix this by setting suffix='.py' in NamedTemporaryFile().
__import__() is not the right way to load a module from a full path. See here: How to import a module given the full path?
You do not flush after writing and before importing, so even if Python does find the file, it may well be empty. Add tmp.flush() after writing to fix this.

Importing can only be done from certain directories which are part of the PYTHON_PATH. You can extend that. Then you will have to use __import__() with a module name (not a path in the file system). You will have to deal with the suffix for the temp file.
I implemented a simple version using the local directory for the temp module file and a version using a proper tempfile:
#!/usr/bin/env python3
import sys
import os
import tempfile
SCRIPT = '''\
def fun():
print("hello world!")
'''
# simple version using the local directory:
with open('bla.py', 'w') as tmp_module_file:
tmp_module_file.write(SCRIPT)
import bla
bla.fun()
# version using the tempfile module:
tmpfile = tempfile.NamedTemporaryFile(suffix='.py', delete=True)
try:
tmpfile.write(SCRIPT.encode('utf8'))
tmpfile.flush()
tmpmodule_path, tmpmodule_file_name = os.path.split(tmpfile.name)
tmpmodule_name = tmpmodule_file_name[:-3] # strip off the '.py'
sys.path.append(tmpmodule_path)
tmpmodule = __import__(tmpmodule_name)
finally:
tmpfile.close()
tmpmodule.fun()

Include .pyd module files in py2exe compilation

I'm trying to compile a python script. On executing the exe I got:-
C:\Python27\dist>visualn.exe
Traceback (most recent call last):
File "visualn.py", line 19, in <module>
File "MMTK\__init__.pyc", line 39, in <module>
File "Scientific\Geometry\__init__.pyc", line 30, in <module>
File "Scientific\Geometry\VectorModule.pyc", line 9, in <module>
File "Scientific\N.pyc", line 1, in <module>
ImportError: No module named Scientific_numerics_package_id
I can see the file Scientific_numerics_package_id.pyd at the location "C:\Python27\Lib\site-packages\Scientific\win32". I want to include this module file into the compilation. I tried to copy the above file in the "dist" folder but no good. Any idea?
Update:
Here is the script:
from MMTK import *
from MMTK.Proteins import Protein
from Scientific.Visualization import VRML2; visualization_module = VRML2
protein = Protein('3CLN.pdb')
center, inertia = protein.centerAndMomentOfInertia()
distance_away = 8.0
front_cam = visualization_module.Camera(position= [center[0],center[1],center[2]+distance_away],description="Front")
right_cam = visualization_module.Camera(position=[center[0]+distance_away,center[1],center[2]],orientation=(Vector(0, 1, 0),3.14159*0.5),description="Right")
back_cam = visualization_module.Camera(position=[center[0],center[1],center[2]-distance_away],orientation=(Vector(0, 1, 0),3.14159),description="Back")
left_cam = visualization_module.Camera(position=[center[0]-distance_away,center[1],center[2]],orientation=(Vector(0, 1, 0),3.14159*1.5),description="Left")
model_name = 'vdw'
graphics = protein.graphicsObjects(graphics_module = visualization_module,model=model_name)
visualization_module.Scene(graphics, cameras=[front_cam,right_cam,back_cam,left_cam]).view()

Py2exe lets you specify additional Python modules (both .py and .pyd) via the includes option:
setup(
...
options={"py2exe": {"includes": ["Scientific.win32.Scientific_numerics_package_id"]}}
)
EDIT. The above should work if Python is able to
import Scientific.win32.Scientific_numerics_package_id

There is a way to work around this types of issues that I have used a number of times. In order to add extra files to the py2exe result you can extend the media collector in order to have a custom version of it. The following code is an example:
import glob
from py2exe.build_exe import py2exe as build_exe
def get_py2exe_extension():
"""Return an extension class of py2exe."""
class MediaCollector(build_exe):
"""Extension that copies Scientific_numerics_package_id missing data."""
def _add_module_data(self, module_name):
"""Add the data from a given path."""
# Create the media subdir where the
# Python files are collected.
media = module_name.replace('.', os.path.sep)
full = os.path.join(self.collect_dir, media)
if not os.path.exists(full):
self.mkpath(full)
# Copy the media files to the collection dir.
# Also add the copied file to the list of compiled
# files so it will be included in zipfile.
module = __import__(module_name, None, None, [''])
for path in module.__path__:
for f in glob.glob(path + '/*'): # does not like os.path.sep
log.info('Copying file %s', f)
name = os.path.basename(f)
if not os.path.isdir(f):
self.copy_file(f, os.path.join(full, name))
self.compiled_files.append(os.path.join(media, name))
else:
self.copy_tree(f, os.path.join(full, name))
def copy_extensions(self, extensions):
"""Copy the missing extensions."""
build_exe.copy_extensions(self, extensions)
for module in ['Scientific_numerics_package_id',]:
self._add_module_data(module)
return MediaCollector
I'm not sure which is the Scientific_numerics_package_id module so I've assumed that you can import it like that. The copy extensions method will get a the different module names that you are having problems with and will copy all their data into the dir folder for you. Once you have that, in order to use the new Media collector you just have to do something like the following:
cmdclass['py2exe'] = get_py2exe_extension()
So that the correct extension is used. You might need to touch the code a little but this should be a good starting point for what you need.

I encountered similar probelm with py2exe and the only solution I can find ,is to use another tool to convert python to exe - pyinstaller
Its very easy tool to use and more important , it works!
UPDATE
As I understood from your comments below , running your script from command line is not working also , due to import error (My recommendation is to first check your code from command line ,and than try to convert it to EXE)
It looks like PYTHONPATH problem.
PYTHONPATH is list of paths (similar of Windows PATH) that python programs use to find import modules.
If your script run from your IDE , that means the PYTHONPATH is set correctly in the IDE ,so all imported modules are found.
In order to set PYTHONPATH you can use :
import sys|
sys.path.append(pathname)
or use the following code that add the all folders under path parameter to PYTHONPATH:
import os
import sys
def add_tree_to_pythonpath(path):
"""
Function: add_tree_to_pythonpath
Description: Go over each directory in path and add it to PYTHONPATH
Parameters: path - Parent path to start from
Return: None
"""
# Go over each directory and file in path
for f in os.listdir(path):
if f == ".bzr" or f.lower() == "dll":
# Ignore bzr and dll directories (optional to NOT include specific folders)
continue
pathname = os.path.join(path, f)
if os.path.isdir(pathname) == True:
# Add path to PYTHONPATH
sys.path.append(pathname)
# It is a directory, recurse into it
add_tree_to_pythonpath(pathname)
else:
continue
def startup():
"""
Function: startup
Description: Startup actions needed before call to main function
Parameters: None
Return: None
"""
parent_path = os.path.normpath(os.path.join(os.getcwd(), ".."))
parent_path = os.path.normpath(os.path.join(parent_path, ".."))
# Go over each directory in parent_path and add it to PYTHONPATH
add_tree_to_pythonpath(parent_path)
# Start the program
main()
startup()

The ImportError is rectified by using "Gil.I" and "Janne Karila" suggestion by setting pythonpath and by using include function. But before this I had to create __init__.py file in the win32 folder of both the modules.
BTW I still got another error for the above script - link