Folder A has more than 100 files, folder B is my destination folder. I want to copy 10 files in folder A to folder B. The 10 files names are in the text file C.
import os
import shutil
from glob import glob
namelist = open('/Users/C.txt').read().splitlines()
input = '/Users/A'
output = '/Users/B'
path = '/Users/A'
files = glob(path)
for path in files:
filedir, filename = os.path.split(path)
for filename in namelist:
shutil.copy2(input,output)
It returns an Error. Please help me to do it in Python, thanks a lot!
There are a lot of things that you can do with your code:
import os
import shutil
from glob import glob
#namelist = open('/Users/C.txt').read().splitlines()
# context manager will take care of closing the file after open
# no need read as one string and do splitlines, readlines take care of that
with open('/Users/C.txt') as fp:
namelist = fp.readlines()
input = '/Users/A'
output = '/Users/B'
path = '/Users/A'
files = os.listdir(path)
# dont need glob import as you already imported os
#files = glob(path)
# loop only through files mentioned in the text file and see if they are available in
# folder A
for file_name in namelist:
file_path = os.path.join(input,file_name)
if file_path in files:
dest_path = os.path.join(output,file_name)
shutil.copy(file_path,dest_path)
#for path in files:
# filedir, filename = os.path.split(path)
# for filename in namelist:
# shutil.copy2(input,output)
I do not have sample data or error message to check. From what i can see in your code,
for path in files:
filedir, filename = os.path.split(path)
if filename in namelist:
shutil.copy2(input,output)
Your paths are from the root folder because of the starting forward slash. Try putting a dot in front of them if the folders and files are relative to the location of your .py file or no preceding slash:
./Users/A or Users/A
I am trying to copy all jpeg files from a directory (with multiple subdirectories) to a single directory. There are multiple files with the same name, so I am trying to rename the files using the name of the parent directory. For example: c:\images\tiger\image_00001.jpg will be moved to a new folder and renamed to c:\images\allimages\tiger_image_00001.jpg. I tried the code below, but nothing happens. The folder gets created, but the files do not move. This is what I have so far:
import os
path = 'source/'
os.mkdir('source/allimages/')
extensions = ['.jpeg']
for folder, _, filenames in os.walk(path):
for filename in filenames:
if folder == path or folder == os.path.join(path, 'allimages'):
continue
folder = folder.strip(path)
extension = os.path.splitext(os.path.splitext(filename)[0])[-1].lower()
if extension in extensions:
infilename = os.path.join(path, folder, filename)
newname = os.path.join(path, 'all_files', "{}-{}".format(folder.strip('./')))
os.rename(infilename, newname)
I would recommend having a function dedicated to resolving a unique filename. A while loop should do the trick. This should work.
import os
import shutil
def resolve_path(filename, destination_dir):
dest = os.path.join(destination_dir, filename)
*base_parts, extension = filename.split('.')
base_name = '.'.join(base_parts)
duplicate_num = 1
while os.path.exists(dest):
new_base = base_name + str(duplicate_num).zfill(5)
new_filename = "{}.{}".format(new_base, extension)
dest = os.path.join(destination_dir, new_filename)
duplicate_num += 1
return dest
That is such that the following is the result....
>>> with open('/path/to/file.extension', 'w') as f:
>>> pass # just create the file
>>> resolve_path('file.extension', '/path/to/')
'/path/to/file00001.extension'
Then put it together with traversing the source...
def consolidate(source, destination, extension='.jpg'):
if not os.path.exists(destination):
os.makedirs(destination)
for root, dirs, files in os.walk(source):
for f in files:
if f.lower().endswith(extension):
source_path = os.path.join(root, f)
destination_path = resolve_path(f, destination)
shutil.copyfile(source_path, destination_path)
You're calling splitext on its own output, which doesn't get what you want:
In [4]: os.path.splitext(os.path.splitext('foo.bar')[0])[-1]
Out[4]: ''
You just want extension = os.path.splitext(filename)[-1].lower(), or if you don't want the dot, then extension = os.path.splitext(filename)[-1].lower()[1:].
(Edited) more seriously, there's a problem with folder.strip(path): this will remove all characters in path from folder. For instance 'source/rescue'.strip('source/') == ''. What you want is folder.replace(path, '').
I'd like to change the files whose extension are '.test.txt' into '.txt'.
As my codes as below, it cannot work cause invalid syntax happened to the place of 'if'.
Could you please figure out it?
Thank you so much.
import sys
import os
path = "Dir"
for(dirpath,dirnames,files)in os.walk(path):
for filename in files:
filepath = os.path.join(dirpath,filename)
if '.test.txt' in filename:
newfilename = filename.replace('.test.txt','.txt')
os.rename(filename,newfilename)
this should work...
import sys
import os
path = r"Dir"
for dirpath,dirnames,files in os.walk(path):
for filename in files:
filepath = os.path.join(dirpath,filename)
if '.test.txt' in filename:
newfilename = filename.replace('.test.txt','.txt')
newfilepath = os.path.join(dirpath, newfilename)
os.rename(filepath, newfilepath)
you did not define the new file path, in renaming action you have to supply the full file path, os.rename(src_path, dest_path)
I've been trying to use the zipfile and shutil.make_archive modules to recursively create a zip file of a directory. Both modules work great--except empty directories do not get added to the archive. Empty directories containing other empty directories are also silently skipped.
I can use 7Zip to create an archive of the same path and empty directories are preserved. Therefore I know this is possible within the file format itself. I just don't know how to do it within Python. Any ideas? Thanks!
There is a example using zipfile:
import os, zipfile
from os.path import join
def zipfolder(foldername, filename, includeEmptyDIr=True):
empty_dirs = []
zip = zipfile.ZipFile(filename, 'w', zipfile.ZIP_DEFLATED)
for root, dirs, files in os.walk(foldername):
empty_dirs.extend([dir for dir in dirs if os.listdir(join(root, dir)) == []])
for name in files:
zip.write(join(root ,name))
if includeEmptyDIr:
for dir in empty_dirs:
zif = zipfile.ZipInfo(join(root, dir) + "/")
zip.writestr(zif, "")
empty_dirs = []
zip.close()
if __name__ == "__main__":
zipfolder('test1/noname/', 'zip.zip')
You'll need to register a new archive format to do that, since the default ZIP archiver does not support that. Take a look at the meat of the existing ZIP archiver. Make your own archiver that creates directories using that currently-unused dirpath variable. I looked for how to create an empty directory and found this:
zip.writestr(zipfile.ZipInfo('empty/'), '')
With that, you should be able to write the necessary code to make it archive empty directories.
This is lifted from Adding folders to a zip file using python but is the only function I have tried that works. The one listed as the answer does not work under Python 2.7.3 (doesn't copy empty directories and is inefficient). The following is tried and tested:
#!/usr/bin/python
import os
import zipfile
def zipdir(dirPath=None, zipFilePath=None, includeDirInZip=True):
if not zipFilePath:
zipFilePath = dirPath + ".zip"
if not os.path.isdir(dirPath):
raise OSError("dirPath argument must point to a directory. "
"'%s' does not." % dirPath)
parentDir, dirToZip = os.path.split(dirPath)
#Little nested function to prepare the proper archive path
def trimPath(path):
archivePath = path.replace(parentDir, "", 1)
if parentDir:
archivePath = archivePath.replace(os.path.sep, "", 1)
if not includeDirInZip:
archivePath = archivePath.replace(dirToZip + os.path.sep, "", 1)
return os.path.normcase(archivePath)
outFile = zipfile.ZipFile(zipFilePath, "w",
compression=zipfile.ZIP_DEFLATED)
for (archiveDirPath, dirNames, fileNames) in os.walk(dirPath):
for fileName in fileNames:
filePath = os.path.join(archiveDirPath, fileName)
outFile.write(filePath, trimPath(filePath))
#Make sure we get empty directories as well
if not fileNames and not dirNames:
zipInfo = zipfile.ZipInfo(trimPath(archiveDirPath) + "/")
#some web sites suggest doing
#zipInfo.external_attr = 16
#or
#zipInfo.external_attr = 48
#Here to allow for inserting an empty directory. Still TBD/TODO.
outFile.writestr(zipInfo, "")
outFile.close()
def zip_dir(src_dir, dst_zip, *, skip_suffixes=None, dry=False):
import logging
from pathlib import Path
from os import walk
from tempfile import TemporaryDirectory
from zipfile import ZipFile, ZipInfo
_log = logging.getLogger(zip_dir.__name__)
_log.addHandler(logging.NullHandler())
_sep = 50 * "-"
skip_suffixes = skip_suffixes or []
src_dir, dst_zip = Path(src_dir), Path(dst_zip)
_log.info("zipping dir: '%s' to: '%s", str(src_dir), str(dst_zip))
if not src_dir.exists():
raise FileNotFoundError(str(src_dir))
if not src_dir.is_dir():
raise NotADirectoryError(str(src_dir))
if dst_zip.exists():
raise FileExistsError(str(dst_zip))
with TemporaryDirectory() as tmp_dir:
tmp_zip_path = Path(tmp_dir).joinpath(dst_zip.name)
with ZipFile(str(tmp_zip_path), mode="w") as zip_out:
for root, dirs, files in walk(src_dir):
root = Path(root)
for folder in dirs:
folder = root.joinpath(folder)
# add empty folders to the zip
if not list(folder.iterdir()):
_log.debug(_sep)
folder_name = f"{str(folder.relative_to(src_dir))}/"
_log.debug("empty dir: '%s'", folder_name)
if dry:
continue
zip_out.writestr(ZipInfo(folder_name), "")
for file in files:
file = root.joinpath(file)
_log.debug(_sep)
_log.debug("adding: '%s'", str(file))
should_skip = None
for suffix in file.suffixes:
if suffix in skip_suffixes:
should_skip = suffix
break
if should_skip:
_log.debug("skipped [%s]: %s", should_skip, str(file))
continue
arcname = str(file.relative_to(src_dir))
_log.debug("arcname: '%s'", arcname)
if dry:
continue
zip_out.write(str(file), arcname=arcname)
if not dry:
dst_zip.write_bytes(tmp_zip_path.read_bytes())
tmp_zip_path.unlink()
if __name__ == '__main__':
import logging
logging.basicConfig(level=logging.DEBUG, format="%(asctime)s | %(levelname)8s | %(module)25s:%(lineno)-5s | %(message)s")
zip_dir("/tmp/opera_profile", "opera_profile.zip", skip_suffixes=[".log"], dry=True)
I have a C++/Obj-C background and I am just discovering Python (been writing it for about an hour).
I am writing a script to recursively read the contents of text files in a folder structure.
The problem I have is the code I have written will only work for one folder deep. I can see why in the code (see #hardcoded path), I just don't know how I can move forward with Python since my experience with it is only brand new.
Python Code:
import os
import sys
rootdir = sys.argv[1]
for root, subFolders, files in os.walk(rootdir):
for folder in subFolders:
outfileName = rootdir + "/" + folder + "/py-outfile.txt" # hardcoded path
folderOut = open( outfileName, 'w' )
print "outfileName is " + outfileName
for file in files:
filePath = rootdir + '/' + file
f = open( filePath, 'r' )
toWrite = f.read()
print "Writing '" + toWrite + "' to" + filePath
folderOut.write( toWrite )
f.close()
folderOut.close()
Make sure you understand the three return values of os.walk:
for root, subdirs, files in os.walk(rootdir):
has the following meaning:
root: Current path which is "walked through"
subdirs: Files in root of type directory
files: Files in root (not in subdirs) of type other than directory
And please use os.path.join instead of concatenating with a slash! Your problem is filePath = rootdir + '/' + file - you must concatenate the currently "walked" folder instead of the topmost folder. So that must be filePath = os.path.join(root, file). BTW "file" is a builtin, so you don't normally use it as variable name.
Another problem are your loops, which should be like this, for example:
import os
import sys
walk_dir = sys.argv[1]
print('walk_dir = ' + walk_dir)
# If your current working directory may change during script execution, it's recommended to
# immediately convert program arguments to an absolute path. Then the variable root below will
# be an absolute path as well. Example:
# walk_dir = os.path.abspath(walk_dir)
print('walk_dir (absolute) = ' + os.path.abspath(walk_dir))
for root, subdirs, files in os.walk(walk_dir):
print('--\nroot = ' + root)
list_file_path = os.path.join(root, 'my-directory-list.txt')
print('list_file_path = ' + list_file_path)
with open(list_file_path, 'wb') as list_file:
for subdir in subdirs:
print('\t- subdirectory ' + subdir)
for filename in files:
file_path = os.path.join(root, filename)
print('\t- file %s (full path: %s)' % (filename, file_path))
with open(file_path, 'rb') as f:
f_content = f.read()
list_file.write(('The file %s contains:\n' % filename).encode('utf-8'))
list_file.write(f_content)
list_file.write(b'\n')
If you didn't know, the with statement for files is a shorthand:
with open('filename', 'rb') as f:
dosomething()
# is effectively the same as
f = open('filename', 'rb')
try:
dosomething()
finally:
f.close()
If you are using Python 3.5 or above, you can get this done in 1 line.
import glob
# root_dir needs a trailing slash (i.e. /root/dir/)
for filename in glob.iglob(root_dir + '**/*.txt', recursive=True):
print(filename)
As mentioned in the documentation
If recursive is true, the pattern '**' will match any files and zero or more directories and subdirectories.
If you want every file, you can use
import glob
for filename in glob.iglob(root_dir + '**/**', recursive=True):
print(filename)
Agree with Dave Webb, os.walk will yield an item for each directory in the tree. Fact is, you just don't have to care about subFolders.
Code like this should work:
import os
import sys
rootdir = sys.argv[1]
for folder, subs, files in os.walk(rootdir):
with open(os.path.join(folder, 'python-outfile.txt'), 'w') as dest:
for filename in files:
with open(os.path.join(folder, filename), 'r') as src:
dest.write(src.read())
TL;DR: This is the equivalent to find -type f to go over all files in all folders below and including the current one:
for currentpath, folders, files in os.walk('.'):
for file in files:
print(os.path.join(currentpath, file))
As already mentioned in other answers, os.walk() is the answer, but it could be explained better. It's quite simple! Let's walk through this tree:
docs/
└── doc1.odt
pics/
todo.txt
With this code:
for currentpath, folders, files in os.walk('.'):
print(currentpath)
The currentpath is the current folder it is looking at. This will output:
.
./docs
./pics
So it loops three times, because there are three folders: the current one, docs, and pics. In every loop, it fills the variables folders and files with all folders and files. Let's show them:
for currentpath, folders, files in os.walk('.'):
print(currentpath, folders, files)
This shows us:
# currentpath folders files
. ['pics', 'docs'] ['todo.txt']
./pics [] []
./docs [] ['doc1.odt']
So in the first line, we see that we are in folder ., that it contains two folders namely pics and docs, and that there is one file, namely todo.txt. You don't have to do anything to recurse into those folders, because as you see, it recurses automatically and just gives you the files in any subfolders. And any subfolders of that (though we don't have those in the example).
If you just want to loop through all files, the equivalent of find -type f, you can do this:
for currentpath, folders, files in os.walk('.'):
for file in files:
print(os.path.join(currentpath, file))
This outputs:
./todo.txt
./docs/doc1.odt
The pathlib library is really great for working with files. You can do a recursive glob on a Path object like so.
from pathlib import Path
for elem in Path('/path/to/my/files').rglob('*.*'):
print(elem)
import glob
import os
root_dir = <root_dir_here>
for filename in glob.iglob(root_dir + '**/**', recursive=True):
if os.path.isfile(filename):
with open(filename,'r') as file:
print(file.read())
**/** is used to get all files recursively including directory.
if os.path.isfile(filename) is used to check if filename variable is file or directory, if it is file then we can read that file.
Here I am printing file.
If you want a flat list of all paths under a given dir (like find . in the shell):
files = [
os.path.join(parent, name)
for (parent, subdirs, files) in os.walk(YOUR_DIRECTORY)
for name in files + subdirs
]
To only include full paths to files under the base dir, leave out + subdirs.
I've found the following to be the easiest
from glob import glob
import os
files = [f for f in glob('rootdir/**', recursive=True) if os.path.isfile(f)]
Using glob('some/path/**', recursive=True) gets all files, but also includes directory names. Adding the if os.path.isfile(f) condition filters this list to existing files only
For my taste os.walk() is a little too complicated and verbose. You can do the accepted answer cleaner by:
all_files = [str(f) for f in pathlib.Path(dir_path).glob("**/*") if f.is_file()]
with open(outfile, 'wb') as fout:
for f in all_files:
with open(f, 'rb') as fin:
fout.write(fin.read())
fout.write(b'\n')
use os.path.join() to construct your paths - It's neater:
import os
import sys
rootdir = sys.argv[1]
for root, subFolders, files in os.walk(rootdir):
for folder in subFolders:
outfileName = os.path.join(root,folder,"py-outfile.txt")
folderOut = open( outfileName, 'w' )
print "outfileName is " + outfileName
for file in files:
filePath = os.path.join(root,file)
toWrite = open( filePath).read()
print "Writing '" + toWrite + "' to" + filePath
folderOut.write( toWrite )
folderOut.close()
os.walk does recursive walk by default. For each dir, starting from root it yields a 3-tuple (dirpath, dirnames, filenames)
from os import walk
from os.path import splitext, join
def select_files(root, files):
"""
simple logic here to filter out interesting files
.py files in this example
"""
selected_files = []
for file in files:
#do concatenation here to get full path
full_path = join(root, file)
ext = splitext(file)[1]
if ext == ".py":
selected_files.append(full_path)
return selected_files
def build_recursive_dir_tree(path):
"""
path - where to begin folder scan
"""
selected_files = []
for root, dirs, files in walk(path):
selected_files += select_files(root, files)
return selected_files
I think the problem is that you're not processing the output of os.walk correctly.
Firstly, change:
filePath = rootdir + '/' + file
to:
filePath = root + '/' + file
rootdir is your fixed starting directory; root is a directory returned by os.walk.
Secondly, you don't need to indent your file processing loop, as it makes no sense to run this for each subdirectory. You'll get root set to each subdirectory. You don't need to process the subdirectories by hand unless you want to do something with the directories themselves.
Try this:
import os
import sys
for root, subdirs, files in os.walk(path):
for file in os.listdir(root):
filePath = os.path.join(root, file)
if os.path.isdir(filePath):
pass
else:
f = open (filePath, 'r')
# Do Stuff
If you prefer an (almost) Oneliner:
from pathlib import Path
lookuppath = '.' #use your path
filelist = [str(item) for item in Path(lookuppath).glob("**/*") if Path(item).is_file()]
In this case you will get a list with just the paths of all files located recursively under lookuppath.
Without str() you will get PosixPath() added to each path.
This worked for me:
import glob
root_dir = "C:\\Users\\Scott\\" # Don't forget trailing (last) slashes
for filename in glob.iglob(root_dir + '**/*.jpg', recursive=True):
print(filename)
# do stuff
If just the file names are not enough, it's easy to implement a Depth-first search on top of os.scandir():
stack = ['.']
files = []
total_size = 0
while stack:
dirname = stack.pop()
with os.scandir(dirname) as it:
for e in it:
if e.is_dir():
stack.append(e.path)
else:
size = e.stat().st_size
files.append((e.path, size))
total_size += size
The docs have this to say:
The scandir() function returns directory entries along with file attribute information, giving better performance for many common use cases.