I'm getting an error while trying to copy files from a single source directory which contains bug fixes: /home/saurabh/testbed/patch_dir/ to multiple destination directories: app_dir_1 and app_dir_2.
Both these directories are exact replicas of each other.
The script does the following:-
Read lines from a text file into a list. Each line contains the name of one component. In this case: ['file1.class', file2.html]
Search value at each index recursively, starting from a particular directory:
/home/saurabh/testbed/dest_dir/
Take a backup of these files wherever they are found by appending ddMonyyyy to their extension.
Copy files from directory which contains patched components: /home/saurabh/testbed/patch_dir/
to the directory where backup was taken earlier
Directory Overview:-
/home/saurabh/testbed/dest_dir/
|--app_dir_1
|--file1.class
|--file2.html
|--file3.jsp
|--file4.xml
|--sub_dir
|--app_dir_2
|--file1.class
|--file2.html
|--file3.jsp
|--file4.xml
|--sub_dir
|--other_directories
/home/saurabh/testbed/patch_dir/
|--file1.class
|--file2.html
Below is my code:
#!/usr/bin/env python
import os
import fnmatch
import datetime
import shutil
with open('filenames.txt') as f:
content = f.readlines()
content = [x.strip() for x in content]
print('File contents:')
print(content)
suffix = datetime.datetime.now().strftime("_%d%b%Y")
approot = '/home/saurabh/testbed/dest_dir/'
source_dir = '/home/saurabh/testbed/patch_dir/'
dir_list = []
print('\n' + 'Renaming files present at:')
for root, dirs, files in os.walk(approot):
for file_list in content:
for filename in fnmatch.filter(files, file_list):
print(os.path.join(root, filename))
dir_list.append(root)
current_file = os.path.join(root, filename)
backup_file = os.path.join(root, filename + suffix)
os.rename(current_file, backup_file)
print("\n" + "Backup of all files complete!")
print('Backup of ' + str(len(dir_list)) + ' files taken recursively')
print('Number of files mentioned in text file: ' + str(len(content)) + '\n')
# 2 instances in UAT
# 12 instances in PROD
if (2*len(content)) == len(dir_list):
print("Retrofitted components will be copied to their respective directories")
for dst_ind in range(0, len(dir_list)):
if filename in fnmatch.filter(files, file_list):
print(source_dir + content[dst_ind] + "\t" + dir_list[dst_ind])
#shutil.copy2(source_dir+content[dst_ind], dir_list[dst_ind])
I'm getting the below error while copying the files (4.)
File contents:
['file1.class', 'file2.html']
Renaming files present at:
/home/saurabh/testbed/dest_dir/app_dir_1/file1.class
/home/saurabh/testbed/dest_dir/app_dir_1/file2.html
/home/saurabh/testbed/dest_dir/app_dir_2/file1.class
/home/saurabh/testbed/dest_dir/app_dir_2/file2.html
Backup of all files complete!
Backup of 4 files taken recursively
Number of files mentioned in text file: 2
Retrofitted components will be copied to their respective directories
/home/saurabh/testbed/patch_dir/file1.class /home/saurabh/testbed/dest_dir/app_dir_1
/home/saurabh/testbed/patch_dir/file2.html /home/saurabh/testbed/dest_dir/app_dir_1
Traceback (most recent call last):
File "./prod_movement.py", line 56, in <module>
print(source_dir + content[dst_ind] + "\t" + dir_list[dst_ind])
IndexError: list index out of range
Expected Output:
File contents:
['file1.class', 'file2.html']
Renaming files present at:
/home/saurabh/testbed/dest_dir/app_dir_1/file1.class
/home/saurabh/testbed/dest_dir/app_dir_1/file2.html
/home/saurabh/testbed/dest_dir/app_dir_2/file1.class
/home/saurabh/testbed/dest_dir/app_dir_2/file2.html
Backup of all files complete!
Backup of 4 files taken recursively
Number of files mentioned in text file: 2
Retrofitted components will be copied to their respective directories
/home/saurabh/testbed/patch_dir/file1.class /home/saurabh/testbed/dest_dir/app_dir_1
/home/saurabh/testbed/patch_dir/file2.html /home/saurabh/testbed/dest_dir/app_dir_1
/home/saurabh/testbed/patch_dir/file1.class /home/saurabh/testbed/dest_dir/app_dir_2
/home/saurabh/testbed/patch_dir/file2.html /home/saurabh/testbed/dest_dir/app_dir_2
Appreciate any help to fix the code.
Figured it out !!
#!/usr/bin/env python
import os
import fnmatch
import datetime
import shutil
with open('filenames.txt') as f:
content = f.readlines()
content = [x.strip() for x in content]
print('File contents:')
print(content)
suffix = datetime.datetime.now().strftime("_%d%b%Y")
approot = '/Users/saurabhm/Desktop/Python/testbed/dest_dir/'
source_dir = '/Users/saurabhm/Desktop/Python/testbed/patch_dir/'
dir_list = []
print('\n' + 'Renaming files present at:')
for root, dirs, files in os.walk(approot):
for file_list in content:
for filename in fnmatch.filter(files, file_list):
print(os.path.join(root, filename))
dir_list.append(root)
current_file = os.path.join(root, filename)
backup_file = os.path.join(root, filename + suffix)
#print(current_file, backup_file)
os.rename(current_file, backup_file)
#print(source_dir + filename + "\t" + root)
shutil.copy2(source_dir + filename, root)
os.chmod(root + '/' + filename, 0o750)
print("\n" + "Backup of all files complete!")
print('Backup of ' + str(len(dir_list)) + ' files taken recursively')
print('Number of files mentioned in text file: ' + str(len(content)) + '\n')
Once patched components are received over email, they are copied to a directory in the respective server. All these files are consolidated in a single directory (patch_dir)
Backup of existing files (having the same name and are present in "dest_dir") are taken wherever they are found, following which each file is copied from "patch_dir" to the directories inside "dest_dir", where their backup was taken.
Related
After copying files from a user provided source and destination, the files need to be renamed based on the creation time. The files have multiple extension types that need to be preserved while being renamed and a record original filenames. The issue arises when attempting to rename the file after it has been copied to the destination. All attempted solutions creates an exception in callback. What's the best approach here?
def filter_copy(self):
current_dir = self.Source.get()
destination_dir = self.Destination.get()
extensions = (['.avi', '.mpg'])
os.chdir(current_dir)
for root, dir, files in os.walk('.'):
for file in files:
if os.path.splitext(file)[1] in extensions:
src = os.path.join(root, file)
time.sleep(0.1)
logging.info("Copying the file %s..." % file)
print("Copying the file %s..." % file)
try:
with open('OriginalFilesReceipt.txt', 'w') as f:
f.write(file)
except FileNotFoundError:
logging.info("The directory does not exist to create to generate a receipt")
shutil.copy(src, destination_dir)
for names in os.listdir('.'):
t = os.path.getmtime(names)
v = datetime.datetime.fromtimestamp(t)
x = v.strftime('%Y%m%d-%H%M%S')
os.rename(names, x)
Found a solution based on the work from yzhong52 here: https://gist.github.com/tvdsluijs/1640613a32416b1197fc36b45e7b4999
The main error that was being received should have been obvious. The working directory wasn't the destination directory. The solution for the original question:
os.chdir(destination_dir)
for names in os.listdir('.'):
#split into filename and extension
filename, extension = os.path.splitext(names)
if (extension in extensions):
create_time = os.path.getctime(names)
format_time = datetime.datetime.fromtimestamp(create_time)
format_time_string = format_time.strftime("%Y-%m-%d %H.%M.%S")
newfile = format_time_string + extension
#if other file with same timestamp
if(newfile in newfilesDictionary.keys()):
index = newfilesDictionary[newfile] + 1
newfilesDictionary[newfile] = index
newfile = format_time_string + '-' + str(index) + extension
else:
newfilesDictionary[newfile] = 0
os.rename(names, newfile)
num_files = num_files + 1
print(names.rjust(35) + ' => ' + newfile.ljust(35))
import os
path = "G:\krunker\mod"
abcde = open("path.txt", "w")
for dirpath, dirnames, filenames in os.walk(path):
directory_level = dirpath.replace(path, "")
directory_level = directory_level.count(os.sep)
indent = " " * 4
print("{}{}/".format(indent*directory_level, os.path.basename(dirpath)), file=abcde)
for f in filenames:
print("{}{}".format(indent*(directory_level+1), f), file=abcde)
abcde.close()
I want it to print the files in every single folder of the path but it does only on the last
The indentation is not correct. The second for loop also has to be inside the first for loop.
Correct code:
import os
path = "/home/user/my_folder/tools"
abcde = open("path.txt", "w")
for dirpath, dirnames, filenames in os.walk(path):
directory_level = dirpath.replace(path, "")
directory_level = directory_level.count(os.sep)
indent = " " * 4
print("{}{}/".format(indent*directory_level, os.path.basename(dirpath)), file=abcde)
for f in filenames:
print("{}{}".format(indent*(directory_level+1), f), file=abcde)
abcde.close()
A part of path.txt content:
tools/
.gitignore
README.md
__init__.py
requirements3.txt
test.py
path.txt
.git/
description
hooks/
commit-msg.sample
info/
exclude
refs/
heads/
master
You can also use a recursive function which gets recursivly all subfolder's content until there's any subfolder:
from os import walk
output = open("path.txt", "w")
def listFiles(path, indent):
for (openedPath, folders, files) in walk(path):
for file in files:
output.write("\t" * (indent) + file + "\n")
for folder in folders:
output.write("\t" * (indent) + folder + "/\n")
listFiles(path + "/" + folder, indent + 1)
break
source = "/my/path/to/my/folder"
print(source + "/")
listFiles(source, 1)
There's an example with a little code project folder.
/my/path/to/my/folder/
input.txt
main.py
output/
error.cpp
trying.cpp
logo.cpp
You can use this kind of code to simplify
import os
folder = r"C:\path\to\find\files"
x = [os.path.join(r,file) for r,d,f in os.walk(folder) for file in f # If you want specific files if file.endswith(".txt")]
y = [os.path.join(r,folder) for r,d,f in os.walk(folder) for folder in d]
print(x) #For files in main directory and subdirectories
print(y) #For files in main directory and subdirectories
I have some textfiles in a folder and I want to search all of them for names between row 4 and 20 and then copy the ones containing one of those names to a different folder. With my code I only get an empty result file even though I know the keywords are in my folder. What could be the problem with this code for Python 3?
from os import system, listdir, path
import codecs
FILE = open('C:\\Users\\Admin\\Desktop\\Test\\Result.txt', 'w')
desktop_dir = path.join('C:\\Users\\Admin\\Desktop\\test\\')
for fn in listdir(desktop_dir):
fn_w_path = path.join(desktop_dir, fn)
if path.isfile(fn_w_path):
with open(fn_w_path, "r") as filee:
for line in filee.readlines():
for word in line.lower().split():
if word in {'James',
'Tim',
'Tom',
'Ian',
'William',
'Dennis',}:
FILE.write(word + "\n")
FILE.close()
import os
import shutil
for root, dirs, files in os.walk("test_dir1", topdown=False):
for name in files:
current_file = os.path.join(root, name)
destination = current_file.replace("test_dir1", "test_dir2")
print("Found file: %s" % current_file)
print("File copy to: %s" % destination)
shutil.copy(current_file, destination)
Currently trying to develop a catch all script that when pointed to a folder it will look through all of the subfolders and anything that is a .gdb will be moved to a new folder. Some GDBS are zipped and have a folder which then contains the GDB inside. If the GDB files have the same name once extracted, they will be appended with the suffix _2, _3, _4 and then moved to the folder with the other GDBs and won't conflict with any filenames.
Everything works except the renaming. Whereby it will rename one ok and then start placing folders inside of another gdb, before eventually failing with. WindowsError: [Error 183] Cannot create a file when that file already exists.
import shutil, errno, re, os, zipfile, os.path, sys
def unzip(source_filename, dest_dir):
with zipfile.ZipFile(source_filename) as zf:
for member in zf.infolist():
words = member.filename.split('/')
path = dest_dir
for word in words[:-1]:
drive, word = os.path.splitdrive(word)
head, word = os.path.split(word)
if word in (os.curdir, os.pardir, ''): continue
path = os.path.join(path, word)
zf.extract(member, path)
results = []
input_path = sys.argv[1]
output_path = sys.argv[2]
if input_path not in output_path:
for path, subdirs, files in os.walk(input_path):
for f in files:
if f.endswith(".zip") or f.endswith(".ZIP"):
unzip(os.path.join(path,f), path)
if input_path not in output_path:
for path, subdirs, files in os.walk(input_path):
counter = 2
for dir_name in subdirs:
if os.path.join(path, dir_name).endswith(".gdb") or os.path.join(path, dir_name).endswith(".GDB"):
if os.path.exists(output_path + "\\" + dir_name):
print "Moving " + os.path.join(path, dir_name) + " To " + output_path + "\\" + str(counter) + dir_name
os.rename(os.path.join(path, dir_name), output_path + "\\" + dir_name[:-4] + "_" + str(counter) + ".gdb")
counter + 1
else:
print "Moving " + os.path.join(path, dir_name) + " To " + output_path
shutil.move(os.path.join(path, dir_name), output_path)
else:
print "########################################"
print "########################################"
print "Please select a different output path"
print "The output path should not be contained"
print " In the input path"
print "########################################"
print "########################################"
Any help anyone could give would be greatly appreciated. Cheers!
I am trying to sort a large number of files based off of their file extension. A lot of the files are .doc, .docx, .xls, etc.
This is what I was thinking in my head, but if there is a simpler way to do things, let me know! I do have multiple files with the same extension, so I don't want it to create a new folder for that extension every time and overwrite the previous file. I also have a much larger list, but for this example I don't believe all of them are needed. The OS is MacOS.
import os, shutil
extList = ['.doc', '.docx', '.xls']
for ext in extList:
os.mkdir(path + '/' + ext +'_folder')
for file in os.listdir(filepath):
if file.endswith(ext): #missing an indent
print(file)
shutil.copyfile(file + '/' + ext +'_folder' + file)
Also, if I run into a file that I do not have on my list, I would like it to go into a folder named 'noextlist'.
Here is what I was able to create quickly
import os, re, shutil
DocFolder = r'...'#Your doc folder path
DocxFolder = r'...'#Your docx folder path
XlsFolder = r'...'#Your xls folder path
MiscFolder = r'...'#Your misc folder path
for root, dirs, files in os.walk(r'...'): #Your folder path you want to sort
for file in files:
if file.endswith(".doc"):
sourceFolder = os.path.join(root,file)
print sourceFolder
shutil.copy2(sourceFolder,DocFolder)
elif file.endswith(".docx"):
sourceFolder = os.path.join(root,file)
print sourceFolder
shutil.copy2(sourceFolder,DocxFolder)
elif file.endswith(".xls"):
sourceFolder = os.path.join(root,file)
print sourceFolder
shutil.copy2(sourceFolder,XlsFolder)
else:
sourceFolder = os.path.join(root,file)
print sourceFolder
shutil.copy2(sourceFolder,MiscFolder)
Edit:The main function here is the for root,dirs,files in os.walk This allows the program to transverse through the provided path to search all files including the ones in the sub folder and sort it out accordingly.
import errno
import shutil
from os import listdir, mkdir
from os.path import splitext, join
# set for fast lookup
extList = set(['.doc', '.docx', '.xls'])
# source path
filepath = ...
# dest path
path = ...
for f in listdir(filepath):
# extract extension from file name
ext = splitext(f)[1]
if ext in extList:
dir_ = join(path, "{}_folder".format(ext))
try:
mkdir(dir_)
except OSError as e:
if ex.errno != errno.EEXIST:
raise # raise if any other error than "already exists"
dest = join(dir_, f)
else:
dest = join(path, "noextlist_folder", f)
shutil.copy2(join(filepath, f), dest)
If I understand correctly, you like your solution but you need a way to rename files with duplicate names so that the extras don't disappear. You can check if the destination file already exists and construct a variant name by adding _1, _2, etc. to the filename until you find something unused.
newpathname = path + '/' + ext +'_folder' + "/" + file
n = 0
while os.path.exists(newpathname):
n += 1
base, ext = os.path.splitext(newpathname)
newpathname = "%s_%d%s" % (base, n, ext)
shutil.copyfile(filepath+"/"+file, newpathname)
But your code has some other glitches, so here's a rewritten scanner. It uses os.walk() to descend into several levels of subdirectories (you don't say if that's needed or not), and it collects files of all extensions in one pass. And it constructs variant names as before.
import os, shutil
extList = ['.doc', '.docx', '.xls']
from os.path import join as joinpath
# Make sure the destination directories exist
for ext in extList:
extdir = joinpath(path, ext[1:]+"_folder")
if not os.path.exists(extdir):
os.mkdir(extdir)
for dirname, _dirs, files in os.walk(filepath):
for file in files:
base, ext = os.path.splitext(file)
if ext not in extList:
continue
destpath = joinpath(path, ext[1:]+"_folder")
n = 0
newpathname = joinpath(destpath, file)
# If the new name is in use, find an unused variant
while os.path.exists(newpathname):
n += 1
newfile = "%s_%d%s" % (base, n, ext)
newpathname = joinpath(path, newfile)
sh.copy(joinpath(dirname, file), newpathname) # or other copy method