Develop Reference

Need help checking for multiple underscores in a collection of files - see code below

I am working on a project where I need to sort .jpg files and folders that contain .jpg files. I have other scripts that are functional which I intend to incorporate into this python script later. First though, I've implemented in the first script below to count the number of underscores in a file and take action based on the result and this works successfully. I need help on creating logic that will go through .jpg image files and if the files have more than one underscore the program will move the files into an error folder. Also any feedback on how to optimize this script would be greatly appreciated!
from pathlib import Path
import shutil, os, time, glob
timestr = time.strftime("%Y%m%d-%H%M%S")
folder = 'D:\\test\\testing'
working_folder = 'DitaTest1'
full_path = Path(os.path.join(folder, working_folder))
test_path = folder + '\\' + working_folder
for file_path in full_path.iterdir():
file_name = file_path.name
result = file_name.count('_')
if file_path.is_file():
os.chdir(test_path)
for file in glob.glob("*.jpg"):
dst=test_path+"\\"+file.replace(" ","_").replace(".jpg","") # .replace("Angle","").replace("Front","").replace("Side","")
os.mkdir(dst)
# print(dst)
shutil.move(file,dst)
elif result != 1:
if not file_path.is_file():
shutil.move(os.path.join(folder, working_folder, file_name), os.path.join(folder, working_folder + ' - dir-ERRORS_' + timestr, file_name))
else:
print('Ignored operation')

You need to explain more so that we can understand it better but from what I have read,
Your if logic seems to be wrong, if you want to check the number of underscores you shouldn't put that logic in elif. You should try sth like this instead.
for file_path in full_path.iterdir():
file_name = file_path.name
result = file_name.count('_')
if os.path.isdir(file_path):
pass
else:
if result == 1:
os.chdir(test_path)
for file in glob.glob("*.jpg"):
dst=test_path+"\\"+file.replace(" ","_").replace(".jpg","") # .replace("Angle","").replace("Front","").replace("Side","")
os.mkdir(dst)
# print(dst)
shutil.move(file,dst)
else:
shutil.move(os.path.join(folder, working_folder, file_name), os.path.join(folder, working_folder + ' - dir-ERRORS_' + timestr, file_name))
What this code does is, iterate over the folder and if it finds a folder it will just pass and when it finds a file it will check if result == 1. If it is it will move it to your desired folder, otherwise it will move it to the error folder. If I made a mistake let me know.

Uncompyle6 convert pyc to py file python 3 (Whole directory)

I have 200 pyc files I need to convert in a folder. I am aware of converting pyc to py files through uncompyle6 -o . 31.pyc however as I have so many pyc files, this would take a long period of time. I've founds lots of documentation but not much in bulk converting to py files. uncompyle6 -o . *.pyc was not supported.
Any idea on how I can achieve this?

Might not be perfect but it worked great for me.
import os
import uncompyle6
your_directory = ''
for dirpath, b, filenames in os.walk(your_directory):
for filename in filenames:
if not filename.endswith('.pyc'):
continue
filepath = dirpath + '/' + filename
original_filename = filename.split('.')[0]
original_filepath = dirpath + '/' + original_filename + '.py'
with open(original_filepath, 'w') as f:
uncompyle6.decompile_file(filepath, f)

This is natively supported by uncompyle6
uncompyle6 -ro <output_directory> <python_directory>
-r tells the tool to recurse into sub directories.
-o tells the tool to output to the given directory.

In operating systems with shell filename expansion, you might be able to use the shell's file expansion ability. For example:
uncompyle6 -o /tmp/unc6 myfiles/*.pyc
If you need something fancier or more control, you could always write some code that does the fancier expansion. Here is the above done in POSIX shell filtering out the single file myfiles/huge.pyc:
cd myfiles
for pyc in *.pyc; do
if [[ $pyc != huge.pyc ]] ; then
uncompyle -o /tmp/unc $pyc
fi
done
Note: It seems this question was also asked in Issue on output directory while executing commands with windows batch command "FOR /R"

thank you for the code, extending it to recursively call, nested sub directories, save as uncompile.py, in the directory to be converted, to run in command prompt type "python uncomple.py" would convert pyc to py in current working directory, with error handling and if rerun skips (recovery) files checking existing py extension match
import os
import uncompyle6
#Use current working directory
your_directory = os.getcwd()
#function processing current dir
def uncompilepath(mydir):
for dirpath, b, filenames in os.walk(mydir):
for d in b:
folderpath = dirpath + '/' + d
print(folderpath)
#recursive sub dir call
uncompilepath(folderpath)
for filename in filenames:
if not filename.endswith('.pyc'):
continue
filepath = dirpath + '/' + filename
original_filename = filename.split('.')[0]
original_filepath = dirpath + '/' + original_filename + '.py'
#ignore if already uncompiled
if os.path.exists(original_filepath):
continue
with open(original_filepath, 'w') as f:
print(filepath)
#error handling
try:
uncompyle6.decompile_file(filepath, f)
except Exception:
print("Error")
uncompilepath(your_directory)

Recursive download with pysftp

I'm trying to fetch from SFTP with the following structure:
main_dir/
dir1/
file1
dir2/
file2
I tried to achieve this with commands below:
sftp.get_r(main_path + dirpath, local_path)
or
sftp.get_d(main_path + dirpath, local_path)
The local path is like d:/grabbed_files/target_dir, and the remote is like /data/some_dir/target_dir.
With get_r I am getting FileNotFound exception. With get_d I am getting empty dir (when target dir have files not dirs, it works fine).
I'm totally sure that directory exists at this path. What am I doing wrong?

This one works for me, but when you download directory it create full path locally.
pysftp.Connection.get_r()
I also created simple download and upload methods:
def download_r(sftp, outbox):
tmp_dir = helpers.create_tmpdir()
assert sftp.isdir(str(outbox))
assert pathlib.Path(tmp_dir).is_dir()
sftp.get_r(str(outbox), str(tmp_dir))
tmp_dir = tmp_dir / outbox
return tmp_dir
def upload_r(sftp, inbox, files):
assert sftp.isdir(str(inbox))
if pathlib.Path(files).is_dir():
logger.debug(list(files.iterdir()))
sftp.put_r(str(files), str(inbox))
else:
logger.debug('No files here.')

I didn't understand why it doesn't work so I ended with my own recursive solution:
def grab_dir_rec(sftp, dirpath):
local_path = target_path + dirpath
full_path = main_path + dirpath
if not sftp.exists(full_path):
return
if not os.path.exists(local_path):
os.makedirs(local_path)
dirlist = sftp.listdir(remotepath=full_path)
for i in dirlist:
if sftp.isdir(full_path + '/' + i):
grab_dir_rec(sftp, dirpath + '/' + i)
else:
grab_file(sftp, dirpath + '/' + i)

In the event that you want a context manager wrapper around pysftp that does this for you, here is a solution that is even less code (after you copy/paste the github gist) that ends up looking like the following when used
path = "sftp://user:password#test.com/path/to/file.txt"
# Read a file
with open_sftp(path) as f:
s = f.read()
print s
# Write to a file
with open_sftp(path, mode='w') as f:
f.write("Some content.")
The (fuller) example: http://www.prschmid.com/2016/09/simple-opensftp-context-manager-for.html
This context manager happens to have auto-retry logic baked in in the event you can't connect the first time around (which surprisingly happens more often than you'd expect in a production environment...).
Oh, and yes, this assumes you are only getting one file per connection as it will auto-close the ftp connection.
The context manager gist for open_sftp: https://gist.github.com/prschmid/80a19c22012e42d4d6e791c1e4eb8515

Python: How to copy specific files from one location to another and keep directory structure

Hi I'm struggling with some python code for to copy specific files in a folder to another folder whilst keeping the directory structure.
I'm learning so this code is put together using various code snippets I've found, I couldn't find anything that exactly matched my circumstance and I don't understand python enough yet to understand where I've gone wrong
def filtered_copy(src_dir, dest_dir, filter):
print 'Copying files named ' + filter + ' in ' + src_dir + ' to ' + dest_dir
ignore_func = lambda d, files: [f for f in files if isfile(join(d, f)) and f != filter]
if os.path.exists(dest_dir):
print 'deleting existing data'
shutil.rmtree(dest_dir)
copytree(src_dir, dest_dir, ignore=ignore_func)
Executing this code like this
filtered_copy(c:\foldertosearch, c:\foldertocopyto, 'settings.xml')
does copy across the file I want but does not copy across the parent folder i.e. the src_dir, so the result I'm trying to achieve is:
c:\foldertocopyto\foldertosearch\settings.xml
*** Edit - to clarify this is a script that will be used on multiple operating systems
So if the folder structure was more complex i.e.
Parent folder
-subfolder
--subsubfolder
----subsubsubfolder
------settings.xml
and I ran
filtered_copy(subsubsubfolder, foldertocopyto, 'settings.xml')
I would want the new folder structure to be
foldertocopyto (there could be more parent folders above this or not)
--subsubsubfolder
----settings.xml
In other words the folder I search for a specific file in should also be copied across and if that folder already exists it should be deleted before the folder and file is copied across
I assumed copytree() would do this part - but obviously not!
*** end of Edit
*** Latest code changes
This works but I'm sure it's a long-winded way, also it copies blank folders, presumably because of the copytree() execution, I'd prefer just the folder that's being searched and the filtered file...
def filtered_copy(src_dir, dest_dir, filter):
foldername = os.path.basename(os.path.normpath(src_dir))
print 'Copying files named ' + filter + ' in ' + src_dir + ' to ' + dest_dir + '/' + foldername
ignore_func = lambda d, files: [f for f in files if isfile(join(d, f)) and f != filter]
if os.path.exists(dest_dir + '/' + foldername):
print 'deleting existing data'
shutil.rmtree(dest_dir)
copytree(src_dir, dest_dir + '/' + foldername, ignore=ignore_func)
*** end of latest code changes

You can use distutils.dir_util.copy_tree. It works just fine and you don't have to pass every argument, only src and dst are mandatory.
However in your case you can't use a similar tool like shutil.copytree because it behaves differently: as the destination directory must not exist this function can't be used for overwriting its contents
Give a try to a sample code below:
def recursive_overwrite(src, dest, ignore=None):
if os.path.isdir(src):
if not os.path.isdir(dest):
os.makedirs(dest)
files = os.listdir(src)
if ignore is not None:
ignored = ignore(src, files)
else:
ignored = set()
for f in files:
if f not in ignored:
recursive_overwrite(os.path.join(src, f),
os.path.join(dest, f),
ignore)
else:
shutil.copyfile(src, dest)

recursive file copying into subdirectory

I need to copy all the files and folders to the current folder to a subdirectory. What would be the best way to do so? I tried the following snippet but it fails as it fails if the destination directory already exists.
def copy(d=os.path.curdir):
dest = "t"
for i in os.listdir(d):
if os.path.isdir(i):
shutil.copytree(i, dest)
else:
shutil.copy(i, dest)
I have the feeling that the same task can be done in a better and easier manner. How do i do it?

I would never do it on python, but the following solution came to mind. It doesn't look simple, but it should work and can be simplified (haven't checked, sorry, no access to the computer now):
def copyDirectoryTree(directory, destination, preserveSymlinks=True):
for entry in os.listdir(directory):
entryPath = os.path.join(directory, entry)
if os.path.isdir(entryPath):
entrydest = os.path.join(destination, entry)
if os.path.exists(entrydest):
if not os.path.isdir(entrydest):
raise IOError("Failed to copy thee, the destination for the `" + entryPath + "' directory exists and is not a directory")
copyDirectoryTree(entrypath, entrydest, preserveSymlinks)
else:
shutil.copytree(entrypath, entrydest, preserveSymlinks)
else: #symlinks and files
if preserveSymlinks:
shutil.copy(entryPath, directory)
else:
shutil.copy(os.path.realpath(entryPath), directory)

See the code in http://docs.python.org/library/shutil.html, then tweak it a little (e.g. try: around os.makedirs(dst)).

To extend mamnun's answer,
If you want to use the direct call to the os, I'd advise using cp -r since you seem to want a recursive copy for directories.

Do you really need to use python? Because shutil functions cannot copy all file metadata and group permissions. Why don't you try built-in OS commands like cp in linux and xcopy in windows?
You can even try to run these commands from python
import os
os.system("cp file1 file2")
Hope this helps.

Here is my version of a recursive copy method for python, seems to work :)
def copy_all(fr, to, overwrite=True):
fr = os.path.normpath(fr)
to = os.path.normpath(to)
if os.path.isdir(fr):
if (not os.path.exists(to + os.path.basename(fr)) and not
os.path.basename(fr) == os.path.basename(to)):
to += "/" + os.path.basename(fr)
mkdirs(to)
for file in os.listdir(fr):
copy_all(fr + "/" + file, to + "/")
else: #symlink or file
dest = to
if os.path.isdir(to):
dest += "/"
dest += os.path.basename(fr)
if overwrite and (os.path.exists(dest) or os.path.islink(dest)
rm(dest)
if os.path.isfile(fr):
shutil.copy2(fr, dest)
else: #has to be a symlink
os.symlink(os.readlink(fr), dest)
def mkdirs(path):
if not os.path.isdir(path):
os.makedirs(path)
def rm(path):
if os.path.isfile(path) or os.path.islink(path):
os.remove(path)
elif os.path.isdir(path):
for file in os.listdir(path):
fullpath = path+"/"+file
os.rmdir(fullpath)