Develop Reference

How to Account and Drop a Specific String before another string in Python

So I have an application that generally does a good job at collecting information and moving stuff from one AWS s3 bucket to another, and then processing it, but it doesn't really do a good job when people name their file with a pretext string.
Currently, I look for glob:
dump_files = glob.glob('docker-support*.zip')
What this does is I have logic built to only look for things that account for file names that utilize docker-support as the main identifier.
However, I need it to account for times when people do something like
super_Secret123-Production-whatever-docker-support*.zip
Basically, I would like for the function to rename it using that variable dump_files
Should I just set the variable to something like this:
dump_files = glob.glob('*docker-support*.zip')
or
dump_files = glob.glob('/^(.*?)\docker-support*.zip')
The main thing is I am going to want to pick it up, rename it and then strip the part of the file name that is before the actual file name needed for processing: docker-support*.zip as the application needs to look for files in S3 just named in that format.
Code that handles this:
#!/usr/bin/env python3
# main execution loop for dump analysis tool
# Author: Bryce Ryan, Mirantis Inc.
#
# checks for new files in dump_originals, when found, runs run-parts against that file
# v1.1
# pause.main.loop check
# improved error handling
# escape file name to run-parts to avoid metacharacters
#
#
import os
import tempfile
import time
import zipfile
import logging
import shutil
import glob
from datetime import date
import sys
from os import path
logging.basicConfig(filename='/dump/logs/analyzer_logs.txt', level=logging.DEBUG, format='%(asctime)s %(message)s', datefmt='%Y-%m-%dT%H:%M:%S%z' )
ROOT_DIR = os.path.abspath('..')
logging.debug("ROOT_DIR: {}".format(ROOT_DIR))
DUMP_DIR = os.path.join(ROOT_DIR, 'dump_originals')
logging.debug("DUMP_DIR: {}".format(DUMP_DIR))
WORK_DIR = os.path.join(ROOT_DIR, 'work_dir')
logging.debug("WORK_DIR: {}".format(WORK_DIR))
# can we actually create a file? just because we have perms, or think we do, doesn't mean there are
# enough inodes or capacity to do basic stuff.
with open(os.path.join(DUMP_DIR, "testfile"), 'w'):
pass
logging.info("Beginning event loop for lodestone. Looking for new files in {}".format(DUMP_DIR))
print("Beginning event loop for lodestone.")
sys.stdout.flush()
os.chdir(DUMP_DIR)
logging.basicConfig(filename="analyzer.logs", level=logging.DEBUG)
while True:
# here at the top of the loop, check to see if we should wait for a bit
# typically, because of testing or maintenance
# if the magic, undocumented file exists, wait for 5 sec and check again
# do this forever
while path.exists("/dump/pause.main.loop"):
print("Pausing main loop for 60s, waiting on /dump/pause.main.loop")
time.sleep(60)
dump_files = glob.glob('docker-support*.zip')
try:
if dump_files[0] != '':
logging.debug("files found")
print("================== BEGIN PROCESSING NEW FILE ========= ")
print("File found:", dump_files[0])
print(" ")
logging.info("Processing new file: " + dump_files[0] )
sys.stdout.flush()
support_dump_file = dump_files[0]
# check that it's an actual zip; if not, ignore it
if not zipfile.is_zipfile(support_dump_file):
print("File: " + str(support_dump_file))
print("Inbound file is not recognized as a zip file.\n\n")
logging.info("Inbound file not recognized as a zip file.")
# now move it out of the way so we don't see it again;
# ok if exists on destination and we ignore the error
shutil.move( support_dump_file, "../dump_complete/" )
# no further processing, so back to the top
sys.stdout.flush()
continue
temp_dir = tempfile.mkdtemp(prefix='dump.', dir=WORK_DIR)
os.chmod(temp_dir, 0o777)
logging.info("temp_dir is: " + temp_dir)
# cmd = ROOT_DIR + "/utilities/run-parts --exit-on-error --arg=analyze --arg=" + DUMP_DIR + " --arg=" + support_dump_file + " --arg=" + temp_dir + " " + ROOT_DIR + "/analysis"
cmd = ROOT_DIR + "/utilities/run-parts --arg=analyze --arg=" + DUMP_DIR + " --arg=\'" + support_dump_file + "\' --arg=" + temp_dir + " " + ROOT_DIR + "/analysis"
print(cmd)
logging.info("Will execute: " + cmd )
sys.stdout.flush()
try:
retcode = os.system(cmd)
tempdir =temp_dir
if retcode == 1:
print("Removing temporary work_dir")
logging.debug("Removing temporary work_dir", tempdir)
shutil.rmtree(tempdir, ignore_errors=True)
sys.stdout.flush()
finally:
print("Finally block for cmd. . .")
print("Removing temporary work_dir")
logging.debug("Removing work_dir " + tempdir)
print(tempdir)
sys.stdout.flush()
# shutil.rmtree(tempdir, ignore_errors=True)
os.system('/bin/rm -rf' + tempdir)
sys.stdout.flush()
except:
pass
# pause for a moment; save some processor cycles
sys.stdout.flush()
time.sleep(1)
Right now I do not have the function that will rename this in there.

Split PDF file in color and BW pages for printing

I've wanted a program to split a pdf file into black and white and color pages, as BW pages as cheaper to print.
I've seen this code that seems to do the job (as extracted from github.com/dalemyers/9547783)
The main advantages I see with it are that the script takes in a PDF and creates two new PDFs. One has the black and white pages and the other has the colour pages. It also takes duplex printing
into account. So a black and white side which is on the same sheet as a colour side will be placed into the colour PDF.
However, I've not been able to reproduce the code.
Can someone help out?
For example, please use a pdf file from here that contains color and BW pages
Code bellow:
#!/usr/bin/env python
# Python 2 and 3 compatible.
#This script takes in a PDF and creates two new PDFs. One has the black and
#white pages and the other has the colour pages. It also takes duplex printing
#into account. So a black and white side which is on the same sheet as a colour
#side will be placed into the colour PDF.
#This is from a script created by Iain Murray. The original comment is below.
#This version simply has some different defaults and removes the PDFtoPPM.
#Original ######################################################################
# Python program to take a pdf file, and split it into color and black
# and white part(s). Requires pdftk and one of gs and pdftoppm.
#
# Iain Murray, February 2010.
#
# Inspired by dvicoloursplit.py, Jeremy Sanders 2001, although written
# from scratch.
#
# 2011-09-19 fixed bug with odd numbers of pages reported by Richard Shaw
# 2012-06-11 tweaked to run in Python 3 as well as 2.
#End Original ##################################################################
## This program is free software; you can redistribute it and/or modify
## it under the terms of the GNU General Public License as published by
## the Free Software Foundation; either version 2 of the License, or
## (at your option) any later version.
## This program is distributed in the hope that it will be useful,
## but WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
## GNU General Public License for more details.
import os, os.path, sys, string, re, tempfile, shutil, getopt
def a2b(x):
"""Turn ascii into bytes for Python 3, in way that works with Python 2"""
try:
return bytes(x)
except:
return bytes(x, 'ascii')
def iscolorppm(filename):
"""Does the PPM file contain any non-grayscale colors?"""
file = open(filename, 'rb')
# Ugly: I read the whole file into RAM, and copy it needlessly a lot
data = file.read()
file.close()
# PPM is a *very* liberal file format. It allows comments anywhere in the
# header, even in the middle of tokens.
comments_re = re.compile(a2b('^([^ \t\n]*)#[^\n]*\n'))
split_re = re.compile(a2b('^([ \t\n]|#[^\n]*\n)+([^ \t\n#])'))
tok_re = re.compile(a2b('^([^ \t\n]*)([ \t\n].*)'), re.DOTALL)
toks = []
while len(toks) < 4:
while split_re.match(data):
data = split_re.sub(r'\2', data)
while comments_re.match(data):
data = comments_re.sub(r'\1', data)
(tok, data) = tok_re.match(data).groups()
toks.append(tok)
magic = toks[0]
(width, height, max_color) = map(int, toks[1:])
data = data[1:]
if magic == b'P3':
binary = False
elif magic == b'P6':
binary = True
else:
print("%s is not a valid PPM file" % filename)
sys.exit(1)
# Massage data so adjacent triples should have the same value in b/w images
data_len = width*height*3
if binary:
if int(max_color) > 255:
# Untested. Each intensity is in two bytes.
data_len *= 2
data = data[1:data_len:2] + data[:data_len:2]
else:
data = [int(x) for x in data.split()]
if len(data) < data_len:
print('PPM file is truncated?')
sys.exit(1)
triples = zip(data[0:data_len:3], data[1:data_len:3], data[2:data_len:3])
black_and_white = all((a==b and a==c for (a,b,c) in triples))
return not black_and_white
def pdfcolorsplit(file, doublesided, merge, verbose):
# Work out which pages are color
if verbose:
print('Analyzing %s...' % file)
tmpdir = tempfile.mkdtemp(prefix = 'pdfcs_')
gs_opts = '-sDEVICE=ppmraw -dBATCH -dNOPAUSE -dSAFE -r20'
if not verbose:
gs_opts += ' -q'
os.system('gs ' + gs_opts + ' -sOutputFile="%s" "%s"' \
% (os.path.join(tmpdir, 'tmp%06d.ppm'), file))
PPMs = os.listdir(tmpdir)
PPMs.sort()
iscolor = [iscolorppm(os.path.join(tmpdir, x)) for x in PPMs]
num_pages = len(iscolor)
shutil.rmtree(tmpdir)
if doublesided:
# Treat as color those b/w pages that share a sheet with a color page
iscolorpair = [x or y for (x,y) in zip(iscolor[::2], iscolor[1::2])]
iscolor[:2*len(iscolorpair):2] = iscolorpair
iscolor[1::2] = iscolorpair
# Construct page range strings
flips = [x for x in range(2,num_pages+1) if iscolor[x-1] != iscolor[x-2]]
if not flips:
if verbose:
print('No splitting needs to be done, skipping %s' % file)
return
edges = [1] + flips + [num_pages+1]
ranges = ['%d-%d' % (x,y-1) for (x,y) in zip(edges[:-1], edges[1:])]
# Finally output split files
if verbose:
print('Outputing splits as new pdf files...')
base_name = file
if base_name.lower().endswith('.pdf'):
base_name = base_name[:-4]
suffixes = ['_bwsplit.pdf', '_colorsplit.pdf']
# jobs is a seq of (range, filename) pairs, e.g. ('1-3', 'colorbits.pdf')
if merge:
jobs = ((' '.join(ranges[0::2]), base_name + suffixes[iscolor[0]]),\
(' '.join(ranges[1::2]), base_name + suffixes[not iscolor[0]]))
else:
jobs = [(r, '%s_%03d%s' % (base_name,n+1,suffixes[(n+iscolor[0])%2])) \
for (n,r) in enumerate(ranges)]
for (pages, name) in jobs:
if verbose:
print('pdftk "%s" cat %s output "%s"' % (file, pages, name))
os.system('pdftk "%s" cat %s output "%s"' % (file, pages, name))
def usage():
progname = os.path.basename(sys.argv[0])
print('Usage: %s [OPTIONS] <PDF-file(s)>' % progname)
print('')
print('Splits PDF files into color and black and white sections.')
print('')
print('Options:')
print(' -m Write out the file in multiple parts rather than a PDF for')
print(' each different section')
print(' -s option chooses simplex rather than duplex output')
print(' -v verbose.')
def main():
try:
opt_pairs, filenames = getopt.gnu_getopt(sys.argv[1:], "hvpms", ["help"])
except getopt.GetoptError as err:
print(str(err))
usage()
sys.exit(1)
if opt_pairs:
opts = list(zip(*opt_pairs))[0]
else:
opts = []
if ('-h' in opts) or ('--help' in opts) or (not filenames):
usage()
sys.exit()
verbose = '-v' in opts
use_pdftoppm = '-p' in opts
merge = '-m' not in opts
doublesided = '-s' not in opts
for file in filenames:
pdfcolorsplit(file, doublesided, merge, verbose)
if __name__ == "__main__":
main()

Solution:
Open your command prompt (in my case: Anaconda Prompt as administrator).
Install dependencies: pypdftk, gs and pdf2image
Go to your target folder containing the target file (which I've named filename)
(base) C:\Users\folder>pdfcoloursplitter.py
Usage: pdfcoloursplitter.py [OPTIONS] <PDF-file(s)>
Splits PDF files into color and black and white sections.
Options:
-m Write out the file in multiple parts rather than a PDF for
each different section
-s option chooses simplex rather than duplex output
-v verbose.
C:\Users\folder>pdfcoloursplitter.py -m filename.pdf

Moving Files by creation/modification date then moving with Python

I am new to programming, even more so with Python. So please excuse any ignorance on my part. I am trying to write a script for myself that will move files that have been modified in the last 24 hours. So far I have came up with this:
import datetime
import os
import shutil
src = "C:\Users\Student\Desktop\FolderA"
dst = "C:\Users\Student\Desktop\FolderB"
now = dt.datetime.now()
before = now - dt.timedelta(hours=24)
def mins_since_mod(fname):
return (os.path.getmtime(fname))
for fname in os.listdir(src):
if mins_since_mod > before:
src_fname = os.path.join(src,fname)
os.path.join(dst,fname)
shutil.move(src_fname, dst)
I know i'm close to the solution, but I can't seem to figure out how to get this to work. I looked around here on the community and was not able to find a solution to my problem. Thank you for any leads or suggestions.

There are a few things to change. First, you can't compare the datetime in before to the Unix timestamp that getmtime() returns. It's easier to just use that directly. Also, you actually need to pass the (full) filename to mins_since_mod() for it to do anything.
Here's something that should work, changing the name of mins_since_mod() to reflect what it does better:
import time
import os
import shutil
SECONDS_IN_DAY = 24 * 60 * 60
src = "C:\Users\Student\Desktop\FolderA"
dst = "C:\Users\Student\Desktop\FolderB"
now = time.time()
before = now - SECONDS_IN_DAY
def last_mod_time(fname):
return os.path.getmtime(fname)
for fname in os.listdir(src):
src_fname = os.path.join(src, fname)
if last_mod_time(src_fname) > before:
dst_fname = os.path.join(dst, fname)
shutil.move(src_fname, dst_fname)

Hey mate I have actually just done something like this myself. I found that there will be a few issues will the time comparison as well as some issues in comparing and moving folders.
Try this:
import os
import shutil
import datetime
def filter_by_date(src_folder, archive_date):
os.chdir(src_folder)
delay_time = 24 * 60 * 60
archive_period = archive_date - delay_time
return [
name for name in os.listdir(u'.')
if os.path.isdir(name)
and datetime.datetime.fromtimestamp(os.path.getmtime(name)) < archive_period
]
if __name__ == '__main__':
folders = filter_by_date("C:/Users/Student/Desktop/FolderA", time.time())
for files in folders:
print files
try:
shutil.copytree(files, os.path.join("C:/Users/Student/Desktop/New", files))
except OSError as e:
print('\nDirectory not copied. Error: %s' % e)
except shutil.Error as e:
try:
files = files.encode('UTF-8')
dst_path = os.path.join('C:/Users/Student/Desktop/FolderB/', files)
shutil.copytree(files, dst_path)
finally:
print('\nDirectory not copied. Error: %s' % e)
print "\Completed"
This is going to ensure any file name (including Chinese, Russian and Japanese will be copied) and any folder (directory or sub-directory) is copied. It will also keep all file attributes.

Downloading, writing, converting and saving with imagemagick in python (corrupt images)

Okay so I've been trying to solve this for about six hours now and it's just not happening.
I've got this function that grabs a bunch of image (GIF) files from a URL based on a timestamp (using the Requests library. The images get saved to my desktop in a specific directory just fine.
When I try to open that image, rename it and process it then everything breaks.
Here's the initial method that sets everything up:
def createImage():
AB_CODES = ["WHK", "WHN", "WWW", "XBU", "XSM"]
BASE_URL = "http://url_where_I_get_images_from"
orig_dir = "originals/"
new_dir = "processed/"
# Add new image for each code
for code in AB_CODES:
radar_dir = BASE_URL + code
url = requests.head(radar_dir)
#parseUrl creates a valid timestamp corresponding to the latest image.
timestamp = parseUrl(url)
filename = timestamp + "_" + code + "_PRECIP_RAIN.gif"
radar = BASE_URL + code + "/" + filename
radar_img = requests.get(radar, timeout=30.000)
# This is where the file from original source gets saved to my desktop, works.
# Image gets saved in path/originals/img.gif
if (radar_img.status_code == requests.codes.ok):
image = radar_img.content
filepath = os.path.join(orig_dir, filename)
imgfile = open(filepath, "wb")
imgfile.write(image)
imgfile.close()
# This is where I create a new image to be saved and worked on in
# path/processed/new_img.gif
image = radar_img.content
filename = code + "_radar.gif"
convpath = os.path.join(new_dir, filename)
convimg = open(convpath, "wb")
convimg.write(image)
# This is the call to the function where I use imagemagick
# which is not working
image = processImage(convimg.name, "processed/XSM_radar_output.gif")
convimg.close()
Here are the two methods that make up my processing function. Right now it's in more of a testing phase because it won't work.
def formatArg(arg):
if arg.startswith("#") or " " in arg:
return repr(arg)
return arg
def processImage(input, output, verbose=True, shell=True):
args = [
"convert", input,
"-resize", "50x50",
output
]
if verbose:
print("Image: %s" % input)
print(" ".join(formatArg(a) for a in args))
print
if os.path.exists(input):
try:
result = subprocess.check_call(args)
except subprocess.CalledProcessError:
result = False
if result:
print("%s : SUCCESS!" % output)
else:
print("%s : FAIL!" % output)
I've tried it without shell=true. The error message I get is that the image is corrupted:
Image: processed/XSM_radar.gif
convert processed/XSM_radar.gif -resize 50x50 processed/XSM_radar.gif
convert.im6: corrupt image `processed/XSM_radar.gif' # error/gif.c/ReadGIFImage/1356.
convert.im6: no images defined `processed/XSM_radar_output.gif' # error/convert.c/ConvertImageCommand/3044.
processed/XSM_radar.gif : FAIL!
I don't understand why it's telling me that it's corrupted. I've run these exact same commands from the command line and it works just fine.
(I have imported subprocess)

Someone helped me determine the error.
def processImage(input, output, verbose=True, shell=True):
args = [
"convert", input,
"-resize", "50x50",
output
]
if verbose:
print("Image: %s" % input)
print(" ".join(formatArg(a) for a in args))
print
if os.path.exists(input):
try:
**result = subprocess.check_call(args)**
**except subprocess.CalledProcessError:**
**result = False**
**if result:**
print("%s : SUCCESS!" % output)
else:
print("%s : FAIL!" % output)
That section with the results should have been
result_code = subprocess.check_call(args)
and if it returned an error, result_code should have been set equal to 1.
So:
if result_code == 0:
do stuff
Thank you for your attention!
Have a lovely day.

IOError opening an existing file with Python

Running the following code:
import os
import datetime
import ftplib
currdate = datetime.datetime.now()
formatdate = currdate.strftime("%m-%d-%Y %H%M")
def log():
fqn = os.uname()[1]
ext_ip = urllib2.urlopen('http://whatismyip.org').read()
log = open ('/Users/admin/Documents/locatelog.txt','w')
log.write(str("Asset: %s " % fqn))
log.write(str("Checking in from IP#: %s" % ext_ip))
smush = str(fqn +' # ' + formatdate)
os.rename('/Users/admin/Documents/locatelog.txt','/Users/admin/Documents/%s.txt' % smush )
s = ftplib.FTP('10.7.1.71','username','password')
f = open('/Users/admin/Documents/%s.txt' % smush,'r')
s.storbinary("STOR /Users/admin/Documents/%s.txt" % smush,f)
Generates the following error:
ftplib.error_perm: 550 /Users/admin/Documents/678538.local # 02-24-2010 1301.txt: No such file or directory
I have a feeling something is amiss in this line :
s.storbinary("STOR /Users/admin/Documents/%s.txt" % smush,f)
678538 is the host I am testing on...using Mac OS X 10.5 and Python 2.5.1

Shouldn't it bef = open('/Users/admin/Documents/%s.txt' % smush,'r') ? notice the / in front of Users
If you dont put the first /, the script will think the path to the file is relative to the current directory (where the script is run from)
Edit:
I m not too familiar with Python (I wish) but shouldnt it be:
s.storbinary('STOR /Users/admin/Documents/%s.txt' % smush,f) ?
In your example, Python will treat your string as literal and you want to interpolate the value of smush with %s
Edit 2:
Does the directory /Users/admin/Documents/ exist on your server? If not, I think you will have to create them before copying anything. (Since the error message is about some files/folders missing).
You can create them yourself first. Run your script. If the file is copied successfully, then you can add the creation of the directories from within your script.

remove all spaces from file name .eg in smush = str(fqn +' # ' + formatdate), you are putting a space in front of and after "#". you path looks like
/Users/admin/Documents/something # something
and when you pass it to ftplib, it may have problem. another way is to try putting quotes, eg
s.storbinary("STOR '/Users/admin/Documents/%s.txt'" % smush,f)

Edit:
This version works: Problem was that I was writing to buffer, and not to file.
import os
import urllib2
import datetime
import ftplib
currdate = datetime.datetime.now()
formatdate = currdate.strftime("%m-%d-%Y-%H%M")
def log():
fqn = os.uname()[1]
ext_ip = urllib2.urlopen('http://whatismyip.org').read()
smush = str(fqn + formatdate)
s = ftplib.FTP('10.7.1.71','username','password')
f = open('/Users/admin/Documents/%s.txt' % smush,'w')
f.write(str("Asset: %s " % fqn))
f.write('\n')
f.write(str("Checking in from IP#: %s" % ext_ip))
f.write('\n')
f.write(str("On: %s" % formatdate))
f.close
f = open('/Users/admin/Documents/%s.txt' % smush,'rb')
s.storbinary('STOR %s.txt' % smush , f)
s.close
f.close