Develop Reference

Python on Crontab does not execute bash script

import subprocess as sub
import re
import os
from datetime import datetime as influx_timestap
from influxdb import InfluxDBClient
from collections import OrderedDict
insert_json = []
hostname = str(sub.check_output('hostname')).strip()
location = str(sub.check_output(['ps -ef | grep mgr'], shell=True)).split()
current_dir = os.getcwd()
print("script executed")
gg_location_pattern = re.compile(r'mgr\.prm$')
gg_process_pattertn = re.compile(r'^REPLICAT|^EXTRACT')
for index in location:
if gg_location_pattern.search(index) != None:
gg_location = index[:-14]
os.chdir(gg_location)
print("checkpoint1")
get_lag = sub.check_output(str(current_dir) + '/ggsci_test.sh', shell=True)
print("checkpoint2")
processes = get_lag.split("\n")
for process in processes:
if gg_process_pattertn.search(process) != None:
lag_at_chkpnt = int((process.split()[3]).split(":")[0]) * 3600 + int((process.split()[3]).split(":")[1]) *60 + int((process.split()[3]).split(":")[2])
time_since_chkpnt = int((process.split()[4]).split(":")[0]) * 3600 + int((process.split()[4]).split(":")[1]) *60 + int((process.split()[4]).split(":")[2]
)
process_dict = OrderedDict({"measurement": "GoldenGate_Mon_" + str(hostname) + "_Graph",
"tags": {"hostname": hostname, "process_name": process.split()[2]},
"time": influx_timestap.now().isoformat('T'),
"fields": {"process_type": process.split()[0], "process_status": process.split()[1],
"lag_at_chkpnt": lag_at_chkpnt, "time_since_chkpnt": time_since_chkpnt}})
insert_json.append(process_dict)
host = 'xxxxxxxx'
port = 'x'
user = 'x'
password = 'x'
dbname = 'x'
print("before client")
client = InfluxDBClient(host, port, user, password, dbname)
client.write_points(insert_json)
print("after client")
This code works manually perfect, but on the crontab it is not working. After searching on the internet I found that they say change or set your "PATH" variable on the crontab. I changed my "PATH" variable and it is still not working.
Crontab log file write "checkpoint1" after that there is nothing. So, line not working is "get_lag = sub.check_output(str(current_dir) + '/ggsci_test.sh', shell=True)"
What can I do here afterwards?
Take care,

it looks like your external script (ggsci_test.sh) has some issues with the paths / general failure.
From the Python subprocess documentation about subprocess.check_output:
If the return code was non-zero it raises a CalledProcessError. The
CalledProcessError object will have the return code in the returncode
attribute and any output in the output attribute.
So thats the reason why you see the error when catching it, but not being able to continue.
You should check therefore if your shell script has any issues that need to be solved before.

How to insert a "missing" page as blank page in PDF with Python?

Say you have to join some pages that are number 2, 4 and 5… (the files are named test_002.pdf, test_004.pdf and test_005.pdf), then we could say there is a page 3 missing.
What I try to do is having a result from those commands :
pdfjam --nup 2 --papersize '{47cm,30cm}' --scale 1.0 test_002.pdf test_003.pdf --outfile joined_002-003.pdf
pdfjam --nup 2 --papersize '{47cm,30cm}' --scale 1.0 test_004.pdf test_005.pdf --outfile joined_004-005.pdf
that will join even and odd page in one unique page, with a blank page (3) in place of the missing page.
I guess it should:
check incoming files from the beginning to the end looking for what page is missing (in this case from 2 to 5 missing #3)
on-the-fly generate blank '23.5cm,30cm' pdf pages (using pyPdf maybe)
classify them 'even' and 'odd' as couples to be able to join every even with odd page (using pdfjam)…
Am I right?
Is that possible with some lines of Python?
Or is there a easier way?
Because here's what I started to do, making it work like an hotfolder, but I'm really completely lost in the even and odd management and missing "files/pages" :
#!/usr/bin/python
# -*- coding: UTF8 -*-
import os
import os.path
import re
import time
import datetime
CODEFILE = re.compile("^(TES|EXA).*\.pdf$")
WHERE = "/tmp/TEST/"
STORAGE = "/tmp/WORK/"
DBLSIZE = "{47cm,30cm}"
def time_stamp():
now = datetime.datetime.now()
return now.strftime("%Y-%m-%d %H:%M:%S")
print(time_stamp()+" : Starting.")
def files_list(path):
this_files = list()
root, dires, files = os.walk(path).next()
for f in files:
if CODEFILE.match(f):
this_files.append(os.path.join(root, f))
return this_files
def file_sizes(filename):
meta = os.lstat(filename)
return meta.st_size
def files_to_handle(path):
this_files = list()
ft1 = dict()
ft2 = dict()
for f in files_list(WHERE):
ft1[f] = file_sizes(f)
time.sleep(10)
for f in files_list(WHERE):
ft2[f] = file_sizes(f)
for f, t in ft2.items():
try:
if ft1[f] == t:
this_files.append(f)
except:
pass
return this_files
r = files_to_handle(WHERE)
print(time_stamp()+" : Files available :")
print(r)
for f in r:
rc = os.system("pdfjam --batch --nup 2 --papersize {1} --scale 1.0 --outfile . {2}".format(
DBLSIZE, f))
if rc != 0:
print(time_stamp()+" : an ERROR as occured with the file {0}.".format(f))
else:
print(time_stamp()+" : files {0} OK.".format(f))
os.system("mv {0} {1}".format(f, STORAGE))
print(time_stamp()+" : Stopping.")
Thanks in advance!

Python program crashing

So I've designed a program that runs on a computer, looks for particular aspects of files that have been plaguing us, and deletes the files if a flag is passed. Unfortunately the program seems to be almost-randomly shutting down/crashing. I say almost-randomly, because the program always exits after it deletes a file, though it will commonly stay up after a success.
I've run a parallel Python program that counts upwards in the same intervals, but does nothing else. This program does not crash/exit, and stays open.
Is there perhaps a R/W access issue? I am running the program as administrator, so I'm not sure why that would be the case.
Here's the code:
import glob
import os
import time
import stat
#logging
import logging
logging.basicConfig(filename='disabledBots.log')
import datetime
runTimes = 0
currentPhp = 0
output = 0
output2 = 0
while runTimes >= 0:
#Cycles through .php files
openedProg = glob.glob('*.php')
openedProg = openedProg[currentPhp:currentPhp+1]
progInput = ''.join(openedProg)
if progInput != '':
theBot = open(progInput,'r')
#Singles out "$output" on this particular line and closes the process
readLines = theBot.readlines()
wholeLine = (readLines[-4])
output = wholeLine[4:11]
#Singles out "set_time_limit(0)"
wholeLine2 = (readLines[0])
output2 = wholeLine2[6:23]
theBot.close()
if progInput == '':
currentPhp = -1
#Kills the program if it matches the code
currentTime = datetime.datetime.now()
if output == '$output':
os.chmod(progInput, stat.S_IWRITE)
os.remove(progInput)
logging.warning(str(currentTime) +' ' + progInput + ' has been deleted. Please search for a faux httpd.exe process and kill it.')
currentPhp = 0
if output2 == 'set_time_limit(0)':
os.chmod(progInput, stat.S_IWRITE)
os.remove(progInput)
logging.warning(str(currentTime) +' ' + progInput + ' has been deleted. Please search for a faux httpd.exe process and kill it.')
currentPhp = 0
else:
currentPhp = currentPhp + 1
time.sleep(30)
#Prints the number of cycles
runTimes = runTimes + 1
logging.warning((str(currentTime) + ' botKiller2.0 has scanned '+ str(runTimes) + ' times.'))
print('botKiller3.0 has scanned ' + str(runTimes) + ' times.')

Firstly, it'll be hell of a lot easier to work out what's going on if you base your code around something like this...
for fname in glob.glob('*.php'):
with open(fname) as fin:
lines = fin.readlines()
if '$output' in lines[-4] or 'set_time_limit(0)' in lines[0]:
try:
os.remove(fname)
except IOError as e:
print "Couldn't remove:", fname
And err, that's not actually a secondly at the moment, your existing code is just too tricky to follow fullstop, let alone all the bits that could cause a strange error that we don't know yet!

if os.path.exists(progInput):
os.chmod(progInput, stat.S_IWRITE)
os.remove(progInput)
ALSO:
You never reset the output or output2 variables in the loop?
is this on purpose?

"Not implemented" Exception when using pywin32 to control Adobe Acrobat

I have written a script in python using pywin32 to save pdf files to text that up until recently was working fine. I use similar methods in Excel. The code is below:
def __pdf2Txt(self, pdf, fileformat="com.adobe.acrobat.accesstext"):
outputLoc = os.path.dirname(pdf)
outputLoc = os.path.join(outputLoc, os.path.splitext(os.path.basename(pdf))[0] + '.txt')
try:
win32com.client.gencache.EnsureModule('{E64169B3-3592-47d2-816E-602C5C13F328}', 0, 1, 1)
adobe = win32com.client.DispatchEx('AcroExch.App')
pdDoc = win32com.client.DispatchEx('AcroExch.PDDoc')
pdDoc.Open(pdf)
jObject = pdDoc.GetJSObject()
jObject.SaveAs(outputLoc, "com.adobe.acrobat.accesstext")
except:
traceback.print_exc()
return False
finally:
del jObject
pdDoc.Close()
del pdDoc
adobe.Exit()
del adobe
However this code has suddenly stopped working and I get the following output:
Traceback (most recent call last):
File "C:\Documents and Settings\ablishen\workspace\HooverKeyCreator\src\HooverKeyCreator.py", line 38, in __pdf2Txt
jObject.SaveAs(outputLoc, "com.adobe.acrobat.accesstext")
File "C:\Python27\lib\site-packages\win32com\client\dynamic.py", line 505, in __getattr__
ret = self._oleobj_.Invoke(retEntry.dispid,0,invoke_type,1)
com_error: (-2147467263, 'Not implemented', None, None)
False
I have similar code written in VB that works correctly so I'm guessing that it has something to do with the COM interfaces not binding to the appropriate functions correctly? (my COM knowledge is patchy).

Blish, this thread holds the key to the solution you are looking for: https://mail.python.org/pipermail/python-win32/2002-March/000260.html
I admit that the post above is not the easiest to find (probably because Google scores it low based on the age of the content?).
Specifically, applying this piece of advice will get things running for you: https://mail.python.org/pipermail/python-win32/2002-March/000265.html
For reference, the complete piece of code that does not require you to manually patch dynamic.py (snippet should run pretty much out of the box):
# gets all files under ROOT_INPUT_PATH with FILE_EXTENSION and tries to extract text from them into ROOT_OUTPUT_PATH with same filename as the input file but with INPUT_FILE_EXTENSION replaced by OUTPUT_FILE_EXTENSION
from win32com.client import Dispatch
from win32com.client.dynamic import ERRORS_BAD_CONTEXT
import winerror
# try importing scandir and if found, use it as it's a few magnitudes of an order faster than stock os.walk
try:
from scandir import walk
except ImportError:
from os import walk
import fnmatch
import sys
import os
ROOT_INPUT_PATH = None
ROOT_OUTPUT_PATH = None
INPUT_FILE_EXTENSION = "*.pdf"
OUTPUT_FILE_EXTENSION = ".txt"
def acrobat_extract_text(f_path, f_path_out, f_basename, f_ext):
avDoc = Dispatch("AcroExch.AVDoc") # Connect to Adobe Acrobat
# Open the input file (as a pdf)
ret = avDoc.Open(f_path, f_path)
assert(ret) # FIXME: Documentation says "-1 if the file was opened successfully, 0 otherwise", but this is a bool in practise?
pdDoc = avDoc.GetPDDoc()
dst = os.path.join(f_path_out, ''.join((f_basename, f_ext)))
# Adobe documentation says "For that reason, you must rely on the documentation to know what functionality is available through the JSObject interface. For details, see the JavaScript for Acrobat API Reference"
jsObject = pdDoc.GetJSObject()
# Here you can save as many other types by using, for instance: "com.adobe.acrobat.xml"
jsObject.SaveAs(dst, "com.adobe.acrobat.accesstext")
pdDoc.Close()
avDoc.Close(True) # We want this to close Acrobat, as otherwise Acrobat is going to refuse processing any further files after a certain threshold of open files are reached (for example 50 PDFs)
del pdDoc
if __name__ == "__main__":
assert(5 == len(sys.argv)), sys.argv # <script name>, <script_file_input_path>, <script_file_input_extension>, <script_file_output_path>, <script_file_output_extension>
#$ python get.txt.from.multiple.pdf.py 'C:\input' '*.pdf' 'C:\output' '.txt'
ROOT_INPUT_PATH = sys.argv[1]
INPUT_FILE_EXTENSION = sys.argv[2]
ROOT_OUTPUT_PATH = sys.argv[3]
OUTPUT_FILE_EXTENSION = sys.argv[4]
# tuples are of schema (path_to_file, filename)
matching_files = ((os.path.join(_root, filename), os.path.splitext(filename)[0]) for _root, _dirs, _files in walk(ROOT_INPUT_PATH) for filename in fnmatch.filter(_files, INPUT_FILE_EXTENSION))
# patch ERRORS_BAD_CONTEXT as per https://mail.python.org/pipermail/python-win32/2002-March/000265.html
global ERRORS_BAD_CONTEXT
ERRORS_BAD_CONTEXT.append(winerror.E_NOTIMPL)
for filename_with_path, filename_without_extension in matching_files:
print "Processing '{}'".format(filename_without_extension)
acrobat_extract_text(filename_with_path, ROOT_OUTPUT_PATH, filename_without_extension, OUTPUT_FILE_EXTENSION)
I have tested this on WinPython x64 2.7.6.3, Acrobat X Pro

makepy.py is a script that comes with the win32com python package.
Running it for your installation "wires" python into the COM/OLE object in Windows. The following is an excerpt of some code I used to talk to Excel and do some stuff in it. This example gets the name of sheet 1 in the current workbook. It automatically runs makepy if it has an exception:
import win32com;
import win32com.client;
from win32com.client import selecttlb;
def attachExcelCOM():
makepyExe = r'python C:\Python25\Lib\site-packages\win32com\client\makepy.py';
typeList = selecttlb.EnumTlbs();
for tl in typeList:
if (re.match('^Microsoft.*Excel.*', tl.desc, re.IGNORECASE)):
makepyCmd = "%s -d \"%s\"" % (makepyExe, tl.desc);
os.system(makepyCmd);
# end if
# end for
# end def
def getSheetName(sheetNum):
try:
xl = win32com.client.Dispatch("Excel.Application");
wb = xl.Workbooks.Item(sheetNum);
except Exception, detail:
print 'There was a problem attaching to Excel, refreshing connect config...';
print Exception, str(detail);
attachExcelCOM();
try:
xl = win32com.client.Dispatch("Excel.Application");
wb = xl.Workbooks.Item(sheetNum);
except:
print 'Could not attach to Excel...';
sys.exit(-1);
# end try/except
# end try/except
wsName = wb.Name;
if (wsName == 'PERSONAL.XLS'):
return( None );
# end if
print 'The target worksheet is:';
print ' ', wsName;
print 'Is this correct? [Y/N]',;
answer = string.strip( sys.stdin.readline() );
answer = answer.upper();
if (answer != 'Y'):
print 'Sheet not identified correctly.';
return(None);
# end if
return( (wb, wsName) );
# end def
# -- Main --
sheetInfo = getSheetName(sheetNum);
if (sheetInfo == None):
print 'Sheet not found';
sys.exit(-1);
else:
(wb, wsName) = sheetInfo;
# end if

IOError opening an existing file with Python

Running the following code:
import os
import datetime
import ftplib
currdate = datetime.datetime.now()
formatdate = currdate.strftime("%m-%d-%Y %H%M")
def log():
fqn = os.uname()[1]
ext_ip = urllib2.urlopen('http://whatismyip.org').read()
log = open ('/Users/admin/Documents/locatelog.txt','w')
log.write(str("Asset: %s " % fqn))
log.write(str("Checking in from IP#: %s" % ext_ip))
smush = str(fqn +' # ' + formatdate)
os.rename('/Users/admin/Documents/locatelog.txt','/Users/admin/Documents/%s.txt' % smush )
s = ftplib.FTP('10.7.1.71','username','password')
f = open('/Users/admin/Documents/%s.txt' % smush,'r')
s.storbinary("STOR /Users/admin/Documents/%s.txt" % smush,f)
Generates the following error:
ftplib.error_perm: 550 /Users/admin/Documents/678538.local # 02-24-2010 1301.txt: No such file or directory
I have a feeling something is amiss in this line :
s.storbinary("STOR /Users/admin/Documents/%s.txt" % smush,f)
678538 is the host I am testing on...using Mac OS X 10.5 and Python 2.5.1

Shouldn't it bef = open('/Users/admin/Documents/%s.txt' % smush,'r') ? notice the / in front of Users
If you dont put the first /, the script will think the path to the file is relative to the current directory (where the script is run from)
Edit:
I m not too familiar with Python (I wish) but shouldnt it be:
s.storbinary('STOR /Users/admin/Documents/%s.txt' % smush,f) ?
In your example, Python will treat your string as literal and you want to interpolate the value of smush with %s
Edit 2:
Does the directory /Users/admin/Documents/ exist on your server? If not, I think you will have to create them before copying anything. (Since the error message is about some files/folders missing).
You can create them yourself first. Run your script. If the file is copied successfully, then you can add the creation of the directories from within your script.

remove all spaces from file name .eg in smush = str(fqn +' # ' + formatdate), you are putting a space in front of and after "#". you path looks like
/Users/admin/Documents/something # something
and when you pass it to ftplib, it may have problem. another way is to try putting quotes, eg
s.storbinary("STOR '/Users/admin/Documents/%s.txt'" % smush,f)

Edit:
This version works: Problem was that I was writing to buffer, and not to file.
import os
import urllib2
import datetime
import ftplib
currdate = datetime.datetime.now()
formatdate = currdate.strftime("%m-%d-%Y-%H%M")
def log():
fqn = os.uname()[1]
ext_ip = urllib2.urlopen('http://whatismyip.org').read()
smush = str(fqn + formatdate)
s = ftplib.FTP('10.7.1.71','username','password')
f = open('/Users/admin/Documents/%s.txt' % smush,'w')
f.write(str("Asset: %s " % fqn))
f.write('\n')
f.write(str("Checking in from IP#: %s" % ext_ip))
f.write('\n')
f.write(str("On: %s" % formatdate))
f.close
f = open('/Users/admin/Documents/%s.txt' % smush,'rb')
s.storbinary('STOR %s.txt' % smush , f)
s.close
f.close