I'm trying to run a python script in crontab.
5 0 * * * python /home/hadoop/import_openapp.py >> /home/hadoop/openapp.out 2>&1
The python script is something like below:
import sys
import datetime
from fabric.api import local
ystd = datetime.date.today() - datetime.timedelta(days=1)
c = ystd.strftime('%Y-%m-%d')
print(c)
print('Start to format file ...')
......
print('Start to upload on HDFS ...')
local("/home/hadoop/hadoop/bin/hadoop fs -put " + finalfile + " /user/hadoop/yunying/openapp")
print('Start to upload on MaxCompute ...')
......
When the crontab is called, the log file is like:
2016-07-01
Start to format file ...
Start to upload on HDFS ...
[localhost] local: /home/hadoop/hadoop/bin/hadoop fs -put /data/uxin/nsq_client_active_collect/hadoop/openappfinal.log /user/hadoop/yunying/openapp
And then, the process is over. I cannot find it in ps -ef|grep python
Why it comes to an end while meeting local()?
It is likely that the PYTHONPATH it not set up correctly for whatever the user that CRON is using to run the script. Print out the path to a debug file to check:
with open('/path/to/debug_file.txt', 'wt') as f:
f.write(str(sys.path))
Try adding the line:
sys.path.append("/path/to/fabric.api")
before importing local
You can also dynamically get the location of the file that is being run using
import os
os.path.realpath(__file__)
this will allow you to use relative paths if you need them
Related
NOTE:
using cmd in my profile -working
using sch.cmd in my profile -working
using task scheduler running sch.cmd by choosing user is logged in- working
using task scheduler running sch.cmd by choosing user is logged in or not-NOT WORKING. And also A.py run for some 1 or 2 seconds and then shut down without giving result.
NOTE:
All files are running using cmd in my profile and fetching correct results.But not with the windows task scheduler in my profile.
All the permission is already given. I want to list the filename and both methods glob.glob and listdir is not working giving-[ ] empty set or printing nothing at all.
Task scheduler setting:(elastic search is running already and I want to run A.py)
program script:
cmd
arguments:
/c sche.cmd > yash.txt
Start in:
D:\path\
sche.cmd contains:
#echo on
cmd /k "cd /d D:\path\env\Scripts\ & activate & pythonw.exe & cd /d D:\path\files & python A.py"
Now,
NOTE: There is gap in path: '\11.11.11.11\d$\E*ELE In A5*\S\A' but restriction is I just can't change my path as this is path of production.
NOTE: Run with when the user is logged or not AND the highest privileged is enabled.
A.py is:
from elasticsearch import Elasticsearch
import os, glob
import datetime as dt
from datetime import datetime
from dateutil import parser
ES_HOST = {"host": "localhost", "port": 9250}
es = Elasticsearch(hosts=[ES_HOST])
n = 0
a=b=c=d=0
ip = '11.11.11.11'
username = 'abcdefgh'
password = '12345678'
use_dict = {}
use_dict['remote'] = ('\\11.11.11.11\d$\E\ELE In A5\S\A')
use_dict['password'] = ('12345678')
use_dict['username'] = ('abcdefgh')
folder_path = r'\\11.11.11.11\d$\E\ELE In A5\S\A'
print("############", folder_path)
dirContents = os.listdir(r'\\11.11.11.11\d$\E\ELE In A5\S\A')
print("yahoo")
if len(dirContents) == 0:
print('Folder is Empty')
n = 0
else:
print('Folder is Not Empty')
for filename in glob.glob(os.path.join(folder_path, '*')):
with open(filename, 'r', encoding="ISO-8859-1") as f:
text = f.read()
{DO SOMETHING}
**more details are already in unanswered question:
**Running python script in Service account by using windows task scheduler****
I have a bash script for performing the passive checks i.e., external agent/application. I tried converting the bash script into python but when I execute the file I don't see any kind of responses on my nagios core interface regarding my passive check result.
import os
import datetime
CommandFile='/usr/local/nagios/var/rw/nagios.cmd'
datetime = datetime.datetime.now()
os.stat(CommandFile)
f = open(CommandFile, 'w')
f.write("/bin/echo " + str(datetime) + " PROCESS_SERVICE_CHECK_RESULT;compute-1;python dummy;0;I am dummy python")
f.close()
my bash script code is:
#!/bin/sh
# Write a command to the Nagios command file to cause
# it to process a service check result
echocmd="/bin/echo"
CommandFile="/usr/local/nagios/var/rw/nagios.cmd"
# get the current date/time in seconds since UNIX epoch
datetime=`date +%s`
# create the command line to add to the command file
cmdline="[$datetime] PROCESS_SERVICE_CHECK_RESULT;host-name;dummy bash;0;I am dummy bash"
# append the command to the end of the command file
`$echocmd $cmdline >> $CommandFile`
Changed my code, now its working perfectly fine. I can see the response in the Nagios interface.
import time
import sys
HOSTNAME = "compute-1"
service = "python dummy"
return_code = "0"
text = "python dummy is working .....I am python dummy"
timestamp = int(time.time())
nagios_cmd = open("/usr/local/nagios/var/rw/nagios.cmd", "w")
nagios_cmd.write("[{timestamp}] PROCESS_SERVICE_CHECK_RESULT;{hostname};{service};{return_code};{text}\n".format
(timestamp = timestamp,
hostname = HOSTNAME,
service = service,
return_code = return_code,
text = text))
nagios_cmd.close()
I have a shell script from which I am creating some logfile in a directory. And this shell script I am executing in python as below.
cmd = "sh xyz.sh"
try:
subprocess.call(cmd,shell=True)
except OSError:
print "Failed to run the script.:
I want to attach the timestamp on those created logfile which is getting created by xyz.sh script. For this I want to use below python code.
b=time.strftime("%x")
c=time.strftime("%X")
ts=b+"_"+c
I want to attach the result of 'ts' variable on created log file. For example : if logfile is name as abc.log and output of ts is : 6/19/2016_10:20:16 then my logfile should renamed as 6/19/2016_10:20:16_abc.log
Please help me to achieve this.
If you already know the name of the generated logfile, you can use os.rename to rename it. Something along the lines of :
import time, os
b = time.strftime("%x")
c = time.strftime("%X")
ts = b + "_" + c
os.rename("abc.log", ts + "_abc.log")
If you don't know the name of the logfile, you can find it using os.listdir(os.getcwd()), which will give you a list with all the files of the current directory (i.e. the one from where you are running your script).
I've been trying to trouble this for days now, and would appreciate some help --
Basically, I wrote the following Python script
import os, sys
# =__=__=__=__=__=__=__ START MAIN =__=__=__=__=__=__=__
if __name__ == '__main__':
# initialize variables
all_files = []
# directory to download data siphon files to
dDir = '/path/to/download/directory/'
# my S3 bucket
s3bucket = "com.mybucket/"
foldername = "test"
# get a list of available feeds
feeds = <huge JSON object with URLs to feeds>
for item in range(feeds['count']):
# ...check if the directory exists, and if not, create the directory...
if not os.path.exists(folderName):
os.makedirs(folderName)
... ... ...
# Loop through all the splits
for s in dsSplits:
... ... ...
location = requestFeedLocation(name, timestamp)
... ... ...
downloadFeed(location[0], folderName, nameNotGZ)
# THIS IS WHERE I AM HAVING PROBLEMS!!!!!!!!!!!
cmd = 's3cmd sync 'dDir+folderName+'/ s3://'+s3bucket+'/'
os.system(cmd)
Everything in my code works...when I run this straight from the command line, everything runs as expected...however, when I have it executed via cron -- the following DOES NOT execute (everything else does)
# THIS IS WHERE I AM HAVING PROBLEMS!!!!!!!!!!!
cmd = 's3cmd sync 'dDir+folderName+'/ s3://'+s3bucket+'/'
os.system(cmd)
To answer a few questions, I am running the cron as root, s3cmd is configured for the root user, OS is Ubuntu 12.04, python version is 2.7, all of the necessary directories have Read / Write permissions...
What am I missing?
First check variable 'foldername' in command you have used N in variable.
In command syntax you need to add plus (+) sign in prefix like +dDir+folderName+
So I hope command would be like below..
cmd = 's3cmd sync '+dDir+foldername+'/ s3://'+s3bucket+'/'
os.system(cmd)
Here I found some more s3cmd sync help: http://tecadmin.net/s3cmd-file-sync-with-s3bucket/
OpenShift has these default dir's:
# $_ENV['OPENSHIFT_INTERNAL_IP'] - IP Address assigned to the application
# $_ENV['OPENSHIFT_GEAR_NAME'] - Application name
# $_ENV['OPENSHIFT_GEAR_DIR'] - Application dir
# $_ENV['OPENSHIFT_DATA_DIR'] - For persistent storage (between pushes)
# $_ENV['OPENSHIFT_TMP_DIR'] - Temp storage (unmodified files deleted after 10 days)
How do reference them in a python script?
Example script "created a log file in log directory and log in data directory?
from time import strftime
now= strftime("%Y-%m-%d %H:%M:%S")
fn = "${OPENSHIFT_LOG_DIR}/test.log"
fn2 = "${OPENSHIFT_DATA_DIR}/test.log"
#fn = "test.txt"
input = "appended text " + now + " \n"
with open(fn, "ab") as f:
f.write(input)
with open(fn2, "ab") as f:
f.write(input)
Can these script be used with cron?
EDIT the BASH File:
#! /bin/bash
#date >> ${OPENSHIFT_LOG_DIR}/new.log
source $OPENSHIFT_HOMEDIR/python-2.6/virtenv/bin/activate
python file.py
date >> ${OPENSHIFT_DATA_DIR}/new2data.log
import os
os.getenv("OPENSHIFT_INTERNAL_IP")
should work.
So with your example, modify to:-
import os
OPENSHIFT_LOG_DIR = os.getenv("OPENSHIFT_LOG_DIR")
fn = os.path.join(OPENSHIFT_LOG_DIR, "test.log")
And, yes, you can call this python script with a cron by referencing your bash script if you want... Like this for example:-
#!/bin/bash
date >> ${OPENSHIFT_LOG_DIR}/status.log
chmod +x status
cd ${OPENSHIFT_REPO_DIR}/wsgi/crawler
nohup python file.py 2>&1 &
Those variables OPENSHIFT_* are provided as environment variables on OpenShift -- so the $_ENV["OPENSHIFT_LOG_DIR"] is an example to get the value inside a php script.
In python, the equivalent would just be os.getenv("OPENSHIFT_LOG_DIR").
Made edits to Calvin's post above and submitted 'em.
Re: the question of where file.py exists -- use os.getenv("OPENSHIFT_REPO_DIR") as the base directory where all your code would be located on the gear where you app is running.
So if your file is located in .openshift/misc/file.py -- then just use:
os.path.join(os.getenv("OPENSHIFT_REPO_DIR"), ".openshift", "misc", "file.py")
to get the full path.
Or in bash, the equivalent would be:
$OPENSHIFT_REPO_DIR/.openshift/misc/file.py
HTH