I have a python script that is triggered by CRON every 5 mins. This python script then does some checks and then calls another python script many times with different configurations. Those other python scripts are trading bots and they need to be called separately for each trading pair so I end up calling this second script about 50 times (each time with a different config).
The problem I have is that between my first python script being started by CRON and then the subsequent python scripts being called, about 5 to 8 seconds pass (I am using subprocess.Popen to open the secondary scripts) and this 8 seconds makes a huge difference in trading.
So what I would like to know, is if there is a way to offset my system clock by say, 5 seconds so that the secondary scripts are called as close as possible to the actual start of the minute?
Related
I have created python action which runs for 30 minutes. When I try to invoke and run that locally, action gets invoked but I'm not able to see the result. I have tried to extend execution timeout using --timeout but I think threshold value is 5 minutes. Can I execute action for more than 5 minutes using open whisk? I'm new to open whisk, any help would be appreciated.
My (simplified and generalized) need is the following:
Given is a main program which starts my Python script every minute. By design, this script is intended to be a short-runner.
Its goal is to call different web hooks - not on every call, but in individual intervals, e.g. every 5 minutes. (The script can read the configuration from a file.)
So it is not possible for me to use apscheduler in a permanent running program.
On the contrary, the program itself must determine on each run which API calls are overdue and have to be made now.
Is it possible to use apscheduler for this?
I am writing a python script to transfer large files via sftp with the pysftp module. I have a massive amount of data to transfer, a total of around 36Tb, divided in 54 runs, or batches.
I want only to carry out these transfers between certain hours of the day, for this example, between 6pm and 7am. So my idea is to use a for loop to iterate over all the runs/ batches. Upon each iteration, I would check what hour it is. If it is between 6pm and 7am I would transfer. Else the script would sleep until it is 6pm minimum. The code that I wrote looks like so:
runsList = 'runA runB runC'.split() # these are directories
# time constraints
bottomLimit = 7
upperLimit = 18
doNotUploadRange = range(bottomLimit, upperLimit)
for run in runsList:
hour = dt.datetime.now().hour
while hour in doNotUploadRange:
print('do not upload now')
time.sleep(1800)
hour = dt.datetime.now().hour
# when I leave the while condition above
# do the transfer via pysftp (large amount of data) per run
The question here does not concern the code itself not I want to check whether or not the script is running (which can be checked with htop), but I am concerned that my script will crash, for whatever reason, before it finishes (perhaps it would be running for a full week if nothing crashes).
I do sometimes call scripts that run for a very long time and they do crash sometimes, with no obvious reasons for crash.
So my question is whether it is, for whatever reason, obvious that the script will crash after running for 6-7 days of can I expect it to finish provided that there is no error in the code itself? My idea is to call this script on the background, inside tmux I would python script.py &
I am currently working on a program that needs to run every 14 days. I have looked into Schedule which works fine, but I have a few doubts about how to go about this.
I will create a service which will handle the execution of the python program itself on a CentOS 7 system.
The issue here is that every 14 days I will run a function that generates a lot of email addresses and send them to a support entity. I am afraid that if something unintended happens, and the program restart - the support entity will get spammed with emails outside the time frame in which they should receive emails.
As far as I can tell, Schedule does not have any way of determining if the program has restarted, and therefore a reboot of either the system or the service will cause this behaviour.
Would it be a correct solution to write a date to a text file after each completed function run, and then check that text file once a day to determine whether the function should run or not? This method would survive a service and/or system reboot, but is it a "correct" way of doing it?
****UPDATE**** Having the cronjob run on specific days of the month (for example 1st and 15th.) is not sufficient. This could cause gaps in the data which the program processes. The script makes a call which pulls data from 14 days back, and this is the maximum number of days supported by the script (licensing and stuff, can't be changed so not that important except that it is a limitation). So it need to run on lets say odd or even week numbers (to get 14 days).
Any ideas on how to accomplish this given this new information?.
You should look into the use of cron (or google it yourself if you dont like the link).
I suggest creating a simple Python script that is called by cron every 14 days. The crontab entry could look like the following:
# this will run at 00:01 on the 15th and 30th of every month
1 0 */15 * * /path/to/python/script.py
# this will run at 00:01 on the 1st and 15th of every month
1 0 1,15 * * /path/to/python/script.py
You still could make your script write some sort of result (with maybe a timestamp) to a file, so that you could easily check that file to see if it ran correctly (or log some error info).
# this will run at 00:01 on the 1st and 15th of every month
1 0 1,15 * * /path/to/python/script.py >> /path/to/logfile.log 2>&1
EDIT
You can also configure cron to run every Monday (or another day) if the 1st and 15th of every month are not sufficient. And the script could check a log file to see if it was run the previous Monday to assure it only executes your business logic every 2 weeks.
# this will run at 00:01 once a week on Mondays
1 0 * * 1 /path/to/python/script.py >> /path/to/logfile.log 2>&1
I have a Python program I run at all times of the day that alerts me when something I am looking for online is posted. I want to give this to my employees but I only want to have it email them during business hours.
I have a Ubuntu server and use a .sh. I have a command in crontab that runs on startup.
How do I make my command run from 9-5?
You could create a cronjob that starts the script every 5 minutes (or whatever often you want it to run) and additionally modify the script such that it creates a .lock file which it removes on exiting, but if it encounters it at the beginning it won't do anything (this way you don't have a long-running script active multiple times).