The schedule library allows you to schedule jobs at specific times of the day. For example:
schedule.every().day.at("10:30").do(job)
However its documentation doesn't mention the time reference it uses.
Is it based on UTC time?
Or is it based on the time zone of the server that runs my python script?
If you look at the code of the library, you will find that the function datetime.datetime.now() is used (here fore example).
When looking at the now documentation, you will see that without argument, it is the local time that it is used.
Thus, schedule.every().day.at("10:30").do(job) will use the server time zone.
Related
I'm trying to implement SLA in my airflow DAG.
I know how SLAs work, you set a timedelta object and if the task does not get done in that duration, it will send an email and notifies that the task is not done yet.
I want some similar functionality, but instead of giving duration, I want to set specific time in SLA. For example, if the task is not done due to 8:00 AM, it sends the email and notifies the manager. Something like this:
'sla': time(hour=8, minute=0, second=0)
I have searched a lot, but nothing found.
Is there any solution for this specific problem? or any other solutions than SLA?
Thanks in advance.
SLA param of BaseOperator expects a datetime.timedelta object, so there is nothing more to do there. Take into consideration that SLA represents a time delta after the scheduled period is over. The example from the docs supposes a DAG scheduled daily:
For example if you set an SLA of 1 hour, the scheduler would send an email soon after 1:00AM on the 2016-01-02 if the 2016-01-01 instance has not succeeded yet.
The point is, it's always a time delta from the schedule period which is not what you are looking for.
So I think you should take another approach, like schedule your DAG whenever you need it, execute the tasks you want and then add a sensor operator to check if the condition you are looking for is met or not. There are a few types of sensors depending on the context you have you could choose from them.
Another option could be, create a new DAG dedicated to check if your tasks executed in the original DAG were successfully executed or not, and act accordingly (for example, send emails, etc.). To do this you could use an ExternalTaskSensor, check online for tutorials on how to implement it, although it may be simpler to avoid cross DAG dependencies as stated in the docs.
Hope that this could point you into the right direction.
Currently making use of datetime.datetime.now() from python code to obtain the date/time stamp. However this is the date/timezone that is set on the system which can be changed/altered.
How can i retrieve the real time i.e RTC time from python. Any help will be greatly appreciated.
If you are using Linux/Unix, you can get the system-wide real time clock with the time module from the standard library, as follows:
import time
rtc = time.clock_gettime(time.CLOCK_REALTIME)
print("System RTC = {rtc}")
> System RTC = 1549619678.899073
For Windows, RTC is not available. There are a variety of other clocks you could use, depending on your application, which wouldn't be affected by updates to system time. For instance, if you are trying to measure time between two separate calls, and don't want this to be affected by changes to system datetime/timezone, you can use time.monotonic() which is available on Windows as well. However, is only useful relative to another call to time.monotonic() (i.e. for measuring duration) and does not have a defined reference point, so you can't do a call to time.monotonic() to ask "what time is it?"
I have a scheduling function and a scheduler with a queue of future events ordered by time. I'm using UNIX timestamps and the regular time.time(). One fragment of the scheduler is roughly equivalent to this:
# select the nearest event (eventfunc at eventime)
sleeptime = eventtime - time.time()
# if the sleep gets interrupted,
# the whole block will be restarted
interruptible_sleep(sleeptime)
eventfunc()
where the eventtime could be computed either based on a delay:
eventtime = time.time() + delay_seconds
or based on an exact date and time, e.g.:
eventtime = datetime(year,month,day,hour,min).timestamp()
Now we have the monotonic time in Python. I'm considering to modify the scheduler to use the monotonic time. Schedulers are supposed to use the monotonic time they say.
No problem with delays:
sleeptime = eventtime - time.monotonic()
where:
eventtime = time.monotonic() + delay_seconds
But with the exact time I think the best way is to leave the code as it is. Is that correct?
If yes, I would need two event queues, one based on monotonic time and one based on regular time. I don't like that idea much.
As I said in the comment, your code duplicates the functionality of the sched standard module - so you can as well use solving this problem as a convenient excuse to migrate to it.
That said,
what you're supposed to do if system time jumps forward or backward is task-specific.
time.monotonic() is designed for cases when you need to do things with set intervals between them regardless of anything
So, if your solution is expected to instead react to time jumps by running scheduled tasks sooner or later than it otherwise would, in accordance with the new system time, you have no reason to use monotonic time.
If you wish to do both, then you either need two schedulers, or tasks with timestamps of the two kinds.
In the latter case, the scheduler will need to convert one type to the other (every time it calculates how much to wait/whether to run the next task) - for which time provides no means.
I need a function to execute every time the date changes. Currently I'm checking in a loop to see if the date changed, but I'm looking for a more effective method....in Python
Any help appreciated
What you really want to do is schedule a function to be run at a certain time. You need to do this with a scheduling mechanism. You could, of course, write one yourself, but probably the best way to go would be to use a library that does this for you.
APScheduler is a very mature good library for just this sort of thing.
Docs: http://apscheduler.readthedocs.org/en/latest/
Pypi: https://pypi.python.org/pypi/APScheduler/3.0.0
Example
Here is a quick little example
from apscheduler.schedulers.background import BlockingScheduler
scheduler = BlockingScheduler()
#scheduler.scheduled_job('interval', seconds=5, timezone='UTC')
def hello():
print('Hello!')
scheduler.start()
This will run the function hello every five seconds. You can change seconds=5 to days=1 to have it run once a day. There is much more configuration you can do, so you'll probably want to read the documentation. It is able to express just about any date time format you could want, including cron.
It also supports different types of schedulers, for instance I chose a BlockingScheduler because wanted the entire program to run as a function of the scheduling mechanism (so you could try this out easily on your own system). You can also use, for instance, a BackgroundScheduler which will allow you to schedule tasks from within your program in an efficient manner that will not block the main thread (fixes your going in a loop forever problem).
How do I schedule a task to run once every six hours (on repeat)?
I am trying to implement a Redis queue for the first time.
I went through Heroku's tutorial : https://devcenter.heroku.com/articles/python-rq
But the tutorial did not explain how to run a task repeatedly with a timeframe (such as checking a couple of websites for info, once every six hours)
Also, since I am new to do this, if I should not be using Redis for such a task, please let me know what I should be using to check a couple of websites for info once every six hours
Thanks
You don't need Redis for this functionality at all.
Take a look at the Heroku Scheduler here: https://devcenter.heroku.com/articles/scheduler
You can set this to run your code every hour, and have your code check if the current hour is 0,5,11,17 (or whatever other interval you may need).