I was working with the Tornado webserver and wanted to understand on how to determine if the server has been idle for a while and then restart it.
I was trying to determine the time difference between self.requests to determine if no request has been received for a while. But is there a better way to do it?
One simple way is to use a global variable to hold the timestamp of the last request and update it whenever you process requests. Something like:
#startup initialization code
_last_request = datetime.datetime.now()
#we will use these variables later
interval = datetime.timedelta(seconds=30)
idle_timeout = datetime.timedelta(seconds=1800)
...
#update timestamp in handler(s)
global _last_request
_last_request = datetime.datetime.now()
You will then add a function which checks from time to time the _last_request variable:
def idle_check():
if _last_request - datetime.datetime.now() > idle_timeout:
#stop the IOLoop
tornado.ioloop.IOLoop.instance().stop()
else:
#schedule to run again
tornado.ioloop.IOLoop.instance().add_timeout(interval, idle_check)
Do not forget to call the idle_check function before starting the IOLoop.
Related
I'm creating HMI web application which visualise realtime data from simulation.
It means I'm updating data every 0.25/0.5 seconds. I decided to use socket.io.
So I want my server after connection to emit data every some period of time. I thought the best option would be emitting data in something similar like setInterval in JavaScript. However, in python this has not proved too easy. I tried a lot of options from stackoverflow f.e:
Python Equivalent of setInterval()?
But most of them are causing errors. Here are some methods which I tried.
#socketio.on('signalList')
def start_polling_signals():
global poll_signals
poll_signals = threading.Timer(1000, start_polling_signals())
poll_signals.start()
list_signals_v2()
print('polling')
#socketio.on('stopSignalList')
def stop_polling_signals():
global poll_signals
poll_signals.cancel()
poll_signals = None
Causes maximum recursion depth exceeded
#socketio.on('signalList')
def start_polling_signals():
starttime = time.time()
while True:
list_signals_v2()
time.sleep(1 - ((time.time() - starttime) % 1))
Causes that other socket events don't work, and I have no idea how to stop that polling.
Any ideas how to do it in optimal way? Have in mind I need to be able to start and stop the interval.
You should start a background task using the
start_background_task(target, *args, **kwargs)
This is the correct way to init something like what you want to do. Pass as target a function that will send data in the period of time you want it to send data.
I have a Django web app which is used by embedded systems to upload regular data, currently every 2 minutes, to the server where Django just pops it into a database.
I'd like to create an alert system where by if there's no data uploaded from the remote system in a time period, say 10 minutes for example, I raise an alarm on the server, via email or something.
In other programming languages/environments I'd create a 10 minute timer to execute a function in 10 minutes, but every time data is uploaded I'd restart the timer. Thus hopefully the timer would never expire and the expiry function would never get called.
I might well have missed something obvious but if there is something I have missed it. This just does not seem possible in Python. Have I missed something?
At present looks like I need an external daemon monitoring the database :-(
You could use the time module for this:
import time
def didEventHappen():
# insert appropriate logic here to check
# for what you want to check for every 10 minutes
value = True # this is just a placeholder so the code runs
return value
def notifyServer():
print("Hello server, the event happened")
start = time.clock()
delay = 10 * 60 # 10 minutes, converted to seconds
while True:
interval = time.clock() - start
eventHappened = False
if interval >= delay:
eventHappened = didEventHappen()
start = time.clock() # reset the timer
if eventHappened:
notifyServer()
else:
print("event did not happen")
Alternatively, you could use the sched module.
I have a python app where user can initiate a certain task.
The whole purpose of a task is too execute a given number of POST/GET requests with a particular interval to a given URL.
So user gives N - number of requests, V - number of requests per second.
How is it better to design such a task taking into account that due to a I/O latency the actual r/s speed could bigger or smaller.
First of all I decided to use Celery with Eventlet because otherwise I would need dozen of works which is not acceptable.
My naive approach:
Client starts a task using task.delay()
Inside task I do something like this:
#task
def task(number_of_requests, time_period):
for _ in range(number_of_requests):
start = time.time()
params_for_concrete_subtask = ...
# .... do some IO with monkey_patched eventlet requests library
elapsed = (time.time() - start)
# If we completed this subtask to fast
if elapsed < time_period / number_of_requests:
eventlet.sleep(time_period / number_of_requests)
A working example is here.
if we are too fast we try to wait to keep the desired speed. If we are too slow it's ok from client's prospective. We do not violate requests/second requirement. But will this resume correctly if I restart Celery?
I think this should work but I thought there is a better way.
In Celery I can define a task with a particular rate limit which will almost match my needs guarantee. So I could use Celery group feature and write:
#task(rate_limit=...)
def task(...):
#
task_executor = task.s(number_of_requests, time_period)
group(task_executor(params_for_concrete_task) for params_for_concrete_task in ...).delay()
But here I hardcode the the rate_limit which is dynamic and I do not see a way of changing it. I saw an example:
task.s(....).set(... params ...)
But I tried to pass rate_limit to the set method it it did not work.
Another maybe a bettre idea was to use Celery's periodic task scheduler. With the default implementation periods and tasks to be executed periodically is fixed.
I need to be able to dynamically create tasks, which run periodically a given number of times with a specific rate limit. Maybe I need to run my own Scheduler which will take tasks from DB? But I do not see any documentation around this.
Another approach was to try to use a chain function, but I could not figure out is there a delay between tasks parameter.
If you want to adjust the rate_limit dynamically you can do it using the following code. It is also creating the chain() at runtime.
Run this you will see that we successfully override the rate_limit of 5/sec to 0.5/sec.
test_tasks.py
from celery import Celery, signature, chain
import datetime as dt
app = Celery('test_tasks')
app.config_from_object('celery_config')
#app.task(bind=True, rate_limit=5)
def test_1(self):
print dt.datetime.now()
app.control.broadcast('rate_limit',
arguments={'task_name': 'test_tasks.test_1',
'rate_limit': 0.5})
test_task = signature('test_tasks.test_1').set(immutable=True)
l = [test_task] * 100
chain = chain(*l)
res = chain()
I also tried to override the attribute from within the class, but IMO the rate_limit is set when the task is registered by the worker, that is why the .set() has no effects. I'm speculating here, one would have to check the source code.
Solution 2
Implement your own waiting mechanism using the end time of the previous call, in the chain the return of the function is passed to the next one.
So it would look like this:
from celery import Celery, signature, chain
import datetime as dt
import time
app = Celery('test_tasks')
app.config_from_object('celery_config')
#app.task(bind=True)
def test_1(self, prev_endtime=dt.datetime.now(), wait_seconds=5):
wait = dt.timedelta(seconds=wait_seconds)
print dt.datetime.now() - prev_endtime
wait = wait - (dt.datetime.now() - prev_endtime)
wait = wait.seconds
print wait
time.sleep(max(0, wait))
now = dt.datetime.now()
print now
return now
#app.control.rate_limit('test_tasks.test_1', '0.5')
test_task = signature('test_tasks.test_1')
l = [test_task] * 100
chain = chain(*l)
res = chain()
I think this is actually more reliable than the broadcast.
I would like to implement a cron-like behaviour with my twisted application.
I want to trigger a periodic call (let's say every week) but running at a precise time only, not when i start my application.
My use case is the following:
my python application is started at any time in the week. I want the calls to be performed every monday at 8am.
But I don't want to perorm active waiting (using a time.sleep()), i would like to use callLater to trigger the call next monday and then start a looping call from that date.
any idea/advice?
thanks,
J.
If you are absolutely in love with cron-style specifiers, you could also consider using parse-crontab
Then your code looks basically like:
from crontab import CronTab
monday_morning = CronTab("0 8 * * 1")
def do_something():
reactor.callLater(monday_morning.next(), do_something)
# do whatever you want!
reactor.callLater(monday_morning.next(), do_something)
reactor.run()
If I understood your question correctly you are thinking of first time execution of a scheduled task and how to supply initial start time for the app. If this is a case, you just need to calculate timedelta value in seconds to be passed to callLater.
import datetime
from twisted.internet import reactor
def cron_entry():
full_weekseconds = 7*24*60*60
print "I was called at a specified time, now you can add looping task with a full weekseconds frequency"
def get_seconds_till_next_event(isoweekday,hour,minute,second):
now = datetime.datetime.now()
full_weekseconds = 7*24*60*60
schedule_weekseconds = ((((isoweekday*24)+hour)*60+minute)*60+second)
now_weekseconds=((((now.isoweekday()*24)+now.hour)*60+now.minute)*60+now.second)
if schedule_weekseconds > now_weekseconds:
return schedule_weekseconds - now_weekseconds
else:
return now_weekseconds - schedule_weekseconds + full_weekseconds
initial_execution_timedelta = get_seconds_till_next_event(3,2,25,1)
"""
This gets a delta in seconds between now and next Wednesday -3, 02 hours, 25 minutes and 01 second
"""
reactor.callLater(initial_execution_timedelta,cron_entry)
reactor.run()
I'm working on a python script that needs to run between two given times. I'm required to use the build in sched module as this script needs to be able to run directly on any machine that has python 2.7 as to reduce configuration time. (SO CRON IS NOT AN OPTION)
A few variables define the settings for the time to run, here set_timer_start=0600 and set_timer_end=0900 are written in HHMM. I'm able to stop the script at the right time.
I don't know exactly how sched works (the python doc page doesn't make to much sense to me), but as far as I understand It runs at a date/time (epoch) while I only want it to run at a given time (HHMM).
Can anyone give me an example (or link) on how to use the scheduler and perhaps calculate the next run date/time?
If I got your requirements right, what you need is probably a loop, that will re-enter a task in the queue every time it will be executed. Something along the lines of:
# This code assumes you have created a function called "func"
# that returns the time at which the next execution should happen.
s = sched.scheduler(time.time, time.sleep)
while True:
if not s.queue(): # Return True if there are no events scheduled
time_next_run = func()
s.enterabs(time_next_run, 1, <task_to_schedule_here>, <args_for_the_task>)
else:
time.sleep(1800) # Minimum interval between task executions
However, using the scheduler is - IMO - overkilling. Using datetime objects could suffice, for example a basic implementation would look like:
from datetime import datetime as dt
while True:
if dt.now().hour in range(start, stop): #start, stop are integers (eg: 6, 9)
# call to your scheduled task goes here
time.sleep(60) # Minimum interval between task executions
else:
time.sleep(10) # The else clause is not necessary but would prevent the program to keep the CPU busy.
HTH!