I am trying to use the APScheduler and SendGrid on Heroku to create a cron job for sending an email.
Even though the add_job method call seems to be executing correctly, I am getting the following error.
Below are the logs from heroku
2016-09-11T22:33:37.776867+00:00 heroku[clock.1]: State changed from crashed to starting
2016-09-11T22:33:40.672563+00:00 heroku[clock.1]: Starting process with command `python clock.py`
2016-09-11T22:33:41.353373+00:00 heroku[clock.1]: State changed from starting to up
2016-09-11T22:33:43.527949+00:00 app[clock.1]: created background scheduler
2016-09-11T22:33:43.528848+00:00 app[clock.1]: started background scheduler
2016-09-11T22:33:43.572751+00:00 app[clock.1]: added cron job
2016-09-11T22:33:43.585801+00:00 app[clock.1]: Exception in thread APScheduler (most likely raised during interpreter shutdown):
2016-09-11T22:33:43.585807+00:00 app[clock.1]: Traceback (most recent call last):
2016-09-11T22:33:43.585808+00:00 app[clock.1]: File "/app/.heroku/python/lib/python2.7/threading.py", line 801, in __bootstrap_inner
2016-09-11T22:33:43.585810+00:00 app[clock.1]: File "/app/.heroku/python/lib/python2.7/threading.py", line 754, in run
2016-09-11T22:33:43.585827+00:00 app[clock.1]: File "/app/.heroku/python/lib/python2.7/site-packages/apscheduler/schedulers/blocking.py", line 29, in _main_loop
2016-09-11T22:33:43.585829+00:00 app[clock.1]: File "/app/.heroku/python/lib/python2.7/threading.py", line 614, in wait
2016-09-11T22:33:43.585848+00:00 app[clock.1]: File "/app/.heroku/python/lib/python2.7/threading.py", line 364, in wait
2016-09-11T22:33:43.585851+00:00 app[clock.1]: <type 'exceptions.ValueError'>: list.remove(x): x not in list
2016-09-11T22:33:43.695569+00:00 heroku[clock.1]: Process exited with status 0
2016-09-11T22:33:43.719265+00:00 heroku[clock.1]: State changed from up to crashed
I am running one clock process, which is in the clock.py file below.
from apscheduler.schedulers.background import BackgroundScheduler
import sendgrid
import os
from sendgrid.helpers.mail import *
def send_email():
try:
sg = sendgrid.SendGridAPIClient(apikey=os.environ.get('SENDGRID_API_KEY'))
print("created send grid api client")
from_email = Email("ohta.g#husky.neu.edu")
print("created from email")
subject = "Weekly update"
to_email = Email("ohta.g#husky.neu.edu")
print("created to email")
content = Content("text/plain", "Hello, Email!")
print("created content")
mail = Mail(from_email, subject, to_email, content)
print("created mail")
response = sg.client.mail.send.post(request_body=mail.get())
except Exception as e:
return e
try:
sched = BackgroundScheduler()
print("created background scheduler")
sched.start()
print("started background scheduler")
sched.add_job(send_email, 'cron', day_of_week=6, hour=22, minute=20)
print("added cron job")
except Exception as e:
print e.message
Here is my Procfile.
clock: python clock.py
Here is my requirements.txt file.
APScheduler==3.1.0
sendgrid==3.4.0
Could someone please tell me what I'm doing wrong?
You started a background scheduler but then allowed your main thread to exit, which also exits the clock process. This is the entire reason why BlockingScheduler exists. Have you not read Heroku's APScheduler instructions?
Related
I am using multiprocessing.pool to work with the http server in python - it works great, but when I terminate, I get a slew of errors from all the spawnpoolworkers - and I'm just wondering how I avoid this.
My main code:
def run(self):
global pool
port = self.arguments.port
pool = multiprocessing.Pool( processes= self.arguments.threads)
with http.server.HTTPServer( ("", port), Handler ) as daemon:
print(f"serving on port {port}")
while True:
try:
daemon.handle_request()
except KeyboardInterrupt:
print("\nexiting")
pool.terminate()
pool.join()
return 0
I've tried doing nothing to the pool, I've tried doing pool.close() - I've tried not joining. But even if I just run that - never even access the port or call anything onto the pool, I still get a random list of things like this when I press control-c
Process SpawnPoolWorker-8:
Process SpawnPoolWorker-4:
Traceback (most recent call last):
File "/opt/homebrew/Cellar/python#3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/homebrew/Cellar/python#3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/opt/homebrew/Cellar/python#3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/pool.py", line 114, in worker
task = get()
File "/opt/homebrew/Cellar/python#3.10/3.10.1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/queues.py", line 365, in get
with self._rlock:
File "/opt/homebrew/Cellar/python#3.10/3.10.1/F
how do I exit the pool cleanly, with no errors, and with no output?
ok - I'm stupid - the control-c was also interrupting all the child processes. This fixed it:
def ignore_control_c():
signal.signal(signal.SIGINT, signal.SIG_IGN)
pool = multiprocessing.Pool( processes = self.arguments.threads, initializer = ignore_control_c )
I have redis installed and running (Redis version=6.0.9 on a Ubuntu 18.04 machine) and followed this simple guide to schedule a job with rq but as soon as I launch app.py redis worker fails with:
10:31:17 RQ worker u'rq:worker:c73feeafc20f4f0c8baff1e5225e73f1' started, version 1.0
10:31:17 *** Listening on default...
10:31:18 Worker c73feeafc20f4f0c8baff1e5225e73f1: found an unhandled exception, quitting...
Traceback (most recent call last):
File "/home/enrico/.local/lib/python2.7/site-packages/rq/worker.py", line 470, in work
result = self.dequeue_job_and_maintain_ttl(timeout)
File "/home/enrico/.local/lib/python2.7/site-packages/rq/worker.py", line 514, in dequeue_job_and_maintain_ttl
job_class=self.job_class)
File "/home/enrico/.local/lib/python2.7/site-packages/rq/queue.py", line 468, in dequeue_any
job = job_class.fetch(job_id, connection=connection)
File "/home/enrico/.local/lib/python2.7/site-packages/rq/job.py", line 287, in fetch
job.refresh()
File "/home/enrico/.local/lib/python2.7/site-packages/rq/job.py", line 428, in refresh
self.started_at = to_date(as_text(obj.get('started_at')))
File "/home/enrico/.local/lib/python2.7/site-packages/rq/job.py", line 411, in to_date
return utcparse(as_text(date_str))
File "/home/enrico/.local/lib/python2.7/site-packages/rq/utils.py", line 175, in utcparse
return datetime.datetime.strptime(string, '%Y-%m-%dT%H:%M:%SZ')
File "/usr/lib/python2.7/_strptime.py", line 332, in _strptime
(data_string, format))
ValueError: time data '' does not match format '%Y-%m-%dT%H:%M:%SZ'
Here tasks.py:
from datetime import datetime, timedelta
import time
def print_task(seconds):
print("Starting task")
for num in range(seconds):
print(num, ". Hello World!")
time.sleep(1)
print("Task completed")
def print_numbers(seconds):
print("Starting num task")
for num in range(seconds):
print(num)
time.sleep(1)
print("Task to print_numbers completed")
Here app.py:
from datetime import datetime, timedelta
import time
from redis import Redis
from rq import Queue
import tasks
queue = Queue(connection=Redis())
def queue_tasks():
queue.enqueue(tasks.print_task, 5)
queue.enqueue_in(timedelta(seconds=10), tasks.print_numbers, 5)
def main():
queue_tasks()
if __name__ == "__main__":
main()
Here the commands I used:
python3.8 -m venv venv
source venv/bin/activate
pip install rq ### Successfully installed click-7.1.2 redis-3.5.3 rq-1.7.0
python app.py
I open a second window and run rq worker and I got the fail in the first part of this post.
I tried to google around but this post just said that he updated his windows OS, this that was a resolved bug...does anybody got a clue on how to solve this?
I am adding job in redis and on completion of job I have added an event handler.
In eventhandler I am returning value based on which I am removing job id from jobstore. It is removed successfully but immediately it throws an exception.
Code
from datetime import datetime
from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.events import EVENT_JOB_EXECUTED
import logging
logging.basicConfig()
scheduler = BackgroundScheduler()
scheduler.add_jobstore('redis')
scheduler.start()
def tick():
print('Tick! The time is: %s' % datetime.now())
return 'success'
def removing_jobs(event):
if event.retval == 'success':
scheduler.remove_job(event.job_id)
scheduler.add_listener(removing_jobs, EVENT_JOB_EXECUTED)
try:
count = 0
while True:
count += 1
time.sleep(10)
job_ret = scheduler.add_job(tick, 'interval', id = str(count), seconds=10)
except (KeyboardInterrupt, SystemExit):
scheduler.shutdown()
Exception
Exception in thread APScheduler:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/.virtualenvs/py3/lib/python3.5/site-packages/apscheduler/schedulers/blocking.py", line 30, in _main_loop
wait_seconds = self._process_jobs()
File "/.virtualenvs/py3/lib/python3.5/site-packages/apscheduler/schedulers/base.py", line 995, in _process_jobs
jobstore.update_job(job)
File "/.virtualenvs/py3/lib/python3.5/site-packages/apscheduler/jobstores/redis.py", line 91, in update_job
raise JobLookupError(job.id)
apscheduler.jobstores.base.JobLookupError: 'No job by the id of 1 was found'
In short: you are removing the job while it is processed;
so you should remove the job outside its execution.
That's because the scheduler doesn't know what the job's execution will do; so it launches tick and sends a job object to the redis jobstore thinking it will be executed again. Before that, the EVENT_JOB_LISTENER launches removing_jobs.
The problem is that when the redis' jobstore gets the job for update its status, it is already deleted so it raises the JobLookupError.
I installed RabbitMQ, Celery, Flask and Python but when I tried to run celery worker to test. It does not work, these are the error that was pop out in the cmd.
[2019-01-18 09:56:37,443: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
File "c:\users\ansonkho\anaconda3\lib\site-packages\celery\worker\consumer\consumer.py", line 317, in start
blueprint.start(self)
File "c:\users\ansonkho\anaconda3\lib\site-packages\celery\bootsteps.py", line 119, in start
step.start(parent)
File "c:\users\ansonkho\anaconda3\lib\site-packages\celery\worker\consumer\mingle.py", line 40, in start
self.sync(c)
File "c:\users\ansonkho\anaconda3\lib\site-packages\celery\worker\consumer\mingle.py", line 44, in sync
replies = self.send_hello(c)
File "c:\users\ansonkho\anaconda3\lib\site-packages\celery\worker\consumer\mingle.py", line 57, in send_hello
replies = inspect.hello(c.hostname, our_revoked._data) or {}
below is my code:
from celery import Celery
app = Celery('test_celery', broker='amqp://myuser:mypassword#localhost/myvhost',backend='rpc://')
[2019-01-18 09:56:37,443: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
As mentioned in the error, there is no broker running. You need to start Rabbitmq before making a connection to it. That is why the consumer is throwing Connection to broker lost as the broker is not running.
I am trying to read meta info from celery task in case of timeout (if task is not finished in given time). I have 3 celery workers. When I execute tasks on 3 workers serially my timeout logic (getting meta info from redis backend) works fine. But, when I execute tasks in parallel using threads, I get error 'AttributeError: 'DisabledBackend' object has no attribute '_get_task_meta_for''.
main script.
from threading import Thread
from util.tasks import app
from celery.exceptions import TimeoutError
# from celery.task.control import revoke
from celery.result import AsyncResult
def run(cmd, workerName, async=False, timeout=9999999):
print "Executing Celery cmd: ", cmd
ret = app.send_task(workerName+'.run_cmd', args=[cmd], kwargs={}, queue=workerName)
if async:
return ret
else:
try:
return ret.get(timeout=timeout)
except TimeoutError:
task = AsyncResult(ret.task_id)
# print task.info
out = task.info['PROGRESS']
# stop_task(ret.task_id)
print 'TIMEOUT', out
return 'TIMEOUT', out
cmd = r'ping 10.10.10.10'
threads = []
# this block works
print "This block works"
run(cmd, 'MH_VTF203', timeout=10)
run(cmd, 'MH_VTF1661', timeout=10)
run(cmd, 'MH_VTF106', timeout=10)
# this block errors
print "This block erros"
for vtf in ['MH_VTF203', 'MH_VTF1661', 'MH_VTF106']:
t = Thread(target=run, args=[cmd, vtf], kwargs={'timeout': 10})
t.start()
threads.append(t)
for t in threads:
t.join()
util.tasks.py
from celery import Celery
import subprocess
app = Celery('tasks', backend='redis://', broker='redis://localhost:6379/0')
app.conf.CELERY_IGNORE_RESULT = False
app.conf.CELERY_RESULT_BACKEND = 'redis://localhost:6379/0'
#app.task()
def run_cmd(*args, **kwargs):
cmd = " ".join(args)
print "executing command :",cmd
try:
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
out = ""
while p.poll() is None:
l = p.stdout.readline()
print l
out += l
run_cmd.update_state(
state='PROGRESS',
meta={'PROGRESS': out}
)
l = p.stdout.read()
print l
out += l
return out
except subprocess.CalledProcessError, e:
print 'Error executing command: ', cmd
return str(e)
Output.
C:\Python27\python.exe C:/Users/mkr/Documents/work/New_RoD/testing/run.py
This block works
Executing Celery cmd: ping 10.10.10.10
TIMEOUT
Pinging 10.10.10.10 with 32 bytes of data:
Request timed out.
Request timed out.
Executing Celery cmd: ping 10.10.10.10
TIMEOUT
Pinging 10.10.10.10 with 32 bytes of data:
Request timed out.
Request timed out.
Executing Celery cmd: ping 10.10.10.10
TIMEOUT
Pinging 10.10.10.10 with 32 bytes of data:
Request timed out.
Request timed out.
This block erros
Executing Celery cmd: ping 10.10.10.10
Executing Celery cmd: ping 10.10.10.10
Executing Celery cmd: ping 10.10.10.10
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
self.run()
File "C:\Python27\lib\threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "C:/Users/mkr/Documents/work/New_RoD/testing/run.py", line 18, in run
out = task.info['PROGRESS']
File "C:\Python27\lib\site-packages\celery\result.py", line 356, in result
return self._get_task_meta()['result']
File "C:\Python27\lib\site-packages\celery\result.py", line 339, in _get_task_meta
return self._maybe_set_cache(self.backend.get_task_meta(self.id))
File "C:\Python27\lib\site-packages\celery\backends\base.py", line 292, in get_task_meta
meta = self._get_task_meta_for(task_id)
AttributeError: 'DisabledBackend' object has no attribute '_get_task_meta_for'
Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
self.run()
File "C:\Python27\lib\threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "C:/Users/mkr/Documents/work/New_RoD/testing/run.py", line 18, in run
out = task.info['PROGRESS']
File "C:\Python27\lib\site-packages\celery\result.py", line 356, in result
return self._get_task_meta()['result']
File "C:\Python27\lib\site-packages\celery\result.py", line 339, in _get_task_meta
return self._maybe_set_cache(self.backend.get_task_meta(self.id))
File "C:\Python27\lib\site-packages\celery\backends\base.py", line 292, in get_task_meta
meta = self._get_task_meta_for(task_id)
AttributeError: 'DisabledBackend' object has no attribute '_get_task_meta_for'
Exception in thread Thread-3:
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
self.run()
File "C:\Python27\lib\threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "C:/Users/mkr/Documents/work/New_RoD/testing/run.py", line 18, in run
out = task.info['PROGRESS']
File "C:\Python27\lib\site-packages\celery\result.py", line 356, in result
return self._get_task_meta()['result']
File "C:\Python27\lib\site-packages\celery\result.py", line 339, in _get_task_meta
return self._maybe_set_cache(self.backend.get_task_meta(self.id))
File "C:\Python27\lib\site-packages\celery\backends\base.py", line 292, in get_task_meta
meta = self._get_task_meta_for(task_id)
AttributeError: 'DisabledBackend' object has no attribute '_get_task_meta_for'
Process finished with exit code 0
using app.AsyncResult worked for me
Works for me as suggested by https://stackoverflow.com/users/2682417/mylari in one of the comments above
celery1 = Celery('mytasks', backend='redis://localhost:6379/1', broker='redis://localhost:6379/0')
def t_status(id):
c = celery1.AsyncResult(id)
return c
Calling method:
#app.route("/tasks/<task_id>", methods=["GET"])
def get_status(task_id):
task_result = t_status(task_id)
result = {
"task_id": task_id,
"task_status": task_result.status,
"task_result": task_result.result
}
return jsonify(result), 200
Celery operations are not thread safe - you probably want to wrap the call to task.info in a lock.
Also mixing celery and threads like that is a little odd.
Try this:
from celery.result import AsyncResult
from iota_celery.app_iota import app as celery_app
AsyncResult(x, app=celery_app).revoke(terminate=True, signal='SIGKILL')
celery.AsyncResult work for me:
celery = make_celery(flask_app)
task_result = celery.AsyncResult(task_id)