i Have a django project which works fine. I need a rq worker to do a job. I got a redis-server running.
Whis is my worker.py file:
import os
import redis
from rq import Worker, Queue, Connection
listen = ['high', 'default', 'low']
redis_url = os.getenv('REDISTOGO_URL', 'redis://localhost:6379')
conn = redis.from_url(redis_url)
if __name__ == '__main__':
with Connection(conn):
os.environ['DJANGO_SETTINGS_MODULE']='ak.settings.prod_lt'
worker = Worker(map(Queue, listen))
worker.work()
I run the worker from my ubuntu terminal using the following commandline:
/opt/crm/env/bin/python /opt/crm/ak/ak/worker.py
The worker starts fine.
The job I am giving it is getting data from the database and writing the data to an excel file but I am getting the following error:
rq.exceptions.UnpickleError: (
u'Could not unpickle',
ImportError('No module named ak.settings.prod_lt',
<function model_unpickle at 0x7fcba6b67938>,
(('crm', 'EmailExport'), [], <function simple_class_factory at 0x7fcba6b678c0>)))
Can anybody tell me what could be wrong?
Related
Problem Statement
After booting the GUnicorn worker processes, I want the worker processes still be able to receive data from another process. Currently, I'm trying to use multiprocessing.Queue to achieve this. Specifically, I start a data management process before forking the workers and use two queues to connect it with the workers. One queue is for the workers to request data from the data management process, the other to receive the data. In the post_fork hook, a worker sends out a request to the request queue and receives a response on the response queue, and only then proceeds to serving the application.
This works fine at first. However, when I manually terminate the workers and gunicorn restarts it, it will get stuck in the post_fork method and never receive a response from the data management process.
Minimal Example
The following code shows a minimal example (config.py):
import logging
import os
import multiprocessing
logging.basicConfig(level=logging.INFO)
bind = "localhost:8080"
workers = 1
def s(req_q: multiprocessing.Queue, resp_q: multiprocessing.Queue):
while True:
logging.info("Waiting for messages")
other_pid = req_q.get()
logging.info("Got a message from %d", other_pid)
resp_q.put(os.getpid())
m = multiprocessing.Manager()
q1 = m.Queue()
q2 = m.Queue()
proc = multiprocessing.Process(target=s, args=(q1, q2), daemon=True)
proc.start()
def post_fork(server, worker):
logging.info("Sending request")
q1.put(os.getpid())
logging.info("Request sent")
other_pid = q2.get()
logging.info("Got response from %d", other_pid)
My application module (app.py) is:
from flask import Flask
app = Flask(__name__)
And I start the server via
$ gunicorn -c config.py app:app
INFO:root:Waiting for messages
[2023-01-31 14:20:46 +0800] [24553] [INFO] Starting gunicorn 20.1.0
[2023-01-31 14:20:46 +0800] [24553] [INFO] Listening at: http://127.0.0.1:8080 (24553)
[2023-01-31 14:20:46 +0800] [24553] [INFO] Using worker: sync
[2023-01-31 14:20:46 +0800] [24580] [INFO] Booting worker with pid: 24580
INFO:root:Sending request
INFO:root:Request sent
INFO:root:Got a message from 24580
INFO:root:Waiting for messages
INFO:root:Got response from 24574
The log shows that the messages were successfully exchanged. Now, we'll stop the worker process and let gunicorn restart it:
$ kill 24580
[2023-01-31 14:22:40 +0800] [24580] [INFO] Worker exiting (pid: 24580)
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/util.py", line 319, in _exit_function
p.join()
File "/usr/lib/python3.6/multiprocessing/process.py", line 122, in join
assert self._parent_pid == os.getpid(), 'can only join a child process'
AssertionError: can only join a child process
[2023-01-31 14:22:40 +0800] [24553] [WARNING] Worker with pid 24574 was terminated due to signal 15
[2023-01-31 14:22:40 +0800] [29497] [INFO] Booting worker with pid: 29497
INFO:root:Sending request
INFO:root:Request sent
Question
Why doesn't s receive the message from the worker after re-starting?
Besides, why am I getting this 'can only join a child process' error thrown? Does it has something to do with the problem?
Environment
Python: 3.8.0
GUnicorn: 20.1.0
OS: Ubuntu 18.04
Related Questions
In this question, a similar problem is presented, and the solution was to use "multiprocessing.manager.queue". However, this didn't solved the issue in my case.
Side Note
I already considered the following alternative designs:
Use HTTP/gRPC/... to share the data: The data that I need to share isn't serializable
Use threading.Thread instead of multiprocessing.Process for the data management process: The data management process initializes an object that will throw an error when it is forked, so I cannot initialize this object within the GUnicorn master process.
Gunicorn Issue #1621 somewhat answers my question. As far as I understand this short statement, this is because Gunicorn uses os.fork and not multiprocessing, so the utilities in multiprocessing apparently aren't guaranteed to work with Gunicorn.
So, instead of directly using multiprocessing.Queue, I replicate the behavior of Queue with another IPC library. Internally, Queue is using a ForkingPickler to serialize the data and this serialized data can also be sent via other IPC libraries, such as ZeroMQ. So, I don't necessarily need the multiprocessing module for this. Unfortunately, directly replacing the Queues with corresponding zeromq code in the above code exhibits the same behavior than in the question.
This problem can be eliminated by putting the complete multiprocessing related code into another script, so the service process s isn't a child process of Gunicorn anymore. This leads to the following code:
config.py:
import logging
import os
import pickle
import zmq
logging.basicConfig(level=logging.INFO)
bind = "localhost:8080"
workers = 1
zmq_url = "tcp://127.0.0.1:5555"
def post_fork(server, worker):
logging.info("Connecting")
context = zmq.Context()
with context.socket(zmq.REQ) as socket:
socket.connect(zmq_url)
logging.info("Sending request")
socket.send(pickle.dumps(os.getpid()))
logging.info("Waiting for a response")
other_pid = pickle.loads(socket.recv())
logging.info("Got response from %d", other_pid)
server.py:
import logging
import os
import pickle
import zmq
def serve(url):
context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind(url)
while True:
logging.info("Waiting for requests on %s", url)
message = socket.recv()
logging.info("Got a message from %d", pickle.loads(message))
socket.send(pickle.dumps(os.getpid()))
if __name__ == '__main__':
logging.basicConfig(level=logging.INFO)
serve("tcp://127.0.0.1:5555")
The startup script looks somewhat like this:
#!/usr/bin/env bash
set -euo pipefail
python server.py &
gunicorn -c config.py app:app
This worked reliably during my testing also for killed and restarting workers.
I have a Flask webapp running on Heroku. There are functions that require more than 30 seconds to process data and for those tasks I using heroku background jobs with Redis with 20 connections limit. However, these tasks are only available for specific users.
My understanding is that Redis opens connection after I initiate the Queue, no matter if the job was queued and processed or not.
Here's my import and Queue initiation:
from rq import Queue
from rq.job import Job
from worker import conn as rconn
q = Queue(connection=rconn)
And here's my worker file:
import os
import urllib
from redis import Redis
from rq import Worker, Queue, Connection
listen = ['high', 'default', 'low']
redis_url = os.getenv('REDIS_URL')
urllib.parse.uses_netloc.append('redis')
url = urllib.parse.urlparse(redis_url)
conn = Redis(host=url.hostname, port=url.port, db=0, password=url.password)
if __name__ == '__main__':
with Connection(conn):
worker = Worker(map(Queue, listen))
worker.work()
I am looking for a way to initiate redis connection only for users with specific access level, so the app won't reach connection error.
Does it make sense to initiate Queue from user_login function as global variable, like this:
if check_password_hash(db_pwd, pwd) and acces_level==4:
q global
q = Queue(connection=rconn)
I'm trying to extend the flask-base project https://github.com/hack4impact/flask-base/tree/master/app which comes with a user model only. I'm trying to add the ability to run a background task on redis using rq. I've found https://devcenter.heroku.com/articles/python-rq which is helpful.
this app has support for redis queues with a background redis queue being implemented by running :
#manager.command
def run_worker():
"""Initializes a slim rq task queue."""
listen = ['default']
conn = Redis(
host=app.config['RQ_DEFAULT_HOST'],
port=app.config['RQ_DEFAULT_PORT'],
db=0,
password=app.config['RQ_DEFAULT_PASSWORD'])
with Connection(conn):
worker = Worker(map(Queue, listen))
worker.work()
using:
$ python manage.py run_worker
In my views I have:
#main.route('/selected')
def background_selected():
from rq import Queue
from manage import run_worker.conn
q = Queue(connection=conn)
return q.enqueue(selected)
The problem is I don't know how to import the connection created in run_worker() into my view. I've tried variations of :
from manage import run_worker.conn
but I'm getting:
SyntaxError: invalid syntax.
How can I get access to the conn variable in the background task?
from the documentation, python-rq Configuration
Can you try by making the below changes:
manager.py
import redis
"""Initializes a slim rq task queue."""
listen = ['default']
conn = redis.Redis(host=app.config['RQ_DEFAULT_HOST'],
port=app.config['RQ_DEFAULT_PORT'],
db=0,
password=app.config['RQ_DEFAULT_PASSWORD'])
#manager.command
def run_worker():
with Connection(conn):
worker = Worker(map(Queue, listen))
worker.work()
and from view:
from rq import Queue
from manage import conn
q = Queue(connection=conn)
I contacted the developer who provided the following:
I'm attempting to get RQ/RQ-Worker running on my Flask application. I've tried to get it down to a very simple test case. Here's the general idea:
The user visits the /test page. Which triggers a job to be queued and returns the queued job's job_key
The worker (worker.py) processes the queued job.
The user can then visit the /retrieve/<job_key> page to retrieve the result. [This is not shown.]
The current job is just to add 2 + 2.
Here is the application code:
from rq import Queue
from rq.job import Job
# import conn from worker.py
from worker import conn
app = Flask(__name__)
q = Queue(connection=conn)
def add():
return 2+2
#app.route('/test')
def test():
job = q.enqueue_call(func="add", args=None, result_ttl=5000)
return job.get_id()
if __name__ == "__main__":
app.run()
The worker.py source code looks like this:
from redis import StrictRedis
from rq import Worker, Queue, Connection
listen = ['default']
redis_url = 'redis://localhost:6379'
conn = StrictRedis.from_url(redis_url)
if __name__ == "__main__":
with Connection(conn):
worker = Worker(list(map(Queue, listen)))
worker.work()
To my knowledge, the application code isn't the issue. I can visit the /test page which will enqueue the job. However, once I run the worker, I get the following error:
Traceback (most recent call last):
File "/home/<>/dev/sched/venv/lib/python3.5/site-packages/rq/worker.py", line 588, in perform_job
rv = job.perform()
File "/home/<>/dev/sched/venv/lib/python3.5/site-packages/rq/job.py", line 498, in perform
self._result = self.func(*self.args, **self.kwargs)
File "/home/<>/dev/sched/venv/lib/python3.5/site-packages/rq/job.py", line 206, in func
return import_attribute(self.func_name)
File "/home/<>/dev/sched/venv/lib/python3.5/site-packages/rq/utils.py", line 149, in import_attribute
module_name, attribute = name.rsplit('.', 1)
ValueError: not enough values to unpack (expected 2, got 1)
I feel like the line:
worker = Worker(list(map(Queue, listen)))
is the problem just b/c of the nature of the error, but I have no idea how to fix it. Especially b/c I've seen other projects that seem to use the exact same worker source code.
My technology stack is:
Flask (0.11.1)
Redis (2.10.5)
RQ (0.6.0)
RQ-Worker (0.0.1)
EDIT:
Beginning to think this is a bug. Check out this issue ticket in RQ's source: issue #531.
For me the issue was caused by RQ not being able to resolve my worker module.
The solution was to supply the "qualified" name to enqueue, e.g:
job = q.enqueue("app.worker.add", data)
I'm trying to run multiple process in Tornado and I tried the suggestions made on this thread : run multiple tornado processess
But the error hasn't gone for me. This is the server file.
server.py
import os
import sys
import tornado
#import pymongo
from tornado import ioloop, web, httpserver, websocket
from tornado.options import options
#Loading default setting files
import settings
#Motorengine - ODM for mongodb
#from motorengine import connect
app = tornado.web.Application(handlers=[
(r'/', MainHandler),
(r'/ws', WSHandler),
(r'/public/(.*)', tornado.web.StaticFileHandler, {'path': options.public_path})],
template_path=os.path.join(os.path.dirname(__file__), "app/templates"),
static_path= options.static_path,
autoreload=True,
#images=os.path.join(os.path.dirname(__file__), "images"),
debug=False)
if __name__ == '__main__':
#read settings from commandline
options.parse_command_line()
server = tornado.httpserver.HTTPServer(app, max_buffer_size=1024*1024*201)
server.bind(options.port)
# autodetect cpu cores and fork one process per core
server.start(0)
#app.listen(options.port,xheaders=True)
try:
ioloop = tornado.ioloop.IOLoop.instance()
#connect("attmlplatform", host="localhost", port=27017, io_loop=ioloop)
print("Connected to database..")
ioloop.start()
print ('Server running on http://localhost:{}'.format(options.port))
except KeyboardInterrupt:
tornado.ioloop.IOLoop.instance().stop()
I've commented out the 'connect' import based on the anticipation that it may be triggering the instance and I'm not connecting to the database at all. This is just trying to get the server up.
This is the entire trace :
File "server.py", line 52, in <module>
server.start(0)
File "/home/vagrant/anaconda3/envs/py34/lib/python3.4/site-packages/tornado/tcpserver.py", line 200, in start
process.fork_processes(num_processes)
File "/home/vagrant/anaconda3/envs/py34/lib/python3.4/site-packages/tornado/process.py", line 126, in fork_processes
raise RuntimeError("Cannot run in multiple processes: IOLoop instance "
RuntimeError: Cannot run in multiple processes: IOLoop instance has already been initialized. You cannot call IOLoop.instance() before calling start_processes()
Any suggestions much appreciated!
Thanks!
autoreload is incompatible with multi-process mode. When autoreload is enabled you must run only one process.