Develop Reference

Using Pika with Django (Event-based microservice using django rest framework)

anyone here has experience implementing pika with Django?
I am basically running an event-based microservice using django rest framework. And using RabbitMQ as the message bus. I know the default library to use in this case would be Celery, but I am looking for a lighter version where I can just implement a simple pub / sub on the messages.
Has anyone implemented such a service using pika before?
My question is more how do you spawn pika as a separate process together with Django? Or is there a more elegant solution out there?
Thanks in advance.
--- UPDATE ---
What we ended up doing was:
For the publisher:
We spawn a separate thread (or many if you need to publish a high volume/ sec) that keeps a pika connection alive.
For the subscriber:
We spawn a separate worker process (in a separate container) that has the django context (using django.setup()) which consumes the messages from RabbitMQ

How to set up a distributed worker pool with Celery and RabbitMQ

I'm still really new to this kind of thing so it's entirely possible that I've got this wrong.
I am trying to set up a distributed task system. I have a Django webapp that is generating tasks using Celery. Right now, I have the webapp, the worker, and RabbitMQ running all on the same server. I would like to distribute this out to several servers.
As I currently understand it, I should be able to have my webapp generating tasks, handing them off to the message queue -- which is its own server -- and then workers distributed across any number of servers will consume tasks from that queue. I know how to tell my Django app which server is the broker, but how do I start worker threads on the worker servers and instruct them where to consume tasks from? I'm totally lost -- I don't even know where to look.

You can run your worker code (async_tasks.py) like this:
from celery import Celery
app = Celery('tasks', broker=broker_url)
#app.task(queue='queue_name')
def async_compute_something(input):
# do something
return "Result"
on other machines using this command :
celery -A async_tasks worker -Q queue_name
Note that you have set the url of the broker correctly and not localhost

Django background RabbitMQ Connection

Alright, so I have a somewhat unique issue that I'm dealing with. I'm writing a small web application in Django and I would like it to interface with RabbitMQ directly over the backend. IE, I have a server task that I'm developing a web interface for, and I want to interact with it's API directly through the Message Queue. I'm struggling with how to spawn a long-running connection to the Message Queue. To state the obvious, it's expensive to construct/teardown TCP connections to RMQ for every new request! So, is it possible to create some long-running process that I can use to interact with this AMQP API?
As a note regarding Celery, since I'm sure it will be mentioned, my understanding of Celery is that it is great at distributing time-intensive tasks to other processes and/or nodes, but does not expose a long-running AMQP connection to the Django application. So I don't think it will help here. That being said, I also haven't done much more than the tutorial and read parts of the documentation.

Need queue module to be shared between two applications

I need to share some queue between two applications on same machine, one is Tornado which is going to occasionally add message to that queue and another is python script runs from cron which is going in every iteration add new messages. Can anyone suggest me module for this ?
(Can this be solved with redis usage, I avoid to use mysql for this purpose )

I would use redis with a list. You can push a element top, and rpop to remove from the tail.
See redis rpop
and redis lpushx

The purest way I can think of to do this is with IPC. Python has very good support for IPC between two processes when one process spawns another, but not in your scenario. There are python modules for ipc such as sysv_ipc and posix_ipc. But if you are going to have your main application built in tornado, why not just have it listen on a zeromq socket for published messages.
Here is a link with more information. You want the Publisher-Subscriber model.
http://zeromq.github.io/pyzmq/eventloop.html#tornado-ioloop
Your cron job will start and publish messages a to zeromq socket. Your already running application will receive them as subscriber.

Try RabbitMQ for hosting the queue independent of your applications, then access using Pika, which even comes with a Tornado adapter. Just pick the appropriate model: queue/exchange/topic and protocol of the message you want (strings, json, xml, yaml) and you are set.

Python Server, Job Queue, Launch Multiprocessing Job

I need to create a python server that can accept multiple job requests. Then from those it requests, it processes each Job one at a time but the server can still accept new Jobs while processing a task.
Does anyone have an suggestions on how to do this?
Thanks

Sure. Create a multiprocessing.Pool which will by default spawn one process per core. Then use the original process to run an HTTP service or something else that accepts jobs via some protocol. The main process then listens for new requests and submits them to the pool for async processing.

Use twisted. Twisted is an event-driven networking engine. Twisted also supports many common network protocols, including SMTP, POP3, IMAP, SSHv2, and DNS.