I've got a django project with simple form to take users details in. I want to use python bot running in the background and constantly checking django database for any changes. Is it Celery the right tool for this job? Any other solution? Thank you
I don't think Celery is really what you want here - Celery is primarily for moving tasks that don't need to be dealt with in the same process to a separate worker, such as sending registration emails.
For this situation I'd be inclined to use Django's signals to trigger the required functionality whenever the appropriate changes are made to the database. For instance, if it needed to be triggered when a particular type of object was created, such as a new user, then you might use the post_save signal of the user model.
The bot would be in a separate process, but it's not too hard to communicate between processes using Redis. Just have the signal publish a message to Redis, and have the bot listen for that message and carry out the required action on that event.
I don't have the details of your needs but, there are a few ways to achieve such things:
The Constantly checking approach:
A crontab which launch your python script every minute.
Like you said, you could use Celery beat, to achieve what a crontab would do, in your python environment
"On change" approach:
Probably the best, if you have control of the Django project, you could have your script run on the form validation/save! For this, You can add a celery task, run the python script, use Django signals...
Related
I have a DJango app and where users upload many images. What I want is to trigger a management command using the call_command function from management once a new object is created but the problem is I want to do it asynchronously. I don't want to keep the user waiting for the management command to finish. Yes I am aware about third party services like Celery but I want to keep it simple. I also know that I can schedule a cron job but the thing is I want the change to reflect instantly. Is there any other way to do so?
I'm using signals for post-processing data. Because a lot needs to happen, and later I want to run that logic on the background so the user doesn't have to wait for this, I want to run this code in a separate class.
I want to run the code in my Post Save event
But I get the following error:
ImportError: cannot import name 'ActivityDetail' from 'ryf_app.models'
The model definitely exists in my models.py file
What am I missing here?
If you want to run a task asynchronously or in the background, you might use task queue like celery. For a broker or cache db there are options for redis, rabbitmq, amazon sqs. Celery have a good documentation with rabbitmq supporting broker. You can follow this link-here.
I have two servers, A primary server that provide REST API to accept data from user and maintain a product details list. This server is also responsible to share product list (a subset of product data) with secondary server as soon as product is updated/created.
also note that secondary url depends on product details, not a fix server.
Primary server written in Django. I have used django model db signal as product update, create and delete event.
Now problem is that I don’t want to bock my primary server REST call until it populates detail to secondary server. I need some scheduler stuff to do that, i.e. create a task to populate data in background without blocking my current thread.
I found python asyncio module comes with a function 'run_in_executor', and its working till now, But I don’t have a knowledge of the side effect over django run in wsgi server, can anyone explain ? or any other alternate ?
I found django channel, but it need extra stuff like run worker thread separately, redis cache.
You should use Django Celery for running Tasks asynchronously or in the background.
Celery is a task queue with batteries included. It’s easy to use so that you can get started without learning the full complexities of the problem it solves.
You can get more information on celery from http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html#first-steps
I need to execute a command on a simple button press event in my Django project (for which I'm using "subprocess.Popen()" in my views.py ).
After I execute this script it may take anywhere from 2 minutes to 5 minutes to complete. So while the script executes I need to disable the html button but I want the users to continue using other web pages while the script finishes in the background. Now the real problem is that I want to enable the html button back, when the process finishes!
I'm stuck at this from many days. Any help or suggestion is really really appreciated.
I think you have to use some "realtime" libraries for django. I personally know django-realtime (simple one) and swampdragon (less simple, but more functional). With both of this libraries you can create web-socket connection and send messages to clients from server that way. It may be command for enabling html button or javascript alert or whatever you want.
In your case I advice you first option, because you can send message to client directly from any view. And swampdragon needs model to track changes as far I know.
Like valentjedi suggested, you should be using swampdragon for real time with django.
You should take the first tutorial here: http://swampdragon.net/tutorial/part-1-here-be-dragons-and-thats-a-good-thing/
Then read this as it holds knowledge required to accomplish what you want:
http://swampdragon.net/tutorial/building-a-real-time-server-monitor-app-with-swampdragon-and-django/
However there is a difference between your situation and the example given above, in your situation:
Use Celery or any other task queue, since the action you wait for takes long time to finish, you will need to pass it to the background. (You can also make these tasks occur one after another if you don't want to freeze your system with enormous memory usage).
Move the part of code that runs the script to your celery task, in this case, Popen should be called in your Celery task and not in your view (router in swampdragon).
You then create a channel with the user's unique identifier, and add relevant swampdragon javascript code in your html file for the button to subscribe to that user's channel (also consider disabling the feature on your view (router) since front-end code can be tempered with.
The channel's role will be to pull the celery task state, you
then disable or enable the button according to the state of
the task.
overview:
Create celery task for your script.
Create a user unique channel that pulls the task state.
Disable or enable the button on the front-end according to the state of the taks, consider displaying failure message in case the script fails so that the user restart again.
Hope this helps!
I have two sites running essentially the same codebase, with only slight differences in settings. Each site is built in Django, with a WordPress blog integrated.
Each site needs to import blog posts from WordPress and store them in the Django database. When a user publishes a post, WordPress hits a webhook URL on the Django side, which kicks off a Celery task that grabs the JSON version of the post and imports it.
My initial thought was that each site could run its own instance of manage.py celeryd, each is in its own virtualenv, and the two sites would stay out of each other's way. Each is daemonized with a separate upstart script.
But it looks like they're colliding somehow. I can run one at a time successfully, but if both are running, one instance won't receive tasks, or tasks will run with the wrong settings (in this case, each has a WORDPRESS_BLOG_URL setting).
I'm using a Redis queue, if that makes a difference. What am I doing wrong here?
Have you specified the name of the default queue that celery should use? If you haven't set CELERY_DEFAULT_QUEUE the both sites will be using the same queue and getting each other's messages. You need to set this setting to a different value for each site to keep the message separate.
Edit
You're right, CELERY_DEFAULT_QUEUE is only for backends like RabbitMQ. I think you need to set a different database number for each site, using a different number at the end of your broker url.
If you are using django-celery then make sure you don't have an instance of celery running outside of your virtualenvs. Then start the celery instance within your virtualenvs using manage.py celeryd like you have done. I recommend setting up supervisord to keep track of your instances.