I want to use dynamic scheduler management for celery. I know djcelery have that functionality with database support.
But currently I do not use Django, but Flask. I couldn't find out Flask project or implementation with djcelery.schedulers.
Is it possible to use djcelery and implement dynamic scheduler management system without Django?
Short answer: No, but...
You have to use django. The scheduler's entries are instances of django models so you would have to setup djcelery app somehow (see this code: https://github.com/celery/django-celery/blob/master/djcelery/schedulers.py) Also you won't have the admin interface to add scheduler's entries.
This is just a guess, but you can try setting django's ORM standalone and syncing djcelery's models. (see: Use Django ORM as standalone)
You can also implement your own scheduler following the structure of djcelery/schedulers.py
Also see: Can celery celerybeat use a Database Scheduler without Django?
You can check out this flask-djcelery. It configures djcelery with flask, allows using django admin and also provide a browseable rest api for managing tasks.
Related
Lets say we have one django project that has two apps - FooApp & BarApp.
Each app talks to its own database. Meaning they both manage their own set of models. Is it possible to manage the deployments of these apps independently? It's okay for them to be deployed on same server within the same nginx process as long as I am able to make changes to the apps without bringing down the other as well.
I understand that these can very well be separate projects and they can communicate with each other using RESTful APIs. For my needs, I want to avoid that REST interaction for the time being.
The django documentation [here][1] describes a django project and django app as follows
Django project:
The term project describes a Django web application.
Django App:
The term application describes a Python package that provides some set
of features.
So if the app is simply a python package, and the django project is really the one that defines how these apps are managed via the settings module, then I suppose there is no straight forward way of accomplishing what I want. Or is there anything I can do?
TIA
[1]: https://docs.djangoproject.com/en/dev/ref/applications/
I am developing a small independent python application which uses Celery. I have built this using django framework but my application is back end only. This means that the users do not need to visit my site and my application is built only for the purpose of receiving tasks queue from celery and performing operations on the database. In order to perform operations on the database, I need to use Django modules.
What I am trying to do is eliminate the rest of my django application and use ONLY celery and django models modules (including the dependencies required to run these).
In short, my simple celery application will be running receiving instructions from my redis broker and perform operations in database using django models.
Is is possible to do this? If so, how?
Here is my project structure:
myproject/
--manage.py
--myproject/
----celery.py
----models.py
----settings.py
----tasks.py
----urls.py
----wsgi.py
Here is my settings.py:
In your project's settings.py, just add this at beginning.
import django
import os
sys.path.insert(0, your_project_path) # Ensure python can find your project
os.environ['DJANGO_SETTINGS_MODULE'] = 'myproject.settings'
django.setup()
Then you can use django orm, remember to delete the middleware you don't need in django settings.
you just need
env['DJANGO_SETTING_MODULE'] = 'myproject.settings'
django.setup()
(assuming you setup your database and installed_apps stuff in settings.py)
You have have a python script that requires some celery tasks and you need Django ORM too for the database interactions.
You can setup the django project
create an app for your purpose, include in settings.py and inside your app in models.py create the required models.
ref : What minimal files i need to use django ORM
Set up the environment for executing celery. Ie, redis server. integrate "djcelery" with django project. for the celery task purpose.
you can use celery beats for the periodic tasks. or delay.
ref: http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html
You can import and use the django models like normal inside the celery tasks.
And The celery tasks you can run using
i. celery -A tasks worker --loglevel=info
ii. celery -A tasks beat -l info. use beats if you want the tasks which are written for periodic execution.
If the tasks need to be executed asynchronously just immediately or after a time interval , you can use task_name.delay()
call the tasks inside the python script using delay()
i think to use djcelery in your script you may need to set up the django env inside the script.
just Do django.setup().
i think this will help you to solve your problem.
OK.. i am starting a project in django 1.4 and i want MongoDB as my backend. after a half a day of google search, i figured out that mongoengine is a best option(as it is an active project and provides a django like orm)
Now the problem is
1. I cant find any good step-by-step setup guide to integrate mongoengine with a django project.
I understand, using mongoengine means that i am replacing django orm and there is no need to do syncdb. now this project have a multi-tenant architecture (*.domain.com) which i am gonna resolve using a middleware..also a considerable part of this project will work on django admin.
Question: will replacing django orm with mongoengine going to affect django admin and other operations(such as middleware, authentication etc.) ?
I am open to suggestions and criticism as well.
Django Admin is designed to work with the Django ORM only. Using MongoEngine and no Django ORM will mean you don't get the automatic admin interface. Other middleware might use the Django ORM or be sufficiently abstracted enough to allow you to plugin MongoEngine - eg: Sessions and Authentication.
There are some helpers for Django in MongoEngine - but its by no means complete or designed to be a drop in replacement for the Django ORM.
For more information see this presentation from Django Conf Finland: http://staltz.github.io/djangoconfi-mongoengine
Just in case, situation has changed and there is a solution now for this problem, namely django-mongoadmin.
A Guide to integrating Django with MongoDB
The way to connect Django with MongoDB by adding just one line of code:
First install djongo:
pip install djongo
Then run your migrations:
manage.py make migrations
manage.py migrate
and finally add to your settings file:
DATABASES = {
‘default’: {
‘ENGINE’: ‘djongo’,
‘NAME’: ‘your-db-name’,
}
}
It is as simple as that!
If you want to manipulate MongoDB using Django Admin, simply fire it up:
manage.py runserver
Goto: http://localhost:8000/admin/
Manipulate your embedded models as shown in this screenshot:
For more information do checkout the djongo documentation.
You should definitely consider the pros and cons of using a NEW framework (like MongoEngine) vs using inbuilt Django ORM. Do read at this tutorial before considering to adopt MongoEngine as suggested by other knowledgeable members! No offence!
Let me know if you agree with this approach in the comments :)
What are the different options, with pros and cons, for periodically adding records to a Django app hosted on GAE?
Use a custom django management command on the remote datastore
Write an API in Django that exposes the datastore to be updated
Use a cron task on GAE to update
(am I missing anything else?)
1: Custom Django management command on "remote"
I'm currently using #1: django-nonrel on GAE and using custom management/django-admin commands for my models. For example, this is how I call my custom management command on the remote datastore:
manage.py remote mycommand
The advantage of this command is ease of development: I can test the the management command locally and simply add "remote" to use it on GAE.
2: Write an API in Django that exposes the datastore
I would have to use an extra server with cron to update.
3: Use a cron task in Google
I don't know how GAE likes having its users run a scraper periodically. Also, GAE doesn't have a real cron -- it simply hits a URL at a set intervals.
Use a cron job. That's what they're designed for. Whether or not scraping is okay depends on the terms of service on the site you're scraping.
I have a Django project that uses SQLAlchemy to use some legacy ORM objects. This application also hits up an ldap server for user authentication. I was getting sick of moving from development to production servers for both ldap and the database. I was hoping to create a IS_DEVELOPMENT variable in the settings.py. One problem I am running into is that the ORM modules are not in the Django project.
I know django.conf does a great job of parsing that settings file and making those settings available to you throughout your application. What I can not figure out is how to make those same settings available to myself outside of that app.
Anyone done this one before?
"Standalone Django Scripts"