I have my own templatetag:
#register.inclusion_tag('path_to_module.html', takes_context=True)
def getmodule(context, token):
try:
return slow_function(params)
except Exception, e:
return None
And it is very slow. Template waiting for this tags.
Can I call them asynchonously?
If it's cacheable (doesn't need to be unique per page view); then cache it. Either using Django's cache API in your templatetag, or template fragment caching directly in your template. As #jpic says, if it's something that takes a while to recalculate - pass it off to a task queue like Celery.
If you need this function to run every page view for whatever reason; then separate it out in to a new view and load it in to some container in your main template asynchronously using JavaScript.
You can execute python functions in a background process:
django-ztask
celery (django kombu for database transport)
uWSGI spooler (if using uWSGI for deployment)
You could create a background task that renders path_to_module and caches the output. When the cache should be invalidated: run slow_function in the background again.
Related
I have one DB query that takes a couple of seconds in production. I have also a DRF ViewSet action that returns this query.
I'm already caching this action using cache_page.
#method_decorator(cache_page(settings.DEFAULT_CACHE_TIMEOUT))
#action(detail=False)
def home(self, request) -> Response:
articles = Article.objects.home()
return Response(serializers.ArticleListSerializer(articles, many=True).data,
headers={'Access-Control-Allow-Origin': '*'})
The problem is that after 15 minutes, at least one user needs to wait 15 seconds for the response. I want to pre-cache this every 5 minutes in background so that no user will need to wait.
I use the default caching mechanism.
My idea is to create a management command that will be executed using crontab. Every 5 minutes it will call the Article.objects.home() or the ViewSet.action and change it's value in the cache.
As this is only one entry, I don't hesitate to use database caching.
How would you do that?
EDIT: as the default LocMemCache is single-threaded, I'll go with the database caching. I just don't know how to manually cache the view or QuerySet.
A cron or Celery beat task (if you already use celery) looks like the best option.
Calling Article.objects.home() would not do much unless you cache in home() method of the manager (which could be a valid option that could simplify automated cache refresh).
To automate the refresh of view cache you better send actual requests to the URL from the management command. You will also want to invalidate the cache before sending the request, in order to update it.
Also, keep in mind the cache timeout when planning the job frequency. You wouldn't want to refresh too early nor too late.
I have an already existing Django app. I would like to add a system that sends data from my Django application to another Python application hosted on another server, so that the Python application receives data from the Django App in json format, possibly.
So for example, i would need to create a view that every tot seconds sends the data from a DB table to this application, or when a form is hit, the data is sent to this external application.
How can i do this? Is there an example for this particular matter? I don't know what tools i'd need to use to create this system, i only know that i would need to use Celery to perform asynchronous tasks, but nothing else; should i use Webhooks maybe? Or Django channels?
Edit: adding some more context:
I have my Django client. Then i have one or two Python applications running on another server. On my Django client i have some forms. Once the form is submitted, the data is saved on the db, but i also want this data to be sent instantly to my Python applications. The Python applications should receive the data from Django in Json format and perform some tasks according to the values submitted by users. Then the application should send a response to Django.
Come on! I'll call your Django app here "DjangoApp" and your Python apps, in Flask or another framework by "OtherApp".
First as you predicted you will need a framework that is capable of performing tasks, the new **Django 3.0 allows this, but I haven't used it yet ... I will pass on to you something that you are using and fully functional with Django 2.8 and Python 3.8**.
On your DjangoApp server you will need to structure the communication well with your Celery, let's leave the tasks to him. You can read Celery Docs and this post, its very ok to make this architecture.
Regardless of how your form or Django App looks, when you want it to activate a task in celery, it is basically the function to transmit data but in the background.
from .tasks import send_data
...
form.save()
# Create a function within the form to get the data the way you want it
# or do it the way you want.
values = form.new_function_serializedata()
send_data.delay(values) # [CALL CELERY TASKS][1]
...
Read too CALL CELERY TASKS
In all your other applications you will need to have a POST route to receive and serialize this data, do this with lightweight frameworks like Pyramid
This way, every time a form is submitted, you will have this data sent to the server within the send_data function.
In my experience, but not knowing much about your problem I would use a similar architecture but using Celery Beat.
CELERY_BEAT_SCHEDULE = {
'send_data': {
'task': 'your_app.tasks.send_data',
'schedule': crontab(), # CONFIGURE YOUR CRON
},
}
Not only is the above code added, but it is something like that.
Within your models I would create one field for sent. And every 2 seconds, 10 seconds .. as long as I wish I would filter all objects with sent = false, and pass all objects for the send_data task.
I don't know if you got confused, that's a lot to explain. But I hope I can help and answer your questions.
import requests
from django import http
def view(request):
url = 'python.app.com' # replace with other python app url or ip
request_data = {'key': 'value'} # replace with data to be sent to other app
response = requests.post(url, json=request_data)
response_data = response.json() # data returned by other app
return http.JsonResponse(response_data)
This is an example of a function based view that uses the requests library to hit an external service. The request lib takes care of encoding/decoding your data to/from json.
Yeah, webhook would be one of the options, but there are other options available too.
-> You can use Rest Apis to send data from one app to another. but In their case, you need to think about synchronization. That depends on your requirement, If you don't want data in synchronize manner then you may use RabbiMq or other async tools. Just push your rest API request in Rabbitmq and Rabbitmq will handle.
I'm a newbie to google app engine. I want the security restriction for url of cron so that it shouldn't be accessible by url directly. For this I've already read the docs and some of Q&As ([Google app engine: security of cron jobs).
I implemented the login : admin solution suggested in this link. But I failed to implement security as self.request.headers.get('X-AppEngine-Cron') is always None, whether it is cron or accessed via url directly.
So I don't know from where is the request coming (from cron or direct access)
def cron_method(BaseRestHandler):
def check_if_cron(self, *args, **kwargs):
if self.request.headers.get('X-AppEngine-Cron') is None:
logging.info("error-- not cron")
self.UNAUTH = "cron"
self.raise_exception()
else:
return BaseRestHandler(self, *args, **kwargs)
return check_if_cron
I used customized handler BaseRestHandler for other authentications.
#cron_method
def put(self):
logging.info("inside put--")
This is called via taskqueue from the get method of the class.
The problem is I didn't get header X-AppEngine-Cron
Any other logic or method will be appreciated.
Thanks In Advance.
It seems you attempted to make the check a decorator.
But your code shows the decorator applied to a put() method, not a get() method - the cron executes only on a get().
Also your decorator doesn't look quite right to me. Shouldn't a decorator take as argument a function and return some locally defined function which executes (not returns) the function received as argument?
I'd suggest you go back to basics - try to make the header check in the get method of the handler itself and only after you get that working consider further, more complex changes like the pulling the check in a decorator.
It is more likely that your decorator is not working than GAE's documented infra to not be working. Keeping things simple (at first) would at least help your investigation effort be pointed in a better direction.
Try this:
def cron_method(handler_method):
def check_if_cron(self, *args, **kwargs):
if self.request.headers.get('X-AppEngine-Cron') is None:
logging.info("error-- not cron")
self.UNAUTH = "cron"
self.raise_exception()
else:
handler_method(self, *args, **kwargs)
return check_if_cron
As for the invocations from the task queue - those requests are no longer cron requests, even if the tasks are created and enqueued by a cron request.
From Securing task handler URLs:
If a task performs sensitive operations (such as modifying data), you
might want to secure its worker URL to prevent a malicious external
user from calling it directly. You can prevent users from accessing
task URLs by restricting access to App Engine administrators.
Task requests themselves are issued by App Engine and can always
target restricted URL.
You can restrict a URL by adding the login: admin element to the
handler configuration in your app.yaml file.
If you want to also prevent manual access to those URLs (i.e. restrict it only to task queue requests) you can perform header checks similar to the cron one. The header values are listed in Reading request headers. Personally I picked X-AppEngine-TaskName.
I'm using scrapy to perform test on an internal web app.
Once all my tests are done, I use CrawlSpider to check everywhere, and I run for each response a HTML validator and I look for 404 media files.
It work very well except for this: the crawl at the end, GET things in a random order...
So, URL that perform DELETE operation are being executed before other operations.
I would like to schedule all delete at the end. I tried many way, with such kind of scheduler:
from scrapy import log
class DeleteDelayer(object):
def enqueue_request(self, spider, request):
if request.url.find('delete') != -1:
log.msg("delay %s" % request.url, log.DEBUG)
request.priority = 50
But it does not work... I see delete being "delay" in the log but they are executed during the execution.
I thought of using a middleware that can pile up in memory all the delete URL and when the spider_idle signal is called to put them back in, but I'm not sure on how to do this.
What is the best way to acheive this?
default priority for request is 0, so you set priority to 50 will not work
you can use a middleware to collect (insert the requests into your own queue, e.g, redis set) and ignore (return IngnoreRequest Exception) those 'delete' request
start a 2nd crawl with requests load from your queue in step 2
I have the following code:
#task()
def handle_upload(title, temp_file, user_id):
.
.
.
photo.save()
#if i insert here "photo2 = Photo.objects.get(pk=photo.pk)" it works, including the view function
return photo.pk
#view function
def upload_status(request):
task_id = request.POST['task_id']
async_result = AsyncResult(task_id)
photo_id = async_result.get()
if async_result.successful():
photo = Photo.objects.get(pk=photo_id)
I use an ajax request to check for the uploaded file but after the celery task finishes i get a Photo matching query does not exist. The photo pk does exist and gets returned. If i query the database manually it works. Is this some sort of database lag? How can I fix it?
I'm using Django 1.4 and Celery 3.0
You can confirm if it is a lag issue by adding a delay to your django view to wait after the task has successfully finished for a a few seconds. If that resolves the problem you might want to wrap the handle_upload in a transaction to block until the db has completely confirmed it has finished before returning.
Beside Django, DB too has its own caches. When django invokes the queryset, it gets stale data either from its own caches (unlikely unless you were reusing querysets, which I didn't see in the portion of the code you posted) or the DB is caching results for the same Django connection.
For example if you were to invoke post processing after the celery task has finished in a completely new django request/view you would probably see the new changes in DB just fine. However, since your view was blocked while the task was executing (which defeats the purpose of celery btw) internally django only keeps the snapshot of the DB at the time the view was entered. Therefore your get fails and you confirmed this behavior directly when simply entering the django shell.
You can fix this like you already did by either:
invoking transactional management which will refresh the snapshot
changing on your DB endpoint caching and autocommit policies
have celery make a callback to django (web request) once it is done to finalize processing (which is likely what you want to do anyway because blocking django defeats the purpose)