I am using Django and am making some long running processes that I am just interacting with through my web user interface. Such as, they would be running all the time, checking a database value every few minutes and stopping only if this has changed (would be boolean true false). So, I want to be able to use Django to interact with these, however am unsure of the way to do this. When I used to use PHP I had some method of doing this, figure it would be even easier to do in Python but am not able to find anything on this with my searches.
Basically, all I want to be able to do is to execute python code without waiting for it to finish, so it just begins execute then goes on to do whatever else it needs for django, quickly returning a new page to the user.
I know that there are ways to call an external program, so I suppose that may be the only way to go? Is there a way to do this with just calling other python code?
Thanks for any advice.
Can't vouch for it because I haven't used it yet, but "Celery" does pretty much what you're asking for and was originally built specifically for Django.
http://celeryproject.org/
Their example showing a simple task adding two numbers:
from celery.decorators import task
#task
def add(x, y):
return x + y
You can execute the task in the background, or wait for it to finish:
>>> result = add.delay(8, 8)
>>> result.wait() # wait for and return the result
16
You'll probably need to install RabbitMQ also to get it working, so it might be more complicated of a solution than you're looking for, but it will achieve your goals.
You want an asynchronous message manager. I've got a tutorial on integrating Gearman with Django. Any pickleable Python object can be sent to Gearman, which will do all the work and post the results wherever you want; the tutorial includes examples of posting back to the Django database (it also shows how to use the ORM outside of Django).
Related
I am trying to set up a small server providing various functionalities implemented as python scripts. So far, I only wrote scripts so far that I had to run manually, and the result was done once they had finished.
Obviously, I could still implement a script on the OS side to run the actual python script. For instance, a cronjob would be a solution.
However I am now wondering how to implement this behaviour inside the actual python script. Frameworks like Flask for instance can be used to just start a python-based web server and will continue running and answering requests etc. continuously. Or if I where to implement some kind of API with python, I would have to keep this script running permanently to allow api-calls to be received at all time.
So, I wonder how this behaviour is actually implemented using python? I have seen some “easy” solution by basically wrapping the whole script into a never ending while loop, constantly evaluating If statements and this does work, however I imagine this not really the “professional” way to go and might create various problems (memory leak etc.) over time?
Okay, so basically I am creating a website. The data I need to display on this website is delivered twice daily, where I need to read the delivered data from a file and store this new data in the database (instead of the old data).
I have created the python functions to do this. However, I would like to know, what would be the best way to run this script, while my flask application is running? This may be a very simple answer, but I have seen some answers saying to incorporate the script into the website design (however these answers didn't explain how), and others saying to run it separately. The script needs to run automatically throughout the day with no monitoring or input from me.
TIA
Generally it's a really bad idea to put a webserver to handle such tasks, that is the flask application in your case. There are many reasons for it so just to name a few:
Python's Achilles heel - GIL.
Sharing system resources of the application between users and other operations.
Crashes - it happens, it could be unlikely but it does. And if you are not careful, the web application goes down along with it.
So with that in mind I'd advise you to ditch this idea and use crontabs. Basically write a script that does whatever transformations or operations it needs to do and create a cron job at a desired time.
I have to setup a program which reads in some parameters from a widget/gui, calculates some stuff based on database values and the input, and finally sends some ascii files via ftp to remote servers.
In general, I would suggest a python program to do the tasks. Write a Qt widget as a gui (interactively changing views, putting numbers into tables, setting up check boxes, switching between various layers - never done something as complex in python, but some experience in IDL with event handling etc), set up data classes that have unctions, both to create the ascii files with the given convention, and to send the files via ftp to some remote server.
However, since my company is a bunch of Windows users, each sitting at their personal desktop, installing python and all necessary libraries on each individual machine would be a pain in the ass.
In addition, in a future version the program is supposed to become smart and do some optimization 24/7. Therefore, it makes sense to put it to a server. As I personally rather use Linux, the server is already set up using Ubuntu server.
The idea is now to run my application on the server. But how can the users access and control the program?
The easiest way for everybody to access something like a common control panel would be a browser I guess. I have to make sure only one person at a time is sending signals to the same units at a time, but that should be doable via flags in the database.
After some google-ing, next to QtWebKit, django seems to the first choice for such a task. But...
Can I run a full fledged python program underneath my web application? Is django the right tool to do so?
As mentioned previously, in the (intermediate) future ( ~1 year), we might have to implement some computational expensive tasks. Is it then also possible to utilize C as it is within normal python?
Another question I have is on the development. In order to become productive, we have to advance in small steps. Can I first create regular python classes, which later on can be imported to my web application? (Same question applies for widgets / QT?)
Finally: Is there a better way to go? Any standards, any references?
Django is a good candidate for the website, however:
It is not a good idea to run heavy functionality from a website. it should happen in a separate process.
All functions should be asynchronous, I.E. You should never wait for something to complete.
I would personally recommend writing a separate process with a message queue and the website would only ask that process for statuses and always display a result immediatly to the user
You can use ajax so that the browser will always have the latest result.
ZeroMQ or Celery are useful for implementing the functionality.
You can implement functionality in C pretty easily. I recomment however that you write that functionality as pure c with a SWIG wrapper rather that writing it as an extension module for python. That way the functionality will be portable and not dependent on the python website.
Question for Python 2.6
I would like to create an simple web application which in specified time interval will run a script that modifies the data (in database). My problem is code for infinity loop or some other method to achieve this goal. The script should be run only once by the user. Next iterations should run automatically, even when the user leaves the application. If someone have idea for method detecting apps breaks it would be great to show it too. I think that threads can be the best way to achive that. Unfortunately, I just started my adventure with Python and don't know yet how to use them.
The application will have also views showing database and for control of loop script.
Any ideas?
You mentioned that you're using Google App Engine. You can schedule recurring tasks by placing a cron.yaml file in your application folder. The details are here.
Update: It sounds like you're not looking for GAE-specific solutions, so the more general advice I'd give is to use the native scheduling abilities of whatever platform you're using. Cron jobs on a *nix host, scheduled tasks on Windows, cron.yaml on GAE, etc.
In your other comments you've suggested wanting something in Python that doesn't leave your script executing, and I don't think there's any way to do this. Some process has to be responsible for kicking off whatever it is you need done, so either you do it in Python and keep a process executing (even if it's just sleeping), or you use the platform's scheduling tools. The OS is almost guaranteed to do a better job of this than your code.
i think you'd want to use cron. write your script, and have cron run it every X minutes / hours.
if you really want to do this in Python, you can do something like this:
while(True):
<your app logic here>
sleep(TIME_INTERVAL)
Can you use cron to schedule the job to run at certain intervals? It's usually considered better than infinite loops, and was designed to help solve this sort of problem.
There's a very primitive cron in the Python standard library: import sched. There's also threading.Timer.
But as others say, you probably should just use the real cron.
I have a website I am looking to stay updated with and scrape some content from there every day. I know the site is updated manually at a certain time, and I've set cron schedules to reflect this, but since it is updated manually it could be 10 or even 20 minutes later.
Right now I have a hack-ish cron update every 5 minutes, but I'd like to use the deferred library to do things in a more precise manner. I'm trying to chain deferred tasks so I can check if there was an update and defer that same update a for couple minutes if there was none, and defer again if need be until there is finally an update.
I have some code I thought would work, but it only ever defers once, when instead I need to continue deferring until there is an update:
(I am using Python)
class Ripper(object):
def rip(self):
if siteHasNotBeenUpdated:
deferred.defer(self.rip, _countdown=120)
else:
updateMySite()
This was just a simplified excerpt obviously.
I thought this was simple enough to work, but maybe I've just got it all wrong?
The example you give should work just fine. You need to add logging to determine if deferred.defer is being called when you think it is. More information would help, too: How is siteHasNotBeenUpdated set?