I have a Django project which parses XML feeds. The thing is that I often need to write a custom script to parse or download a new feed/s for a new client.
I don't think the best way is to modify Django code every time I need a custom parsing/downloading pipeline.
My idea is to create a standard. For example, I upload myscript.py through Admin, which must have the class class Downloader() with function download(self) and this function will be called every time that source has to be downloaded.
So what is the most common way to do that?
The only thing which comes to my mind is to upload myscript.py which has if __name__ == '__main__' function and call it by for example Popen('python filename.py')
or
from subprocess import call
call(["python", "filename.py"])
model:
class Source(..):
custom_downloader = FileField(... # if needed
...
Related
I am building an app with Django to automate a reservation which I need to make every day. Django is used to provide a web interface to monitor and control this.
However, I cannot figure out where to put my code within the Django files. Before trying Django I just had a while True loop that made the reservation as soon as a condition was true. To do this in Django, I need to create a main object on startup and reference this from different views in the views.py file. And I need to run some kind of schedule that checks the condition if the reservation is to be made. All of this in parallel to the actual website code.
Using celery seems complicated and overkill for this. Is there a main Django object that is created on startup and can be accessed from other files (such as views.py)? Then I could create a parent class of this class that holds my main code and allows for starting and managing parallel jobs.
E.g., I looked at AppConfig, but don't know how to reference this object from other files...
class ReservationAppConfig(AppConfig):
default_auto_field = 'django.db.models.BigAutoField'
name = 'reservation_app'
driver = None
def ready(self):
# python manage.py runserver runs the ready method twice — once in each of two processes —
# but we only want to run it once.
if os.environ.get('RUN_MAIN', None) != 'true':
self.driver = Safari()
self.driver.get(LIB_ADRESS)
I suggest the next approach:
You create management command with your while True: script.
from django.core.management import BaseCommand
class Command(BaseCommand):
def handle(self, *args, **options):
while True:
... # your previous script + storing data into DB via Django models
This script changes state in database. So after each cycle it saves or updates some [Django models])https://docs.djangoproject.com/en/4.0/topics/db/models/).
Django View get these models from DB and displays data on web UI.
It seems more stable, than directly accessing script's object from web threads (you risk to trap into threading-related errors).
I have a python script written that takes an input from one model, queries it, and appends the various results from the query to another model through a ForeignKey relationship. It works great from the python shell, but I was wondering if there is a way to run it from the admin webpage so that every time a new object for the first model is submitted, it runs the script and updates the database automatically for the other model. I'm using the Django admin interface as part of development for staff to do data entry since I've found it's a very flexible interface. The script is written specifically for this app, so it is on the app's folder.
I was surprised that this wasn't already answered.
Either wrap the existing script as a management command, or import it into a management command.
Once you've done that, you can override the Admin view in question, like this..
from django.contrib.admin import AdminSite
from django.views.decorators.cache import never_cache
class MyAdminSite(AdminSite):
#never_cache
def index(self, request, extra_context=None):
# do stuff
Then, you create an instance of this class, and use this instance, rather than admin.site to register your models.
admin_site = MyAdminSite()
Then, later:
from somewhere import admin_site
class MyModelAdmin(ModelAdmin):
...
admin_site.register(MyModel, MyModelAdmin)
Lastly, in that overriding view, you can use management.call_command from your code to call the management command. This lets you use it both from the commandline, and from inside your code - and if you need to, you can schedule it from cron, too. :)
I have a function based view function in django that receives an ID from a model, retrieve a file address and delete it using os.remove
image = Images.objects.get(id=image_id)
os.remove(image.file)
the image_id is valid and is a part of my fixture.
what's the best way to write a test for this view, without manually creating a file each time I'm testing the code?
Is there a way to change the behavior of os.remove function for test?
Yes. It's called mocking, and there is a Python library for it: mock. Mock is available in the standard library as unittest.mock for Python 3.3+, or standalone for earlier versions.
So you would do something like this:
from mock import patch
...
#patch('mymodel_module.os.remove')
def test_my_method(self, mocked_remove):
call_my_model_method()
self.assertTrue(mocked_remove.called)
(where mymodel_module is the models.py where your model is defined, and which presumably imports os.)
I have two modules. One is the core of the website based on web.py (let's name it code.py), and other is an add-on module (addon.py). Using web.py, for each page that the website server there should be class definition in a core, like that:
class Page:
def GET(self):
variable = "Hello!"
return render.page_template(variable) #Here, it returns the rendered template to user
def POST(self):
post_variables = web.input()
pass #Doing something with those variables, maybe, writing in a database...
Now I really need to move that class definition from code.py to addon.py. I can refer to the class definition as a addon.Page instead of simply Page. The Page.GET function still works well... But there's one problem with POST. It seems like at the each call of POST function web.input() in a core module is being set as a storage object storing all the variables. And if my class definition is being stored in addon, the core simply calls addon.Page.POST() (I see no way to change this behaviour). The POST() tries to get web.input()... And fails, of course - web is not imported in addon.py, and even if it was, there wouldn't be any value web.py web-server is getting - just empty dictionary, it would be just another instance of the module. So i don't know...
One solution would be: putting some kind of function in addon.Page.POST(). This function would go one level down, to code.py and execute web.input() there, and return it back, to addon.py, some kind of accessing parent module namespace (like doing import __main__ and accessing __main__.web.input() (which, as I know, is discouraged) ).
Or, for example, putting some kind of C-like pointer that would be shared between the modules, like:
* in code.py there's definition that all the calls to code.addon.web_input() get routed to code.web.input()
* in addon.py - there's simply need to call addon.web_input to get info from code.web.input()
What do I do in this situation? There will be multiple addons, each with the class definition stored in this addon, and I should be able to add new modules, connect and disconnect existing modules easily, without any need to modify code.py. I believe this is possible in Python... Maybe web.py source needs modifying then?
I guess I'll turn my comment into an answer, since it seems to have solved your issue.
Modules that are imported are cached in Python. That means that when you import a module like web (the main web.py module) from multiple other modules, they'll all get the same module object, with the same contents.
So, probably all you need to do is import web at the top of your addon.py module.
What is the best idea to fill up data into a Django model from an external source?
E.g. I have a model Run, and runs data in an XML file, which changes weekly.
Should I create a view and call that view URL from a curl cronjob (with the advantage that that data can be read anytime, not only when the cronjob runs), or create a python script and install that script as a cron (with DJANGO _SETTINGS _MODULE variable setup before executing the script)?
There is excellent way to do some maintenance-like jobs in project environment- write a custom manage.py command. It takes all environment configuration and other stuff allows you to concentrate on concrete task.
And of course call it directly by cron.
You don't need to create a view, you should just trigger a python script with the appropriate Django environment settings configured. Then call your models directly the way you would if you were using a view, process your data, add it to your model, then .save() the model to the database.
I've used cron to update my DB using both a script and a view. From cron's point of view it doesn't really matter which one you choose. As you've noted, though, it's hard to beat the simplicity of firing up a browser and hitting a URL if you ever want to update at a non-scheduled interval.
If you go the view route, it might be worth considering a view that accepts the XML file itself via an HTTP POST. If that makes sense for your data (you don't give much information about that XML file), it would still work from cron, but could also accept an upload from a browser -- potentially letting the person who produces the XML file update the DB by themselves. That's a big win if you're not the one making the XML file, which is usually the case in my experience.
"create a python script and install that script as a cron (with DJANGO _SETTINGS _MODULE variable setup before executing the script)?"
First, be sure to declare your Forms in a separate module (e.g. forms.py)
Then, you can write batch loaders that look like this. (We have a LOT of these.)
from myapp.forms import MyObjectLoadForm
from myapp.models import MyObject
import xml.etree.ElementTree as ET
def xmlToDict( element ):
return dict(
field1= element.findtext('tag1'),
field2= element.findtext('tag2'),
)
def loadRow( aDict ):
f= MyObjectLoadForm( aDict )
if f.is_valid():
f.save()
def parseAndLoad( someFile ):
doc= ET.parse( someFile ).getroot()
for tag in doc.getiterator( "someTag" )
loadRow( xmlToDict(tag) )
Note that there is very little unique processing here -- it just uses the same Form and Model as your view functions.
We put these batch scripts in with our Django application, since it depends on the application's models.py and forms.py.
The only "interesting" part is transforming your XML row into a dictionary so that it works seamlessly with Django's forms. Other than that, this command-line program uses all the same Django components as your view.
You'll probably want to add options parsing and logging to make a complete command-line app out of this. You'll also notice that much of the logic is generic -- only the xmlToDict function is truly unique. We call these "Builders" and have a class hierarchy so that our Builders are all polymorphic mappings from our source documents to Python dictionaries.