I have a Django project where users can register and add their XML/CSV feeds.
What the project does is (everyday morning):
download the feed
parse the feed and store it to the model Product
generate XML/CSV export from these products in a different format
Now the problem is that users can register anytime during the project lifetime and many feeds have to be either downloaded, parsed or exported in a specific way.
So I need to be able to react quickly when the feed is added and write custom functions/classes/scripts to download/parse or export these sources.
Let's say it would be enough to add or extend some base classes like Downloader,Parser or Exporter
I'm looking for the most common way to do this.
I tried multiple approaches. For example, I've created a package parsers and there is a __init__.py with CHOICES attribute. When a user adds new feed, I create a parser__sourcename.py file which I add into parsers package which contains Parser class which extends BaseParser class with parse method where is the custom code.
Then I add import into __init__.py file so it looks like this:
from feeds_core.parsers.parser__source1 import Parser as ParserForSource1
from feeds_core.parsers.parser__source2 import Parser as ParserForSource2
from feeds_core.parsers.parser__source3 import Parser as ParserForSource3
PARSER__SOURCE1 = 'source1'
PARSER__SOURCE2 = 'source2'
PARSER__SOURCE3 = 'source3'
CHOICES = {
PARSER__SOURCE1: ParserForSource1,
PARSER__SOURCE2: ParserForSource2,
PARSER__SOURCE3: ParserForSource3,
}
def get_parser(choice):
return CHOICES[choice]
Then I have a model Source with this field:
PARSER_CHOICES = [(x, x) for x in CHOICES.keys()]
parser = models.CharField(max_length=128, choices=PARSER_CHOICES, null=True, blank=True)
def get_parser(self):
...
elif self.parser:
return self.get_parser_class()(self.get_last_filepaths(), self.user_eshop, self)
else:
raise self.NoParserDefinedException
def parse(self): # TODO raise spec exc
self.parse_started()
self.get_parser().parse()
self.parse_ended()
def get_parser_class(self) -> BaseParser:
return get_parser(self.parser)
And when there is a new feed, I create the file, modify the __init__.py and choose the parser for this source in the Django admin interface.
But it is little bit complicated and moreover, I'm afraid I have to restart production server every time.
Do you have any ideas/experiences? Is there some best practice how to do such things?
A very basic implementation where you would add a file to a directory (settings.PARSER_FILE_DIR), it would then be an available choice for Source.parser_file, then using importlib you would load the file and extract the Profile class out of it. settings.PARSER_FILE_DIR would probably have to be in your PYTHONPATH
This is an extremely basic example and you would need a lot of error handling and would need to make sure it was very secure
import importlib
class Source(models.Model):
parser_file = models.FilePathField(path=settings.PARSER_FILE_DIR, match='.*\.py$')
def get_parser_class(self):
return importlib.import_module(self.parser_file.replace('.py', '')).Parser
Related
can anyone please help me
I have created a model, like so
class TickerOHLC(models.Model):
date = models.DateField()
open = models.FloatField()
close = models.FloatField()
low = models.FloatField()
high = models.FloatField()
volume = models.FloatField()
def __str__(self):
return str(self.date), str(self.open)
and I can insert data into the database by uploading a file in the admin panel using import-export, like so
Screenshot of admin panel
Here are the admin.py content
from django.contrib import admin
from .models import *
from import_export.admin import ImportExportModelAdmin
#admin.register(Task, TickerOHLC)
class ViewAdmin(ImportExportModelAdmin):
pass
How can I import data to the database from a .py file? I have tried this
from tasks.models import TickerOHLC #This import gives an error: No module named 'tasks'
dataframe = generateDataframe() #function that returns a dataframe
datas = [
TickerOHLC(
date = dataframe.iloc[row]['Date'],
open = dataframe.iloc[row]['Open'],
close = dataframe.iloc[row]['Close'],
low = dataframe.iloc[row]['Low'],
high = dataframe.iloc[row]['High'],
volume = dataframe.iloc[row]['Volume'],
)
for row in dataframe.iterrows()
]
TickerOHLC.objects.bulk_create(datas)
Here are my folder structure
Screenshot of folder structure
I am new to Django and I don't even know if my approach is possible
My goal is to be able to run a script periodically that inserts into the database
Any kind of help is greatly appreciated, thank you
You are getting a ModuleNotFoundError because your project directory wasn't included in the PATH variable, which Django normally does for you via the python3 manage.py command.
If you were to set the PATH variable manually, you would end up getting an ImproperlyConfigured error because your DJANGO_SETTINGS_MODULE variable was not set (again, Django normally does this for you).
Instead of trying to configure everything outside of django, it's much easier to use the python3 manage.py utility.
In your case, the the easiest fix is to put this script in a custom Management Command, in the directory tasks/management/commands/my_command.py:
from django.core.management.base import BaseCommand
from tasks.models import TickerOHLC
class Command(BaseCommand):
def handle(self, **options):
dataframe = generateDataframe()
datas = [
TickerOHLC(
date = dataframe.iloc[row]['Date'],
open = dataframe.iloc[row]['Open'],
close = dataframe.iloc[row]['Close'],
low = dataframe.iloc[row]['Low'],
high = dataframe.iloc[row]['High'],
volume = dataframe.iloc[row]['Volume'],
)
for row in dataframe.iterrows()
]
TickerOHLC.objects.bulk_create(datas)
Now to execute this command, simply call python3 manage.py my_command, and Django will configure everything for you.
This good for testing and for tasks you typically use during development, but isn't suitable for periodic tasks that you mentioned in your question. For that I would recommend starting with django-background-tasks.
You need an empty __init__.py file in every folder so that python is able to import modules. the parent directory doesn't have an __init__.py file and that's why the import is failing.
Secondly, for periodic tasks, you should look into celery.
If you are executing this code outside of Django, please add below two lines of code at the beginning.
import django
django.setup()
If you are planning to schedule automatic execution of the file, it is better to set up Job Scheduling using django-extensions.
In that case above 2 lines of code are not required.
I want to make Django accept url that consists of infinite number of parameters. Something like dropbox has going on. Each parameter for each folder the file is in. There could be infinite number of subfolders. Parameter is alphanumeric.
You can create a URL that accepts an arbitrarily long parameter than contains some form of separator and then in your view split the parameter by this separator. For a path like parameter:
url(r'^prefix/(?P<path>[a-zA-Z\/]*)/$', your_view),
def your_view(request, path):
folders = path.split('/')
Now any request to this URL like prefix/folder1/folder2/folder3/ and folders will contain ['folder1', 'folder2', 'folder3']
I don't think so that you can define infinite url parameters with django URLS. Because with django you have to declare the urls that you site will gonna use.
But if you are talking specifically of folder and files, you can do this with FileField in a model, that allows you to decide where you can save files and save the path in instance.filefield.url. then insted of search django urls, search nginx url, something like this. {site_url}/static/{any}/{folder}/{you}/{want}/{etc}/.
Inclusive, you can use models like this.
class Folder(models.Model):
name = models.CharField(...)
parent = mdoels.ForeignKey('self')
...
def get_path_to_here(self):
# here you have the full url getting all parents folders.
...
def func_to_declare_where_save_it(instance, filename):
return os.path.join(
f'{instance.folder.get_path_to_here()}',
f'{filename}')
class File(models.Model):
name = models.CharField(...)
folder = models.ForeignKey(Folder)
file = models.FileField(upload_to=func_to_declare_where_save_it)
I'm working on a Django project that utilizes customized greetings (like in voicemail). The whole functionality is implemented, i have created a custom model:
class Greeting(models.Model):
audio_file = models.FileField(upload_to='greetings/')
description = models.CharField(max_length=128)
uploaded_at = models.DateTimeField(auto_now_add=True)
The next thing that i wanted to do is to make sure that the uploaded file has all the expected properties (is a WAV file, has one channel, has low bitrate etc). But i don't even know where to start. These files will be only added via django admin. In regular FormView i would utilize server-sided validation in View, and only then add it to model. How to do it in django admin?
To summarize what i expect my app to do:
1) Add file to a model in django admin
2) Server checks file properties, and if requirements are not met, tries to convert it to proper format 3) If the file is in proper format, only then it saves the object.
You need to register a ModelAdmin with a custom form.
ModelAdmin has a form property which is by default set to forms.ModelForm class, you can replace that by assigining that property to your Admin class.
# app_dir/admin.py
from django.contrib import admin
from .forms import GreetingAdminForm
from .models import Greeting
#admin.register(models.Greeting)
class GreetingAdmin(admin.ModelAdmin):
form = GreetingAdminForm
readonly_fields = ['uploaded_at']
Than you need to define your GreetingAdminForm in forms.py. with custom validation Logic.
The way I would do it is add a ModelForm with overridden audo_file field with added validators. You can check the django documentation for writing your validation logic here
Probaly you want to use file extension validation, and add a clean_{fieldname} method on the form.
The clean_{fieldname} method does not take any arguments but the return value of this method must replace the existing value in cleaned_data. You will need an external library that suits your needs, accepts audio formats that you intend to allow, and outputs processed file in desired format. Docs on cleaning specific attribiutes are here
# app_dir/forms.py
from django import forms
from django.core.exceptions import ValidationError
from .validators import validate_file_extension
from .models import Greeting
class GreetingAdminForm(forms.ModelForm):
audio_file = forms.FileField(validators=[validate_file_extension])
def clean_audio_file(self):
data = self.cleaned_data
processed_audio_file = None
# audio file processing logic goes here,
if not processed_audio_file:
raise ValidationError('error message')
data['audio_file'] = processed_audio_file
return data
class Meta:
model = Greeting
fields = [
'audio_file',
'description'
]
# app_dir/validators.py
def validate_file_extension(value):
# validate file extension logic here,
you can find a example of file extension validation
here
Another angle to approach this could be also
- writing a custom form field which subclasses the FileField,you can find documentation on writing your own field here, this class should override w methods validate() - which handles validation logic, and to python where you would prepare output to be available in your python code
During the import procedure I made sure to mark the imgs editable, and put several content sections into multiline constructs. The imported templates show up in the client's template set, but when I try to edit them, I only presented with the zip/HTML upload option instead of the actual editor:
From my code base, my own template object has the exporting code:
class EmailBodyTemplate(models.Model):
...
def migrate_email_template(self, cm_client_id, cm_token):
cm_template = Template(cm_token)
# Basically I expose a specially tailored HTML version of my template
# for CM: adding editable to imgs, introducing multiline wrappers and
# adding unsubscribe, view in browser and other special links
html_url = settings.HOME_URL + reverse('email_import_view', args=(self.id,))
template_id = cm_template.create(cm_client_id, self.name, html_url, None)
self.template_id = template_id
self.save()
I reached out to Campaign Monitor support on the matter, and got the info that their editor is not applicable for imported HTML templates from an external source. In that case you need to keep editing those outside of the system and re-upload after edit.
I am writing a migration script to parse an old html website into Django CMS pages.
The thing I need is to understand on how to nest plugins programatically.
In particular case I need to have html < a> tags converted into CMS LinkPlugin objects, nested inside text that is edited by standard ckeditor TextPlugin of Django-CMS.
How to programmatically nest plugins inside other plugins of Django CMS. In my case I need to nest a CMS Link plugin inside of the TextPlugin in the text.
I know on how to parse text. I do not understand on how to do it from nested CMS plugins perspective?
I can not interconnect the Link CMS plugin object instance and CMSPlugin object instance that I insert into the ancestor TextPlugin.
More context:
Note I really know how to do this from UI perspective. I need to emulate this in a script.
I have dumped the database into JSON and noticed there are certain things there.
First I have a CMSPlugin class instance that is placed into a page placeholder. (Sotle this part from placeholderadmin.py of the CMS)
position = CMSPlugin.objects.filter(language=lang, parent=parent).count()
plugin = CMSPlugin(
language='en',
position=position,
plugin_type=plugin_type,
placeholder=placeholder,
)
plugin.insert_at(parent, position='last-child', save=False)
plugin.save()
# ?????
plugin.link = Link(
name='Link text',
page_link=target_page,
placeholder=placeholder,
)
plugin.save()
This creates a nested plugin in a proper placeholder and attaches it into a text plugin. However it is added with a blank LinkPlugin instance. I'm later creating an instance of a Link plugin in the CMS.
The thing is I do not know on how to do this properly.
From UI perspective the CMS plugin is added nested but contains no real plugin instance. SO the Admin plugins tree for that placeholder is rendered with empty Link plugins.
CMSPlugins are added Link < Empty>.
I can edit this created Link plugin through admin and add a text and target link. How to do this programatically. E.g. inside of a script? Script must do 1000-s of pages so I can not do it manually
Sorry just to be consistent on this. This is far more complicated in logic then it seems.
I did an article on this.
Django CMS Adding plugins inside plugins programmatically
In general the solution is to mimic the CMS way of doing this.
# Getting an site admin instance
admin_site = AdminSite()
instance, plugin_admin = plugin.get_plugin_instance(admin_site)
plugin_admin.cms_plugin_instance = plugin
plugin_admin.placeholder = plugin.placeholder
# Triggering the Django Admin add view with our request.
# That's how Django-CMS does this action itself.
response = plugin_admin.add_view(request)
Look for the full snippet in article. Hope this helps someone with similar problems.
To add nested plugins you need to do this:
add_plugin(
placeholder=placeholder,
plugin_type='TextPlugin',
language=translation.get_language(),
)
target = placeholder.get_plugins().get(plugin_type='TextPlugin')
add_plugin(
placeholder=placeholder, #same placeholder as the parent plugin
plugin_type='LinkPlugin',
language=translation.get_language(),
target=target, #the parent plugin
#here comes the params from the selected plugin
name='Google',
url='http://www.google.com'
)
This also works with custom plugins.
Did you tried to save Link plugin that you've created?
plugin.link = Link(
name='Link text',
page_link=target_page,
placeholder=placeholder,
)
maybe try to add
plugin.link.save()
I hope that is the case.
To create nested plugins in your cms_plugins.py file
from .models import ParentPlugin, ChildPlugin
#plugin_pool.register_plugin
class ParentCMSPlugin(CMSPluginBase):
render_template = 'parent.html'
name = 'Parent'
model = ParentPlugin
allow_children = True # This enables the parent plugin to accept child plugins
# You can also specify a list of plugins that are accepted as children,
# or leave it away completely to accept all
# child_classes = ['ChildCMSPlugin']
def render(self, context, instance, placeholder):
context = super().render(context, instance, placeholder)
return context
#plugin_pool.register_plugin
class ChildCMSPlugin(CMSPluginBase):
render_template = 'child.html'
name = 'Child'
model = ChildPlugin
require_parent = True # Is it required that this plugin is a child of another plugin?
# You can also specify a list of plugins that are accepted as parents,
# or leave it away completely to accept all
# parent_classes = ['ParentCMSPlugin']
def render(self, context, instance, placeholder):
context = super(ChildCMSPlugin, self).render(context, instance, placeholder)
return context
In your plugin template file of parent plugin
{% load cms_tags %}
<div class="plugin parent">
{% for plugin in instance.child_plugin_instances %}
{% render_plugin plugin %}
{% endfor %}
</div>
For detailed documentation, go through
https://docs.django-cms.org/en/latest/how_to/custom_plugins.html#nested-plugins