can anyone please help me
I have created a model, like so
class TickerOHLC(models.Model):
date = models.DateField()
open = models.FloatField()
close = models.FloatField()
low = models.FloatField()
high = models.FloatField()
volume = models.FloatField()
def __str__(self):
return str(self.date), str(self.open)
and I can insert data into the database by uploading a file in the admin panel using import-export, like so
Screenshot of admin panel
Here are the admin.py content
from django.contrib import admin
from .models import *
from import_export.admin import ImportExportModelAdmin
#admin.register(Task, TickerOHLC)
class ViewAdmin(ImportExportModelAdmin):
pass
How can I import data to the database from a .py file? I have tried this
from tasks.models import TickerOHLC #This import gives an error: No module named 'tasks'
dataframe = generateDataframe() #function that returns a dataframe
datas = [
TickerOHLC(
date = dataframe.iloc[row]['Date'],
open = dataframe.iloc[row]['Open'],
close = dataframe.iloc[row]['Close'],
low = dataframe.iloc[row]['Low'],
high = dataframe.iloc[row]['High'],
volume = dataframe.iloc[row]['Volume'],
)
for row in dataframe.iterrows()
]
TickerOHLC.objects.bulk_create(datas)
Here are my folder structure
Screenshot of folder structure
I am new to Django and I don't even know if my approach is possible
My goal is to be able to run a script periodically that inserts into the database
Any kind of help is greatly appreciated, thank you
You are getting a ModuleNotFoundError because your project directory wasn't included in the PATH variable, which Django normally does for you via the python3 manage.py command.
If you were to set the PATH variable manually, you would end up getting an ImproperlyConfigured error because your DJANGO_SETTINGS_MODULE variable was not set (again, Django normally does this for you).
Instead of trying to configure everything outside of django, it's much easier to use the python3 manage.py utility.
In your case, the the easiest fix is to put this script in a custom Management Command, in the directory tasks/management/commands/my_command.py:
from django.core.management.base import BaseCommand
from tasks.models import TickerOHLC
class Command(BaseCommand):
def handle(self, **options):
dataframe = generateDataframe()
datas = [
TickerOHLC(
date = dataframe.iloc[row]['Date'],
open = dataframe.iloc[row]['Open'],
close = dataframe.iloc[row]['Close'],
low = dataframe.iloc[row]['Low'],
high = dataframe.iloc[row]['High'],
volume = dataframe.iloc[row]['Volume'],
)
for row in dataframe.iterrows()
]
TickerOHLC.objects.bulk_create(datas)
Now to execute this command, simply call python3 manage.py my_command, and Django will configure everything for you.
This good for testing and for tasks you typically use during development, but isn't suitable for periodic tasks that you mentioned in your question. For that I would recommend starting with django-background-tasks.
You need an empty __init__.py file in every folder so that python is able to import modules. the parent directory doesn't have an __init__.py file and that's why the import is failing.
Secondly, for periodic tasks, you should look into celery.
If you are executing this code outside of Django, please add below two lines of code at the beginning.
import django
django.setup()
If you are planning to schedule automatic execution of the file, it is better to set up Job Scheduling using django-extensions.
In that case above 2 lines of code are not required.
Related
I have a Django project where users can register and add their XML/CSV feeds.
What the project does is (everyday morning):
download the feed
parse the feed and store it to the model Product
generate XML/CSV export from these products in a different format
Now the problem is that users can register anytime during the project lifetime and many feeds have to be either downloaded, parsed or exported in a specific way.
So I need to be able to react quickly when the feed is added and write custom functions/classes/scripts to download/parse or export these sources.
Let's say it would be enough to add or extend some base classes like Downloader,Parser or Exporter
I'm looking for the most common way to do this.
I tried multiple approaches. For example, I've created a package parsers and there is a __init__.py with CHOICES attribute. When a user adds new feed, I create a parser__sourcename.py file which I add into parsers package which contains Parser class which extends BaseParser class with parse method where is the custom code.
Then I add import into __init__.py file so it looks like this:
from feeds_core.parsers.parser__source1 import Parser as ParserForSource1
from feeds_core.parsers.parser__source2 import Parser as ParserForSource2
from feeds_core.parsers.parser__source3 import Parser as ParserForSource3
PARSER__SOURCE1 = 'source1'
PARSER__SOURCE2 = 'source2'
PARSER__SOURCE3 = 'source3'
CHOICES = {
PARSER__SOURCE1: ParserForSource1,
PARSER__SOURCE2: ParserForSource2,
PARSER__SOURCE3: ParserForSource3,
}
def get_parser(choice):
return CHOICES[choice]
Then I have a model Source with this field:
PARSER_CHOICES = [(x, x) for x in CHOICES.keys()]
parser = models.CharField(max_length=128, choices=PARSER_CHOICES, null=True, blank=True)
def get_parser(self):
...
elif self.parser:
return self.get_parser_class()(self.get_last_filepaths(), self.user_eshop, self)
else:
raise self.NoParserDefinedException
def parse(self): # TODO raise spec exc
self.parse_started()
self.get_parser().parse()
self.parse_ended()
def get_parser_class(self) -> BaseParser:
return get_parser(self.parser)
And when there is a new feed, I create the file, modify the __init__.py and choose the parser for this source in the Django admin interface.
But it is little bit complicated and moreover, I'm afraid I have to restart production server every time.
Do you have any ideas/experiences? Is there some best practice how to do such things?
A very basic implementation where you would add a file to a directory (settings.PARSER_FILE_DIR), it would then be an available choice for Source.parser_file, then using importlib you would load the file and extract the Profile class out of it. settings.PARSER_FILE_DIR would probably have to be in your PYTHONPATH
This is an extremely basic example and you would need a lot of error handling and would need to make sure it was very secure
import importlib
class Source(models.Model):
parser_file = models.FilePathField(path=settings.PARSER_FILE_DIR, match='.*\.py$')
def get_parser_class(self):
return importlib.import_module(self.parser_file.replace('.py', '')).Parser
I had forked my shipping app into a folder called forked_apps using oscar_fork_app command and also added in settings.py get_core_apps(['forked_apps.shipping']), I just want to create two shipping methods mentioned, standard and express in the docs in this link:https://django-oscar.readthedocs.io/en/latest/howto/how_to_configure_shipping.html.
In the init.py I have this code pre-existing:
default_app_config = 'forked_apps.shipping.config.ShippingConfig'
In repository.py I have written like this:
from oscar.apps.shipping import repository
from .methods import *
class Repository(repository.Repository):
def get_available_shipping_methods(
self, basket, user=None, shipping_addr=None,
request=None, **kwargs):
methods = (Standard())
print("\n\nFetch availble shipping methods")
if shipping_addr:
# Express is only available in the UK
methods = (Standard(), Express())
return methods
And in the methods.py I had written:
from decimal import Decimal as D
from oscar.apps.shipping import methods
from oscar.core import prices
class Standard(methods.FixedPrice):
code = 'standard'
name = 'Standard shipping'
charge_excl_tax = D('5.00')
class Express(methods.FixedPrice):
code = 'express'
name = 'Express shipping'
charge_excl_tax = D('10.00')
What should happen is, the shipping_methods.html page should show up, but instead, after entering the shipping address it goes to the payment details page directly; this would usually happen only if there are no shipping methods defined, but I have implemented two shipping methods, standard and Express in the above code.I can't figure out how to make this work, even the print statement isn't working.
Is there any other additional code that I must write?
Can someone provide a solution with some code, if you have implemented it?
Remove oscar apps from settings.
For example:
#oscar.apps.checkout
#oscar.apps.shipping
etc.
This section gives me error. I can't fix that.
get_available_shipping_methods(
self, basket, user=None, shipping_addr=None,
request=None, **kwargs):
...
Django ver. > 2.1 | Oscar ver. > Latest
I using it like this;
mkdir customapp
touch customapp/__init__.py
python manage.py oscar_fork_app shipping customapp/
Edit settings.py
from oscar import get_core_apps
INSTALLED_APPS = INSTALLED_APPS + get_core_apps(
['customapp.shipping'])
In our customapp/shipping directory added new file, called (repository.py)
from oscar.apps.shipping import repository
from . import methods
class Repository(repository.Repository):
methods = (methods.Standard(),)
Then add new file in same directory, customapp/shipping, called (methods.py)
from oscar.apps.shipping import methods
from oscar.core import prices
from decimal import Decimal as D
class Standard(methods.Base):
code = 'standard'
name = 'Shipping (Standard)'
def calculate(self, basket):
return prices.Price(
currency=basket.currency,
excl_tax=D('5.00'), incl_tax=D('5.00'))
You can add more methods.
Then run these commands;
python manage.py makemigrations
python manage.py migrate
python manage.py runserver
Hope this helps.
I'm trying to access my Django database from within a regular Python script. So far what I did is:
import os
import django
from django.db import models
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "m_site.settings")
django.setup()
from django.apps import apps
ap=apps.get_model('py','Post')
q=ap.objects.all()
for a in q:
print(a.title)
There is an application in Django called py where I have many posts (models.py, which contains class Post(models.Model)).
I would like to have the possibility to access and update this database from a regular Python script. So far script above works fine, I can insert, update or query but it looks like it is a separate database and not the database from Django's py application. Am I doing something wrong or missing something? Any help would be very appreciated.
Consider writing your script as a custom management command. This will let you run it via manage.py, with Django all properly wired up for you, e.g.
python manage.py print_post_titles
Something like this should be a good start:
from django.core.management.base import BaseCommand
from py.models import Post
class Command(BaseCommand):
help = 'Prints the titles of all Posts'
def handle(self, *args, **options):
for post in Post.objects.all():
print(a.title)
I'm trying to turn a Python script into a Django management command. My script is in an application folder called sites. Folder structure:
project
|--sites
|scanner.py
|--management
|__init__.py
|--commands
|__init__.py
|getdeals.py
I'm trying to have getdeals.py run as a management command. It finds objects in my Site model and then uses them to create an instance of the SiteDeals class, which is in the scanner.py file.
getdeals.py:
from django.core.management.base import BaseCommand
from sites.models import Site
class Command(BaseCommand):
help = "Scans all sites for deals"
def handle(self, *args, **options):
site_set = Site.objects.all()
for site in site_set:
scraper = SiteDeals(site)
When I run python manage.py getdeals it says NameError: name 'SiteDeals' is not defined.
I thought of taking the code from handle and writing it as a main() function in scanner.py, and then accessing it from getdeals.py, but can't work out how to access it that way.
How do I access SiteDeals from the scanner.py file, given that it is in another folder from my management/commands folder, so that I can pass my objects to it?
You need to import SiteDeals, just as you import the Site model. Try:
from sites.scanner import SiteDeals
I'd like to have these lines of code executed on server startup (both development and production):
from django.core import management
management.call_command('syncdb', interactive=False)
Putting it in settings.py doesn't work, as it requires the settings to be loaded already.
Putting them in a view and accessing that view externally doesn't work either, as there are some middlewares that use the database and those will fail and not let me access the view.
Putting them in a middleware would work, but that would get called each time my app is accessed. An possible solution might be to create a middleware that does all the job and then removes itself from MIDDLEWARE_CLASSES so it's not called anymore. Can I do that without too much monkey-patching?
Write middleware that does this in __init__ and afterwards raise django.core.exceptions.MiddlewareNotUsed from the __init__, django will remove it for all requests :). __init__ is called at startup by the way, not at the first request, so it won't block your first user.
There is talk about adding a startup signal, but that won't be available soon (a major problem for example is when this signal should be sent)
Related Ticket: https://code.djangoproject.com/ticket/13024
Update: Django 1.7 includes support for this. (Documentation, as linked by the ticket)
In Django 1.7+ if you want to run a startup code and,
1. Avoid running it in migrate, makemigrations, shell sessions, ...
2. Avoid running it twice or more
A solution would be:
file: myapp/apps.py
from django.apps import AppConfig
def startup():
# startup code goes here
class MyAppConfig(AppConfig):
name = 'myapp'
verbose_name = "My Application"
def ready(self):
import os
if os.environ.get('RUN_MAIN'):
startup()
file: myapp/__init__.py
default_app_config = 'myapp.apps.MyAppConfig'
This post is using suggestions from #Pykler and #bdoering
If you were using Apache/mod_wsgi for both, use the WSGI script file described in:
http://blog.dscpl.com.au/2010/03/improved-wsgi-script-for-use-with.html
Add what you need after language translations are activated.
Thus:
import sys
sys.path.insert(0, '/usr/local/django/mysite')
import settings
import django.core.management
django.core.management.setup_environ(settings)
utility = django.core.management.ManagementUtility()
command = utility.fetch_command('runserver')
command.validate()
import django.conf
import django.utils
django.utils.translation.activate(django.conf.settings.LANGUAGE_CODE)
# Your line here.
django.core.management.call_command('syncdb', interactive=False)
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
You can create a custom command and write your code in the handle function. details here https://docs.djangoproject.com/en/dev/howto/custom-management-commands/
Then you can create a startup script that runs the django server then executes your new custom command.
If you are using mod_wsgi you can put it in the wsgi start app
Here is how I work around the missing startup signal for Django:
https://github.com/lsaffre/djangosite/blob/master/djangosite/models.py
The code that is being called there is specific to my djangosite project, but the trick to get it called by writing a special app (based on an idea by Ross McFarland) should work for other environments.
Luc