I'm trying to create a web interface for a data analysis pipeline using Pyramid. I'm using deform and colander to make the forms. I've been adapting this example:
http://pyramid-tutorials.readthedocs.org/en/latest/humans/security/step02/
Most of the work gets done when a form gets submitted, but there are a couple of generic steps that only need to run once. Can I load some things into memory when the server starts up so that they are accessible from within a view?
You can define some module level variables in the main file of your application (or may be somewhere else) and then use them by importing them as per your requirement.
I use this method to create settings like database connection string for SQLAlchemy from environment variables.
By default, a module will be parsed only once in Python, so you module level code will run only once.
Update 1
Let's assume directory structure of a pyramid project looks like this:
.
├── __init__.py
├── models
│ ├── __init__.py
│ ├── meta
│ │ ├── base.py
│ │ ├── __init__.py
│ │ ├── orm.py
│ │ ├── schema.py
│ │ ├── types.py
│ ├── users.py
├── security.py
├── settings
│ ├── database.py
│ ├── email.py
│ ├── __init__.py
│ ├── redis.py
│ ├── security.py
├── static
│ ├── css
│ │ └── main.css
│ └── js
│ ├── app.js
│ ├── app-services.js
│ ├── controllers
│ │ └── excel_preview.js
├── templates
│ ├── auth
│ │ └── login.html
│ ├── base.html
│ ├── home.html
├── views
│ ├── auth.py
│ ├── home.py
│ ├── __init__.py
Let's say we've following code in settings/redis.py:
import os
import redis
def get_redis_client():
# Read settings from environment variables
redis_db_name = os.environ.get('REDIS_NAME')
redis_host = os.environ.get('REDIS_HOST')
redis_port = os.environ['REDIS_PORT']
# create a redis connection
redis_client = redis.StrictRedis(
host=redis_host,
port=redis_port,
db=redis_db_name,
)
# return newly created redis connection
return redis_client
redis_client = get_redis_client()
SOME_SETTING_STORED_IN_REDIS = redis_client.get('some_setting_stored_in_redis')
You can use this SOME_SETTING_STORED_IN_REDIS variable from anywhere. If name of your app is example_app then in example_app/views/home.py you can use it like this:
from pyramid.view import view_config
from example_app.settings.redis import SOME_SETTING_STORED_IN_REDIS
def includeme(config):
config.add_route('home', '/')
#view_config(
route_name='home',
renderer='home.html',
permission='authenticated'
)
def home_view(request):
return {
"some_setting": SOME_SETTING_STORED_IN_REDIS,
}
I think you're trying to achieve something similar.
If by "things which only need to run once" you mean something like a database connection, some configuration data etc. - in other words something which never changes during the lifecycle of the process then defining them as globals and re-using across the application is fine. Example:
APP_TITLE = 'Data Analysis Pipeline using Pyramid'
#view_config(...)
def home_view(request):
return "Welcome to %s" % APP_TITLE
If you're thinking about storing some global state and re-using it across the views then it would not be a good idea. Example (bad, do not do this):
subscription_step = 1
#view_config(...)
def next_subscription_step(request):
global subscription_step
subscription_step += 1
return HTTPFound('/subscription/step_%s' % subscription_step)
The above code may appear to work for you locally, but things will start fall apart as soon as there's more than one user accessing the app, or if the webserver spawns another worker process, or if the webserver restarts, or a million of other reasons.
Related
I would like to know how Django's imports work. For example, in this code:
# in some_app/views.py
from another_app.models import UserModel
# another_app is another app of the same project
there's an import statement that imports UserModel from models.py in another_app (another app of the same project).
I was just wondering how Django handles this importing because usual Django project's directory structure looks like this:
.
├── another_app
│ ├── __init__.py
│ ├── admin.py
│ ├── apps.py
│ ├── migrations
│ │ └── __init__.py
│ ├── models.py
│ ├── tests.py
│ └── views.py
├── manage.py
├── some_app
│ ├── __init__.py
│ ├── admin.py
│ ├── apps.py
│ ├── migrations
│ │ └── __init__.py
│ ├── models.py
│ ├── tests.py
│ └── views.py
└── some_project
├── __init__.py
├── asgi.py
├── settings.py
├── urls.py
└── wsgi.py
some_app and another_app are separate directories.
I want to know how the importing works because I want to find a workaround for one of my projects that have separate directories but requires each other's functions.
If you register the app name in installed app in django settings you can easily handle most migrations and other stuff other wise you can just branch off from the folder of the app name.
What i recommend is making a separate directory named app and you can place all your app inside app director so the app and project directory gets separated easily and your import will also be easy
you can import in this manner after wards
from app.some_new_app_name.models import MyModel
from app.some_new_app_name_2.models import MyModel2
I had a Flask app that worked fine until, due to growth, I decided to modularize. The new structure is as follows:
├── Dockerfile
├── README.md
├── app
│ ├── __init__.py
│ ├── config.py
│ ├── cron.py
│ ├── database.ini
│ ├── db.py
│ ├── db.sqlite3
│ └── webapp
│ ├── __init__.py
│ └── routes.py
├── db.sqlite3
├── docker-compose.yml
├── error.log
├── gunicorn_config.py
├── rd.py
├── requirements.txt
├── static
│ ├── css
│ │ └── bootstrap.min.css
│ ├── data.html
│ ├── fonts
│ │ ├── glyphicons-halflings-regular.eot
│ │ ├── glyphicons-halflings-regular.svg
│ │ ├── glyphicons-halflings-regular.ttf
│ │ ├── glyphicons-halflings-regular.woff
│ │ └── glyphicons-halflings-regular.woff2
│ ├── images
│ │ ├── ajax-loader.gif
│ │ ├── gear.png
│ │ ├── gear_small.png
│ │ └── logo.png
│ ├── index.html
│ ├── js
│ │ ├── app.js
│ │ ├── bootbox.min.js
│ │ ├── bootstrap.js
│ │ ├── bootstrap.min.js
│ │ └── npm.js
│ ├── login.html
│ ├── manage_users.html
│ ├── register.html
│ ├── reset_password.html
│ ├── settings.html
│ └── upload.html
└── templates
└── base.html
Here is the __init__.py for webapp:
from flask import Flask
from app.config import DebugConfig
from flask_sqlalchemy import SQLAlchemy
from importlib import import_module
from logging import basicConfig, DEBUG, getLogger, StreamHandler
import os
import uuid
def register_blueprints(app):
for module_name in (app.config['MODULES']):
module = import_module('app.{}.routes'.format(module_name))
app.register_blueprint(module.blueprint)
def configure_logs(app):
basicConfig(filename='error.log', level=DEBUG)
logger = getLogger()
logger.addHandler(StreamHandler())
def create_app():
file = (__file__)
app = Flask(__name__)
app.secret_key = str(uuid.uuid4())
app.config.from_object(DebugConfig)
register_blueprints(app)
configure_logs(app)
return app
Here is login page route code:
#blueprint.route("/login", methods=["GET", "POST"])
def login():
if request.method == 'GET':
return render_template(url_for('static', filename='login.html')
Unfortunately this new structure leads to this error when I attempt to serve the app:
builtins.FileNotFoundError FileNotFoundError: [Errno 2] No such file
or directory: '/static/login.html'
I have been messing with this for a while now and just can't figure out how to get the app to work and manage to locate the files. I tried setting the static_url_path value when instantiating the flask app but, although that manages to locate the HTML files, I'm unable to load the other static files like CSS and images because I have my paths defined relative to the static folder as below in the HTML:
<link href="css/bootstrap.min.css?version=24" > or
<link rel="icon" href="/static/images/gear.png">
Please assist, I feel like I'm going to pull my hair out as I've spent too much time on this issue. Thanks in advance.
What your are looking for is the Flask jinja_loader.
In your create_app method, you could overwrite the jinja_loader provided by default for Flask, which look into the template_folder directory to search for file using render_template method.
def create_app():
file = (__file__)
app = Flask(__name__)
app.secret_key = str(uuid.uuid4())
app.config.from_object(DebugConfig)
register_blueprints(app)
configure_logs(app)
# Overwrite Flask jinja_loader, using ChoiceLoader
template_loader = jinja2.ChoiceLoader([
app.jinja_loader,
jinja2.FileSystemLoader('/path/to/static'),
])
app.jinja_loader = template_loader
return app
Jinja2 ChoiceLoader look for template in the list of loaders.
So first in app.jinja_loader : the template_folder directory.
Then using the FileSystemLoader in the /static directory.
Be warn that if you have a template something.html in both the template and the static directory, the first one will be returned.
But Jinja2 has more loader than just the CoiceLoader and you should try to find one that suite your desires. For exemple the DictLoader with a key 'static' could be fine also
whenever I run Django tests, I fall into a double import trap with my models.
Every time I get
RuntimeError: Conflicting 'userdevice' models in application 'cart': <class 'cart.models.devices.UserDevice'> and <class 'fridge.libs.cart.models.devices.UserDevice'>
Minimal failing example is at https://www.dropbox.com/sh/m802q544cet5dui/AACRGOMQXWqxUzi92WfqfvYea?dl=0
My folder structure is
.
├── apps
│ ├── __init__.py
│ └── milk
│ ├── __init__.py
│ └── tests
│ ├── __init__.py
│ └── test_commands.py
├── __init__.py
├── libs
│ ├── cart
│ │ ├── __init__.py
│ │ └── models
│ │ ├── devices.py
│ │ └── __init__.py
│ └── __init__.py
├── Makefile
├── manage.py
└── settings
├── __init__.py
└── settings.py
I have apps and libs folders in INSTALLED_APPS.
My models/__init__.py file contains
from .devices import *
From test_commands.py I include model with
from cart.models import UserDevice
I read multiple questions here at SO and some articles on web. I was also debugging this for a few hours, so I know the problem is that my model is imported twice. Once during the Django project initialization as cart.models.devices.UserDevice and once by the unittest library as fridge.libs.cart.models.devices.UserDevice but I simply do not know how to fix this situation.
This is also causing more problems in files, which can be initialized only once during the app run, but I guess that the solution will be same as to this one.
I am using Django 1.10.8 as this is a legacy project but I experience the same on Django 2.0.7.
Python version is 3.5. OS is Ubuntu 16.04.
Thanks for any advice or hint. Nothing I found so far did not help.
Edit
So I experimented a bit and the problem persist even when no import are made in tests and even if there are no tests at all. Just running the tests will make the double import and RuntimeError happen
No-tests version: https://www.dropbox.com/sh/qqzg842rihbh5im/AADzay8up3d_rFKqy2OAsx1za?dl=0
New folder structure
.
├── __init__.py
├── libs
│ ├── cart
│ │ ├── __init__.py
│ │ └── models
│ │ ├── devices.py
│ │ └── __init__.py
│ └── __init__.py
├── Makefile
├── manage.py
└── settings
├── __init__.py
└── settings.py
Edit:
For the future's sake - content of libs/cart/models/devices.py is just simple Django model class
from django.db import models
class UserDevice(models.Model):
is_active = models.BooleanField(default=True)
class Meta:
app_label = 'cart'
I somehow "solved" the problem by deleting libs/__init__.py file. I am not sure why it was present in the project in the first place.
I want to prepopulate my db using an external script. The script is the following
# -*- coding:utf-8-*-
import os, sys
from random import choice
PROJECT_DIR = os.path.abspath(os.path.join(os.path.abspath(__file__),'..','..'))
sys.path.append(PROJECT_DIR)
os.environ['DJANGO_SETTINGS_MODULE']='geoedu.settings'
from school.models import School
from student.models import Student
if __name__=='__main__':
student = Student(first_name=choice(first_names_males), last_name=choice(last_names_males),
father_name=choice(first_names_males), mother_name=choice(first_names_females),
mobile=choice(mobiles), telephone=choice(telephones))
student.save()
where the arguments in choice are lists with names and telephones. When i run it though i get the following error
django.db.utils.DatabaseError: no such table: student_student
The directory tree
geoedu
├── geoedu
│ ├── __init__.py
│ ├── __init__.pyc
│ ├── settings.py
│ ├── settings.pyc
│ ├── urls.py
│ ├── urls.pyc
│ ├── wsgi.py
│ └── wsgi.pyc
├── geoedu.db
├── geoedu.sublime-project
├── geoedu.sublime-workspace
├── manage.py
├── school
│ ├── admin.py
│ ├── admin.pyc
│ ├── __init__.py
│ ├── __init__.pyc
│ ├── models.py
│ ├── models.pyc
│ ├── tests.py
│ └── views.py
├── scripts
│ └── populate.py
└── student
├── admin.py
├── admin.pyc
├── __init__.py
├── __init__.pyc
├── models.py
├── models.pyc
├── tests.py
└── views.py
What it does is it creates a new geoedu.db inside the script folder and uses that(which is empty and has no student_student table ofcourse)
Why is that happening?The PROJECT_DIR is the correct one(printed it out) and is the root of the tree that is the geodedu on the top of all.
EDIT: new geoedu.db gets created when creating new student. If I comment out those lines geoedu.db doesn't get created.
Defining models in Django helps you to manipulate instances.
But, before, you have to create the table "skeleton" in your sqlite db
For this, you just need to synchronize the db with Django by doing :
python manage.py syncdb
This has to be done one time only.
In the future, you will probably needs to make your models evolve (thus needing similar db evolutions), at this time South - or adhoc schema evolution queries - will be useful (http://south.aeracode.org/) but as far as you're prototyping, you can just remove db and recreate it with good table by executing syncdb
The problem was that i hadn't used an absolute path for the database name in the settings.py file. I added this in the NAME of the database engine
'NAME':os.path.join(SETTINGS_DIR,'geoedu.db')
and everything worked as it should. PROJECT_DIR is
SETTINGS_DIR = os.path.dirname(__file__)
so the database is created inside settings folder. If you wanted inside projects folder you should do
'NAME':os.path.abspath(os.path.join(SETTINGS_DIR,'..','geoedu.db'))
i have the following file structure:
ihe/
├── dcmt
│ ├── actions
│ ├── calendar_observer
│ ├── cms
│ ├── consumption
│ ├── data_mining
│ ├── dcmt
│ ├── dcmt_db
│ ├── dcmt_db.bak.bak
│ ├── dcmt_db.sqlite
│ ├── devices
│ ├── d.py
│ ├── gadgets
│ ├── history
│ ├── houses
│ ├── hwc_settings
│ ├── __init__.py
│ ├── __init__.pyc
│ ├── manage.py
│ ├── notifications
│ ├── profitable
│ ├── rules
│ └── schedule
├── hwc
│ ├── configuration
│ ├── daemons
│ ├── database
│ ├── __init__.py
│ ├── __init__.pyc
│ ├── utils
│ └── wrapper
├── __init__.py
├── __init__.pyc
dcmt is a django project. hwc is pure python. however for instance in hwc/daemons there is a runme.py script. in that runme.py script i want to be able to import the models from the django project. Now as i understand it i have to have the correct python path and then somehow set the django settings. My question is how do i best do this so that for the whole hwc modules I only have to do that once?
Your project structure seems a bit confused.
It's probably not a good idea to have a Django project inside another package hierarchy. A lot of the import paths assume your project is in a top-level package and the only reason you're probably not running into issues already is that Python 2.x still supports relative imports (which have been removed in 3.x). This makes references to packages very ambiguous and can cause weird bugs.
From what I can see your settings package is actually called (fully-qualified) ihe.dcmt.hwc_settings. If ihe is in your Python path (check the value of sys.path in the script you're trying to run), that (i.e. the fully-qualified path) is probably what DJANGO_SETTINGS_MODULE should point at.
If you want to hook into Django's functionality in your scripts, you might want to look into the documentation for writing manage.py commands. This would let you write Django-related scripts more consistently and save you the worry about referencing and initialising Django's settings correctly yourself.