I have a bunch of string and integer constants that I use at various places in my app.
I am planning to put them up in a centralized location, so that it is easier to change them in the future. I could think of the following approaches:
1) Have them as individual variables, stored in the model db.py
settings_title = "My Amazing App"
settings_ver = 2.0
settings_desc = "Moar and Moar cats"
2) Have them as a dict, stored in db.py
settings = { "title": "My Amazing App",
"ver" = 2.0,
"desc" = "Moar and Moar cats"
}
Is it a good idea to use the model db.py? I've heard that it is evaluated for every request. Could putting settings there have a noticeable overhead?
Is there any difference, performance-wise between the two approaches?
Is there a better way of doing this?
You can directly put your config variable in .py file and use that file by import in your module as django used setting.py. If you want to combine the variable on some section bases then you can use ConfigParser which can ready the .cfg file.
Your db.py model file will be executed on every request in any case, so adding a few setting assignments to the code will add negligible overhead. Instead of putting the settings in db.py, for better organization you might consider creating a separate model file. Note, model files are executed in alphabetical order, so if the settings have to be available in subsequent model files, name the settings file something like 0_settings.py to ensure it is executed before any other model files.
If you prefer, you can instead put the settings in a module (e.g., settings.py) in the application's /modules folder and import the settings object in your application code wherever you need it (in that case, the module will only be loaded once by the interpreter). If any of the settings need to be set dynamically based on the incoming request, though, you are probably better off keeping the settings in a model file.
Finally, rather than a standard dictionary, you might consider using a web2py Storage object, which is like a dictionary but allows you to access values as attributes and returns None rather than a KeyError if you try to access a key/attribute that doesn't exist:
from gluon.storage import Storage
settings = Storage()
settings.title = 'My Amazing App'
or
settings = Storage({'title': 'My Amazing App'})
Note, the web2py request, response, and session objects are all instances of the Storage class.
Django, for instance, uses a file settings.py.
It's not a model, but just a collection of variables of all types, strings/ints/dicts/whatever, and you import settings or from settings import * in every module that needs access to them.
Since it is not a single model, there's no overhead on access.
Related
I initially started a small python project (Python, Tkinter amd PonyORM) and became larger that is why I decided to divide the code (used to be single file only) to several modules (e.g. main, form1, entity, database). Main acting as the main controller, form1 as an example can contain a tkinter Frame which can be used as an interface where the user can input data, entity contains the db.Enttiy mappings and database for the pony.Database instance along with its connection details. I think problem is that during import, I'm getting this error "pony.orm.core.ERDiagramError: Cannot define entity 'EmpInfo': database mapping has already been generated". Can you point me to any existing code how should be done.
Probably you import your modules in a wrong order. Any module which contains entity definitions should be imported before db.generate_mapping() call.
I think you should call db.generate_mapping() right before entering tk.mainloop() when all imports are already done.
A good approach to avoid this is rather than having your db.generate_mapping() call happening at a module's top-level code, have a function that a module exports that calls db.generate_mapping() after all other modules have been imported.
The pattern I use is to put all of my db.Entity subclasses into a single module named model, and then at the bottom of model.py is:
def setup():
""" Set up the database """
db.bind(**database_config, create_db=True)
db.generate_mapping(create_tables=True)
This function is called by my application's own startup (which is also responsible for setting up database_config). This way the correct import and setup order can be guaranteed.
The db object itself is also owned by this model module; if I need to use it somewhere else I import model and use model.db.
If you want to further separate things out (with different model classes living in different modules) you can have a module that owns db, then your separate model modules, and then a third module that imports db and the models and provides the setup function. For example, your directory structure could look like this:
model/
__init__.py -- imports all of the model sub-modules and provides a setup function
db.py -- provides the db object itself and any common entities objects that everyone else needs
form1.py, form2.py, etc. -- imports db and uses its database object to define the entities
Then your main app can do something like:
import model
model.setup()
I'm now using django 1.6
As I know, if I want to use some model classes (using ORM) I should put them under folder myapp/models.py.
So what if I want to write some non-model classes (not using ORM), i.e. class Util, and only contains some globally used methods, maybe date formatting or something else.
So, where should these code locate? Is there a generally used way to this case?
You can really put them anywhere you like, but I follow the pattern Django itself uses (in admin, cache, etc.) and put such things in a utils.py file. I have one (if necessary) in each app folder, as well as one for the whole project.
This question isn't entirely App Engine specific, but it might help knowing the context: I have a kind of "static site generator" on App Engine that renders pages and allows them to be styled via various themes and theme settings. The themes are currently stored directly on the App Engine filesystem and uploaded with the application. A theme consists of a few templates and yaml configuration data.
To encapsulate working with themes, I have a Theme class. theme = Theme('sunshine'), for example, constructs a Theme instance that loads and parses the configuration data of the theme called 'sunshine', and allows calls like theme.render_template('index.html') that automatically load and render the correct file on the filesystem.
Problem is, loading and especially parsing a Theme's (yaml) configuration data every time a new request comes in and instantiates a Theme is expensive. So, I want to cache the data within the processes/App Engine instances and maybe later within memcached.
Until now, I've used very simple caches like so:
class Theme(object):
_theme_variables_cache = {}
def __init__(self, name):
self.name = name
if name not in Theme._theme_variables_cache:
Theme._theme_variables[name] = self.load_theme_variables()
...
(I'm aware that the config could be read multiple times when several requests hit the constructor at the same time. I don't think it causes problems though.)
But that kind of caching gets ugly really quickly. I have several different things I want to read from config files and all of the caches are dictionaries because every different theme 'name' also points to a different underlying configuration.
The last idea I had was creating a function like Theme._cached_func(func) that will only execute func when the functions result isn't already cached for the specific template (remember, when the object represents a different template, the cached value can also be different). So I could use it like: self.theme_variables = Theme._cached_func(self.load_theme_variables()), but, I have a feeling I'm missing something obvious here as I'm still pretty new to Python.
Is there an obvious and clean Python caching pattern that will work for such a situation without cluttering up the entire class with cache logic? I think I can't just memoize function results via decorators or something because different templates will have to have different caches. I don't even need any "stale" cache handling because the underlying configuration data doesn't change while a process runs.
Update
I ended up doing it like that:
class ThemeConfig(object):
__instances_cache = {}
#classmethod
def get_for(cls, theme_name):
return cls.__instances_cache.setdefault(
theme_name, ThemeConfig(theme_name))
def __init__(self, theme_name):
self.theme_name = theme_name
self._load_assets_urls() # those calls load yaml files
self._load_variables()
...
class Theme(object):
def __init__(self, theme_name):
self.theme_name = theme_name
self.config = ThemeConfig.get_for(theme_name)
...
So ThemeConfig stores all the configuration stuff that's read from the filesystem for a theme and the factory method ThemeConfig.get_for will always hand out the same ThemeConfig instance for the same theme name. The only caching logic I have is the one line in the factory method, and Theme objects are still as temporary and non-shared as they always were, so I can use and abuse them however I wish.
I will take a shot at this. Basically, a factory pattern can be used here to maintain a clean boundary between your Theme object and the creation of the Theme instance with a particular way.
The factory itself can also maintain a simple caching strategy by storing a mapping between the Theme name and the corresponding Theme object. I would go with a following implementation:
#the ThemeFactory class instantiates a Theme with a particular name if not present within it's cache
class ThemeFactory(object) :
def __init__(self):
self.__theme_variables_cache = {}
def createTheme(self, theme_name):
if not self.__theme_variables_cache.contains(name):
theme = Theme(theme_name)
self.__theme_variables_cache[name] = theme.load_theme_variables()
return self.__theme_variables_cache[name]
The definition of the Theme class is now very clean and simple and will not contain any caching complications
class Theme(object):
def __init__(self, name):
self.__theme_name = name
def load_theme_variables(self):
#contain the logic for loading theme variables from theme files
The approach has the advantages of code maintainability and clear segregation of responsibilities ( although not completely so , the factory class still maintains the simple cache. Ideally it should simply have a reference to a caching service or another class that handles caching .. but you get the point).
Your Theme class does what it does the best - loading theme variables. Since you have a factory pattern, you are keeping the client code ( the one that consumes the Theme class instance) encapsulated from the logic of creating the Theme instances. As your application grows, you can extend this factory to control the creation of various Theme objects (including classes derived fron Theme)
Note that this is just one way of achieving simple caching behavior as well as instance creation encapsulation.
One more point - you could store Theme objects within the cache instead of the theme variables. This way you could read the theme variables from templates only on first use(lazy loading). However, in this case you would need to make sure that you store the theme variables as an instance variable of the Theme class. The method load_theme_variables(self) now needs to be written this way:
def load_theme_variables(self):
#let the theme variables be stored in an instance variable __theme_variable
if not self.__theme_variables is None:
return self.__theme_variables
#__read_theme_file is a private function that reads the theme files
self__theme_variables = self.__read_theme_file(self.__theme_name)
Hopefully, this gives you an idea on how to go about achieving your use case.
What is the best idea to fill up data into a Django model from an external source?
E.g. I have a model Run, and runs data in an XML file, which changes weekly.
Should I create a view and call that view URL from a curl cronjob (with the advantage that that data can be read anytime, not only when the cronjob runs), or create a python script and install that script as a cron (with DJANGO _SETTINGS _MODULE variable setup before executing the script)?
There is excellent way to do some maintenance-like jobs in project environment- write a custom manage.py command. It takes all environment configuration and other stuff allows you to concentrate on concrete task.
And of course call it directly by cron.
You don't need to create a view, you should just trigger a python script with the appropriate Django environment settings configured. Then call your models directly the way you would if you were using a view, process your data, add it to your model, then .save() the model to the database.
I've used cron to update my DB using both a script and a view. From cron's point of view it doesn't really matter which one you choose. As you've noted, though, it's hard to beat the simplicity of firing up a browser and hitting a URL if you ever want to update at a non-scheduled interval.
If you go the view route, it might be worth considering a view that accepts the XML file itself via an HTTP POST. If that makes sense for your data (you don't give much information about that XML file), it would still work from cron, but could also accept an upload from a browser -- potentially letting the person who produces the XML file update the DB by themselves. That's a big win if you're not the one making the XML file, which is usually the case in my experience.
"create a python script and install that script as a cron (with DJANGO _SETTINGS _MODULE variable setup before executing the script)?"
First, be sure to declare your Forms in a separate module (e.g. forms.py)
Then, you can write batch loaders that look like this. (We have a LOT of these.)
from myapp.forms import MyObjectLoadForm
from myapp.models import MyObject
import xml.etree.ElementTree as ET
def xmlToDict( element ):
return dict(
field1= element.findtext('tag1'),
field2= element.findtext('tag2'),
)
def loadRow( aDict ):
f= MyObjectLoadForm( aDict )
if f.is_valid():
f.save()
def parseAndLoad( someFile ):
doc= ET.parse( someFile ).getroot()
for tag in doc.getiterator( "someTag" )
loadRow( xmlToDict(tag) )
Note that there is very little unique processing here -- it just uses the same Form and Model as your view functions.
We put these batch scripts in with our Django application, since it depends on the application's models.py and forms.py.
The only "interesting" part is transforming your XML row into a dictionary so that it works seamlessly with Django's forms. Other than that, this command-line program uses all the same Django components as your view.
You'll probably want to add options parsing and logging to make a complete command-line app out of this. You'll also notice that much of the logic is generic -- only the xmlToDict function is truly unique. We call these "Builders" and have a class hierarchy so that our Builders are all polymorphic mappings from our source documents to Python dictionaries.
I'd like to move my models to a separate directory, similar to the way it's done with Rails to cut down on code clutter. Is there any way to do this easily?
Thanks,
Collin
I assume you're using the basic webkit and not Django or something fancy. In that case just create a subdirectory called models. Put any python files you use for your models in here. Create also one blank file in this folder called __init__.py.
Then in your main.py or "controller" or what have you, put:
import models
at the top.
You just created a python package.
Brandon's answer is what I do. Furthermore, I rather like Rails's custom of one model per file. I don't stick to it completely but that is my basic pattern, especially since Python tends to encourage more-but-simpler lines of code than Ruby.
So what I do is I make models a package too:
models/
models/__init__.py
models/user.py
models/item.py
models/blog_post.py
In the main .py files I put my basic class definition, plus perhaps some helper functions (Python's module system makes it much safer to keep quickie helper functions coupled to the class definition). And my __init__.py stitches them all together:
"""The application models"""
from user import User
from item import Item
from blog_post import BlogPost
It's slightly redundant but I have lots of control of the namespace.