Develop Reference

Sharing PonyORM's db session across different python module

I initially started a small python project (Python, Tkinter amd PonyORM) and became larger that is why I decided to divide the code (used to be single file only) to several modules (e.g. main, form1, entity, database). Main acting as the main controller, form1 as an example can contain a tkinter Frame which can be used as an interface where the user can input data, entity contains the db.Enttiy mappings and database for the pony.Database instance along with its connection details. I think problem is that during import, I'm getting this error "pony.orm.core.ERDiagramError: Cannot define entity 'EmpInfo': database mapping has already been generated". Can you point me to any existing code how should be done.

Probably you import your modules in a wrong order. Any module which contains entity definitions should be imported before db.generate_mapping() call.
I think you should call db.generate_mapping() right before entering tk.mainloop() when all imports are already done.

A good approach to avoid this is rather than having your db.generate_mapping() call happening at a module's top-level code, have a function that a module exports that calls db.generate_mapping() after all other modules have been imported.
The pattern I use is to put all of my db.Entity subclasses into a single module named model, and then at the bottom of model.py is:
def setup():
""" Set up the database """
db.bind(**database_config, create_db=True)
db.generate_mapping(create_tables=True)
This function is called by my application's own startup (which is also responsible for setting up database_config). This way the correct import and setup order can be guaranteed.
The db object itself is also owned by this model module; if I need to use it somewhere else I import model and use model.db.
If you want to further separate things out (with different model classes living in different modules) you can have a module that owns db, then your separate model modules, and then a third module that imports db and the models and provides the setup function. For example, your directory structure could look like this:
model/
__init__.py -- imports all of the model sub-modules and provides a setup function
db.py -- provides the db object itself and any common entities objects that everyone else needs
form1.py, form2.py, etc. -- imports db and uses its database object to define the entities
Then your main app can do something like:
import model
model.setup()

Pony ORM define Entity subclass before creating database

I'm creating a library that contains Bot class. I want it to be able to store messages logs and some other info in the sql database with the help of the pony orm. Idea is for the bot to accept the path to the database and create it or connect to it if it already exists.
Sadly, it seems that with pony I can only define classes after creation of the database object, needing to inherit from database.Entity class to bind my class to the database.
Obvious solution is to define all classes in the Bot class constructor after the creation of the database, but it seems quite ugly in terms of structure, because I plan those classes to be quite large, and wanted to store them in separate files.
Other hypothetical way is to do following (but I don't know whether that functionality is supported):
from pony import orm
class Message(orm.Entity):
text = orm.Required(unicode)
class Database(orm.Database):
def __init__(self, path):
# super(..).__init__(..)
# bind Message to self
class Bot(object):
def __init__(self, path):
self.database = Database(path)
Yet another possible hypothetical way would be if I could inherit from Database.Entity.
Any ideas? Maybe I can achieve this with another ORMs like SQLAlchemy?

How does Flask-SQLAlchemy create_all discover the models to create?

Flask-SQLAlchemy's db.create_all() method creates each table corresponding to my defined models. I never instantiate or register instances of the models. They're just class definitions that inherit from db.Model. How does it know which models I have defined?

Flask-SQLAlchemy does nothing special, it's all a standard part of SQLAlchemy.
Calling db.create_all eventually calls db.Model.metadata.create_all. Tables are associated with a MetaData instance as they are defined. The exact mechanism is very circuitous within SQLAlchemy, as there is a lot of behind the scenes bookkeeping going on, so I've greatly simplified the explanation.
db.Model is a declarative base class, which has some special metaclass behavior. When it is defined, it creates a MetaData instance internally to store the tables it generates for the models. When you subclass db.Model, its metaclass behavior records the class in db.Model._decl_class_registry as well as the table in db.Model.metadata.
Classes are only defined when the modules containing them are imported. If you have a module my_models written somewhere, but it is never imported, its code never executes so the models are never registered.
This may be where some confusion about how SQLAlchemy detects the models comes from. No modules are "scanned" for subclasses, db.Model.__subclasses__ is not used, but importing the modules somewhere is required for the code to execute.
Module containing models is imported and executed.
Model class definition is executed, subclasses db.Model
Model's table is registered with db.Model.metadata

You need to call create_all() in the same module as all the models are in. If they are in separate modules, you need to import them all before calling create_all(). SQLAlchemy looks at what models have been subclassed from db.Model, and it will only see models that have been imported. It will create the corresponding table for each model. Also explained here.

Where to put helper functions when not using models and an ORM?

I have to use PyMongo in a Django project, althought I have always used either Django's ORM or Mongoengine. With the old setup, every model had its own method which did some actual work. This time, having to use PyMongo, I don't have the models, and I don't know where to put the helper functions.
Currently I'm writing them inside the models.py file, but I'm not even halfway done that already I feel that this way I'm cluttering the models file (too many little functions).
Do you know a better solution to the problem?

You are not using django ORM, you don't have any models - you should leave models.py empty.
Here's one option you may consider to use.
Create a package called, for example, db. Divide your helper functions into separate scripts according to the entity/collection they are working with. E.g.:
db/
__init__.py
user.py
customer.py
role.py
Other generic helper function you can leave in __init__.py or create a separate module for them.
Additionally, while splitting your current models.py file into these python modules you may notice that most of your helper functions are similar - in that case, think about going with OOP approach, create a python class for each collection, define generic methods etc (and it will look like inventing your own ORM layer).
Hope that helps.

You can organize the helper methods as classmethods of "model"-like classes. After re-organization, the models file may look something like this:
# models.py
class BaseModel(object):
connection = Connection()
db = connection['test']
collection = None # override in subclasss
#classmethod
def get(cls, object_id):
return cls.collection.findOne({"_id": object_id})
class Person(BaseModel):
collection = BaseModel.db['person']
#classmethod
def get_old_people(cls):
return cls.collection.find(...)

MVC model structure in Python

I'm having problems structuring classes in the Model part of an MVC pattern in my Python app. No matter how I turn things, I keep running into circular imports. Here's what I have:
Model/__init__p.y
should hold all Model class names so
I can do a "from Model import User"
e.g. from a Controller or a unit
test case
Model/Database.py
holds Database class
needs to import all Model classes to do ORM
initialization should be performed on first module import, i.e. no extra init calls or instantiations (all methods on Database class are #classmethods)
Model/User.py
contains User model class
needs access to Database class to do queries
should inherit from base class common to all Model classes to share functionality (database persistency methods, parameter validation code etc.)
I have yet to see a real world Python app employing MVC, so my approach is probably un-Pythonic (and possibly a language-agnostic mess on top of that...) - any suggestions on how to solve this?
Thanks, Simon

There is an inconsistency in your specification. You say Database.py needs to import all Model classes to do ORM but then you say the User class need access to the Database to do queries.
Think of these as layers of an API. The Database class provides an API (maybe object-oriented) to some physical persistence layer (such as DB-API 2.0). The Model classes, like User, use the Database layer to load and save their state. There is no reason for the Database.py class to import all the Model classes, and in fact you wouldn't want that because you'd have to modify Database.py each time you created a new Model class - which is a code smell.

Generally, we put it all in one file. This isn't Java or C++.
Start with a single file until you get some more experience with Python. Unless your files are gargantuan, it will work fine.
For example, Django encourages this style, so copy their formula for success. One module for the model. A module for each application; each application imports a common model.
Your Database and superclass stuff can be in your __init__.py file, since it applies to the entire package. That may reduce some of the circularity.

I think you have one issue that should be straightened. Circular references often result from a failure to achieve separation of concerns. In my opinion, the database and model modules shouldn't know much about each other, working against an API instead. In this case the database shouldn't directly reference any specific model classes but instead provide the functionality the model classes will need to function. The model in turn, should get a database reference (injected or requested) that it would use to query and persist itself.