I am setting up a Django application with a lot of databases, and some of them are using the same models (they are not replicas). I already configured my routers and everything is working great. The problem appears when making the tests as I want to use factory-boy.
On a different project I could setup the database inside Meta but now, I have to select on which database to create an instance dynamically (if not, I would have to create a DjangoModelFactory for each database, which wouldn't be pretty).
Is there an (easier) way to specify the database dynamically for each creation?
As far as I know factory_boy (version <=2.10.0) doesn't provide anything like that.
Though, your problem is the perfect use case to use a context manager. It will allow you to set the database dynamically wherever you need and only under the desired scope, and also DRY!:
# factoryboy_utils.py
#classmethod
def _get_manager(cls, model_class):
return super(cls, cls)._get_manager(model_class).using(cls.database)
class DBAwareFactory(object):
"""
Context manager to make model factories db aware
Usage:
with DBAwareFactory(PersonFactory, 'db_qa') as personfactory_on_qa:
person_on_qa = personfactory_on_qa()
...
"""
def __init__(self, cls, db):
# Take a copy of the original cls
self.original_cls = cls
# Patch with needed bits for dynamic db support
setattr(cls, 'database', db)
setattr(cls, '_get_manager', _get_manager)
# save the patched class
self.patched_cls = cls
def __enter__(self):
return self.patched_cls
def __exit__(self, type, value, traceback):
return self.original_cls
and then, in your tests, you can do something like:
from factoryboy_utils import DBAwareFactory
class MyTest(TestCase):
def test_mymodel_on_db1(self):
...
with DBAwareFactory(MyModelFactory, 'db1') as MyModelFactoryForDB1:
mymodelinstance_on_db1 = MyModelFactoryForDB1()
# whatever you need with that model instance
...
# something else here
def test_mymodel_on_db2(self):
...
with DBAwareFactory(MyModelFactory, 'db2') as MyModelFactoryForDB2:
mymodelinstance_on_db2 = MyModelFactoryForDB2()
# whatever you need with that model instance
...
# something else here
Related
I have a Python class where the constructor creates a MySQL database connection as follows:
class MySQL:
def __init__(self):
self.client = self.get_client()
def get_client():
client = pymysql.connect(**mysql_credentials)
return client
The problem with this implementation is that the connection never ends. So I want to modify the class to create the database connection in the __enter__ method and close the connection on __exit__ method as follows:
class MySQL:
def __enter__(self):
self.client = self.get_client()
def __exit__(self, exc_type, exc_val, exc_tb):
self.client.close()
def get_client():
client = pymysql.connect(**mysql_credentials)
return client
def execute_query(self, sql_query: str):
with self.client.cursor() as cursor:
cursor.execute(sql_query)
Now the question. How can instantiate MySQL class with __enter__ and __exit__ inside the constructor of another class?
Can't be do like this because it calls MySQL __init__ method and it will not open the connection:
class AnotherClass:
def __init__(self):
self.mysql_cli = MySQL()
def run_etl(self):
self.mysql_cli.execute_query('''SELECT VERSION();''')
Any suggestions?
Many thanks!
There are three different ways for a class to properly use a context manager inside its own code.
One option is for the new class to be a context manager itself, so its users can do the managing of the connection lifetimes through it. This is pretty straight forward, just call the __enter__ and __exit__ methods of your contained context manager from your own versions of those methods. That might look like this:
class AnotherClass:
def __init__(self):
self.db = MySQL()
def __enter__(self):
self.db.__enter__()
return self
def __exit__(self, *exc_args):
return self.db.__exit__(*exc_args)
def do_stuff(self):
# do stuff with the database
The burden of managing the connection is delegated to the user of the class, who can use with statements on their AnotherClass instance:
with AnotherClass() as another:
another.do_stuff()
This approach can get tedious though if you have lots of layers of aggregation, and need all the intermediate containers to become context managers just because they contain one at some much lower level.
Another approach is to make separate database connections for each operation that needs to use the database for something. This lets you use with statements for the connection management, but may require that you do a lot more connecting and reconnecting than you'd like:
class AnotherClass:
def __init__(self):
self.db = MySQL()
def do_stuff(self):
with self.db:
# do the actual stuff here, using the database
With this design, the user of AnotherClass doesn't need to use the context manager protocol because the database connections are only alive during the runtime of the methods you call.
another = AnotherClass()
another.do_stuff() # creates and destroys the db connection internally
The third option is to say that AnotherClass should not be involved with the connection management of the MySQL class it uses. That's the responsibility of some other part of the code, and we don't even need to know about it here. For this approach, you probably want the MySQL instance to be created elsewhere (wherever it is being managed) and have it be passed in as an argument to the AnotherClass constructor:
class AnotherClass:
def __init__(self, db):
self.db = db
def do_stuff(self):
# do stuff with the database
The caller would do something like this:
db = MySQL()
with db:
another = AnotherClass(db)
another.do_stuff()
This has the advantage that the db object passed in to AnotherClass can be whatever type you want, as long as it has the APIs that AnotherClass expects. If you need to change your database from MySQL to Postgres, you don't need to change AnotherClass, only the higher level code.
I want to have my database implementation in a separate module or class. But I am struggling with a few details. A simple example:
from peewee import *
db = SqliteDatabase(':memory:')
class BaseModel(Model):
class Meta:
database = db
class User(BaseModel):
name = CharField()
db.connect()
db.create_tables([User,])
db.commit()
#db.atomic()
def add_user(name):
User.create(name=name).save()
#db.atomic()
def get_user(name):
return User.get(User.name == name)
So far this is working fine. I can implement my interface to the database here and import this as a module.
Now I want to be able to choose the database file at runtime. So I need a way to define the Model classes without defining SqliteDatabase('somefile') before. I tried to encapsulate everything in a new Database class, which I can later import and create an instance from:
from peewee import *
class Database:
def __init__(self, dbfile):
self.db = SqliteDatabase(dbfile)
class BaseModel(Model):
class Meta:
database = self.db
class User(BaseModel):
name = CharField()
self.User = User
self.db.connect()
self.db.create_tables([User,])
self.db.commit()
#self.db.atomic() # Error as self is not known on this level
def add_user(self, name):
self.User.create(name=name).save()
#self.db.atomic() # Error as self is not known on this level
def get_user(self, name):
return self.User.get(self.User.name == name)
Now I can call for example database = Database('database.db') or choose any other file name. I can even use multiple database instance in the same program, each with its own file.
However, there are two problems with this approach:
I still need to specify the database driver (SqliteDatabase) before defining the Model classes. To solve this I define the Model classes within the __init__() method, and then create an alias to with self.User = User. I don't really like this approach (it just doesn't feel like neat code), but at least it works.
I cannot use the #db.atomic() decorator since self is not known at class level, I would an instance here.
So this class approach does not seem to work very well. Is there some better way to define the Model classes without having to choose where you want to store your database first?
If you need to change database driver at the runtime, then Proxy is a way to go
# database.py
import peewee as pw
proxy = pw.Proxy()
class BaseModel(pw.Model):
class Meta:
database = proxy
class User(BaseModel):
name = pw.CharField()
def add_user(name):
with proxy.atomic() as txn:
User.create(name=name).save()
def get_user(name):
with proxy.atomic() as txn:
return User.get(User.name == name)
From now on each time you load the module, it won't need a database to be initialized. Instead, you can initialize it at the runtime and switch between multiple as follows
# main.py
import peewee as pw
import database as db
sqlite_1 = pw.SqliteDatabase('sqlite_1.db')
sqlite_2 = pw.PostgresqlDatabase('sqlite_2.db')
db.proxy.initialize(sqlite_1)
sqlite_1.create_tables([db.User], safe=True)
db.add_user(name="Tom")
db.proxy.initialize(sqlite_2)
sqlite_2.create_tables([db.User], safe=True)
db.add_user(name="Jerry")
But if the connection is the only thing that matters, then init() method will be enough.
Now I want to be able to choose the database file at runtime. So I
need a way to define the Model classes without defining
SqliteDatabase('somefile') before. I tried to encapsulate everything
in a new Database class, which I can later import and create an
instance from
Peewee uses the meta class to define the name of the table (Model.Meta.db_table) and database( Model.Meta.database)
set these attribute before calling a Model specific code ( either to create a table or to DML statements)
'Allow to define database dynamically'
Question: I cannot use the #db.atomic() decorator since self is not known at class level
Do it, as you do it with self.User.
I wonder about atomic() instead of atomic, but you tell is working fine.
class Database:
def __init__(self, dbfile):
self.db = SqliteDatabase(dbfile)
...
#self.db.atomic()
def __add_user(self, name):
self.User.create(name=name).save()
self.add_user = __add_user
#self.db.atomic()
def __get_user(self, name):
return self.User.get(self.User.name == name)
self.get_user = __get_user
Related: Define models separately from Database() initialization
I am writing some python gui app (PySide to be exact) and I am using my own class to handling DB. What's the correct way to use models? Currently I have something like this:
class DB(object):
def __init__(self, dbfile):
some db connect work
def updateEntry(entryid):
some update query etc
def getEntry(entryid):
fetching entry from db
def createEntry(entryvalue):
insert entry
class EntryModel(object):
def __init__(db,entryid=None,entryvalue=None):
self.db=db
self.entryid=entryid
self.entryvalue=entryvalue
if entryid is None:
self.db.createEntry(self.entryvalue)
elif self.entryvalue is None:
self.db.getEntry(self.entryid)
def some_func(self):
some other work
And it's working just fine... But I have a feeling that something is wrong here... I mean, I have to pass DB to each model, I don't think that's correct way. How to do it in proper way without using frameworks like SQLAlchemy and so on?
You can at least create a base class, let's called it Model (like in Django, or Base as it is called in SQLAlchemy)
We'll keep a reference to the db object as a class attribute so it is the same for all instances, and inherited so you don't have to pass it around
class Model(object):
db = None # This var is a class attribute
#classmethod
def init_db(cls):
cls.db = your_code_to_create_db()
class Entry(Model):
def __init__(self, entry_id, entry_value):
self.entry_id = entry_id
self.entry_value = entry_value
super(Entry, self).__init__()
def save(self):
# Use db here
self.db
# To use
Model.init_db() # Inits the one db var for the class
entry = Entry(...)
entry.save()
I hope you see the idea and adapt it to your needs!
i'm trying to build sort of a "mini django model" for working with Django and MongoDB without using the norel Django's dist (i don't need ORM access for these...).
So, what i'm trying to do is to mimic the standart behavior or "implementation" of default models of django... that's what i've got so far:
File "models.py" (the base)
from django.conf import settings
import pymongo
class Model(object):
#classmethod
def db(cls):
db = pymongo.Connection(settings.MONGODB_CONF['host'], settings.MONGODB_CONF['port'])
#classmethod
class objects(object):
#classmethod
def all(cls):
db = Model.db() #Not using yet... not even sure if that's the best way to do it
print Model.collection
File "mongomodels.py" (the implementation)
from mongodb import models
class ModelTest1(models.Model):
database = 'mymongodb'
collection = 'mymongocollection1'
class ModelTest2(models.Model):
database = 'mymongodb'
collection = 'mymongocollection2'
File "views.py" (the view)
from mongomodels import ModelTest1, ModelTest2
print ModelTest1.objects.all() #Should print 'mymongocollection1'
print ModelTest2.objects.all() #Should print 'mymongocollection2'
The problem is that it's not accessing the variables from ModelTest1, but from the original Model... what's wrong??
You must give objects some sort of link to class that contains it. Currently, you are just hard-coding it to use Model()s atttributes. Because you are not instantiating these classes, you will either have to use either a decorator or a metaclass to create the object class for you in each subclass of Model().
I'm using Django and want to be able to store classes in a database for things like forms and models so that I can easily make them creatable through a user interface since they are just stored in the database as opposed to a regular file. I don't really know a whole lot about this and am not sure if this is a situation where I need to use exec in python or if there is some other way. My searches on this aren't turning up much of anything.
Basically, it would just be where I do a database call and get the contents of a class, then I want to instantiate it. Any advice is appreciated on how to best do this sort of thing.
EDIT: In response to the idea of a malicious __init__ in the class, these are only for things like forms or models where it is tightly controlled through validation what goes in the class, there would never be an __init__ in the class and it would be basically impossible, since I would validate everything server side, to put anything malicious in the class.
Do not store code in the database!!!
Imagine a class with a malicious __init__ method finding it's way in your "class repository" in the database. This means whoever has write access to those database tables has the ability to read any file from your web server and even nuke it's file system, since they have the ability to execute any python code on it.
Don't store the class itself, store the import path as a string in the database (e.g. 'django.forms.CharField')
I started doing this same thing for another project, and saved off the code in my local repository. To address the security concerns I was going to add an argument to the field constructor of allowed base classes. If you do implement this, let me know, I'd love to have it.
helpers.py
def get_class_from_concrete_classpath(class_path):
# Unicode will throw errors in the __import__ (at least in 2.6)
class_path = str(class_path)
mod_list = class_path.split('.')
module_path = '.'.join(mod_list[:-1])
class_name = mod_list[-1]
base_mod = __import__(module_path, fromlist=[class_name,])
return getattr(base_mod, class_name)
def get_concrete_name_of_class(klass):
"""Given a class return the concrete name of the class.
klass - The reference to the class we're interested in.
Raises a `TypeError` if klass is not a class.
"""
if not isinstance(klass, (type, ClassType)):
raise TypeError('The klass argument must be a class. Got type %s; %s' %
(type(klass), klass))
return '%s.%s' % (klass.__module__, klass.__name__)
fields.py
class ClassFormField(forms.Field):
def to_python(self, value):
return get_concrete_name_of_class(value)
class ClassField(models.CharField):
__metaclass__ = models.SubfieldBase
"""Field used for storing a class as a string for later retrieval"""
MAX_LENGTH = 255
default_error_messages = {
'invalid': _(u'Enter a valid class variable.'),
}
def __init__(self, *args, **kwargs):
kwargs['max_length'] = kwargs.get('max_length', ClassField.MAX_LENGTH)
super(ClassField, self).__init__(*args, **kwargs)
def get_prep_value(self, value):
if isinstance(value, (basestring, NoneType)):
return value
return get_concrete_name_of_class(value)
def to_python(self, value):
if isinstance(value, basestring):
return get_class_from_concrete_classpath(value)
return value
def formfield(self, **kwargs):
defaults = {'form_class' : ClassFormField}
defaults.update(kwargs)
return super(ClassField, self).formfield(**defaults)