Develop Reference

Python - Inherit from a class in a submodule

In order to get a code easier to maintain, I wanted to split classes in different files, classified per role.
Everyting is placed into the same directory, with a init.py file to obtain a submodule.
Something like that :
Resources
|- __init__.py
|- main_code.py
|- resources.py
|--- classes
|- __init__.py
|- solutions.py
|- tests.py
But as in some classes of the tests.py, I inherit from other classes located in solutions.py, I've imported the solutions.py using :
from . import solutions
Here is an example of my code in tests.py :
from . import solutions
class snapshot(solutions.device):
def __init__(self, d):
solutions.device.__init__(self, d)
self.ip = d
But doing that, I've got the following error:
AttributeError: module 'solutions' has no attribute 'device'
I've also tried with :
from resources.classes import solutions
But I've got the same result.
Thanks for your help,
EDIT
Here is the solutions.py :
class device:
def __init__(self, d: str, **kwargs):
self.info = d
username = kwargs.get("username", None)
password = kwargs.get("password", None)
action = kwargs.get("action", None)
vault = kwargs.get("vault", None)
self.init_connection(username, password, vault)
<--- ommitted for visibility --->
When everything was located into the same classes.py file, it worked perfectly.

Maybe you can do
from .solutions import device
in tests.py

Python's importlib and inspect for static class members

In a long-running app I need to dynamically modify static class members based on path to the class' module and the class name.
Ex. I have a class pack1.mod1.Person and by definition I know it has a age property. So utilizing the importlib and inspect I try to load the class using the module path and class name and update the age property. It all seems fine until I read the the age property from my naturally imported Person class and find it's not updated.
Here are some more details:
.
├── app.py
└── pack1
├── __init__.py
└── mod1.py
mod1.py
class Person:
age = 42
app.py
import inspect
import os
from importlib import util
from pack1.mod1 import Person
if __name__ == '__main__':
Person.age = 3
print(Person.age) # => 3
spec = util.spec_from_file_location('pack1.mod1', os.path.join('pack1', 'mod1.py'))
module = util.module_from_spec(spec)
spec.loader.exec_module(module)
members = inspect.getmembers(module)
for x, member in inspect.getmembers(module, lambda i: inspect.isclass(i) and i.__name__ == Person.__name__):
print('Person:', Person.age) # => Person: 3
print('Person from inspect:', member.age) # => Person from inspect: 42
Person.age = 11
member.age = 66
print('Person:', Person.age) # => Person: 11
print('Person from inspect:', member.age) # => Person from inspect: 66
In the app.py I would expect member and Person to be the same thing but as the example shows they aren't.
What am I missing and how to achieve such an update on the static members of a class?

Python has no way of knowing that the regularly imported module and the manually module are "the same": Using util.spec_from_file_location up to spec.loader.exec_module side-steps Python's module registry and explicitly creates a new instance of the module.
Instead, use the native operations of the interpreter (import, ...) or their programmatic equivalents (importlib.load_module, ...)
If the module/class are well-known, one can import it regularly and directly inspect it.
import pack1.mod1
pack1.mod1.Person.age = 66
If module and class are only known by name, one can look them up from the existing modules.
import importlib
module_name, qualname, attribute, value = 'pack1.mod1', 'Person', 'age', 66
obj = importlib.import_module(module_name) # same as `import {module_name}`
for part in qualname.split('.'):
obj = getattr(obj, part) # same as `{obj}.{part}
setattr(obj, attribute, value) # same as `{obj}.{attribute} = {value}`

Python AttributeError: 'Class' object has no attribute 'name' - SetUpClass

I'm writing tests for a model in a Django app, but cannot get them to run. I've tried all I can think of to solve and searched but cannot find the solution.
The error I receive when I run the test is AttributeError: 'UserModelTest' object has no attribute 'firstUser', which would suggest I haven't defined firstUser correctly.
Here are the relevant bits of code.
tests.py
from django.test import TestCase
from .models import ContactForm
from datetime import datetime, timedelta
class UserModelTest(TestCase):
#classmethod
def setUpClass(cls):
cls.firstUser = ContactForm(
email="first#user.com",
name="first",
timestamp=datetime.today() + timedelta(days=2)
)
cls.firstUser.save()
def test_contactform_str_returns_email(self):
self.assertEqual("first#user.com", str(self.firstUser))
models.py
from django.db import models
import datetime
class ContactForm(models.Model):
name = models.CharField(max_length=150)
email = models.EmailField(max_length=250)
timestamp = models.DateTimeField(auto_now_add=True)
def __str__(self):
return self.email
class Meta:
ordering = ['-timestamp']
I'm using Python version 3.5 and Django version 1.10.

setUpClass has been deprecated for Django (I believe since 1.8). In order to set up the class now, I needed to use setUpTestData. The remaining code still works. setUpTestData is then run only once before all test methods in the class (not once before each test).
i.e. the code should be:
class UserModelTest(TestCase):
#classmethod
def setUpTestData(cls):
cls.firstUser = ContactForm(
email="first#user.com",
name="first",
timestamp=datetime.today() + timedelta(days=2)
)
cls.firstUser.save()
def test_contactform_str_returns_email(self):
self.assertEqual("first#user.com", str(self.firstUser))
More information on setUpTestData can be found here.

Python metaClass and import *

Main Goal: Automatically register classes (by a string) in a factory to be created dynamically at run time using that string, classes can be in their own file and not grouped in one file.
I have couple of classes which all inherit from the same base class and they define a string as their type.
A user wants to get an instance of one of these classes but only knows the type at run time.
Therefore I have a factory to create an instance given a type.
I didn't want to hard code an "if then statements" so I have a meta class to register all the sub classes of the base class:
class MetaRegister(type):
# we use __init__ rather than __new__ here because we want
# to modify attributes of the class *after* they have been
# created
def __init__(cls, name, bases, dct):
if not hasattr(cls, 'registry'):
# this is the base class. Create an empty registry
cls.registry = {}
else:
# this is a derived class. Add cls to the registry
interface_id = cls().get_model_type()
cls.registry[interface_id] = cls
super(MetaRegister, cls).__init__(name, bases, dct)
The problem is that for this to work the factory has to import all the subclass (So the meta class runs).
To fix this you can use from X import *
But for this to work you need to define an __all__ var in the __init__.py file of the package to include all the sub classes.
I don't want to hard code the sub classes because it beats the purpose of using the meta class.
I can go over the file in the package using:
import glob
from os.path import dirname, basename, isfile
modules = glob.glob(dirname(__file__) + "/*.py")
__all__ = [basename(f)[:-3] for f in modules if isfile(f)]
Which works great, but the project needs to compile to a single .so file, which nullifies the use of the file system.
So how could I achieve my main goal of creating instances at run time without hard codding the type?
Is there a way to populate an __all__ var at run time without touching the filesystem?
In Java I'd probably decorate the class with an annotation and then get all the classes with that annotation at run time, is there something similar on python?
I know there are decorators in python but I'm not sure I can use them in this way.
Edit 1:
Each subclass must be in a file:
- Models
-- __init__.py
-- ModelFactory.py
-- Regression
--- __init__.py
--- Base.py
--- Subclass1.py
--- Subclass2ExtendsSubclass1.py
Edit 2: Some code to Illustrate the problem:
+ main.py
|__ Models
|__ __init__.py
|__ ModelFactory.py
|__ Regression
|__ init__.py
|__ Base.py
|__ SubClass.py
|__ ModelRegister.py
main.py
from models.ModelFactory import ModelFactory
if __name__ == '__main__':
ModelFactory()
ModelFactory.py
from models.regression.Base import registry
import models.regression
class ModelFactory(object):
def get(self, some_type):
return registry[some_type]
ModelRegister.py
class ModelRegister(type):
# we use __init__ rather than __new__ here because we want
# to modify attributes of the class *after* they have been
# created
def __init__(cls, name, bases, dct):
print cls.__name__
if not hasattr(cls, 'registry'):
# this is the base class. Create an empty registry
cls.registry = {}
else:
# this is a derived class. Add cls to the registry
interface_id = cls().get_model_type()
cls.registry[interface_id] = cls
super(ModelRegister, cls).__init__(name, bases, dct)
Base.py
from models.regression.ModelRegister import ModelRegister
class Base(object):
__metaclass__ = ModelRegister
def get_type(self):
return "BASE"
SubClass.py
from models.regression.Base import Base
class SubClass(Base):
def get_type(self):
return "SUB_CLASS"
Running it you can see only "Base" it printed.
Using a decorator gives the same results.

A simple way to register classes as runtime is to use decorators:
registry = {}
def register(cls):
registry[cls.__name__] = cls
return cls
#register
class Foo(object):
pass
#register
class Bar(object):
pass
This will work if all of your classes are defined in the same module, and if that module is imported at runtime. Your situation, however, complicates things. First, you want to define your classes in different modules. This means that we must be able to dynamically determine which modules exist within our package at runtime. This would be straightforward using Python's pkgutil module, however, you also state that you are using Nuitka to compile your package into an extension module. pkgutil doesn't work with such extension modules.
I cannot find any documented way of determining the modules contained within an Nuitka extension module from within Python. If one does exist, the decorator approach above would work after dynamically importing each submodule.
As it is, I believe the most straightforward solution is to write a script to generate an __init__.py before compiling. Suppose we have the following package structure:
.
├── __init__.py
├── plugins
│   ├── alpha.py
│   └── beta.py
└── register.py
The "plugins" are contained within the plugins directory. The contents of the files are:
# register.py
# -----------
registry = {}
def register(cls):
registry[cls.__name__] = cls
return cls
# __init__.py
# -----------
from . import plugins
from . import register
# ./plugins/alpha.py
# ------------------
from ..register import register
#register
class Alpha(object):
pass
# ./plugins/beta.py
# ------------------
from ..register import register
#register
class Beta(object):
pass
As it stands, importing the package above will not result in any of the classes being registered. This is because the class definitions are never run, since the modules containing them are never imported. The remedy is to automatically generate an __init__.py for the plugins folder. Below is a script which does exactly this -- this script can be made part of your compilation process.
import pathlib
root = pathlib.Path('./mypkg/plugins')
exclude = {'__init__.py'}
def gen_modules(root):
for entry in root.iterdir():
if entry.suffix == '.py' and entry.name not in exclude:
yield entry.stem
with (root / '__init__.py').open('w') as fh:
for module in gen_modules(root):
fh.write('from . import %s\n' % module)
Placing this script one directory above your package root (assuming your package is called mypkg) and running it yields:
from . import alpha
from . import beta
Now for the test: we compile the package:
nuitka --module mypkg --recurse-to=mypkg
and try importing it, checking to see if all of the classes were properly registered:
>>> import mypkg
>>> mypkg.register.registry
{'Beta': <class 'mypkg.plugins.beta.Beta'>,
'Alpha': <class 'mypkg.plugins.alpha.Alpha'>}
Note that the same approach will work with using metaclasses to register the plugin classes, I simply preferred to use decorators here.

If the reflected classes are using your metaclass, you don't need to use from X import * to get them registered. Only import X should be enough. As soon as the module containing the classes is imported, the classes will be created and available in your metaclass registry.

I would do this with dynamic imports.
models/regression/base.py:
class Base(object):
def get_type(self):
return "BASE"
models/regression/subclass.py:
from models.regression.base import Base
class SubClass(Base):
def get_type(self):
return "SUB_CLASS"
__myclass__ = SubClass
loader.py:
from importlib import import_module
class_name = "subclass"
module = import_module("models.regression.%s" % class_name)
model = module.__myclass__()
print(model.get_type())
And empty __init__.py files in models/ and models/regression/
With:
nuitka --recurse-none --recurse-directory models --module loader.py
The resulting loader.so contains all the modules under the models/ subdirectory.

How to create table during Django tests with managed = False

I have a model with managed = False.
class SampleModel(models.Model):
apple = models.CharField(max_length=30)
orange = models.CharField(max_length=30)
class Meta:
managed = False
I have a unit test which creates a SampleModel, however when I run the test I get:
DatabaseError: no such table: SAMPLE_SAMPLE_MODEL
The django docs - https://docs.djangoproject.com/en/dev/ref/models/options/#managed documents the following:
For tests involving models with managed=False, it's up to you to
ensure the correct tables are created as part of the test setup.
How can I actually "create" the tables during the test setup? Or alternatively, how can I make it so that when I am running tests, this model has "managed = True" for the duration of the test?
In the real application, this model is actually backed by a view in the database. However for the during of the test, I would like to treat this as a table and be able to insert test data in there.

You can use SchemaEditor in TestCase.setUp method to explicitly create models with managed = False.
# models.py
from django.db import models
class Unmanaged(models.Model):
foo = models.TextField()
class Meta:
# This model is not managed by Django
managed = False
db_table = 'unmanaged_table'
And in your tests:
# tests.py
from django.db import connection
from django.test import TestCase
from myapp.models import Unmanaged
class ModelsTestCase(TestCase):
def setUp(self):
super().setUp()
with connection.schema_editor() as schema_editor:
schema_editor.create_model(Unmanaged)
if (
Unmanaged._meta.db_table
not in connection.introspection.table_names()
):
raise ValueError(
"Table `{table_name}` is missing in test database.".format(
table_name=Unmanaged._meta.db_table
)
)
def tearDown(self):
super().tearDown()
with connection.schema_editor() as schema_editor:
schema_editor.delete_model(Unmanaged)
def test_unmanaged_model(self):
with self.assertNumQueries(num=3):
self.assertEqual(0, Unmanaged.objects.all().count())
Unmanaged.objects.create()
self.assertEqual(1, Unmanaged.objects.all().count())

Check out this blog post: http://www.caktusgroup.com/blog/2010/09/24/simplifying-the-testing-of-unmanaged-database-models-in-django/ It describes in detail the creation of a test runner for unmanaged models.
from django.test.simple import DjangoTestSuiteRunner
class ManagedModelTestRunner(DjangoTestSuiteRunner):
"""
Test runner that automatically makes all unmanaged models in your Django
project managed for the duration of the test run, so that one doesn't need
to execute the SQL manually to create them.
"""
def setup_test_environment(self, *args, **kwargs):
from django.db.models.loading import get_models
self.unmanaged_models = [m for m in get_models()
if not m._meta.managed]
for m in self.unmanaged_models:
m._meta.managed = True
super(ManagedModelTestRunner, self).setup_test_environment(*args,
**kwargs)
def teardown_test_environment(self, *args, **kwargs):
super(ManagedModelTestRunner, self).teardown_test_environment(*args,
**kwargs)
# reset unmanaged models
for m in self.unmanaged_models:
m._meta.managed = False

Execute raw SQL to create the table in the test setup:
from django.db import connection
class MyTest(unittest.TestCase):
def setUp(self):
connection.cursor().execute("CREATE TABLE ...")
def tearDown(self):
connection.cursor().execute("DROP TABLE ...")

Nice plug and play solution. Just paste this before your test class definition. (note: django 1.8 used)
from django.db.models.loading import get_models
def change_managed_settings_just_for_tests():
"""django model managed bit needs to be switched for tests."""
unmanaged_models = [m for m in get_models() if not m._meta.managed]
for m in unmanaged_models:
m._meta.managed = True
change_managed_settings_just_for_tests()

A quick fix if you don't have many unmanaged tables:
First add a new variable to the settings.
# settings.py
import sys
UNDER_TEST = (len(sys.argv) > 1 and sys.argv[1] == 'test')
then in the models
# models.py
from django.conf import settings
class SampleModel(models.Model):
apple = models.CharField(max_length=30)
orange = models.CharField(max_length=30)
class Meta:
managed = getattr(settings, 'UNDER_TEST', False)

Create your own test runner using this:
from django.test.simple import DjangoTestSuiteRunner
class NoDbTestRunner(DjangoTestSuiteRunner):
""" A test runner to test without database creation """
def setup_databases(self, **kwargs):
""" Override the database creation defined in parent class """
#set manage=True for that specific database on here
Then on your settings add this class to TEST_RUNNER.

Just to add :django.db.models.loading.get_models will be removed in Django 1.9 (see https://github.com/BertrandBordage/django-cachalot/issues/33).
Below is an updated one for Django 1.10:
class UnManagedModelTestRunner(DiscoverRunner):
'''
Test runner that automatically makes all unmanaged models in your Django
project managed for the duration of the test run.
Many thanks to the Caktus Group
'''
def setup_test_environment(self, *args, **kwargs):
from django.apps import apps
self.unmanaged_models = [m for m in apps.get_models() if not m._meta.managed]
for m in self.unmanaged_models:
m._meta.managed = True
super(UnManagedModelTestRunner, self).setup_test_environment(*args, **kwargs)
def teardown_test_environment(self, *args, **kwargs):
super(UnManagedModelTestRunner, self).teardown_test_environment(*args, **kwargs)
# reset unmanaged models
for m in self.unmanaged_models:
m._meta.managed = False
Note you also need to take care migrations(see Testing django application with several legacy databases)
MIGRATION_MODULES = {
'news': 'news.test_migrations',
'economist': 'economist.test_migrations'
}

Using pytest and pytest-django
To make this work (has been tested with django 3.0.2, pytest 5.3.5 and pytest-django 3.8.0):
Run your pytest with the additional argument --no-migrations.
Put the following code in your conftest.py. This has an unfortunate amount of copypasta, but I could not figure out how to first make my models unmanaged and then call the original django_db_setup. The issue of not being able to call pytest fixtures directly is discussed here: https://github.com/pytest-dev/pytest/issues/3950
conftest.py
# example file
import pytest
from pytest_django.fixtures import _disable_native_migrations
#pytest.fixture(scope="session")
def django_db_setup(
request,
django_test_environment,
django_db_blocker,
django_db_use_migrations,
django_db_keepdb,
django_db_createdb,
django_db_modify_db_settings,
):
# make unmanaged models managed
from django.apps import apps
unmanaged_models = []
for app in apps.get_app_configs():
unmanaged_models = [m for m in app.get_models()
if not m._meta.managed]
for m in unmanaged_models:
m._meta.managed = True
# copypasta fixture code
"""Top level fixture to ensure test databases are available"""
from pytest_django.compat import setup_databases, teardown_databases
setup_databases_args = {}
if not django_db_use_migrations:
_disable_native_migrations()
if django_db_keepdb and not django_db_createdb:
setup_databases_args["keepdb"] = True
with django_db_blocker.unblock():
db_cfg = setup_databases(
verbosity=request.config.option.verbose,
interactive=False,
**setup_databases_args
)
def teardown_database():
with django_db_blocker.unblock():
try:
teardown_databases(db_cfg, verbosity=request.config.option.verbose)
except Exception as exc:
request.node.warn(
pytest.PytestWarning(
"Error when trying to teardown test databases: %r" % exc
)
)
if not django_db_keepdb:
request.addfinalizer(teardown_database)

After spending a few hours testing and researching ways to test my django unmanaged models, I finally came up with a solution that worked for me.
My implementation is in this below snippet. It's working great with local tests using db.sqlite3.
# Example classes Implementation
from django.db import models, connection
from django.db.models.base import ModelBase as DjangoModelBase
from django.db.utils import OperationalError
from django.test import TransactionTestCase
class AbstractModel(models.Model):
name = models.TextField(db_column="FIELD_NAME", blank=True, null=True)
class Meta:
managed = False
abstract = True
class TestModel(AbstractModel):
test_field = models.TextField(db_column="TEST_FIELD", blank=True, null=True)
def test_method(self):
print("just testing")
class Meta(AbstractModel.Meta):
db_table = "MY_UNMANAGED_TABLE_NAME"
# My Custom Django TestCases Implementation for my tests
def create_database(model):
with connection.schema_editor() as schema_editor:
try:
schema_editor.create_model(model)
except OperationalError:
pass
def drop_database(model):
with connection.schema_editor() as schema_editor:
try:
schema_editor.delete_model(model)
except OperationalError:
pass
class BaseModelTestCase(TransactionTestCase):
"""Custom TestCase for testing models not managed by django."""
Model = DjangoModelBase
def setUp(self):
super().setUp()
create_database(self.Model)
def tearDown(self):
super().tearDown()
drop_database(self.Model)
class AbstractBaseModelTestCase(TransactionTestCase):
"""Custom TestCase for testing abstract django models."""
Model = DjangoModelBase
def setUp(self):
# this is necessary for testing an abstract class
self.Model = DjangoModelBase(
"__TestModel__" + self.Model.__name__,
(self.Model,),
{"__module__": self.Model.__module__},
)
create_database(self.Model)
def tearDown(self):
drop_database(self.Model)
# Example of usage
class TestModelTestCase(BaseModelTestCase):
Model = TestModel
def setUp(self):
super().setUp()
self.instance = TestModel.objects.create()
self.assertIsInstance(self.instance, TestModel)
class AbstractModelTestCase(AbstractBaseModelTestCase):
Model = AbstractModel
def setUp(self):
super().setUp()
self.instance = AbstractModel.objects.create()
self.assertIsInstance(self.instance, AbstractModel)
p.s. I am not using any Django migrations as soon as my models are managed=False. So this was not necessary for me.

Simply move your unmanaged models to a dedicated app and delete migrations folder. Details in my answer in Multi db and unmanged models - Test case are failing