On Python 3.6
I want to organize my classes in folders.
ClassX in folder Classes
Subclass_of_ClassX in a subfolder
MyPackage
-__init__.py
-someCode.py
-folder Classes
-__init__.py
-ClassX.py
-Subfolder SubClasses
-__init__.py
-Subclass_of_ClassX.py
Subclass_of_ClassX overriddes elements (functions and variables) from ClassX
The question is: if I import ClassX on any project, and I get any instance of Subclass_of_ClassX, and I want to use any overriden function, do I need to explicitly import the subclass?
Or can I just import ClassX and abstract from any inherited class, using any subclass instance as if it were ClassX?
If you get an instance of Subclass_of_ClassX from somewhere, you don't need to additionally import the class at all. Importing a class just makes the name available in your current scope (module).
You only need to import the class or subclass when you need to...
create a new instance
write a subclass
use it otherwise like in isinstance(obj, ClassX)
If you got instance of class - you already don't need to import anything, because it was imported in some other place already.
What you describe is not the Pythonic way to structure packages.
You'll end up with having to do
from mypackage.Classes.ClassX import ClassX
when you could do with
from mypackage.class_x import ClassX
If your package is, uh, for the sake of example, a zoo where you have animals and foods, I would suggest putting the animals in one package and their foods in another.
myzoo/__init__.py (empty)
myzoo/animals/__init__.py (empty)
myzoo/animals/base.py (containing your Animal base class)
myzoo/animals/cats.py (containing Lion and Tiger, for instance)
myzoo/foods/__init__.py (empty)
myzoo/foods/base.py (containing your Food base class)
myzoo/foods/kibbles.py (containing kibbles for the cats, not that they'll like them)
(You could also have one file per animal/food, and probably should if you anticipate them growing large.)
Related
I have a generated python module file with thousands of data classes, size 6MB.
These classes are generated from database structure, so I can not reduce them or not provide them with my module.
This module cause two main problem:
Very slow importing
Damage performance of IDE
the number of classes I currently used in this module is less than 30, but I have to provide all of them in my module since other users may use some of them.
My current code looks like:
foo.py:
from .models import ClassA, ClassB
class Foo():
def __init__(slef):
self.a = ClassA()
self.b = ClassB()
__init__.py :
from .foo import Foo
from . import models
What's the right way to organize the classes in the module?
You can have the classes available in one module without having to import them all into another module where you intend to use them.
Say "6megModule.py" contains classes class1..class9999
from 6megModule import class1, class2, class3
If you can give more details about how you have designed your classes others may be able to help with structuring them, is there a good reason for having so many classes? Will your users be able to wade through such a massive amount to find and use the classes they actually need?
I have the following (toy) package structure
root/
- package1/
- __init__.py
- class_a.py
- class_b.py
- run.py
In both class_a.py and class_b.py I have a class definition that I want to expose to run.py. If I want to import them this way, I will have to use
from package1.class_a import ClassA # works but doesn't look nice
I don't like that this shows the class_a.py module, and would rather use the import style
from package1 import ClassA # what I want
This is also closer to what I see from larger libraries. I found a way to do this by importing the classes in the __init__.py file like so
from class_a import ClassA
from class_b import ClassB
This works fine if it wasn't for one downside: as soon as I import ClassA as I would like (see above), I also immediately 'import' ClassB as, as far as I know, the __init__.py will be run, importing ClassB. In my real scenario, this means I implicitly import a huge class that I use very situationally (which itself imports tensorflow), so I really want to avoid this somehow. Is there a way to create the nice looking imports without automatically importing everything in the package?
It is possible but require a rather low level customization: you will have to customize the class of your package (possible since Python 3.5). That way, you can declare a __getattr__ member that will be called when you ask for a missing attribute. At that moment, you know that you have to import the relevant module and extract the correct attribute.
The init.py file should contain (names can of course be changed):
import importlib
import sys
import types
class SpecialModule(types.ModuleType):
""" Customization of a module that is able to dynamically loads submodules.
It is expected to be a plain package (and to be declared in the __init__.py)
The special attribute is a dictionary attribute name -> relative module name.
The first time a name is requested, the corresponding module is loaded, and
the attribute is binded into the package
"""
special = {'ClassA': '.class_a', 'ClassB': '.class_b'}
def __getattr__(self, name):
if name in self.special:
m = importlib.import_module(self.special[name], __name__) # import submodule
o = getattr(m, name) # find the required member
setattr(sys.modules[__name__], name, o) # bind it into the package
return o
else:
raise AttributeError(f'module {__name__} has no attribute {name}')
sys.modules[__name__].__class__ = SpecialModule # customize the class of the package
You can now use it that way:
import package1
...
obj = package1.ClassA(...) # dynamically loads class_a on first call
The downside is that clever IDE that look at the declared member could choke on that and pretend that you are accessing an inexistant member because ClassA is not statically declared in package1/__init__.py. But all will be fine at run time.
As it is a low level customization, it is up to you do know whether it is worth it...
Since 3.7 you could also declare a __gettatr__(name) function directly at the module level.
I'm wondering if it is possible to change helper functions in a testing environment in Python. My app is structured as follows:
app/
trader/
__init__.py
strategies/
__init__.py
base_strategy.py
util.py
models/
__init__.py
base_model.py
tests/
__init__.py
strategies/
__init__.py
stub_strategy.py
test_strategies.py
models/
__init__.py
stub_model.py
where each base_*.py is an abstract base class, which are inherited from in each stub_*.py file in the tests directory. In my util.py file, there is a helper function which looks through the trader/strategies/models directory and registers all models that are available:
import inspect
from . import alpha_models
existing_alpha_models = []
alpha_model_dict = {}
for name, obj in inspect.getmembers(alpha_models):
if inspect.isclass(obj):
existing_alpha_models.append(name)
alpha_model_dict[name] = obj
Now to my problem: In my BaseTradingStrategy-class I have a method, which uses the existing_alpha_models list, to check if the model exists.
from abc import ABC, abstractmethod
from .util import existing_alpha_models
class BaseTradingStrategy(ABC):
"""BaseTrading Code comes here"""
#abstractmethod
def some_abs_method(self):
raise NotImplementedError
def register_model(self, model_name):
if model_name not in existing_alpha_models:
raise ValueError
For unit testing, I have created stub classes, which are not in the trader/strategies/models directory, and are, therefore, not registered in the existing_alpha_models list. When I want to test the functionality of the ABC with pytest, many tests fail since the method that checks the availability of the models fails. A simple solution would be to put the stub classes in the trader directory of my app, but I'd rather have my test code separated from the rest of the app. I probably could also make existing_alpha_models a property of the base class, but I don't really see the point in doing so, except making the tests pass. Is there a way to inject the stub classes in the existing_alpha_models for unit testing so that the test of the ABC does not fail without changing the base class too much?
--------------EDIT-------------------
I now have 2 working versions of my test code. One is using the verison of #hoefling, where I simply add the alpha_models to the existing_alpha_models list:
from tests.strategies import StubTradingStrategy
from tests.strategies.alpha_models import StubModel, StubModel2, StubModel3
from trader.strategies.util import existing_alpha_models
existing_alpha_models.extend(["StubModel", "StubModel2", "StubModel3"])
and one version where I add the models to the alpha_models module and reload 2 modules:
import importlib
from trader.strategies import alpha_models
from tests.strategies.alpha_models import StubModel, StubModel2, StubModel3
setattr(alpha_models, "StubModel", StubModel)
setattr(alpha_models, "StubModel2", StubModel2)
setattr(alpha_models, "StubModel3", StubModel3)
from nutrader.strategies import util
import nutrader.strategies.base_strategy
importlib.reload(util)
importlib.reload(nutrader.strategies.base_strategy)
from tests.strategies import StubTradingStrategy
The advantage of the second version is that it allows me to actually test the util code, but it also introduces potential risks in my test code since there exist 2 version of certain modules, which is not the case in the production environment. Is this a good idea, or should I leave it at the first version for my test environment?
I have an abstract base class with a number of derived classes. I'm trying to achieve the same behaviour that I would get by placing all the derived classes in the same file as the base class, i.e. if my classes are Base, DerivedA, DerivedB, DerivedC in the file myclass.py I can write in another file
import myclass
a = myclass.DerivedA()
b = myclass.DerivedB()
c = myclass.DerivedC()
but with each derived class in its own file. This has to be dynamic, i.e. such that I could e.g. delete derived_c.py and everything still works except that now I can no longer call myclass.DerivedC, or that if I add a derived_d.py, I could use it without touching the __init__.py so simply using from derived_c import DerivedC is not an option.
I've tried placing them all in a subdirectory and in that directory's __init__.py use pkgutil.walk_packages() to import all the files dynamically, but I can't get them to then be directly in the module's namespace, i.e. rather than myclass.DerivedC() I have to call myclass.derived_c.DerivedC() because I can't figure out how (or if it's possible) to use importlib to achieve the equivalent of a from xyz import * statement.
Any suggestions for how I could achieve this? Thanks!
Edit: The solutions for Dynamic module import in Python don't provide a method for automatically importing the classes in all modules into the namespace of the package.
I had to make something quite similar a while back, but in my case I had to dynamically create a list with all subclasses from a base class in a specific package, so in case you find it useful:
Create a my_classes package containing all files for your Base class and all subclasses. You should include only one class in each file.
Set __all__ appropriately in __init__.py to import all .py files except for __init__.py (from this answer):
from os import listdir
from os.path import dirname, basename
__all__ = [basename(f)[:-3] for f in listdir(dirname(__file__)) if f[-3:] == ".py" and not f.endswith("__init__.py")]
Import your classes using from my_classes import *, since our custom __all__ adds all classes inside the my_classes package to the namespace.
However, this does not allow us direct access to the subclasses yet. You have to access them like this in your main script:
from my_classes import *
from my_classes.base import Base
subclasses = Base.__subclasses__()
Now subclasses is a list containing all classes that derive from Base.
Since Python 3.6 there exists a method for initializing subclasses. This is done on definition, so before all of your code gets executed. In here you can simply import the sub-class that is initialized.
base.py
class Base:
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
__import__(cls.__module__)
sub1.py
class Sub1(Base):
pass
sub2.py
class Sub2(Base):
pass
I have currently this structure inside a file:
class Foo:
__init__(self):
pass
class FooType(object):
__init__(self, value):
_foo = value
__str__(self):
print ">>%s<<" % self._foo
class FooException(Exception):
pass
All the above classes are tightly related. My master class will have types, structures, enums all declared as a separated class with the prefix Foo. And as usual a custom exception should be declared for Foo. At the end of the day I will have a lot of related classes at the same level of other classes.
Is there a proper way to get a better structure? Perhaps a namespace can help, but I don't know how to use it.
In Python, the idiomatic way to do namespaces is to use different modules. You should name your file foo.py, and then import that. Then you use it with foo.Exception and foo.Type. If you need to have a more complex module that needs more than one file, you should make a folder called foo and put an __init__.py file along with your other components of the module. For further documentation on using modules, see the docs.
Another solution is to use nested classes to provide a namespace. The chosen answer to this question about nested classes recommends them for the purpose of namespacing.
You could organize your classes in modules, just like the Python standard library does.