I name my modules same as class names. So MyClass.py would contain class MyClass.
(Why? I adopted this naming convention from other languages and would rather not change it)
I put it in directory components with __init__.py like this:
from .MyClass1 import MyClass1
from .MyClass2 import MyClass2
I do exactly the same thing for classes in directory controllers.
Then, in my main.py, I try to import them like so:
from components import MyClass1, MyClass2
from controllers import MyClass3, MyClass4
From components it imports fine, from controllers, it ignores my __init__.py and imports a module instead of class, so I have to rewrite my imports like so:
from components import MyClass1, MyClass2
from controllers.MyClass3 import MyClass3
from controllers.MyClass4 import MyClass4
Please tell me what could be the reason? Is there a way to fix this without having to change naming convention? Thank you.
Related
I am struggling with what seems to me a very basic and common problem, and the fact that I could not find any answer after hours of Internet searching tells me that I must be doing something very wrong...
I am simply trying to find an elegent way to handle imports with my package.
The background :
My package is structured like this :
mypackage/
__init__.py
model/
__init__.py
A.py
B.py
controllers/
__init__.py
A.py
B.py
# mypackage/model/A.py
class A:
def __init__(self):
print("This is A's model.")
# mypackage/model/B.py
from mypackage.model.A import A as AModel
class B:
def __init__(self):
self._a_model = AModel()
print("This is B's model.")
# mypackage/controllers/A.py
class A:
def __init__(self):
print("This is A's controller.")
# mypackage/controllers/B.py
from mypackage.controllers.A import A as AController
class B:
def __init__(self):
self._a = AController()
print("This is B's controller.")
The problem :
Two things are really bothering me with this design.
First : I want to use namespaces
I really don't like writing
from mypackage.controllers.A import A as AController
...
self._a = AController()
It feels cumbersome and not very pythonic...
I would prefer using namespaces like in :
from mypackage import controllers
...
self._a = controllers.A.A()
But if I try, I get a AttributeError: module 'mypackage.controllers' has no attribute 'A'
Second : I really don't like typing the class's filename
I really don't like writing
from mypackage.controllers.A import A as AController
I would prefer :
from mypackage.controllers import A as AController
What did not work
Putting everything in one file
I understand that I could get what I want by puting all controller's class (A and B) defininitions in a single file (controllers.py) and do the same with the models...
I read several time that putting several class definitions in a single file is a quite common thing to do in python... But for big separate classes I just can't. If A and B are hundreds of lines and have nothing to do with each other (other than being controllers and models), having their definitions in a single file is unusable.
Using imports in the init.py files
Indeed it would solve all my problems...
Except that :
It leads to circular imports. As you can see I need A's model in B's model... So if all models are imported when I need to access one of them, I'm stuck in vicious circle...
It does not seems very pythonic. If only because it forces the user to load every modules.
Here I am...
What is wrong with my reasoning ?
Using imports in the __init__.py file
That is the way to go.
Sorry if it looks too much boiler plate for you, but if yru project is big, that is what is needed.
As for the circular import problem: it will kick in wether you write your imports
in the __init__ files or not, and is just a matter of logistic.
If in your __init__ file, you write the imports in the correct order, there will be no circularity problem.
I.e. in your myproject/models/__init__.py you have:
from .A import A as AModel
from .B import B as BModel
Of course you naming the .py files the same names as the classes won't help you - if you will at least let go of the casing in the filename you can write:
from .a import A
from .b import B
Otherwise, you can do just:
import myproject.models.A
import myproject.models.B
A = myproject.models.A.A
B = myproject.models.B.B
To be able to use "myproject.models.A" as the class:
The name A inside __init__ will override the module object
with the same name.
One writting
import myproject.models.A will get to the module, but by doing
from myproject.models import A you get the module.
If that feels confusing... try not to use the same name for the module file than the classes. Even because in case-ignoring file systems, like Windows you would
ambiguities anyway. Stick with the convention: module names in snake_case, class names in CamelCase
Back to the circular-imports matter: the point is that in this design, b.py is only read after a.py has been already imported, and no circular-import problem.
That is not always possible - sometimes cross-reference between submodules are needed. What is possible to do in these cases is to move the import lines into
the functions or methods, instead of as a global statement. That way, when the import is executed, the referred module is already initialised.
In your example that would be:
mypackage.models.b.py
# mypackage/model/B.py
class B:
def __init__(self):
from mypackage.model.A import A as AModel
self._a_model = AModel()
print("This is B's model.")
# mypackage/controllers/__init__.py
from A import A
Then you can make a new file outside of mypackage with.
# check.py
from mypackage.controllers import A as AController
from mypackage import controllers
a = controllers.A()
>>> This is A's controller.
let us know if it works for you.
[Off the top of my head, without testing]
I really don't like writing
from mypackage.controllers.A import A as AController
#...
self._a = AController()
It feels cumbersome and not very pythonic... I would prefer using
namespaces like in :
from mypackage import controllers
# ...
self._a = controllers.A.A()
In mypackage/controllers/__init__.py you would need: from . import A.
I really don't like writing
from mypackage.controllers.A import A as AController
I would prefer :
from mypackage.controllers import A as AController
In mypackage/controllers/__init__.py you would need from A import A.
I have the following (toy) package structure
root/
- package1/
- __init__.py
- class_a.py
- class_b.py
- run.py
In both class_a.py and class_b.py I have a class definition that I want to expose to run.py. If I want to import them this way, I will have to use
from package1.class_a import ClassA # works but doesn't look nice
I don't like that this shows the class_a.py module, and would rather use the import style
from package1 import ClassA # what I want
This is also closer to what I see from larger libraries. I found a way to do this by importing the classes in the __init__.py file like so
from class_a import ClassA
from class_b import ClassB
This works fine if it wasn't for one downside: as soon as I import ClassA as I would like (see above), I also immediately 'import' ClassB as, as far as I know, the __init__.py will be run, importing ClassB. In my real scenario, this means I implicitly import a huge class that I use very situationally (which itself imports tensorflow), so I really want to avoid this somehow. Is there a way to create the nice looking imports without automatically importing everything in the package?
It is possible but require a rather low level customization: you will have to customize the class of your package (possible since Python 3.5). That way, you can declare a __getattr__ member that will be called when you ask for a missing attribute. At that moment, you know that you have to import the relevant module and extract the correct attribute.
The init.py file should contain (names can of course be changed):
import importlib
import sys
import types
class SpecialModule(types.ModuleType):
""" Customization of a module that is able to dynamically loads submodules.
It is expected to be a plain package (and to be declared in the __init__.py)
The special attribute is a dictionary attribute name -> relative module name.
The first time a name is requested, the corresponding module is loaded, and
the attribute is binded into the package
"""
special = {'ClassA': '.class_a', 'ClassB': '.class_b'}
def __getattr__(self, name):
if name in self.special:
m = importlib.import_module(self.special[name], __name__) # import submodule
o = getattr(m, name) # find the required member
setattr(sys.modules[__name__], name, o) # bind it into the package
return o
else:
raise AttributeError(f'module {__name__} has no attribute {name}')
sys.modules[__name__].__class__ = SpecialModule # customize the class of the package
You can now use it that way:
import package1
...
obj = package1.ClassA(...) # dynamically loads class_a on first call
The downside is that clever IDE that look at the declared member could choke on that and pretend that you are accessing an inexistant member because ClassA is not statically declared in package1/__init__.py. But all will be fine at run time.
As it is a low level customization, it is up to you do know whether it is worth it...
Since 3.7 you could also declare a __gettatr__(name) function directly at the module level.
my problem is following, I have an application in the following hierarchy:
main.py
package/__init__.py
package/MyClass.py
Clearly there is a package named "package". In the MyClass.py there is a class definition.
Now, while im in "main.py" file, to get the class instance i have to do:
package.MyClass.MyClass()
in order to create and instance of that class.
Now I want to be able to write only
package.MyClass()
To instantiate a class. How do I do this? I saw many apis do it this way, is there some trick to do it?
Add this in __init__.py:
from .MyClass import Myclass
Then, in main.py:
import package
package.MyClass()
Or again:
from package import MyClass
MyClass()
Make __init__.py:
from .package import MyClass
I'm wondering if it is possible to change helper functions in a testing environment in Python. My app is structured as follows:
app/
trader/
__init__.py
strategies/
__init__.py
base_strategy.py
util.py
models/
__init__.py
base_model.py
tests/
__init__.py
strategies/
__init__.py
stub_strategy.py
test_strategies.py
models/
__init__.py
stub_model.py
where each base_*.py is an abstract base class, which are inherited from in each stub_*.py file in the tests directory. In my util.py file, there is a helper function which looks through the trader/strategies/models directory and registers all models that are available:
import inspect
from . import alpha_models
existing_alpha_models = []
alpha_model_dict = {}
for name, obj in inspect.getmembers(alpha_models):
if inspect.isclass(obj):
existing_alpha_models.append(name)
alpha_model_dict[name] = obj
Now to my problem: In my BaseTradingStrategy-class I have a method, which uses the existing_alpha_models list, to check if the model exists.
from abc import ABC, abstractmethod
from .util import existing_alpha_models
class BaseTradingStrategy(ABC):
"""BaseTrading Code comes here"""
#abstractmethod
def some_abs_method(self):
raise NotImplementedError
def register_model(self, model_name):
if model_name not in existing_alpha_models:
raise ValueError
For unit testing, I have created stub classes, which are not in the trader/strategies/models directory, and are, therefore, not registered in the existing_alpha_models list. When I want to test the functionality of the ABC with pytest, many tests fail since the method that checks the availability of the models fails. A simple solution would be to put the stub classes in the trader directory of my app, but I'd rather have my test code separated from the rest of the app. I probably could also make existing_alpha_models a property of the base class, but I don't really see the point in doing so, except making the tests pass. Is there a way to inject the stub classes in the existing_alpha_models for unit testing so that the test of the ABC does not fail without changing the base class too much?
--------------EDIT-------------------
I now have 2 working versions of my test code. One is using the verison of #hoefling, where I simply add the alpha_models to the existing_alpha_models list:
from tests.strategies import StubTradingStrategy
from tests.strategies.alpha_models import StubModel, StubModel2, StubModel3
from trader.strategies.util import existing_alpha_models
existing_alpha_models.extend(["StubModel", "StubModel2", "StubModel3"])
and one version where I add the models to the alpha_models module and reload 2 modules:
import importlib
from trader.strategies import alpha_models
from tests.strategies.alpha_models import StubModel, StubModel2, StubModel3
setattr(alpha_models, "StubModel", StubModel)
setattr(alpha_models, "StubModel2", StubModel2)
setattr(alpha_models, "StubModel3", StubModel3)
from nutrader.strategies import util
import nutrader.strategies.base_strategy
importlib.reload(util)
importlib.reload(nutrader.strategies.base_strategy)
from tests.strategies import StubTradingStrategy
The advantage of the second version is that it allows me to actually test the util code, but it also introduces potential risks in my test code since there exist 2 version of certain modules, which is not the case in the production environment. Is this a good idea, or should I leave it at the first version for my test environment?
I have an abstract base class with a number of derived classes. I'm trying to achieve the same behaviour that I would get by placing all the derived classes in the same file as the base class, i.e. if my classes are Base, DerivedA, DerivedB, DerivedC in the file myclass.py I can write in another file
import myclass
a = myclass.DerivedA()
b = myclass.DerivedB()
c = myclass.DerivedC()
but with each derived class in its own file. This has to be dynamic, i.e. such that I could e.g. delete derived_c.py and everything still works except that now I can no longer call myclass.DerivedC, or that if I add a derived_d.py, I could use it without touching the __init__.py so simply using from derived_c import DerivedC is not an option.
I've tried placing them all in a subdirectory and in that directory's __init__.py use pkgutil.walk_packages() to import all the files dynamically, but I can't get them to then be directly in the module's namespace, i.e. rather than myclass.DerivedC() I have to call myclass.derived_c.DerivedC() because I can't figure out how (or if it's possible) to use importlib to achieve the equivalent of a from xyz import * statement.
Any suggestions for how I could achieve this? Thanks!
Edit: The solutions for Dynamic module import in Python don't provide a method for automatically importing the classes in all modules into the namespace of the package.
I had to make something quite similar a while back, but in my case I had to dynamically create a list with all subclasses from a base class in a specific package, so in case you find it useful:
Create a my_classes package containing all files for your Base class and all subclasses. You should include only one class in each file.
Set __all__ appropriately in __init__.py to import all .py files except for __init__.py (from this answer):
from os import listdir
from os.path import dirname, basename
__all__ = [basename(f)[:-3] for f in listdir(dirname(__file__)) if f[-3:] == ".py" and not f.endswith("__init__.py")]
Import your classes using from my_classes import *, since our custom __all__ adds all classes inside the my_classes package to the namespace.
However, this does not allow us direct access to the subclasses yet. You have to access them like this in your main script:
from my_classes import *
from my_classes.base import Base
subclasses = Base.__subclasses__()
Now subclasses is a list containing all classes that derive from Base.
Since Python 3.6 there exists a method for initializing subclasses. This is done on definition, so before all of your code gets executed. In here you can simply import the sub-class that is initialized.
base.py
class Base:
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
__import__(cls.__module__)
sub1.py
class Sub1(Base):
pass
sub2.py
class Sub2(Base):
pass