I am struggling with what seems to me a very basic and common problem, and the fact that I could not find any answer after hours of Internet searching tells me that I must be doing something very wrong...
I am simply trying to find an elegent way to handle imports with my package.
The background :
My package is structured like this :
mypackage/
__init__.py
model/
__init__.py
A.py
B.py
controllers/
__init__.py
A.py
B.py
# mypackage/model/A.py
class A:
def __init__(self):
print("This is A's model.")
# mypackage/model/B.py
from mypackage.model.A import A as AModel
class B:
def __init__(self):
self._a_model = AModel()
print("This is B's model.")
# mypackage/controllers/A.py
class A:
def __init__(self):
print("This is A's controller.")
# mypackage/controllers/B.py
from mypackage.controllers.A import A as AController
class B:
def __init__(self):
self._a = AController()
print("This is B's controller.")
The problem :
Two things are really bothering me with this design.
First : I want to use namespaces
I really don't like writing
from mypackage.controllers.A import A as AController
...
self._a = AController()
It feels cumbersome and not very pythonic...
I would prefer using namespaces like in :
from mypackage import controllers
...
self._a = controllers.A.A()
But if I try, I get a AttributeError: module 'mypackage.controllers' has no attribute 'A'
Second : I really don't like typing the class's filename
I really don't like writing
from mypackage.controllers.A import A as AController
I would prefer :
from mypackage.controllers import A as AController
What did not work
Putting everything in one file
I understand that I could get what I want by puting all controller's class (A and B) defininitions in a single file (controllers.py) and do the same with the models...
I read several time that putting several class definitions in a single file is a quite common thing to do in python... But for big separate classes I just can't. If A and B are hundreds of lines and have nothing to do with each other (other than being controllers and models), having their definitions in a single file is unusable.
Using imports in the init.py files
Indeed it would solve all my problems...
Except that :
It leads to circular imports. As you can see I need A's model in B's model... So if all models are imported when I need to access one of them, I'm stuck in vicious circle...
It does not seems very pythonic. If only because it forces the user to load every modules.
Here I am...
What is wrong with my reasoning ?
Using imports in the __init__.py file
That is the way to go.
Sorry if it looks too much boiler plate for you, but if yru project is big, that is what is needed.
As for the circular import problem: it will kick in wether you write your imports
in the __init__ files or not, and is just a matter of logistic.
If in your __init__ file, you write the imports in the correct order, there will be no circularity problem.
I.e. in your myproject/models/__init__.py you have:
from .A import A as AModel
from .B import B as BModel
Of course you naming the .py files the same names as the classes won't help you - if you will at least let go of the casing in the filename you can write:
from .a import A
from .b import B
Otherwise, you can do just:
import myproject.models.A
import myproject.models.B
A = myproject.models.A.A
B = myproject.models.B.B
To be able to use "myproject.models.A" as the class:
The name A inside __init__ will override the module object
with the same name.
One writting
import myproject.models.A will get to the module, but by doing
from myproject.models import A you get the module.
If that feels confusing... try not to use the same name for the module file than the classes. Even because in case-ignoring file systems, like Windows you would
ambiguities anyway. Stick with the convention: module names in snake_case, class names in CamelCase
Back to the circular-imports matter: the point is that in this design, b.py is only read after a.py has been already imported, and no circular-import problem.
That is not always possible - sometimes cross-reference between submodules are needed. What is possible to do in these cases is to move the import lines into
the functions or methods, instead of as a global statement. That way, when the import is executed, the referred module is already initialised.
In your example that would be:
mypackage.models.b.py
# mypackage/model/B.py
class B:
def __init__(self):
from mypackage.model.A import A as AModel
self._a_model = AModel()
print("This is B's model.")
# mypackage/controllers/__init__.py
from A import A
Then you can make a new file outside of mypackage with.
# check.py
from mypackage.controllers import A as AController
from mypackage import controllers
a = controllers.A()
>>> This is A's controller.
let us know if it works for you.
[Off the top of my head, without testing]
I really don't like writing
from mypackage.controllers.A import A as AController
#...
self._a = AController()
It feels cumbersome and not very pythonic... I would prefer using
namespaces like in :
from mypackage import controllers
# ...
self._a = controllers.A.A()
In mypackage/controllers/__init__.py you would need: from . import A.
I really don't like writing
from mypackage.controllers.A import A as AController
I would prefer :
from mypackage.controllers import A as AController
In mypackage/controllers/__init__.py you would need from A import A.
Related
I have a generated python module file with thousands of data classes, size 6MB.
These classes are generated from database structure, so I can not reduce them or not provide them with my module.
This module cause two main problem:
Very slow importing
Damage performance of IDE
the number of classes I currently used in this module is less than 30, but I have to provide all of them in my module since other users may use some of them.
My current code looks like:
foo.py:
from .models import ClassA, ClassB
class Foo():
def __init__(slef):
self.a = ClassA()
self.b = ClassB()
__init__.py :
from .foo import Foo
from . import models
What's the right way to organize the classes in the module?
You can have the classes available in one module without having to import them all into another module where you intend to use them.
Say "6megModule.py" contains classes class1..class9999
from 6megModule import class1, class2, class3
If you can give more details about how you have designed your classes others may be able to help with structuring them, is there a good reason for having so many classes? Will your users be able to wade through such a massive amount to find and use the classes they actually need?
What I'd like to do
I'd like to import a Python module without adding it to the local namespace.
In other words, I'd like to do this:
import foo
del foo
Is there a cleaner way to do this?
Why I want to do it
The short version is that importing foo has a side effect that I want, but I don't really want it in my namespace afterwards.
The long version is that I have a base class that uses __init_subclass__() to register its subclasses. So base.py looks like this:
class Base:
_subclasses = {}
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
cls._subclasses[cls.__name__] = cls
#classmethod
def get_subclass(cls, class_name):
return cls._subclasses[class_name]
And its subclasses are defined in separate files, e.g. foo_a.py:
from base import Base
class FooA(Base):
pass
and so on.
The net effect here is that if I do
from base import Base
print(f"Before import: {Base._subclasses}")
import foo_a
import foo_b
print(f"After import: {Base._subclasses}")
then I would see
Before import: {}
After import: {'FooA': <class 'foo_a.FooA'>, 'FooB': <class 'foo_b.FooB'>}
So I needed to import these modules for the side effect of adding a reference to Base._subclasses, but now that that's done, I don't need them in my namespace anymore because I'm just going to be using Base.get_subclass().
I know I could just leave them there, but this is going into an __init__.py so I'd like to tidy up that namespace.
del works perfectly fine, I'm just wondering if there's a cleaner or more idiomatic way to do this.
If you want to import a module without assigning the module object to a variable, you can use importlib.import_module and ignore the return value:
import importlib
importlib.import_module("foo")
Note that using importlib.import_module is preferable over using the __import__ builtin directly for simple usages. See the builtin documenation for details.
I have the following (toy) package structure
root/
- package1/
- __init__.py
- class_a.py
- class_b.py
- run.py
In both class_a.py and class_b.py I have a class definition that I want to expose to run.py. If I want to import them this way, I will have to use
from package1.class_a import ClassA # works but doesn't look nice
I don't like that this shows the class_a.py module, and would rather use the import style
from package1 import ClassA # what I want
This is also closer to what I see from larger libraries. I found a way to do this by importing the classes in the __init__.py file like so
from class_a import ClassA
from class_b import ClassB
This works fine if it wasn't for one downside: as soon as I import ClassA as I would like (see above), I also immediately 'import' ClassB as, as far as I know, the __init__.py will be run, importing ClassB. In my real scenario, this means I implicitly import a huge class that I use very situationally (which itself imports tensorflow), so I really want to avoid this somehow. Is there a way to create the nice looking imports without automatically importing everything in the package?
It is possible but require a rather low level customization: you will have to customize the class of your package (possible since Python 3.5). That way, you can declare a __getattr__ member that will be called when you ask for a missing attribute. At that moment, you know that you have to import the relevant module and extract the correct attribute.
The init.py file should contain (names can of course be changed):
import importlib
import sys
import types
class SpecialModule(types.ModuleType):
""" Customization of a module that is able to dynamically loads submodules.
It is expected to be a plain package (and to be declared in the __init__.py)
The special attribute is a dictionary attribute name -> relative module name.
The first time a name is requested, the corresponding module is loaded, and
the attribute is binded into the package
"""
special = {'ClassA': '.class_a', 'ClassB': '.class_b'}
def __getattr__(self, name):
if name in self.special:
m = importlib.import_module(self.special[name], __name__) # import submodule
o = getattr(m, name) # find the required member
setattr(sys.modules[__name__], name, o) # bind it into the package
return o
else:
raise AttributeError(f'module {__name__} has no attribute {name}')
sys.modules[__name__].__class__ = SpecialModule # customize the class of the package
You can now use it that way:
import package1
...
obj = package1.ClassA(...) # dynamically loads class_a on first call
The downside is that clever IDE that look at the declared member could choke on that and pretend that you are accessing an inexistant member because ClassA is not statically declared in package1/__init__.py. But all will be fine at run time.
As it is a low level customization, it is up to you do know whether it is worth it...
Since 3.7 you could also declare a __gettatr__(name) function directly at the module level.
I have an abstract base class with a number of derived classes. I'm trying to achieve the same behaviour that I would get by placing all the derived classes in the same file as the base class, i.e. if my classes are Base, DerivedA, DerivedB, DerivedC in the file myclass.py I can write in another file
import myclass
a = myclass.DerivedA()
b = myclass.DerivedB()
c = myclass.DerivedC()
but with each derived class in its own file. This has to be dynamic, i.e. such that I could e.g. delete derived_c.py and everything still works except that now I can no longer call myclass.DerivedC, or that if I add a derived_d.py, I could use it without touching the __init__.py so simply using from derived_c import DerivedC is not an option.
I've tried placing them all in a subdirectory and in that directory's __init__.py use pkgutil.walk_packages() to import all the files dynamically, but I can't get them to then be directly in the module's namespace, i.e. rather than myclass.DerivedC() I have to call myclass.derived_c.DerivedC() because I can't figure out how (or if it's possible) to use importlib to achieve the equivalent of a from xyz import * statement.
Any suggestions for how I could achieve this? Thanks!
Edit: The solutions for Dynamic module import in Python don't provide a method for automatically importing the classes in all modules into the namespace of the package.
I had to make something quite similar a while back, but in my case I had to dynamically create a list with all subclasses from a base class in a specific package, so in case you find it useful:
Create a my_classes package containing all files for your Base class and all subclasses. You should include only one class in each file.
Set __all__ appropriately in __init__.py to import all .py files except for __init__.py (from this answer):
from os import listdir
from os.path import dirname, basename
__all__ = [basename(f)[:-3] for f in listdir(dirname(__file__)) if f[-3:] == ".py" and not f.endswith("__init__.py")]
Import your classes using from my_classes import *, since our custom __all__ adds all classes inside the my_classes package to the namespace.
However, this does not allow us direct access to the subclasses yet. You have to access them like this in your main script:
from my_classes import *
from my_classes.base import Base
subclasses = Base.__subclasses__()
Now subclasses is a list containing all classes that derive from Base.
Since Python 3.6 there exists a method for initializing subclasses. This is done on definition, so before all of your code gets executed. In here you can simply import the sub-class that is initialized.
base.py
class Base:
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
__import__(cls.__module__)
sub1.py
class Sub1(Base):
pass
sub2.py
class Sub2(Base):
pass
I'm developing a PyQT4 application, and it's getting pretty hard for me to navigate through all of the code at once. I know of the import foo statement, but I can't figure out how to make it import a chunk of code directly into my script, like the BASH source foo statement.
I'm trying to do this:
# File 'functions.py'
class foo(asd.fgh):
def __init__(self):
print 'foo'
Here is the second file.
# File 'main.py'
import functions
class foo(asd.fgh):
def qwerty(self):
print 'qwerty'
I want to include code or merge class decelerations from two separate files. In PHP, there is import_once('foo.php'), and as I mentioned previously, BASH has source 'foo.sh', but can I accomplish this with Python?
Thanks!
For some reason, my first thought was multiple inheritance. But why not try normal inheritance?
class foo(functions.foo):
# All of the methods that you want to add go here.
Is there some reason that this wont work?
Since you just want to merge class definitions, why don't you do:
# main.py
import functions
# All of the old stuff that was in main.foo is now in this class
class fooBase(asd.fgh):
def qwerty(self):
print 'qwerty'
# Now create a class that has methods and attributes of both classes
class foo(FooBase, functions.foo): # Methods from FooBase take precedence
pass
or
class foo(functions.foo, FooBase): # Methods from functions.foo take precedence
pass
This takes advantage of pythons capability for multiple inheritance to create a new class with methods from both sources.
You want execfile(). Although you really don't, since redefining a class, uh... redefines it.
monkey patching in python doesn't work in nearly the same way. This is normally considered poor form, but if you want to do it anyways, you can do this:
# File 'functions.py'
class foo(asd.fgh):
def __init__(self):
print 'foo'
the imported module remains unchanged. In the importing module, we do things quite differently.
# File 'main.py'
import functions
def qwerty(self):
print 'qwerty'
functions.foo.qwerty = qwerty
Note that there is no additional class definition, just a bare function. we then add the function as an attribute of the class.