Develop Reference

Python using argument with class type creates circular import error [duplicate]

I'm trying to split my huge class into two; well, basically into the "main" class and a mixin with additional functions, like so:
main.py file:
import mymixin.py
class Main(object, MyMixin):
def func1(self, xxx):
...
mymixin.py file:
class MyMixin(object):
def func2(self: Main, xxx): # <--- note the type hint
...
Now, while this works just fine, the type hint in MyMixin.func2 of course can't work. I can't import main.py, because I'd get a cyclic import and without the hint, my editor (PyCharm) can't tell what self is.
I'm using Python 3.4, but I'm willing to move to 3.5 if a solution is available there.
Is there any way I can split my class into two files and keep all the "connections" so that my IDE still offers me auto-completion and all the other goodies that come from it knowing the types?

There isn't a hugely elegant way to handle import cycles in general, I'm afraid. Your choices are to either redesign your code to remove the cyclic dependency, or if it isn't feasible, do something like this:
# some_file.py
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from main import Main
class MyObject(object):
def func2(self, some_param: 'Main'):
...
The TYPE_CHECKING constant is always False at runtime, so the import won't be evaluated, but mypy (and other type-checking tools) will evaluate the contents of that block.
We also need to make the Main type annotation into a string, effectively forward declaring it since the Main symbol isn't available at runtime.
If you are using Python 3.7+, we can at least skip having to provide an explicit string annotation by taking advantage of PEP 563:
# some_file.py
from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from main import Main
class MyObject(object):
# Hooray, cleaner annotations!
def func2(self, some_param: Main):
...
The from __future__ import annotations import will make all type hints be strings and skip evaluating them. This can help make our code here mildly more ergonomic.
All that said, using mixins with mypy will likely require a bit more structure then you currently have. Mypy recommends an approach that's basically what deceze is describing -- to create an ABC that both your Main and MyMixin classes inherit. I wouldn't be surprised if you ended up needing to do something similar in order to make Pycharm's checker happy.

For people struggling with cyclic imports when importing class only for Type checking: you will likely want to use a Forward Reference (PEP 484 - Type Hints):
When a type hint contains names that have not been defined yet, that definition may be expressed as a string literal, to be resolved later.
So instead of:
class Tree:
def __init__(self, left: Tree, right: Tree):
self.left = left
self.right = right
you do:
class Tree:
def __init__(self, left: 'Tree', right: 'Tree'):
self.left = left
self.right = right

The bigger issue is that your types aren't sane to begin with. MyMixin makes a hardcoded assumption that it will be mixed into Main, whereas it could be mixed into any number of other classes, in which case it would probably break. If your mixin is hardcoded to be mixed into one specific class, you may as well write the methods directly into that class instead of separating them out.
To properly do this with sane typing, MyMixin should be coded against an interface, or abstract class in Python parlance:
import abc
class MixinDependencyInterface(abc.ABC):
#abc.abstractmethod
def foo(self):
pass
class MyMixin:
def func2(self: MixinDependencyInterface, xxx):
self.foo() # ← mixin only depends on the interface
class Main(MixinDependencyInterface, MyMixin):
def foo(self):
print('bar')

Since Python 3.5, breaking your classes up into separate files is easy.
It's actually possible to use import statements inside of a class ClassName: block in order to import methods into a class. For instance,
class_def.py:
class C:
from _methods1 import a
from _methods2 import b
def x(self):
return self.a() + " " + self.b()
In my example,
C.a() will be a method which returns the string hello
C.b() will be a method which returns hello goodbye
C.x() will thus return hello hello goodbye.
To implement a and b, do the following:
_methods1.py:
from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from class_def import C
def a(self: C):
return "hello"
Explanation: TYPE_CHECKING is True when the type checker is reading the code. Since the type checker doesn't need to execute the code, circular imports are fine when they occur within the if TYPE_CHECKING: block. The __future__ import enables postponed annotations. This is an optional; without it you must quote the type annotations (i.e. def a(self: "C"):).
We define _methods2.py similarly:
from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from class_def import C
def b(self: C):
return self.a() + " goodbye"
In VS Code, I can see the type detected from self.a() when hovering:
And everything runs as expected:
>>> from class_def import C
>>> c = C()
>>> c.x()
'hello hello goodbye'
Notes on older Python versions
For Python versions ≤3.4, TYPE_CHECKING is not defined, so this solution won't work.
For Python versions ≤3.6, postponed annotations are not defined. As a workaround, omit from __future__ import annotations and quote the type declarations as mentioned above.

Turns out my original attempt was quite close to the solution as well. This is what I'm currently using:
# main.py
import mymixin.py
class Main(object, MyMixin):
def func1(self, xxx):
...
# mymixin.py
if False:
from main import Main
class MyMixin(object):
def func2(self: 'Main', xxx): # <--- note the type hint
...
Note the import within if False statement that never gets imported (but IDE knows about it anyway) and using the Main class as string because it's not known at runtime.

Rather than forcing oneself to engage in typing.TYPE_CHECKING shenanigans, there is a simple way to avoid circular type-hints: don't use from imports, and use either from __future__ import annotations or string annotations.
# foo.py
from __future__ import annotations
import bar
class Foo:
bar: bar.Bar
# bar.py
import foo
class Bar:
foo: "foo.Foo"
This style of import is "lazily evaluated", whereas using from foo import Foo would force Python to run the entire foo module to get the final value of Foo immediately at the import line. It's quite useful if you need to use it at runtime as well e.g. if foo.Foo or bar.Bar needs to be used within a function/method, since your functions/methods should only be called once foo.Foo and bar.Bar can be used.

I would advice refactoring your code, as some other persons suggested.
I can show you a circular error I recently faced:
BEFORE:
# person.py
from spell import Heal, Lightning
class Person:
def __init__(self):
self.life = 100
class Jedi(Person):
def heal(self, other: Person):
Heal(self, other)
class Sith(Person):
def lightning(self, other: Person):
Lightning(self, other)
# spell.py
from person import Person, Jedi, Sith
class Spell:
def __init__(self, caster: Person, target: Person):
self.caster: Person = caster
self.target: Person = target
class Heal(Spell):
def __init__(self, caster: Jedi, target: Person):
super().__init__(caster, target)
target.life += 10
class Lightning(Spell):
def __init__(self, caster: Sith, target: Person):
super().__init__(caster, target)
target.life -= 10
# main.py
from person import Jedi, Sith
Step by step:
# main starts to import person
from person import Jedi, Sith
# main did not reach end of person but ...
# person starts to import spell
from spell import Heal, Lightning
# Remember: main is still importing person
# spell starts to import person
from person import Person, Jedi, Sith
console:
ImportError: cannot import name 'Person' from partially initialized module
'person' (most likely due to a circular import)
A script/module can be imported only by one and only one script.
AFTER:
# person.py
class Person:
def __init__(self):
self.life = 100
# spell.py
from person import Person
class Spell:
def __init__(self, caster: Person, target: Person):
self.caster: Person = caster
self.target: Person = target
# jedi.py
from person import Person
from spell import Spell
class Jedi(Person):
def heal(self, other: Person):
Heal(self, other)
class Heal(Spell):
def __init__(self, caster: Jedi, target: Person):
super().__init__(caster, target)
target.life += 10
# sith.py
from person import Person
from spell import Spell
class Sith(Person):
def lightning(self, other: Person):
Lightning(self, other)
class Lightning(Spell):
def __init__(self, caster: Sith, target: Person):
super().__init__(caster, target)
target.life -= 10
# main.py
from jedi import Jedi
from sith import Sith
jedi = Jedi()
print(jedi.life)
Sith().lightning(jedi)
print(jedi.life)
order of executed lines:
from jedi import Jedi # start read of jedi.py
from person import Person # start AND finish read of person.py
from spell import Spell # start read of spell.py
from person import Person # start AND finish read of person.py
# finish read of spell.py
# idem for sith.py
console:
100
90
File composition is key
Hope it will help :D

I think the perfect way should be to import all the classes and dependencies in a file (like __init__.py) and then from __init__ import * in all the other files.
In this case you are
avoiding multiple references to those files and classes and
also only have to add one line in each of the other files and
the third would be the pycharm knowing about all of the classes that you might use.

Typing an unimported module

I have a Python package that has an optional [extras] dependency, yet I want to adhere to typing on all methods.
The situation is that in my file, I have this
class MyClass:
def __init__(self, datastore: Datastore): # <- Datastore is azureml.core.Datastore
...
def my_func(self):
from azureml.core import Datastore
...
I import from within the function because there are other classes in the same file that should be imported when not using the extras (extras being azureml).
So this obviously fails, because I refer to Datastore before importing it. Removing the Datastore typing from the __init__ method obviously solves the problem.
So in general my question is whether it is possible, and if so how, to use typing when typing an optional (extras) package.
Notice, that importing in the class definition (below the class MyClass statement) is not a valid solution, as this code is called when the module is imported

You can use TYPE_CHECKING:
A special constant that is assumed to be True by 3rd party static type
checkers. It is False at runtime.
It is False at runtime: So it doesn't affect your module's behavior.
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from azureml.core import Datastore
class MyClass:
def __init__(self, datastore: Datastore):
...
def my_func(self):
from azureml.core import Datastore
...
Since I want to show this in action, I will use operator.itemgetter as an instance because it's recognizable for type checkers, but azureml.core is not:
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from operator import itemgetter
class MyClass:
def __init__(self, datastore: itemgetter):
...
def my_func(self):
from operator import itemgetter
...
obj1 = MyClass(itemgetter(1)) # line 16
obj2 = MyClass(10) # line 17
Here is the Mypy error:
main.py:17: error: Argument 1 to "MyClass" has incompatible type "int"; expected "itemgetter[Any]"
Found 1 error in 1 file (checked 1 source file)
Which shows it works as excepted.

Just to add my two cents:
While it is certainly a solution, I consider the use of the TYPE_CHECKING constant a red flag regarding the project structure. It typically (though not always) either shows the presence of circular dependencies or poor separation of concerns.
In your case it seems to be the latter, as you state this:
I import from within the function because there are other classes in the same file that should be imported when not using the extras
If MyClass provides optional functionality to your package, it should absolutely reside in its own module and not alongside other classes that provide core functionality.
When you put MyClass into its own module (say my_class), you can place its dependencies at the top with all the other imports. Then you put the import from my_class inside a function that handles the logic of loading internal optional dependencies.
Aside from visibility and arguably better style, one advantage of such a setup over the one you presented is that the my_class module will be consistent in itself and fail on import, if the extra azureml dependency is missing (or broken/renamed/deprecated), rather than at runtime only when MyClass.my_func is called.
You'd be surprised how easy it is to accidentally forget to install all extra dependencies (even in a production environment). Then you'll thank the stars, when the code fails immediately and transparently, rather than causing errors at some point later at runtime.

Import a Python module without adding it to the local namespace

What I'd like to do
I'd like to import a Python module without adding it to the local namespace.
In other words, I'd like to do this:
import foo
del foo
Is there a cleaner way to do this?
Why I want to do it
The short version is that importing foo has a side effect that I want, but I don't really want it in my namespace afterwards.
The long version is that I have a base class that uses __init_subclass__() to register its subclasses. So base.py looks like this:
class Base:
_subclasses = {}
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
cls._subclasses[cls.__name__] = cls
#classmethod
def get_subclass(cls, class_name):
return cls._subclasses[class_name]
And its subclasses are defined in separate files, e.g. foo_a.py:
from base import Base
class FooA(Base):
pass
and so on.
The net effect here is that if I do
from base import Base
print(f"Before import: {Base._subclasses}")
import foo_a
import foo_b
print(f"After import: {Base._subclasses}")
then I would see
Before import: {}
After import: {'FooA': <class 'foo_a.FooA'>, 'FooB': <class 'foo_b.FooB'>}
So I needed to import these modules for the side effect of adding a reference to Base._subclasses, but now that that's done, I don't need them in my namespace anymore because I'm just going to be using Base.get_subclass().
I know I could just leave them there, but this is going into an __init__.py so I'd like to tidy up that namespace.
del works perfectly fine, I'm just wondering if there's a cleaner or more idiomatic way to do this.

If you want to import a module without assigning the module object to a variable, you can use importlib.import_module and ignore the return value:
import importlib
importlib.import_module("foo")
Note that using importlib.import_module is preferable over using the __import__ builtin directly for simple usages. See the builtin documenation for details.

Python subpackages and namespaces

I am struggling with what seems to me a very basic and common problem, and the fact that I could not find any answer after hours of Internet searching tells me that I must be doing something very wrong...
I am simply trying to find an elegent way to handle imports with my package.
The background :
My package is structured like this :
mypackage/
__init__.py
model/
__init__.py
A.py
B.py
controllers/
__init__.py
A.py
B.py
# mypackage/model/A.py
class A:
def __init__(self):
print("This is A's model.")
# mypackage/model/B.py
from mypackage.model.A import A as AModel
class B:
def __init__(self):
self._a_model = AModel()
print("This is B's model.")
# mypackage/controllers/A.py
class A:
def __init__(self):
print("This is A's controller.")
# mypackage/controllers/B.py
from mypackage.controllers.A import A as AController
class B:
def __init__(self):
self._a = AController()
print("This is B's controller.")
The problem :
Two things are really bothering me with this design.
First : I want to use namespaces
I really don't like writing
from mypackage.controllers.A import A as AController
...
self._a = AController()
It feels cumbersome and not very pythonic...
I would prefer using namespaces like in :
from mypackage import controllers
...
self._a = controllers.A.A()
But if I try, I get a AttributeError: module 'mypackage.controllers' has no attribute 'A'
Second : I really don't like typing the class's filename
I really don't like writing
from mypackage.controllers.A import A as AController
I would prefer :
from mypackage.controllers import A as AController
What did not work
Putting everything in one file
I understand that I could get what I want by puting all controller's class (A and B) defininitions in a single file (controllers.py) and do the same with the models...
I read several time that putting several class definitions in a single file is a quite common thing to do in python... But for big separate classes I just can't. If A and B are hundreds of lines and have nothing to do with each other (other than being controllers and models), having their definitions in a single file is unusable.
Using imports in the init.py files
Indeed it would solve all my problems...
Except that :
It leads to circular imports. As you can see I need A's model in B's model... So if all models are imported when I need to access one of them, I'm stuck in vicious circle...
It does not seems very pythonic. If only because it forces the user to load every modules.
Here I am...
What is wrong with my reasoning ?

Using imports in the __init__.py file
That is the way to go.
Sorry if it looks too much boiler plate for you, but if yru project is big, that is what is needed.
As for the circular import problem: it will kick in wether you write your imports
in the __init__ files or not, and is just a matter of logistic.
If in your __init__ file, you write the imports in the correct order, there will be no circularity problem.
I.e. in your myproject/models/__init__.py you have:
from .A import A as AModel
from .B import B as BModel
Of course you naming the .py files the same names as the classes won't help you - if you will at least let go of the casing in the filename you can write:
from .a import A
from .b import B
Otherwise, you can do just:
import myproject.models.A
import myproject.models.B
A = myproject.models.A.A
B = myproject.models.B.B
To be able to use "myproject.models.A" as the class:
The name A inside __init__ will override the module object
with the same name.
One writting
import myproject.models.A will get to the module, but by doing
from myproject.models import A you get the module.
If that feels confusing... try not to use the same name for the module file than the classes. Even because in case-ignoring file systems, like Windows you would
ambiguities anyway. Stick with the convention: module names in snake_case, class names in CamelCase
Back to the circular-imports matter: the point is that in this design, b.py is only read after a.py has been already imported, and no circular-import problem.
That is not always possible - sometimes cross-reference between submodules are needed. What is possible to do in these cases is to move the import lines into
the functions or methods, instead of as a global statement. That way, when the import is executed, the referred module is already initialised.
In your example that would be:
mypackage.models.b.py
# mypackage/model/B.py
class B:
def __init__(self):
from mypackage.model.A import A as AModel
self._a_model = AModel()
print("This is B's model.")

# mypackage/controllers/__init__.py
from A import A
Then you can make a new file outside of mypackage with.
# check.py
from mypackage.controllers import A as AController
from mypackage import controllers
a = controllers.A()
>>> This is A's controller.
let us know if it works for you.

[Off the top of my head, without testing]
I really don't like writing
from mypackage.controllers.A import A as AController
#...
self._a = AController()
It feels cumbersome and not very pythonic... I would prefer using
namespaces like in :
from mypackage import controllers
# ...
self._a = controllers.A.A()
In mypackage/controllers/__init__.py you would need: from . import A.
I really don't like writing
from mypackage.controllers.A import A as AController
I would prefer :
from mypackage.controllers import A as AController
In mypackage/controllers/__init__.py you would need from A import A.

Circular import dependency in Python [duplicate]

This question already has answers here:
What can I do about "ImportError: Cannot import name X" or "AttributeError: ... (most likely due to a circular import)"?
(17 answers)
Closed 7 months ago.
Let's say I have the following directory structure:
a\
__init__.py
b\
__init__.py
c\
__init__.py
c_file.py
d\
__init__.py
d_file.py
In the a package's __init__.py, the c package is imported. But c_file.py imports a.b.d.
The program fails, saying b doesn't exist when c_file.py tries to import a.b.d. (And it really doesn't exist, because we were in the middle of importing it.)
How can this problem be remedied?

You may defer the import, for example in a/__init__.py:
def my_function():
from a.b.c import Blah
return Blah()
that is, defer the import until it is really needed. However, I would also have a close look at my package definitions/uses, as a cyclic dependency like the one pointed out might indicate a design problem.

If a depends on c and c depends on a, aren't they actually the same unit then?
You should really examine why you have split a and c into two packages, because either you have some code you should split off into another package (to make them both depend on that new package, but not each other), or you should merge them into one package.

I've wondered this a couple times (usually while dealing with models that need to know about each other). The simple solution is just to import the whole module, then reference the thing that you need.
So instead of doing
from models import Student
in one, and
from models import Classroom
in the other, just do
import models
in one of them, then call models.Classroom when you need it.

Circular Dependencies due to Type Hints
With type hints, there are more opportunities for creating circular imports. Fortunately, there is a solution using the special constant: typing.TYPE_CHECKING.
The following example defines a Vertex class and an Edge class. An edge is defined by two vertices and a vertex maintains a list of the adjacent edges to which it belongs.
Without Type Hints, No Error
File: vertex.py
class Vertex:
def __init__(self, label):
self.label = label
self.adjacency_list = []
File: edge.py
class Edge:
def __init__(self, v1, v2):
self.v1 = v1
self.v2 = v2
Type Hints Cause ImportError
ImportError: cannot import name 'Edge' from partially initialized module 'edge' (most likely due to a circular import)
File: vertex.py
from typing import List
from edge import Edge
class Vertex:
def __init__(self, label: str):
self.label = label
self.adjacency_list: List[Edge] = []
File: edge.py
from vertex import Vertex
class Edge:
def __init__(self, v1: Vertex, v2: Vertex):
self.v1 = v1
self.v2 = v2
Solution using TYPE_CHECKING
File: vertex.py
from typing import List, TYPE_CHECKING
if TYPE_CHECKING:
from edge import Edge
class Vertex:
def __init__(self, label: str):
self.label = label
self.adjacency_list: List['Edge'] = []
File: edge.py
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from vertex import Vertex
class Edge:
def __init__(self, v1: 'Vertex', v2: 'Vertex'):
self.v1 = v1
self.v2 = v2
Quoted vs. Unquoted Type Hints
In versions of Python prior to 3.10, conditionally imported types must be enclosed in quotes, making them “forward references”, which hides them from the interpreter runtime.
In Python 3.7, 3.8, and 3.9, a workaround is to use the following special import.
from __future__ import annotations
This enables using unquoted type hints combined with conditional imports.
Python 3.10 (See PEP 563 -- Postponed Evaluation of Annotations)
In Python 3.10, function and variable annotations will no longer be
evaluated at definition time. Instead, a string form will be preserved
in the respective annotations dictionary. Static type checkers
will see no difference in behavior, whereas tools using annotations at
runtime will have to perform postponed evaluation.
The string form is obtained from the AST during the compilation step,
which means that the string form might not preserve the exact
formatting of the source. Note: if an annotation was a string literal
already, it will still be wrapped in a string.

The problem is that when running from a directory, by default only the packages that are sub directories are visible as candidate imports, so you cannot import a.b.d. You can however import b.d. since b is a sub package of a.
If you really want to import a.b.d in c/__init__.py you can accomplish this by changing the system path to be one directory above a and change the import in a/__init__.py to be import a.b.c.
Your a/__init__.py should look like this:
import sys
import os
# set sytem path to be directory above so that a can be a
# package namespace
DIRECTORY_SCRIPT = os.path.dirname(os.path.realpath(__file__))
sys.path.insert(0,DIRECTORY_SCRIPT+"/..")
import a.b.c
An additional difficulty arises when you want to run modules in c as scripts. Here the packages a and b do not exist. You can hack the __int__.py in the c directory to point the sys.path to the top-level directory and then import __init__ in any modules inside c to be able to use the full path to import a.b.d. I doubt that it is good practice to import __init__.py but it has worked for my use cases.

I suggest the following pattern. Using it will allow auto-completion and type hinting to work properly.
cyclic_import_a.py
import playground.cyclic_import_b
class A(object):
def __init__(self):
pass
def print_a(self):
print('a')
if __name__ == '__main__':
a = A()
a.print_a()
b = playground.cyclic_import_b.B(a)
b.print_b()
cyclic_import_b.py
import playground.cyclic_import_a
class B(object):
def __init__(self, a):
self.a: playground.cyclic_import_a.A = a
def print_b(self):
print('b1-----------------')
self.a.print_a()
print('b2-----------------')
You cannot import classes A & B using this syntax
from playgroud.cyclic_import_a import A
from playground.cyclic_import_b import B
You cannot declare the type of parameter a in class B __ init __ method, but you can "cast" it this way:
def __init__(self, a):
self.a: playground.cyclic_import_a.A = a

Another solution is to use a proxy for the d_file.
For example, let's say that you want to share the blah class with the c_file. The d_file thus contains:
class blah:
def __init__(self):
print("blah")
Here is what you enter in c_file.py:
# do not import the d_file !
# instead, use a place holder for the proxy of d_file
# it will be set by a's __init__.py after imports are done
d_file = None
def c_blah(): # a function that calls d_file's blah
d_file.blah()
And in a's init.py:
from b.c import c_file
from b.d import d_file
class Proxy(object): # module proxy
pass
d_file_proxy = Proxy()
# now you need to explicitly list the class(es) exposed by d_file
d_file_proxy.blah = d_file.blah
# finally, share the proxy with c_file
c_file.d_file = d_file_proxy
# c_file is now able to call d_file.blah
c_file.c_blah()