Let me first establish what working scenario.
main.py
module/file1.py
module/file2.py
main.py
import module.file1
print(module.file1)
module/file1.py
import module.file2
module/file2.py
import module.file1
Running python3 main.py gives me the following, which is fine.
<module 'module.file1' from '/project/module/file1.py'>
Now, if I change module/file2.py to have the following:
import module.file1 as testtt
I get this new output (error):
Traceback (most recent call last):
File "main.py", line 1, in <module>
import module.file1
File "/project/module/file1.py", line 1, in <module>
import module.file2
File "/project/module/file2.py", line 2, in <module>
import module.file1 as testtt
AttributeError: module 'module' has no attribute 'file2'
I'm guessing that python doesn't fully evaluate the imported module when simply importing, causing the circular reference to blow up only when you immediately use it within either of the two files.
I'd imagine I also would not get the error if I used the module in a function, since that would be evaluate when the function is actually called, like this:
import module.file1
def test():
print(module.file1)
What is the recommendation here? Should I just work to remove the circular reference? It seems like code smell anyway (existing code base).
Its an implementation detail. The import statement uses the __import__ function to do the work of finding and importing the module and then binds its returned module to the as testtt variable.
When doing a nested import like import module.file1 as testtt, __import__ returns the base module ("module"). Since the importer still needs to bind "file1" to the local namespace, it has to look up the submodule name "file1" on that object. Since the import of file1 is still in progress, it hasn't been bound to the "module" module yet.
It works in the import module.file1 case because file1 isn't bound to the local namespace and doesn't need a lookup.
There are many pitfalls with circular imports that will bedevil you throughout your code's life cycle. Good luck!
"import" is an executable statement, so you can just do the import inside the function
def test():
import module.file1
print(module.file1)
Related
I'm getting this error
Traceback (most recent call last):
File "/Users/alex/dev/runswift/utils/sim2014/simulator.py", line 3, in <module>
from world import World
File "/Users/alex/dev/runswift/utils/sim2014/world.py", line 2, in <module>
from entities.field import Field
File "/Users/alex/dev/runswift/utils/sim2014/entities/field.py", line 2, in <module>
from entities.goal import Goal
File "/Users/alex/dev/runswift/utils/sim2014/entities/goal.py", line 2, in <module>
from entities.post import Post
File "/Users/alex/dev/runswift/utils/sim2014/entities/post.py", line 4, in <module>
from physics import PostBody
File "/Users/alex/dev/runswift/utils/sim2014/physics.py", line 21, in <module>
from entities.post import Post
ImportError: cannot import name Post
and you can see that I use the same import statement further up and it works. Is there some unwritten rule about circular importing? How do I use the same class further down the call stack?
See also What happens when using mutual or circular (cyclic) imports in Python? for a general overview of what is allowed and what causes a problem WRT circular imports. See What can I do about "ImportError: Cannot import name X" or "AttributeError: ... (most likely due to a circular import)"? for techniques for resolving and avoiding circular dependencies.
I think the answer by jpmc26, while by no means wrong, comes down too heavily on circular imports. They can work just fine, if you set them up correctly.
The easiest way to do so is to use import my_module syntax, rather than from my_module import some_object. The former will almost always work, even if my_module included imports us back. The latter only works if my_object is already defined in my_module, which in a circular import may not be the case.
To be specific to your case: Try changing entities/post.py to do import physics and then refer to physics.PostBody rather than just PostBody directly. Similarly, change physics.py to do import entities.post and then use entities.post.Post rather than just Post.
When you import a module (or a member of it) for the first time, the code inside the module is executed sequentially like any other code; e.g., it is not treated any differently that the body of a function. An import is just a command like any other (assignment, a function call, def, class). Assuming your imports occur at the top of the script, then here's what's happening:
When you try to import World from world, the world script gets executed.
The world script imports Field, which causes the entities.field script to get executed.
This process continues until you reach the entities.post script because you tried to import Post
The entities.post script causes physics module to be executed because it tries to import PostBody
Finally, physics tries to import Post from entities.post
I'm not sure whether the entities.post module exists in memory yet, but it really doesn't matter. Either the module is not in memory, or the module doesn't yet have a Post member because it hasn't finished executing to define Post
Either way, an error occurs because Post is not there to be imported
So no, it's not "working further up in the call stack". This is a stack trace of where the error occurred, which means it errored out trying to import Post in that class. You shouldn't use circular imports. At best, it has negligible benefit (typically, no benefit), and it causes problems like this. It burdens any developer maintaining it, forcing them to walk on egg shells to avoid breaking it. Refactor your module organization.
To understand circular dependencies, you need to remember that Python is essentially a scripting language. Execution of statements outside methods occurs at compile time. Import statements are executed just like method calls, and to understand them you should think about them like method calls.
When you do an import, what happens depends on whether the file you are importing already exists in the module table. If it does, Python uses whatever is currently in the symbol table. If not, Python begins reading the module file, compiling/executing/importing whatever it finds there. Symbols referenced at compile time are found or not, depending on whether they have been seen, or are yet to be seen by the compiler.
Imagine you have two source files:
File X.py
def X1:
return "x1"
from Y import Y2
def X2:
return "x2"
File Y.py
def Y1:
return "y1"
from X import X1
def Y2:
return "y2"
Now suppose you compile file X.py. The compiler begins by defining the method X1, and then hits the import statement in X.py. This causes the compiler to pause compilation of X.py and begin compiling Y.py. Shortly thereafter the compiler hits the import statement in Y.py. Since X.py is already in the module table, Python uses the existing incomplete X.py symbol table to satisfy any references requested. Any symbols appearing before the import statement in X.py are now in the symbol table, but any symbols after are not. Since X1 now appears before the import statement, it is successfully imported. Python then resumes compiling Y.py. In doing so it defines Y2 and finishes compiling Y.py. It then resumes compilation of X.py, and finds Y2 in the Y.py symbol table. Compilation eventually completes w/o error.
Something very different happens if you attempt to compile Y.py from the command line. While compiling Y.py, the compiler hits the import statement before it defines Y2. Then it starts compiling X.py. Soon it hits the import statement in X.py that requires Y2. But Y2 is undefined, so the compile fails.
Please note that if you modify X.py to import Y1, the compile will always succeed, no matter which file you compile. However if you modify file Y.py to import symbol X2, neither file will compile.
Any time when module X, or any module imported by X might import the current module, do NOT use:
from X import Y
Any time you think there may be a circular import you should also avoid compile time references to variables in other modules. Consider the innocent looking code:
import X
z = X.Y
Suppose module X imports this module before this module imports X. Further suppose Y is defined in X after the import statement. Then Y will not be defined when this module is imported, and you will get a compile error. If this module imports Y first, you can get away with it. But when one of your co-workers innocently changes the order of definitions in a third module, the code will break.
In some cases you can resolve circular dependencies by moving an import statement down below symbol definitions needed by other modules. In the examples above, definitions before the import statement never fail. Definitions after the import statement sometimes fail, depending on the order of compilation. You can even put import statements at the end of a file, so long as none of the imported symbols are needed at compile time.
Note that moving import statements down in a module obscures what you are doing. Compensate for this with a comment at the top of your module something like the following:
#import X (actual import moved down to avoid circular dependency)
In general this is a bad practice, but sometimes it is difficult to avoid.
For those of you who, like me, come to this issue from Django, you should know that the docs provide a solution:
https://docs.djangoproject.com/en/1.10/ref/models/fields/#foreignkey
"...To refer to models defined in another application, you can explicitly specify a model with the full application label. For example, if the Manufacturer model above is defined in another application called production, you’d need to use:
class Car(models.Model):
manufacturer = models.ForeignKey(
'production.Manufacturer',
on_delete=models.CASCADE,
)
This sort of reference can be useful when resolving circular import dependencies between two applications...."
I was able to import the module within the function (only) that would require the objects from this module:
def my_func():
import Foo
foo_instance = Foo()
If you run into this issue in a fairly complex app it can be cumbersome to refactor all your imports. PyCharm offers a quickfix for this that will automatically change all usage of the imported symbols as well.
I was using the following:
from module import Foo
foo_instance = Foo()
but to get rid of circular reference I did the following and it worked:
import module.foo
foo_instance = foo.Foo()
According to this answer we can import another module's object in the block( like function/ method and etc), without circular import error occurring, for example for import Simple object of another.py module, you can use this:
def get_simple_obj():
from another import Simple
return Simple
class Example(get_simple_obj()):
pass
class NotCircularImportError:
pass
In this situation, another.py module can easily import NotCircularImportError, without any problem.
just check your file name see if it is not the same as library you are importing.
Eg - sympy.py
import sympy as sym
I've been working on a project where I have a file which needs to call a function from a file in a sub package/directory, which in turn is calling a function from another file in the same sub package. As such, I have a main file which is importing a sub file. This sub file is also importing another sub file which is in the same package.
The first sub file has no issue whatsoever importing the second sub file. The main file also has no issue importing the first sub file. However, when I put it all together and run the main file, Python thinks that the second sub file doesn't exist, which I find strange. I've simplified and visualised my problem with an example below:
I have the following file hierarchy:
test_package\
__init__.py
main_file.py
test_sub_package\
__init__.py
subfile1.py
subfile2.py
main_file code:
import test_sub_package.subfile1
subfile1 code:
import subfile2
subfile2 code:
def get_string():
return ("Hello, World!")
So, I would expect main_file to import subfile2 via subfile1. However this doesn't seem to be the case because I get an error:
Traceback (most recent call last):
File "...\Test\main_file.py", line 1, in <module>
import test_package.subfile1
File "...\Test\test_sub_package\subfile1.py", line 1, in <module>
import subfile2
ModuleNotFoundError: No module named 'subfile2'
I was a little surprised that I got this error before I even attempted to call the functionality in subfile2. Either way, I'm confused why this doesn't work. Am I just doing something stupid here or am I trying to do something Python fundamentally doesn't support. If anyone can give me a solution it would be most appreciated.
I suspect this is probably a duplicate but I couldn't find an answer to my specific problem. So, sorry in advance.
When you import a module into another module from the same directory you must use must use a relative import in subfile1.py you will need to write:
from . import subfile2
Note, that doesn't give subfile 1 access to get_string to use it in subfile1, you would need to either write subfile2.get_string() or import it directly with:
from .subfile2 import get_string
I have tried this out and it works, I hope this helps :)
Note: that, if you are running a python script, and you need to import a module in that same directory, you can just say import module_name. It makes a difference if it is a script you are running, or a module that is being used in some other script. For a detailed explanation as to why see here
(I assume from your error message that you want to run main.py, if this is not the case you will need to change import test_sub_package.subfile1 to from . import test_sub_package.subfile1)
main file should be:
from test_sub_package.subfile1 import get_string
get_string()
subfile1.py
import test_sub_package.subfile2
I am trying to do a python script that it is divided in multiple files, so I can maintain it more easily instead of making a very-long single file script.
Here is the directory structure:
wmlxgettext.py
<pywmlx>
|- __init__.py
|- (some other .py files)
|- <state>
|- __init__.py
|- state.py
|- machine.py
|- lua_idle.py
if I reach the main directory of my project (where the wmlxgettext.py script is stored) and if I try to "import pywmlx" I have an import error (Attribute Error: 'module' object has no attribute 'state')
Here is the complete error message:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/user/programmi/my/python/wmlxgettext/true/pywmlx/__init__.py", line 9, in <module>
import pywmlx.state as statemachine
File "/home/user/programmi/my/python/wmlxgettext/true/pywmlx/state/__init__.py", line 1, in <module>
from pywmlx.state.machine import setup
File "/home/user/programmi/my/python/wmlxgettext/true/pywmlx/state/machine.py", line 2, in <module>
from pywmlx.state.lua_idle import setup_luastates
File "/home/user/programmi/my/python/wmlxgettext/true/pywmlx/state/lua_idle.py", line 3, in <module>
import pywmlx.state.machine as statemachine
AttributeError: 'module' object has no attribute 'state'
Since I am in the "project main directory" pywmlx should be on PYTHONPATH (infact I have no troubles when I tried to import pywmlx/something.py)
I'm not able to figure where is my error and how to solve this problem.
Here is the pywmlx/__init__.py source:
# all following imports works well:
from pywmlx.wmlerr import ansi_setEnabled
from pywmlx.wmlerr import wmlerr
from pywmlx.wmlerr import wmlwarn
from pywmlx.postring import PoCommentedString
from pywmlx.postring import WmlNodeSentence
from pywmlx.postring import WmlNode
# this is the import that does not work:
import pywmlx.state as statemachine
Here is the pywmlx/state/__init__.py source:
from pywmlx.state.machine import setup
from pywmlx.state.machine import run
But I think that the real problem is somewhat hidden in the "imports" used by one (or all) python modules stored in pywmlx/state directory.
Here is the pywmlx/state/machine.py source:
# State is a "virtual" class
from pywmlx.state.state import State
from pywmlx.state.lua_idle import setup_luastates
import pywmlx.nodemanip as nodemanip
def addstate(self, name, value):
# code is not important for this question
pass
def setup():
setup_luastates()
def run(self, *, filebuf, fileref, fileno, startstate, waitwml=True):
# to do
pass
Finally here is the pywmlx/state/lua_idle.py source:
import re
import pywmlx.state.machine as statemachine
# State is a "virtual" class
from pywmlx.state.state import State
# every state is a subclass of State
# all proprieties were defined originally on the base State class:
# self.regex and self.iffail were "None"
# the body of "run" function was only "pass"
class LuaIdleState (State):
def __init__(self):
self.regex = re.compile(r'--.*?\s*#textdomain\s+(\S+)', re.I)
self.iffail = 'lua_checkpo'
def run(xline, match):
statemachine._currentdomain = match.group(1)
xline = None
return (xline, 'lua_idle')
def setup_luastates():
statemachine.addstate('lua_idle', LuaIdleState)
Sorry if I posted so much code and so many files... but I fear that the files, in directory, hides more than a single import problem, so I published them all, hoping that I could explain the problem avoiding confusion.
I think that I miss something about how import works in python, so I hope this question can be useful also to other programmers, becouse I think I am not the only one who found the official documentation very difficult to understand when explaining import.
Searches Done:
Not Useful: I am already explicitly using import x.y.z all times I need to import something
Not Useful: Even if the question asks about import errors, it seems not useful for the same reason as (1)
Not Useful: As far as I know, pywmlx should be located into PYTHONPATH since "current working directory" on my tests is the directory that contains the main python script and pywmlx directory. Correct me if I am wrong
Python does several things when importing packages:
Create an object in sys.modules for the package, with the name as key: 'pywmlx', 'pywmlx.state', 'pywmlx.state.machine', etc.
Run the bytecode loaded for that module; this may create more modules.
Once a module is fully loaded and it is located inside another package, set the module as an attribute of the parent module object. Thus the sys.modules['pywmlx.state'] module is set as the state attribute on the sys.modules['pywmlx'] module object.
That last step hasn't taken place yet in your example, but the following line only works when it has been set:
import pywmlx.state.machine as statemachine
because this looks up both state and machine as attributes first. Use this syntax instead:
from pywmlx.state import machine as statemachine
Alternatively, just use
import pywmlx.state.machine
and replace statemachine. everywhere else with pywmlx.state.machine.. This works because all that is added to your namespace is a reference to the sys.modules['pywmlx'] module object and the attribute references won't need to be resolved until you use that reference in the functions and methods.
You are having a circular import in your framework. Circular imports do not work well with aliases. When importing a module with an alias and then, during the circular import, importing it again without an alias, python complains. The solution is to not use aliases (the "import module as" syntax) but always use the full "import module" statement.
I'm getting this error
Traceback (most recent call last):
File "/Users/alex/dev/runswift/utils/sim2014/simulator.py", line 3, in <module>
from world import World
File "/Users/alex/dev/runswift/utils/sim2014/world.py", line 2, in <module>
from entities.field import Field
File "/Users/alex/dev/runswift/utils/sim2014/entities/field.py", line 2, in <module>
from entities.goal import Goal
File "/Users/alex/dev/runswift/utils/sim2014/entities/goal.py", line 2, in <module>
from entities.post import Post
File "/Users/alex/dev/runswift/utils/sim2014/entities/post.py", line 4, in <module>
from physics import PostBody
File "/Users/alex/dev/runswift/utils/sim2014/physics.py", line 21, in <module>
from entities.post import Post
ImportError: cannot import name Post
and you can see that I use the same import statement further up and it works. Is there some unwritten rule about circular importing? How do I use the same class further down the call stack?
See also What happens when using mutual or circular (cyclic) imports in Python? for a general overview of what is allowed and what causes a problem WRT circular imports. See What can I do about "ImportError: Cannot import name X" or "AttributeError: ... (most likely due to a circular import)"? for techniques for resolving and avoiding circular dependencies.
I think the answer by jpmc26, while by no means wrong, comes down too heavily on circular imports. They can work just fine, if you set them up correctly.
The easiest way to do so is to use import my_module syntax, rather than from my_module import some_object. The former will almost always work, even if my_module included imports us back. The latter only works if my_object is already defined in my_module, which in a circular import may not be the case.
To be specific to your case: Try changing entities/post.py to do import physics and then refer to physics.PostBody rather than just PostBody directly. Similarly, change physics.py to do import entities.post and then use entities.post.Post rather than just Post.
When you import a module (or a member of it) for the first time, the code inside the module is executed sequentially like any other code; e.g., it is not treated any differently that the body of a function. An import is just a command like any other (assignment, a function call, def, class). Assuming your imports occur at the top of the script, then here's what's happening:
When you try to import World from world, the world script gets executed.
The world script imports Field, which causes the entities.field script to get executed.
This process continues until you reach the entities.post script because you tried to import Post
The entities.post script causes physics module to be executed because it tries to import PostBody
Finally, physics tries to import Post from entities.post
I'm not sure whether the entities.post module exists in memory yet, but it really doesn't matter. Either the module is not in memory, or the module doesn't yet have a Post member because it hasn't finished executing to define Post
Either way, an error occurs because Post is not there to be imported
So no, it's not "working further up in the call stack". This is a stack trace of where the error occurred, which means it errored out trying to import Post in that class. You shouldn't use circular imports. At best, it has negligible benefit (typically, no benefit), and it causes problems like this. It burdens any developer maintaining it, forcing them to walk on egg shells to avoid breaking it. Refactor your module organization.
To understand circular dependencies, you need to remember that Python is essentially a scripting language. Execution of statements outside methods occurs at compile time. Import statements are executed just like method calls, and to understand them you should think about them like method calls.
When you do an import, what happens depends on whether the file you are importing already exists in the module table. If it does, Python uses whatever is currently in the symbol table. If not, Python begins reading the module file, compiling/executing/importing whatever it finds there. Symbols referenced at compile time are found or not, depending on whether they have been seen, or are yet to be seen by the compiler.
Imagine you have two source files:
File X.py
def X1:
return "x1"
from Y import Y2
def X2:
return "x2"
File Y.py
def Y1:
return "y1"
from X import X1
def Y2:
return "y2"
Now suppose you compile file X.py. The compiler begins by defining the method X1, and then hits the import statement in X.py. This causes the compiler to pause compilation of X.py and begin compiling Y.py. Shortly thereafter the compiler hits the import statement in Y.py. Since X.py is already in the module table, Python uses the existing incomplete X.py symbol table to satisfy any references requested. Any symbols appearing before the import statement in X.py are now in the symbol table, but any symbols after are not. Since X1 now appears before the import statement, it is successfully imported. Python then resumes compiling Y.py. In doing so it defines Y2 and finishes compiling Y.py. It then resumes compilation of X.py, and finds Y2 in the Y.py symbol table. Compilation eventually completes w/o error.
Something very different happens if you attempt to compile Y.py from the command line. While compiling Y.py, the compiler hits the import statement before it defines Y2. Then it starts compiling X.py. Soon it hits the import statement in X.py that requires Y2. But Y2 is undefined, so the compile fails.
Please note that if you modify X.py to import Y1, the compile will always succeed, no matter which file you compile. However if you modify file Y.py to import symbol X2, neither file will compile.
Any time when module X, or any module imported by X might import the current module, do NOT use:
from X import Y
Any time you think there may be a circular import you should also avoid compile time references to variables in other modules. Consider the innocent looking code:
import X
z = X.Y
Suppose module X imports this module before this module imports X. Further suppose Y is defined in X after the import statement. Then Y will not be defined when this module is imported, and you will get a compile error. If this module imports Y first, you can get away with it. But when one of your co-workers innocently changes the order of definitions in a third module, the code will break.
In some cases you can resolve circular dependencies by moving an import statement down below symbol definitions needed by other modules. In the examples above, definitions before the import statement never fail. Definitions after the import statement sometimes fail, depending on the order of compilation. You can even put import statements at the end of a file, so long as none of the imported symbols are needed at compile time.
Note that moving import statements down in a module obscures what you are doing. Compensate for this with a comment at the top of your module something like the following:
#import X (actual import moved down to avoid circular dependency)
In general this is a bad practice, but sometimes it is difficult to avoid.
For those of you who, like me, come to this issue from Django, you should know that the docs provide a solution:
https://docs.djangoproject.com/en/1.10/ref/models/fields/#foreignkey
"...To refer to models defined in another application, you can explicitly specify a model with the full application label. For example, if the Manufacturer model above is defined in another application called production, you’d need to use:
class Car(models.Model):
manufacturer = models.ForeignKey(
'production.Manufacturer',
on_delete=models.CASCADE,
)
This sort of reference can be useful when resolving circular import dependencies between two applications...."
I was able to import the module within the function (only) that would require the objects from this module:
def my_func():
import Foo
foo_instance = Foo()
If you run into this issue in a fairly complex app it can be cumbersome to refactor all your imports. PyCharm offers a quickfix for this that will automatically change all usage of the imported symbols as well.
I was using the following:
from module import Foo
foo_instance = Foo()
but to get rid of circular reference I did the following and it worked:
import module.foo
foo_instance = foo.Foo()
According to this answer we can import another module's object in the block( like function/ method and etc), without circular import error occurring, for example for import Simple object of another.py module, you can use this:
def get_simple_obj():
from another import Simple
return Simple
class Example(get_simple_obj()):
pass
class NotCircularImportError:
pass
In this situation, another.py module can easily import NotCircularImportError, without any problem.
just check your file name see if it is not the same as library you are importing.
Eg - sympy.py
import sympy as sym
I know that Python discourages any situation which can get you into a circular import. But I wanted understand the Python internals of why from-imports are seemingly arbitrarily less forgiving than normal imports in circular import situations.
For example, this code compiles:
# main.py
import CommonUtil
# commonutil.py
import util
class CommonUtil:
# some code that uses util.Util
pass
# util.py
import commonutil
class Util:
# some code that uses commonutil.CommonUtil
pass
But this code does not:
# main.py
import CommonUtil
# commonutil.py
import util
class CommonUtil:
# some code that uses util.Util
pass
# util.py
from commonutil import CommonUtil
class Util:
# some code that uses CommonUtil
pass
Traceback (most recent call last):
File "main.py", line 1, in <module>
import CommonUtil
File "commonutil.py", line 1, in <module>
import util
File "util.py", line 1, in <module>
from commonutil import CommonUtil
ImportError: cannot import name CommonUtil
You don't hit compiler errors as long as you don't try to use the relevant classes before all the imports have completed. But when you try to do some aliasing, then it fails. Can someone explain what's going on internally in the Python that causes this error to rear its head only when from-import is used? And secondarily, is there any easy way around this? (Besides the obvious "pull shared code out to a third module," which I'll likely do anyways.)
Modules are executed from top to bottom. When an import is seen for the first time, execution of the current module is suspended so that the other module can be imported. When the other module attempts to import the first module, it gets a reference to the currently partially-executed module. Since code located after the import of the other module hasn't yet been executed, any names contained within it cannot yet exist.
main.py
import a
a.py
var1 = 'foo'
import b
var2 = 'bar'
b.py
import a
print a.var1 # works
print a.var2 # fails
The way around it is to not access the names in the imported module until its execution has completed.
see http://effbot.org/zone/import-confusion.htm#circular-imports for an explanation of what is happening.
I assume you launch the main.py file. Python will first try to load commonutil. It will create a module object, and start filling it with class and function and global variable when encountering their definition. The first statement is an import, so now python creates the util module and start filling it. The common module exist, but is empty. In the first version, you do not access any commonutil object at load time, so everything is fine. In the second one, you try to fetch a specific variable in commonutil which does not exist at this moment. If you had used something like f(commonutil.CommonUtil) in the first version, it would have also crashed.