Develop Reference

Detect circular imports in Python [duplicate]

I'm working with a project that contains about 30 unique modules. It wasn't designed too well, so it's common that I create circular imports when adding some new functionality to the project.
Of course, when I add the circular import, I'm unaware of it. Sometimes it's pretty obvious I've made a circular import when I get an error like AttributeError: 'module' object has no attribute 'attribute' where I clearly defined 'attribute'. But other times, the code doesn't throw exceptions because of the way it's used.
So, to my question:
Is it possible to programmatically detect when and where a circular import is occuring?
The only solution I can think of so far is to have a module importTracking that contains a dict importingModules, a function importInProgress(file), which increments importingModules[file], and throws an error if it's greater than 1, and a function importComplete(file) which decrements importingModules[file]. All other modules would look like:
import importTracking
importTracking.importInProgress(__file__)
#module code goes here.
importTracking.importComplete(__file__)
But that looks really nasty, there's got to be a better way to do it, right?

To avoid having to alter every module, you could stick your import-tracking functionality in a import hook, or in a customized __import__ you could stick in the built-ins -- the latter, for once, might work better, because __import__ gets called even if the module getting imported is already in sys.modules, which is the case during circular imports.
For the implementation I'd simply use a set of the modules "in the process of being imported", something like (benjaoming edit: Inserting a working snippet derived from original):
beingimported = set()
originalimport = __import__
def newimport(modulename, *args, **kwargs):
if modulename in beingimported:
print "Importing in circles", modulename, args, kwargs
print " Import stack trace -> ", beingimported
# sys.exit(1) # Normally exiting is a bad idea.
beingimported.add(modulename)
result = originalimport(modulename, *args, **kwargs)
if modulename in beingimported:
beingimported.remove(modulename)
return result
import __builtin__
__builtin__.__import__ = newimport

Not all circular imports are a problem, as you've found when an exception is not thrown.
When they are a problem, you'll get an exception the next time you try to run any of your tests. You can change the code when this happens.
I don't see any change required from this situation.
Example of when it's not a problem:
a.py
import b
a = 42
def f():
return b.b
b.py
import a
b = 42
def f():
return a.a

Circular imports in Python are not like PHP includes.
Python imported modules are loaded the first time into an import "handler", and kept there for the duration of the process. This handler assigns names in the local namespace for whatever is imported from that module, for every subsequent import. A module is unique, and a reference to that module name will always point to the same loaded module, regardless of where it was imported.
So if you have a circular module import, the loading of each file will happen once, and then each module will have names relating to the other module created into its namespace.
There could of course be problems when referring to specific names within both modules (when the circular imports occur BEFORE the class/function definitions that are referenced in the imports of the opposite modules), but you'll get an error if that happens.

import uses __builtin__.__import__(), so if you monkeypatch that then every import everywhere will pick up the changes. Note that a circular import is not necessarily a problem though.

Getting Variables from Module Which is Being Imported

main:
import fileb
favouritePizza = "pineapple"
fileb.eatPizza
fileb:
from main import favouritePizza
def eatPizza():
print("favouritePizza")
This is what I tried, however, I get no such attribute. I looked at other problems and these wouldn't help.

This is classic example of circular dependency. main imports fileb, while fileb requires main.
Your case is hard (impossible?) to solve even in theory. In reality, python import machinery does even less expected thing — every time you import anything from some module, whole module is read and imported into global(per process) module namespace. Actually, from module import function is just a syntax sugar that gives you ability to not litter your namespace with everything form particular module (from module import *), but behind the scenes, its the almost the same as import module; module.function(...).
From composition/architecture point of view, basic program structure is:
top modules import bottom
bottom modules get parameters from top when being called
You probably want to use favouritePizza variable somehow in fileb, i.e. in eatPizza functions. Good thing to do is to make this function accept parameter, that will be passed from any place that uses it:
# fileb.py
def eatPizza(name):
print(name)
And call it with
# main.py
import fileb
favouritePizza = "pineapple"
fileb.eatPizza(favouritePizza)

How does the Python executable parse and execute scripts? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
Let's say I have the following script, test.py:
import my_library
bar = 12
def foo():
nested_bar = 21
my_library.do_things()
def nested_foo():
nested_bar += 11
not_a_variable += 1
{$ invalid_syntax
bar = 13
foo()
bar = 14
I'm curious as to what exactly happens when I run python test.py. Obviously Python doesn't just read programs line-by-line - otherwise it wouldn't catch syntax errors before actually executing the program. But this makes the workings of the interpreter seem somewhat nebulous. I was wondering if someone would help clear things up for me. In particular, I would like to know:
At what point does Python realize there is a syntax error on line 13?
At what point does Python read the nested functions and add them to the scope of foo?
Similarly, how does Python add the function foo to its namespace when it encounters it, without executing it?
Suppose my_library were an invalid import. Would Python necessarily raise an ImportError before executing any other commands?
Suppose my_library were a valid module, but it has no function do_things. At what point would Python realize this, during execution of foo() or before?
If anyone could point me to documentation on how Python parses and executes scripts it would be very much appreciated.

There's some information in the tutorial's section on modules, but I don't think the documentation has a complete reference for this. So, here's what happens.
When you first run a script or import a module, Python parses the syntax into an AST and then compiles that into bytecode. It hasn't executed anything yet; it's just compiled your code into instructions for a little stack-based machine. This is where syntax errors are caught. (You can see the guts of all this in the ast module, the token module, the compile builtin, the grammar reference, and sprinkled around various other places.)
You can actually compile a module independently of running the generated code; that's what the builtin compileall method does.
So that's the first phase: compiling. Python only has one other phase, which is actually running the code. Every statement in your module, except those contained within def or lambda, is executed in order. That means that imports happen at runtime, wherever you happen to put them in your module. Which is part of the reason it's good hygiene to put them all at the top. Same for def and class: these are just statements that create a specific type of object, and they're executed as they're encountered, like anything else.
The only tricky bit here is that the phases can happen more than once — for example, an import is only executed at runtime, but if you've never imported that module before, then it has to be compiled, and now you're back in compile time. But "outside" the import it's still runtime, which is why you can catch a SyntaxError thrown by an import.
Anyway, to answer your specific questions:
At compile time. When you run this as a script, or when you import it as a module, or when you compile it with compileall, or otherwise ask Python to make any sense of it. In practical terms, this can happen at any time: if you tried to import this module within a function, you'd only get a SyntaxError when calling that function, which might be halfway through your program.
During the execution of foo, because def and class just create a new object and assign it to a name. But Python still knows how to create the nested function, because it's already compiled all the code within it.
The same way it would add foo = lambda: 1 + 2 to a namespace without executing it. A function is just an object that contains a "code" attribute — literally just a block of Python bytecode. You can manipulate the code type as data, because it is data, independently of executing it. Try looking at a function's .__code__, read the "code objects" section of the data model, or even play around with the disassembler. (You can even execute a code object directly with custom locals and globals using exec, or change the code object a function uses!)
Yes, because import is a plain old statement like any other, executed in order. But if there were other code before the import, that would run first. And if it were in a function, you wouldn't get an error until that function ran. Note that import, just like def and class, is just a fancy form of assignment.
Only during the execution of foo(). Python has no way of knowing whether other code will add a do_things to your module before that point, or even change my_library to some other object entirely. Attribute lookups are always done just-in-time, when you ask for them, never in advance.

As a general rule, python first parses the file, compiles the abstract syntax tree to byte code, then attempt to execute it sequentially. That means all statements are executed line by line. Thus, this means:
Syntax errors are caught at parse time, before anything is executed. If you add some side effect to the script, e.g. create a file, you will see that it never gets executed.
A function becomes defined in the scope after the definition. If you try to call nested_foo right before def nested_foo() you will see that it would fail because nested_foo has not been defined at that point.
Same as 2.
If python cannot import a library, where import means it tries to execute the module, then it fails with an ImportError.
Since you don't try to access do_things at import time (i.e. you are not doing from my_library import do_things), an error only occurs when you attempt to call foo().

Auto-imports or other code-less way of using custom functions

I have written the odd handy function while I've been doing python. (A few methods on lists, couple of more useful input functions ect.)
What I want is to be able to access these functions from a python file without having to import a module through import my_module.
The way I though it would happen is automatically importing a module, or putting these functions in another default python module, but I don't really mind how it's done.
Can anyone shed some light on how I should do this?
(I know import my_module is not a lot, but you could end up with
sys.path.append("c:\fake\path")
from my_module import *
which is getting long...)

The python module site is imported automatically whenever the python interpreter is run. This module attempts to import another, named sitecustomize. You could put your functions in there, adding them to the __builtins__ mapping with:
for func in [foo, bar, baz]:
__builtins__[func.__name__] = func
Note that this only works on cpython, where __builtins__ is a mutable dict. Doing so will automatically make these functions available to all your python code for this installation.
I strongly would discourage you from doing so! Implicit is never better than explicit, and anyone maintaining your code will wonder where the hell these came from.
You'd be better off with a from myglobalutils import * import in your modules.

See import this (it says a lot in a few lines). You can import a module which has imported the the other ones, for example:
import my.main # where main.py contains `import X, Y, Z`
# X, Y, and Z access:
my.main.X
my.main.Y
my.main.Z

Python import trouble

I am having some trouble importing a class from a particular module. The class is in the module my_module1.my_module2.my_module3.my_module4.my_module5.my_module6.my_module7
This code works
import my_module1.my_module2.my_module3.my_module4.my_module5.my_module6.my_module7
which means to access my class I have to do
my_module1.my_module2.my_module3.my_module4.my_module5.my_module6.my_module7.MyClass
But this does not
from my_module1.my_module2.my_module3.my_module4.my_module5.my_module6.my_module7 import MyClass
Neither does this
import my_module1.my_module2.my_module3.my_module4.my_module5.my_module6.my_module7 as my_name
Both Give this error saying
AttributeError: 'module' object has no attribute my_module7'
This has me completely stumped and I have been working on it for a couple of weeks now. Any suggestions?
EDIT - I cant change the structure unfortunately as it is imposed by the system I am using

Looks like a circular import.
Gordon McMillan says:
Circular imports are fine where both modules use the “import ” form of import. They fail when the 2nd module wants to grab a name out of the first (“from module import name”) and the import is at the top level. That’s because names in the 1st are not yet available, because the first module is busy importing the 2nd.

I think you may want to consider an alternate design in the first place (redesigning your module breakdown so it's a flatter setup), but as that isn't your question try:
import my_module1.my_module2.my_modu...<etc>
my_name = my_module1.my_module2.my_modu...<etc>
my_name.MyClass
A module is a first class object in python, so you can alias them with variables.