Develop Reference

how does import in python work? When import does python run the code in it? [duplicate]

I have two specific situations where I don't understand how importing works in Python:
1st specific situation:
When I import the same module in two different Python scripts, the module isn't imported twice, right? The first time Python encounters it, it is imported, and second time, does it check if the module has been imported, or does it make a copy?
2nd specific situation:
Consider the following module, called bla.py:
a = 10
And then, we have foo.py, a module which imports bla.py:
from bla import *
def Stuff ():
return a
And after that, we have a script called bar.py, which gets executed by the user:
from foo import *
Stuff() #This should return 10
a = 5
Stuff()
Here I don't know: Does Stuff() return 10 or 5?

Part 1
The module is only loaded once, so there is no performance loss by importing it again. If you actually wanted it to be loaded/parsed again, you'd have to reload() the module.
The first place checked is sys.modules, the cache of all modules that have been imported previously. [source]
Part 2
from foo import * imports a to the local scope. When assigning a value to a, it is replaced with the new value - but the original foo.a variable is not touched.
So unless you import foo and modify foo.a, both calls will return the same value.
For a mutable type such as a list or dict it would be different, modifying it would indeed affect the original variable - but assigning a new value to it would still not modify foo.whatever.
If you want some more detailed information, have a look at http://docs.python.org/reference/executionmodel.html:
The following constructs bind names: formal parameters to functions, import statements, class and function definitions (these bind the class or function name in the defining block), and targets that are identifiers if occurring in an assignment, for loop header, in the second position of an except clause header or after as in a with statement.
The two bold sections are the relevant ones for you: First the name a is bound to the value of foo.a during the import. Then, when doing a = 5, the name a is bound to 5. Since modifying a list/dict does not cause any binding, those operations would modify the original one (b and foo.b are bound to the same object on which you operate). Assigning a new object to b would be a binding operation again and thus separate b from foo.b.
It is also worth noting what exactly the import statement does:
import foo binds the module name to the module object in the current scope, so if you modify foo.whatever, you will work with the name in that module - any modifications/assignments will affect the variable in the module.
from foo import bar binds the given name(s) only (i.e. foo will remain unbound) to the element with the same name in foo - so operations on bar behave like explained earlier.
from foo import * behaves like the previous one, but it imports all global names which are not prefixed with an underscore. If the module defines __all__ only names inside this sequence are imported.
Part 3 (which doesn't even exist in your question :p)
The python documentation is extremely good and usually verbose - you find answer on almost every possible language-related question in there. Here are some useful links:
http://docs.python.org/reference/datamodel.html (classes, properties, magic methods, etc.) ()
http://docs.python.org/reference/executionmodel.html (how variables work in python)
http://docs.python.org/reference/expressions.html
http://docs.python.org/reference/simple_stmts.html (statements such as import, yield)
http://docs.python.org/reference/compound_stmts.html (block statements such as for, try, with)

To answer your first question:
No, python does not get 'imported' twice. When python loads a module, it checks for the module in sys.modules. If it is not in there, it is put in there, and loaded.
To answer your second question:
Modules can define what names they will export to a from camelot import * scenario, and the behavior is to create names for the existing values, not to reference existing variables (python does not have references).
On a somewhat related topic, doing a from camelot import * is not the same as a regular import.

python: state of imported global variables holding a reference to lambda function

Backstory:
I was trying implementing one way to handle -v parameters to increase the verbosity of an application. To that end, I wanted to use a global variable that is pointing to an empty lambda function initially. If -v is given the variable is changed and gets another lambda function assigned that does print it input.
MWE:
I noticed that this did not work as expected when calling the lambda function from another module after importing it via from x import *...
mwe.py:
from mod import *
import mod
def f():
vprint("test in f")
vprint("test before")
print("before: %d" % foo)
set_verbosity(1)
vprint("test after")
print("after: %d" % foo)
f()
mod.vprint("explicit: %d" % mod.foo)
modf()
mod.py:
vprint = lambda *a, **k: None
foo = 42
def set_verbosity(verbose):
global vprint, foo
if verbose > 0:
vprint = lambda *args, **kwargs: print(*args, **kwargs)
foo = 0
def modf():
vprint("modf: %d" % foo)
The output is
before: 42
after: 42
explicit: 0
modf: 0
where the "explicit" and "modf" outputs are due to the mod.vprint and modf calls at the end of the mwe. All other invocations of vprint (that go through the imported version of vprint) are apparently not using the updated definition. Likewise, the value of foo seems to be imported only once.
Question:
It looks to me as if the from x import * type of imports copies the state of the globals of the imported module. I am not so much interested in workarounds per se but the actual reason for this behavior. Where is this defined in the documentation and what's the rationale?
Workaround:
As a side note, one way to implement this anyway is to wrap the global lambda variables by functions and export only them:
_vprint = lambda *a, **k: None
def vprint(*args, **kwargs):
_vprint(*args, **kwargs)
def set_verbosity(verbose):
global _vprint
if verbose > 0:
_vprint = lambda *args, **kwargs: print(*args, **kwargs)
This makes it work with the import-from way that allows the other module to simply call vprint instead of explicitly deferencing via the module's name.

TL;DR: when you do from module import *, you're copying the names and their associated references; changing the reference associated with the original name does not change the reference associated with the copy.
This deals with the underlying difference between names and references. The underlying reason for the behavior has to do with how python handles such things.
Only one thing is truly immutable in python: memory. You can't change individual bytes directly. However, almost everything in python deals with references to individual bytes, and you can change those references. When you do my_list[2] = 5, you're not changing any memory - rather, you're allocating a new block of memory to hold the value 5, and pointing the second index of my_list to it. The original data that my_list[2] used to be pointing at is still there, but since nothing refers to it any more, the garbage collector will take care of it eventually and free the memory it was using.
The same principle goes with names. Any given namespace in python is comparable to a dict - each name has a corresponding reference. And this is where the problems come in.
Consider the difference between the following two statements:
from module import *
import module
In both cases, module is loaded into memory.
In the latter case, only one thing is added to the local namespace - the name 'module', which references the entire memory block containing the module that was just loaded. Or, well, it references the memory block of that module's own namespace, which itself has references to all the names in the module, and so on all the way down.
In the former case, however, every name in module's namespace is copied into the local namespace. The same block of memory still exists, but instead of having one reference to all of it, you now have many references to small parts of it.
Now, let's say we do both of those statements in succession:
from module import *
import module
This leaves us with one name 'module' referencing all the memory the module was loaded into, and a bunch of other names that reference individual parts of that block. We can verify that they point to the same thing:
print(module.func_name == func_name)
# True
But now, we try to assign something else to module.attribute:
module.func_name = lambda x:pass
print(module.func_name == func_name)
# False
It's no longer the same. Why?
Well, when we did module.func_name = lambda x:pass, we first allocated some memory to store lambda x:pass, and then we changed module's 'func_name' name to reference that memory instead of what it was referencing. Note that, like the example I gave earlier with lists, we didn't change the thing that the module.func_name was previously referencing - it still exists, and the local func_name continues to reference it.
So when you do from module import *, you're copying the names and their associated references; changing the reference associated with the original name does not change the reference associated with the copy.
The workaround for this is to not do import *. In fact, this is pretty much the ultimate reason why using import * is usually considered poor practice, save for a handful of special cases. Consider the following code:
# module.py
variable = "Original"
# file1.py
import module
def func1():
module.variable = "New"
# file2.py
import module
import file1
print(module.variable)
file1.func1()
print(module.variable)
When you run python file2.py, you get the following output:
Original
New
Why? Because file1 and file2 both imported module, and in both of their namespaces 'module' is pointing to the same block of memory. module's namespace contains a name 'variable' referencing some value. Then, the following things happen:
file2 says "okay, module, please give me the value associated with the name 'variable' in your namespace.
file1.func1() says "okay, module, the name 'variable' in your namespace now references this other value.
file2 says "okay, module, please give me the value associated with the name 'variable' in your namespace.
Since file1 and file2 are still both talking to the same bit of memory, they stay coordinated.

Random stab in the dark ahead:
If you look at the docs for __import__, there's the bit:
On the other hand, the statement from spam.ham import eggs, sausage as saus results in
_temp = __import__('spam.ham', globals(), locals(), ['eggs', 'sausage'], 0)
eggs = _temp.eggs
saus = _temp.sausage
I think this is the key. If we infer that from mod import * results in something like:
_temp = __import__('mod', globals(), locals(), [], 0)
printv = _temp.printv
foo = _temp.foo
This shows what the problem is. printv is a reference to the old version of printv; what mod.printv was pointing to at the time of import. Reassigning what the printv in mod is pointing to doesn't effect anything in mwe, because the mwe reference to printv is still looking at the previous lambda.
It's similar to how this doesn't change b:
a = 1
b = a
a = 2
b is still pointing to 1, because reassigning a doesn't effect what b is looking at.
On the other hand, mod.printv does work because we are now using a direct reference to the global in mod instead of a reference that points to printv in mod.
This was a random stab because I think I know the answer based on some random reading I did awhile ago. If I'm incorrect, please let me know and I'll remove this to avoid confusion.

Python cross module variables and imports

I see that there are other questions related to cross module variables, but they don't really fully answer my question.
I have an application that I have split into 3 modules + 1 main application, mainly for ease of readability and maintainability.
2 of these modules have threads with variables that need to be modified from other modules and other module threads.
Whilst I can modify a module's variable from the main code, I don't appear to be able to modify one module's variable from another module unless I import every module into every other module.
The example below where a&b are imported into main and a module a needs to access a variable in module b:
main
module a
var a
module b
var a
main
a.a = 1
b.a = 2
module a
b.a = 3
module b
a.a = 0
without importing module a into module b and importing module b into module a, can this be achieved globally through the main program ?
If I do have to import a and b into main, and then import a into b and b into a, what are the implications in terms of memory and resource usage / speed etc ?
I tried the suggestion from #abarnert:
#moda
vara = 10
#modb
print(str(vara))
#main
import moda
from moda import vara
import modb
however I get "name error vara is not defined"

If the code in the modules are defined as classes, and the main program creates instances of these classes, the main program can pass an instance of one module class to another, and changes to that instance will be reflected everywhere. There would be no need to import a or b into each other, because they would simply have references to each other.

If I do have to import a and b into main, and then import a into b and b into a, what are the implications in terms of memory and resource usage / speed etc ?
Absolutely none for memory—every module that imports a will get a reference to the exact same a module object. All you're doing is increasing its refcount, not creating new objects.
For speed, the time to discover that you're trying to import a module that already exists is almost nothing (it's just looking up the module name in a dictionary). It is slightly slower to access a.a than to just access a. But this is very rarely an issue. If it is, you're almost certainly going to want to copy that value into the locals of whatever function is accessing it over and over, at which point it won't matter which globals it came from.
without importing module a into module b and importing module b into module a, can this be achieved globally through the main program ?
Sure. All you have to do is import (with from a import a or import a.a as aa or whatever) or copy the variables from module a into main.
Note that just makes a new name for each value; it doesn't make references to the variables. There is no such thing as a reference to a variable in Python.
This works if the variables are holding constants, or if they're holding mutable values that you modify. It just doesn't do anything useful if the variables are names that you want to rebind to new values. (If you do need to do that, just wrap the values in something mutable—e.g., turn each variable into a 1-item list, so you can rebind a[0] instead of a, which means anyone else who has a reference to a can see your new a[0] value.)
If you insist on a "true global", even that isn't impossible. See builtins for details. But you almost certainly don't want this.

If you want to be able to modify a module-level variable from a different module then yes, you will need to import the other module. I would question why you need to do this. Perhaps you should be breaking your code into classes instead of separate modules.
For example you could choose to encapsulate the variables that need to be modified by both modules inside a separate class and pass a single instance of that class to all classes (or modules but you should really use classes) that need it.
See Circular (or cyclic) imports in Python for more information about cyclical imports.

How to change a module variable from another module?

Suppose I have a package named bar, and it contains bar.py:
a = None
def foobar():
print a
and __init__.py:
from bar import a, foobar
Then I execute this script:
import bar
print bar.a
bar.a = 1
print bar.a
bar.foobar()
Here's what I expect:
None
1
1
Here's what I get:
None
1
None
Can anyone explain my misconception?

You are using from bar import a. a becomes a symbol in the global scope of the importing module (or whatever scope the import statement occurs in).
When you assign a new value to a, you are just changing which value a points too, not the actual value. Try to import bar.py directly with import bar in __init__.py and conduct your experiment there by setting bar.a = 1. This way, you will actually be modifying bar.__dict__['a'] which is the 'real' value of a in this context.
It's a little convoluted with three layers but bar.a = 1 changes the value of a in the module called bar that is actually derived from __init__.py. It does not change the value of a that foobar sees because foobar lives in the actual file bar.py. You could set bar.bar.a if you wanted to change that.
This is one of the dangers of using the from foo import bar form of the import statement: it splits bar into two symbols, one visible globally from within foo which starts off pointing to the original value and a different symbol visible in the scope where the import statement is executed. Changing a where a symbol points doesn't change the value that it pointed too.
This sort of stuff is a killer when trying to reload a module from the interactive interpreter.

One source of difficulty with this question is that you have a program named bar/bar.py: import bar imports either bar/__init__.py or bar/bar.py, depending on where it is done, which makes it a little cumbersome to track which a is bar.a.
Here is how it works:
The key to understanding what happens is to realize that in your __init__.py,
from bar import a
in effect does something like
a = bar.a
# … where bar = bar/bar.py (as if bar were imported locally from __init__.py)
and defines a new variable (bar/__init__.py:a, if you wish). Thus, your from bar import a in __init__.py binds name bar/__init__.py:a to the original bar.py:a object (None). This is why you can do from bar import a as a2 in __init__.py: in this case, it is clear that you have both bar/bar.py:a and a distinct variable name bar/__init__.py:a2 (in your case, the names of the two variables just happen to both be a, but they still live in different namespaces: in __init__.py, they are bar.a and a).
Now, when you do
import bar
print bar.a
you are accessing variable bar/__init__.py:a (since import bar imports your bar/__init__.py). This is the variable you modify (to 1). You are not touching the contents of variable bar/bar.py:a. So when you subsequently do
bar.foobar()
you call bar/bar.py:foobar(), which accesses variable a from bar/bar.py, which is still None (when foobar() is defined, it binds variable names once and for all, so the a in bar.py is bar.py:a, not any other a variable defined in another module—as there might be many a variables in all the imported modules). Hence the last None output.
Conclusion: it is best to avoid any ambiguity in import bar, by not having any bar/bar.py module (since bar.__init__.py makes directory bar/ a package already, that you can also import with import bar).

To put another way:
Turns out this misconception is very easy to make.
It is sneakily defined in the Python language reference: the use of object instead of symbol. I would suggest that the Python language reference make this more clear and less sparse..
The from form does not bind the module name: it goes through the
list of identifiers, looks each one of them up in the module found in
step (1), and binds the name in the local namespace to the object thus
found.
HOWEVER:
When you import, you import the current value of the imported symbol and add it to your namespace as defined. You are not importing a reference, you are effectively importing a value.
Thus, to get the updated value of i, you must import a variable that holds a reference to that symbol.
In other words, importing is NOT like an import in JAVA, external declaration in C/C++ or even a use clause in PERL.
Rather, the following statement in Python:
from some_other_module import a as x
is more like the following code in K&R C:
extern int a; /* import from the EXTERN file */
int x = a;
(caveat: in the Python case, "a" and "x" are essentially a reference to the actual value: you're not copying the INT, you're copying the reference address)

How to make a cross-module variable?

The __debug__ variable is handy in part because it affects every module. If I want to create another variable that works the same way, how would I do it?
The variable (let's be original and call it 'foo') doesn't have to be truly global, in the sense that if I change foo in one module, it is updated in others. I'd be fine if I could set foo before importing other modules and then they would see the same value for it.

If you need a global cross-module variable maybe just simple global module-level variable will suffice.
a.py:
var = 1
b.py:
import a
print a.var
import c
print a.var
c.py:
import a
a.var = 2
Test:
$ python b.py
# -> 1 2
Real-world example: Django's global_settings.py (though in Django apps settings are used by importing the object django.conf.settings).

I don't endorse this solution in any way, shape or form. But if you add a variable to the __builtin__ module, it will be accessible as if a global from any other module that includes __builtin__ -- which is all of them, by default.
a.py contains
print foo
b.py contains
import __builtin__
__builtin__.foo = 1
import a
The result is that "1" is printed.
Edit: The __builtin__ module is available as the local symbol __builtins__ -- that's the reason for the discrepancy between two of these answers. Also note that __builtin__ has been renamed to builtins in python3.

I believe that there are plenty of circumstances in which it does make sense and it simplifies programming to have some globals that are known across several (tightly coupled) modules. In this spirit, I would like to elaborate a bit on the idea of having a module of globals which is imported by those modules which need to reference them.
When there is only one such module, I name it "g". In it, I assign default values for every variable I intend to treat as global. In each module that uses any of them, I do not use "from g import var", as this only results in a local variable which is initialized from g only at the time of the import. I make most references in the form g.var, and the "g." serves as a constant reminder that I am dealing with a variable that is potentially accessible to other modules.
If the value of such a global variable is to be used frequently in some function in a module, then that function can make a local copy: var = g.var. However, it is important to realize that assignments to var are local, and global g.var cannot be updated without referencing g.var explicitly in an assignment.
Note that you can also have multiple such globals modules shared by different subsets of your modules to keep things a little more tightly controlled. The reason I use short names for my globals modules is to avoid cluttering up the code too much with occurrences of them. With only a little experience, they become mnemonic enough with only 1 or 2 characters.
It is still possible to make an assignment to, say, g.x when x was not already defined in g, and a different module can then access g.x. However, even though the interpreter permits it, this approach is not so transparent, and I do avoid it. There is still the possibility of accidentally creating a new variable in g as a result of a typo in the variable name for an assignment. Sometimes an examination of dir(g) is useful to discover any surprise names that may have arisen by such accident.

Define a module ( call it "globalbaz" ) and have the variables defined inside it. All the modules using this "pseudoglobal" should import the "globalbaz" module, and refer to it using "globalbaz.var_name"
This works regardless of the place of the change, you can change the variable before or after the import. The imported module will use the latest value. (I tested this in a toy example)
For clarification, globalbaz.py looks just like this:
var_name = "my_useful_string"

You can pass the globals of one module to onother:
In Module A:
import module_b
my_var=2
module_b.do_something_with_my_globals(globals())
print my_var
In Module B:
def do_something_with_my_globals(glob): # glob is simply a dict.
glob["my_var"]=3

Global variables are usually a bad idea, but you can do this by assigning to __builtins__:
__builtins__.foo = 'something'
print foo
Also, modules themselves are variables that you can access from any module. So if you define a module called my_globals.py:
# my_globals.py
foo = 'something'
Then you can use that from anywhere as well:
import my_globals
print my_globals.foo
Using modules rather than modifying __builtins__ is generally a cleaner way to do globals of this sort.

You can already do this with module-level variables. Modules are the same no matter what module they're being imported from. So you can make the variable a module-level variable in whatever module it makes sense to put it in, and access it or assign to it from other modules. It would be better to call a function to set the variable's value, or to make it a property of some singleton object. That way if you end up needing to run some code when the variable's changed, you can do so without breaking your module's external interface.
It's not usually a great way to do things — using globals seldom is — but I think this is the cleanest way to do it.

I wanted to post an answer that there is a case where the variable won't be found.
Cyclical imports may break the module behavior.
For example:
first.py
import second
var = 1
second.py
import first
print(first.var) # will throw an error because the order of execution happens before var gets declared.
main.py
import first
On this is example it should be obvious, but in a large code-base, this can be really confusing.

I wondered if it would be possible to avoid some of the disadvantages of using global variables (see e.g. http://wiki.c2.com/?GlobalVariablesAreBad) by using a class namespace rather than a global/module namespace to pass values of variables. The following code indicates that the two methods are essentially identical. There is a slight advantage in using class namespaces as explained below.
The following code fragments also show that attributes or variables may be dynamically created and deleted in both global/module namespaces and class namespaces.
wall.py
# Note no definition of global variables
class router:
""" Empty class """
I call this module 'wall' since it is used to bounce variables off of. It will act as a space to temporarily define global variables and class-wide attributes of the empty class 'router'.
source.py
import wall
def sourcefn():
msg = 'Hello world!'
wall.msg = msg
wall.router.msg = msg
This module imports wall and defines a single function sourcefn which defines a message and emits it by two different mechanisms, one via globals and one via the router function. Note that the variables wall.msg and wall.router.message are defined here for the first time in their respective namespaces.
dest.py
import wall
def destfn():
if hasattr(wall, 'msg'):
print 'global: ' + wall.msg
del wall.msg
else:
print 'global: ' + 'no message'
if hasattr(wall.router, 'msg'):
print 'router: ' + wall.router.msg
del wall.router.msg
else:
print 'router: ' + 'no message'
This module defines a function destfn which uses the two different mechanisms to receive the messages emitted by source. It allows for the possibility that the variable 'msg' may not exist. destfn also deletes the variables once they have been displayed.
main.py
import source, dest
source.sourcefn()
dest.destfn() # variables deleted after this call
dest.destfn()
This module calls the previously defined functions in sequence. After the first call to dest.destfn the variables wall.msg and wall.router.msg no longer exist.
The output from the program is:
global: Hello world!
router: Hello world!
global: no message
router: no message
The above code fragments show that the module/global and the class/class variable mechanisms are essentially identical.
If a lot of variables are to be shared, namespace pollution can be managed either by using several wall-type modules, e.g. wall1, wall2 etc. or by defining several router-type classes in a single file. The latter is slightly tidier, so perhaps represents a marginal advantage for use of the class-variable mechanism.

This sounds like modifying the __builtin__ name space. To do it:
import __builtin__
__builtin__.foo = 'some-value'
Do not use the __builtins__ directly (notice the extra "s") - apparently this can be a dictionary or a module. Thanks to ΤΖΩΤΖΙΟΥ for pointing this out, more can be found here.
Now foo is available for use everywhere.
I don't recommend doing this generally, but the use of this is up to the programmer.
Assigning to it must be done as above, just setting foo = 'some-other-value' will only set it in the current namespace.

I use this for a couple built-in primitive functions that I felt were really missing. One example is a find function that has the same usage semantics as filter, map, reduce.
def builtin_find(f, x, d=None):
for i in x:
if f(i):
return i
return d
import __builtin__
__builtin__.find = builtin_find
Once this is run (for instance, by importing near your entry point) all your modules can use find() as though, obviously, it was built in.
find(lambda i: i < 0, [1, 3, 0, -5, -10]) # Yields -5, the first negative.
Note: You can do this, of course, with filter and another line to test for zero length, or with reduce in one sort of weird line, but I always felt it was weird.

I could achieve cross-module modifiable (or mutable) variables by using a dictionary:
# in myapp.__init__
Timeouts = {} # cross-modules global mutable variables for testing purpose
Timeouts['WAIT_APP_UP_IN_SECONDS'] = 60
# in myapp.mod1
from myapp import Timeouts
def wait_app_up(project_name, port):
# wait for app until Timeouts['WAIT_APP_UP_IN_SECONDS']
# ...
# in myapp.test.test_mod1
from myapp import Timeouts
def test_wait_app_up_fail(self):
timeout_bak = Timeouts['WAIT_APP_UP_IN_SECONDS']
Timeouts['WAIT_APP_UP_IN_SECONDS'] = 3
with self.assertRaises(hlp.TimeoutException) as cm:
wait_app_up(PROJECT_NAME, PROJECT_PORT)
self.assertEqual("Timeout while waiting for App to start", str(cm.exception))
Timeouts['WAIT_JENKINS_UP_TIMEOUT_IN_SECONDS'] = timeout_bak
When launching test_wait_app_up_fail, the actual timeout duration is 3 seconds.