I'm new to Python and there's something that's been bothering me for quite some time. I read in "Learning Python" by Mark Lutz that when we use a from statement to import a name present in a module, it first imports the module, then assigns a new name to it (i.e. the name of the function, class, etc. present in the imported module) and then deletes the module object with the del statement. However what happens if I try to import a name using from that references a name in the imported module that itself is not imported? Consider the following example in which there are two modules mod1.py and mod2.py:
#mod1.py
from mod2 import test
test('mod1.py')
#mod2.py
def countLines(name):
print len(open(name).readlines())
def countChars(name):
print len(open(name).read())
def test(name):
print 'loading...'
countLines(name)
countChars(name)
print '-'*10
Now see what happens when I run or import mod1:
>>>import mod1
loading...
3
44
----------
Here when I imported and ran the test function, it ran successfully although I didn't even import countChars or countLines, and the from statement had already deleted the mod2 module object.
So I basically need to know why this code works even though considering the problems I mentioned it shouldn't.
EDIT: Thanx alot to everyone who answered :)
Every function have a __globals__ attribute which holds a reference for the environment where it search for global variables and functions.
The test function is then linked to the global variables of mod2. So when it calls countLines the interpreter will always find the right function even if you wrote a new one with the same name in the module importing the function.
I think you're wrestling with the way python handles namespaces. when you type from module import thing you are bringing thing from module into your current namespace. So, in your example, when mod1 gets imported, the code is evaluated in the following order:
from mod2 import test #Import mod2, bring test function into current module namespace
test("mod1.py") #run the test function (defined in mod2)
And now for mod2:
#create a new function named 'test' in the current (mod2) namespace
#the first time this module is imported. Note that this function has
#access to the entire namespace where it is defined (mod2).
def test(name):
print 'loading...'
countLines(name)
countChars(name)
print '-'*10
The reason that all of this is important is because python lets you choose exactly what you want to pull into your namespace. For example, say you have a module1 which defines function cool_func. Now you are writing another module (module2) and it makes since for module2 to have a function cool_func also. Python allows you to keep those separate. In module3 you could do:
import module1
import module2
module1.cool_func()
module2.cool_func()
Or, you could do:
from module1 import cool_func
import module2
cool_func() #module1
module2.cool_func()
or you could do:
from module1 import cool_func as cool
from module2 import cool_func as cooler
cool() #module1
cooler() #module2
The possibilities go on ...
Hopefully my point is clear. When you import an object from a module, you are choosing how you want to reference that object in your current namespace.
The other answers are better articulated than this one, but if you run the following you can see that countChars and countLines are actually both defined in test.__globals__:
from pprint import pprint
from mod2 import test
pprint(test.__globals___)
test('mod1')
You can see that importing test brings along the other globals defined in mod2, letting you run the function without worrying about having to import everything you need.
Each module has its own scope. Within mod1, you cannot use the names countLines or countChars (or mod2).
mod2 itself isn't affected in the least by how it happens to be imported elsewhere; all names defined in it are available within the module.
If the webpage you reference really says that the module object is deleted with the del statement, it's wrong. del only removes names, it doesn't delete objects.
From A GUIDE TO PYTHON NAMESPACES,
Even though modules have their own global namespaces, this doesn’t mean that all names can be used from everywhere in the module. A scope refers to a region of a program from where a namespace can be accessed without a prefix. Scopes are important for the isolation they provide within a module. At any time there are a number of scopes in operation: the scope of the current function you’re in, the scope of the module and then the scope of the Python builtins. This nesting of scopes means that one function can’t access names inside another function.
Namespaces are also searched for names inside out. This means that if there is a certain name declared in the module’s global namespace, you can reuse the name inside a function while being certain that any other function will get the global name. Of course, you can force the function to use the global name by prefixing the name with the ‘global’ keyword. But if you need to use this, then you might be better off using classes and objects.
An import statement loads the whole module in memory so that's why the test() function ran successfully.
But as you used from statement that's why you can't use the countLines and countChars directly but test can surely call them.
from statement basically loads the whole module and sets the imported function, variable etc to the global namespace.
for eg.
>>> from math import sin
>>> sin(90) #now sin() is a global variable in the module and can be accesed directly
0.89399666360055785
>>> math
Traceback (most recent call last):
File "<pyshell#2>", line 1, in <module>
math
NameError: name 'math' is not defined
>>> vars() #shows the current namespace, and there's sin() in it
{'__builtins__': <module '__builtin__' (built-in)>, '__file__': '/usr/bin/idle', '__package__': None, '__name__': '__main__', 'main': <function main at 0xb6ac702c>, 'sin': <built-in function sin>, '__doc__': None}
consider a simple file, file.py:
def f1():
print 2+2
def f2():
f1()
import only f2:
>>> from file import f2
>>> f2()
4
though I only imported f2() not f1() but it ran f1() succesfully it's because the module is loaded in memory but we can only access f2(), but f2() can access other parts of the module.
Related
I understand that below are the scoping levels in python (listed in highest to lowest order)
Local(L): Defined inside function/class
Enclosed(E): Defined inside enclosing functions(Nested function concept)
Global(G): Defined at the uppermost level
Built-in(B): Reserved names in Python builtin modules
I understand this concept when we have a single file. I am trying to understand how this concept of namespace, local, enclosed, global, built-in works when using import in python.
I understand we can import using foll. approaches:
import modulename
import modulename.function(function/variable)
import package.module.function(function/variable)
from module import function(function/variable)
from package import module
from package.module import function(function/variable)
from module import *
from package import *
I know that in each of the above the entire module is loaded in the sys modules dictionary. But I am confused with respect to namespace, scope, global, local. I want to understand LEGB principle, namespace, global, localwith respect to the method of import vs from import. I have read that in case of point 1 (import modulename), the modulename is loaded in its own namespace and reference is available to the calling module with which it can change the variable values globally. Does this also mean it is in global namespace? Any if so then global with respect to what? Also I have read that from module import function/variable or from module import * brings function(s)/variable(s) in the calling modules namespace and any changes are not global. Does this mean that it is in local namespace? Local with respect to what?
Example:
Say module foo has a variable bar=10. When we use [import foo] - this brings foo in current namespace and allows us to change the value of bar via foo.bar=1000 and this change is visible to every piece of code that makes use of [import foo print(foo.bar)] after the above assignment was made. Where as in case of [from foo import bar], any change made to bar is only visible to this module and not any other module irrespective of whether they do an [import foo] or [from foo import bar] they would see value of bar as 10.
Any reference/links in this area will be very helpful. I want to understand the basics of how this works internally. Any information will be helpful to understand this. Articles that I found on this topic explain what is import vs from xx import yy. But they don't explain why/how the changes made become global (in case of import) vs local (in case of from import).
Some more examples:
In below code example, change made to a is only seen in the main.py file and does not affect the value of a in module1.py and subsequent modules that import module1.py will still see the value of a as 10.
module1.py:
a=10
main.py:
from module1 import a
print(a)
a=100
Whereas if we used import module1 instead of from module1 import a then the assignment would have changed value of a in module1.py and any subsequent import of module1.py would show the value of a as 100.
Note: Similarly we could also have functions inside the module.py that get and set the value of variable a. And using this approach, once we do a from module1 import getfn, setfn in main.py, we can use these functions which when called make the change to variable a and this change is globally visible.
You're over complicating things. From the point of view of imports, the only thing to know is that import foo brings foo into the current namespace, and from foo import bar brings bar into the current namespace.
Python has only 3 scopes: global local and builtin.
They are relative to the position in the code
Every variable whose value you can change is local
(note that changing global variable results in creating a new local variable instead,
so there are local and global variable of the same name)
Enclosed you speak of are local to that function
Every variable with unchangeable content whose value you can get is global
Builtin variables are exactly like global so they can even be considered global
to see what variables are global and what are local, put this line in your code
print("global:", globals(), "\n\nlocal:", locals())
Given a class or function, is there a way to find the full path of the module where it is originally defined? (I.e. using def xxx or class xxx.)
I'm aware that there is sys.modules[func.__module__]. However, if func is imported in a package's __init__.py, then sys.modules will simply redirect to that __init__.py, because the function has been brought into that namespace, as far as my understanding goes.
A concrete example:
>>> import numpy as np
>>> import sys
>>> np.broadcast.__module__
'numpy'
>>> sys.modules[np.broadcast.__module__]
<module 'numpy' from '/Users/brad/.../site-packages/numpy/__init__.py'>
Obviously, broadcast is not defined in __init__.py; it is just brought into the namespace with one of these from module import * statements.
It would be nice to see where in the source np.broadcast is defined (regardless of the file extension, be it .c or .py). Is this possible?
Your understanding:
However, if func is imported in a package's __init__.py, then
sys.modules will simply redirect to that __init__.py, because the
function has been brought into that namespace, as far as my
understanding goes.
is wrong. __init__.py importing a thing has no effect on that thing's __module__.
The behavior you're seeing with numpy.broadcast happens because C types don't really have a "defining module" the same way types written in Python do. numpy.broadcast.__module__ == 'numpy' because numpy.broadcast is written in C and declares its name to be "numpy.broadcast", and a C type's __module__ is determined from its name.
As for how to get a class or function's "module of original definition", the best you really have is __module__ and other functions that go through __module__.
Similar Question: Understanding A Chain of Imports in Python
NB: I'm using Python 3.3
I have setup the following two files in the same directory to explain importing to myself, however I still don't get exactly what it's doing. I understand function and class definitions are statements that need to run.
untitled.py:
import string
class testing:
def func(self):
try:
print(string.ascii_lowercase)
except:
print('not imported')
class second:
x=1
print('print statement in untitled executed')
stuff.py:
from untitled import testing
try:
t=testing()
t.func()
except NameError:
print('testing not imported')
try:
print(string.ascii_uppercase)
except NameError:
print('string not imported')
try:
print(untitled.string.ascii_uppercase)
except NameError:
print('string not imported in untitled')
try:
s=second()
print(s.x)
except NameError:
print('second not imported')
This is the output I get from running stuff.py:
print statement in untitled executed
abcdefghijklmnopqrstuvwxyz
string not imported
string not imported in untitled
second not imported
The print statement in untitled.py is executed despite the import in stuff.py specifying only the testing class. Moreover what is the string module's relation inside stuff.py, as it can be called from within the testing class yet not from the outside.
Could somebody please explain this behaviour to me, what exactly does a "from import" statment do (what does it run)?
You can think of python modules as namespaces. Keep in mind that imports are not includes:
modules are only imported once
the first time, the top level code is executed
any imports, variable, function or class declarations affects only the module local namespace
Suppose you have a module called foo.py:
import eggs
bar = "Lets drink, it's a bar'
So when you do a from foo import bar in another module, you will make bar available in the current namespace. The module eggs will be available under foo.eggs if you do an import foo. If you do a from foo import *, then eggs, bar and everything else in the module namespace will be also in the current namespace - but never do that, wildcard imports are frowned upon in Python.
If you do a import foo and then import eggs, the top level code at eggs will be executed once and the module namespace will be stored in the module cache: if another module imports it the information will be pulled from this cache. If you are going to use it, then import it - no need to worry about multiple imports executing the top level code multiple times.
Python programmers are very fond of namespaces; I always try to use import foo and then foo.bar instead of from foo import bar if possible - it keeps the namespace clean and prevent name clashes.
That said, the import mechanism is hackable, you can make python import statement work even with files that are not python.
The from statement isn't any different to import with regard to loading behaviour. Always the top level code is executed, when loading the module. from just controls which parts of the loaded module are being added to the current scope (the first point is most important):
The from form uses a slightly more complex process:
find the module specified in the from clause loading and initializing it if necessary;
for each of the identifiers specified in the import clauses:
check if the imported module has an attribute by that name
if not, attempt to import a submodule with that name and then check the imported module again for that attribute
if the attribute is not found, ImportError is raised.
otherwise, a reference to that value is bound in the local namespace, using the name in the as clause if it is present, otherwise using the attribute name
Thus you can access the contents of a module partially imported with from with this inelegant trick:
print(sys.modules['untitled'].string.ascii_uppercase)
In your first file (untitled.py), when python compiler parses(since you called it in import) this file It will create 2 class code objects and execute the print statement. Note that it will even print it if you run untitled.py from command line.
In your second file(stuff.py), to add to #Paulo comments, you have only imported testing class in your namspace, so only that will be available, from the 2 code objects from untitled.py
However if you just say
import untitled
your 3rd "try" statement will work, since it will have untitled in its namespace.
Next thing. try importing untitled.testing :)
I am reading http://docs.python.org/2/tutorial/modules.html#more-on-modules and wonder if the following is correct:
Modules can import other modules. It is customary but not required to
place all import statements at the beginning of a module (or script,
for that matter). The imported module names are placed in the
importing module’s global symbol table.
Apparently not:
>>> def foo(): import sys
...
>>> foo()
>>> sys.path
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'sys' is not defined
See http://ideone.com/cLK09v for an online demo.
So, is it a bug in the Python's documentation or I don't understand something?
Yes, this is a documentation error. The import statement imports the names to the current namespace. Usually import is used outside of functions and classes, but as you've discovered, it does work within them. In your example function, the module is imported into the function's local namespace when the function is called. (Which you didn't do, but that wouldn't make it available outside the function anyway.)
The global keyword does work here, however:
def foo():
global sys
import sys
foo()
sys.path
I don't think this is actually an error in the documentation, but more of a mis-interpretation. You simply have a scope issue. You are importing it in the scope of the function foo(). You could certainly do as the documentation suggests and put the import at the bottom of the file or somewhere else in the file that would still have the same global scope as your module. The problem is "The imported module names are placed in the importing module’s global symbol table", where the scope of the module you are importing into is contained in the function foo(), not at the module's global level.
I am currently doing a python tutorial, but they use IDLE, and I opted to use the interpreter on terminal. So I had to find out how to import a module I created. At first I tried
import my_file
then I tried calling the function inside the module by itself, and it failed. I looked around and doing
my_file.function
works. I am very confused why this needs to be done if it was imported. Also, is there a way around it so that I can just call the function? Can anyone point me in the right direction. Thanks in advance.
If you wanted to use my_file.function by just calling function, try using the from keyword.
Instead of import my_file try from my_file import *.
You can also do this to only import parts of a module like so :
from my_file import function1, function2, class1
To avoid clashes in names, you can import things with a different name:
from my_file import function as awesomePythonFunction
EDIT:
Be careful with this, if you import two modules (myfile, myfile2) that both have the same function inside, function will will point to the function in whatever module you imported last. This could make interesting things happen if you are unaware of it.
This is a central concept to python. It uses namespaces (see the last line of import this). The idea is that with thousands of people writing many different modules, the likelihood of a name collision is reasonably high. For example, I write module foo which provides function baz and Joe Smith writes module bar which provides a function baz. My baz is not the same as Joe Smiths, so in order to differentiate the two, we put them in a namespace (foo and bar) so mine can be called by foo.baz() and Joe's can be called by bar.baz().
Of course, typing foo.baz() all the time gets annoying if you just want baz() and are sure that none of your other modules imported will provide any problems... That is why python provides the from foo import * syntax, or even from foo import baz to only import the function/object/constant baz (as others have already noted).
Note that things can get even more complex:
Assume you have a module foo which provides function bar and baz, below are a few ways to import and then call the functions contained inside foo...
import foo # >>> foo.bar();foo.baz()
import foo as bar # >>> bar.bar();bar.baz()
from foo import bar,baz # >>> bar(); baz()
from foo import * # >>> bar(); baz()
from foo import bar as cow # >>> cow() # This calls bar(), baz() is not available
...
A basic import statement is an assignment of the module object (everything's an object in Python) to the specified name. I mean this literally: you can use an import anywhere in your program you can assign a value to a variable, because they're the same thing. Behind the scenes, Python is calling a built-in function called __import__() to do the import, then returning the result and assigning it to the variable name you provided.
import foo
means "import module foo and assign it the name foo in my namespace. This is the same as:
foo = __import__("foo")
Similarly, you can do:
import foo as f
which means "import module foo and assign it the name f in my namespace." This is the same as:
f = __import__("foo")
Since in this case, you have only a reference to the module object, referring to things contained by the module requires attribute access: foo.bar etc.
You can also do from foo import bar. This creates a variable named bar in your namespace that points to the bar function in the foo module. It's syntactic sugar for:
bar = __import__("foo").bar
I don't really understand your confusion. You've imported the name my_file, not anything underneath it, so that's how you reference it.
If you want to import functions or classes inside a module directly, you can use:
from my_file import function
I'm going to incorporate many of the comments already posted.
To have access to function without having to refer to the module my_file, you can do one of the following:
from my_file import function
or
from my_file import *
For a more in-depth description of how modules work, I would refer to the documentation on python modules.
The first is the preferred solution, and the second is not recommended for many reasons:
It pollutes your namespace
It is not a good practice for maintainability (it becomes more difficult to find where specific names reside.
You typically don't know exactly what is imported
You can't use tools such as pyflakes to statically detect errors in your code
Python imports work differently than the #includes/imports in a static language like C or Java, in that python executes the statements in a module. Thus if two modules need to import a specific name (or *) out of each other, you can run into circular referencing problems, such as an ImportError when importing a specific name, or simply not getting the expected names defined (in the case you from ... import *). When you don't request specific names, you don't run into the, risk of having circular references, as long as the name is defined by the time you actually want to use it.
The from ... import * also doesn't guarantee you get everything. As stated in the documentation on python modules, a module can defined the __all__ name, and cause from ... import * statements to miss importing all of the subpackages, except those listed by __all__.