Suppose I have a module foo.py and a package foo/. If I call
import foo
which one will be loaded? How can I specify I want to load the module, or the package?
I believe the package will always get loaded. You can't work around this, as far as I know. So change either the package or the module name. Docs: http://docs.python.org/tutorial/modules.html#the-module-search-path
Actually, it is possible, by manually guiding the import machinery to use a .py file instead of directory. (This code is not well tested, but seems to work). UPDATE 2020: Note that this requires using custom import_module() function instead of normal import statement. However, with modern Python3 and its importlib, it might be possible to make the bare import statement to work the same way too. (Note that this answer shows flexibility which Python offers. It's not an encouragement to use this in your applications. Use this only if you know what you're doing.)
File foo.py
print "foo module loaded"
File foo/__init__.py
print "foo package loaded"
File test1.py
import foo
File test2.py
import os, imp
def import_module(dir, name):
""" load a module (not a package) with a given name
from the specified directory
"""
for description in imp.get_suffixes():
(suffix, mode, type) = description
if not suffix.startswith('.py'): continue
abs_path = os.path.join(dir, name + suffix)
if not os.path.exists(abs_path): continue
fh = open(abs_path)
return imp.load_module(name, fh, abs_path, (description))
import_module('.', 'foo')
Running
$ python test1.py
foo package loaded
$ python test2.py
foo module loaded
Maybe you want to move your classes from foo.py module to __init__.py.
This way you'll be able to import them from the package as well as importing optional subpackages:
File foo/__init__.py:
class Bar(object):
...
File foo/subfoo.py:
class SubBar(object):
...
File mymodule.py:
from foo import Bar
from foo.subfoo import SubBar
Related
Do I have to take out all the spaces in the file name to import it, or is there some way of telling import that there are spaces?
You should take the spaces out of the filename. Because the filename is used as the identifier for imported modules (i.e. foo.py will be imported as foo) and Python identifiers can't have spaces, this isn't supported by the import statement.
If you really need to do this for some reason, you can use the __import__ function:
foo_bar = __import__("foo bar")
This will import foo bar.py as foo_bar. This behaves a little bit different than the import statement and you should avoid it.
If you want to do something like from foo_bar import * (but with a space instead of an underscore), you can use execfile (docs here):
execfile("foo bar.py")
though it's better practice to avoid spaces in source file names.
You can also use importlib.import_module function, which is a wrapper around __import__.
foo_bar_mod = importlib.import_module("foo bar")
or
foo_bar_mod = importlib.import_module("path.to.foo bar")
More info: https://docs.python.org/3/library/importlib.html
Just to add to Banks' answer, if you are importing another file that you haven't saved in one of the directories Python checks for importing directories, you need to add the directory to your path with
import sys
sys.path.append("absolute/filepath/of/parent/directory/of/foo/bar")
before calling
foo_bar = __import__("foo bar")
or
foo_bar = importlib.import_module("foo bar")
This is something you don't have to do if you were importing it with import <module>, where Python will check the current directory for the module. If you are importing a module from the same directory, for example, use
import os,sys
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
foo_bar = __import__('foo_bar')
Hope this saves someone else trying to import their own weirdly named file or a Python file they downloaded manually some time :)
I have a program that creates a module called "cool" using file operations. I later say import cool and then uses the cool module that was created.
Here is my directory
project/
main.py
modules/
maker.py
cool/ #this folder and its contents was made by maker.py
__init__.py
coolm.py
If I want to make my program into the .exe format, this strategy will not work anymore. Does anyone know another technique?
Note: I cannot use exec to use the cool module..
Import your module when you need it like this:
coolmod = __import__('coolm')
coolm.someproperty
Alternatively you could try:
import importlib
coolmod = importlib.import_module('coolm', 'cool')
This allows you to specify the package name as a second argument.
Do I have to take out all the spaces in the file name to import it, or is there some way of telling import that there are spaces?
You should take the spaces out of the filename. Because the filename is used as the identifier for imported modules (i.e. foo.py will be imported as foo) and Python identifiers can't have spaces, this isn't supported by the import statement.
If you really need to do this for some reason, you can use the __import__ function:
foo_bar = __import__("foo bar")
This will import foo bar.py as foo_bar. This behaves a little bit different than the import statement and you should avoid it.
If you want to do something like from foo_bar import * (but with a space instead of an underscore), you can use execfile (docs here):
execfile("foo bar.py")
though it's better practice to avoid spaces in source file names.
You can also use importlib.import_module function, which is a wrapper around __import__.
foo_bar_mod = importlib.import_module("foo bar")
or
foo_bar_mod = importlib.import_module("path.to.foo bar")
More info: https://docs.python.org/3/library/importlib.html
Just to add to Banks' answer, if you are importing another file that you haven't saved in one of the directories Python checks for importing directories, you need to add the directory to your path with
import sys
sys.path.append("absolute/filepath/of/parent/directory/of/foo/bar")
before calling
foo_bar = __import__("foo bar")
or
foo_bar = importlib.import_module("foo bar")
This is something you don't have to do if you were importing it with import <module>, where Python will check the current directory for the module. If you are importing a module from the same directory, for example, use
import os,sys
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
foo_bar = __import__('foo_bar')
Hope this saves someone else trying to import their own weirdly named file or a Python file they downloaded manually some time :)
I am trying to parse a given path for python source files, import each file and DoStuff™ to each imported module.
def ParsePath(path):
for root, dirs, files in os.walk(path):
for source in (s for s in files if s.endswith(".py")):
name = os.path.splitext(os.path.basename(source))[0]
m = imp.load_module(name, *imp.find_module(name, [root]))
DoStuff(m)
The above code works, but packages aren't recognized ValueError: Attempted relative import in non-package
My question is basically, how do I tell imp.load_module that a given module is part of a package?
You cannot directly tell Importer Protocol method load_module that the module given is part of the package. Taken from PEP 302 New Import Hooks
The built-in __import__ function
(known as PyImport_ImportModuleEx
in import.c) will then check to see whether the module doing the
import is a package or a submodule of a package. If it is indeed a
(submodule of a) package, it first tries to do the import relative
to the package (the parent package for a submodule). For example if
a package named "spam" does "import eggs", it will first look for
a
module named "spam.eggs". If that fails, the import continues as an
absolute import: it will look for a module named "eggs". Dotted
name imports work pretty much the same: if package "spam" does
"import eggs.bacon" (and "spam.eggs" exists and is itself a
package), "spam.eggs.bacon" is tried. If that fails "eggs.bacon" is
tried. (There are more subtleties that are not described here, but
these are not relevant for implementers of the Importer
Protocol.)
Deeper down in the mechanism, a dotted name import is split up by
its components. For "import spam.ham", first an "import spam" is
done, and only when that succeeds is "ham" imported as a submodule
of "spam".
The Importer Protocol operates at this level of individual
imports. By the time an importer gets a request for
"spam.ham",
module "spam" has already been imported.
You must then simulate what the built-in import does and load parent packages before loading sub modules.
The function imp.find_module always takes a plain module name without dots, but the documentation of imp.load_module says
The name argument indicates the full module name (including the package name, if this is a submodule of a package).
So you could try this:
def ParsePath(path):
for root, dirs, files in os.walk(path):
for source in (s for s in files if s.endswith(".py")):
name = os.path.splitext(os.path.basename(source))[0]
full_name = os.path.splitext(source)[0].replace(os.path.sep, '.')
m = imp.load_module(full_name, *imp.find_module(name, [root]))
DoStuff(m)
I had the same problem. Good news is that there is a way of doing it, but you have to use a combination of imp and importlib. Here's an illustrative example:
import imp
import importlib
package_path = r"C:\path_to_package"
package_name = "module"
module_absolute_name = "module.sub_module"
module_relative_name = ".sub_module"
# Load the package first
package_info = imp.find_module(package_name, [package_path])
package_module = imp.load_module(package_name, *package_info)
# Try an absolute import
importlib.import_module(module_absolute_name, package_name)
# Try a relative import
importlib.import_module(module_relative_name, package_name)
This will allow sub_module to import using relative module paths because we've already loaded the parent package and the submodule has been loaded correctly by importlib to know what it's being imported relative to.
I believe this solution is only necessary for those of us stuck in Python 2.*, but would need someone to confirm that.
In a big application I am working, several people import same modules differently e.g.
import x
or
from y import x
the side effects of that is x is imported twice and may introduce very subtle bugs, if someone is relying on global attributes
e.g. suppose I have a package mypakcage with three file mymodule.py, main.py and init.py
mymodule.py contents
l = []
class A(object): pass
main.py contents
def add(x):
from mypackage import mymodule
mymodule.l.append(x)
print "updated list",mymodule.l
def get():
import mymodule
return mymodule.l
add(1)
print "lets check",get()
add(1)
print "lets check again",get()
it prints
updated list [1]
lets check []
updated list [1, 1]
lets check again []
because now there are two lists in two different modules, similarly class A is different
To me it looks serious enough because classes itself will be treated differently
e.g. below code prints False
def create():
from mypackage import mymodule
return mymodule.A()
def check(a):
import mymodule
return isinstance(a, mymodule.A)
print check(create())
Question:
Is there any way to avoid this? except enforcing that module should be imported one way onyl. Can't this be handled by python import mechanism, I have seen several bugs related to this in django code and elsewhere too.
Each module namespace is imported only once. Issue is, you're importing them differently. On the first you're importing from the global package, and on the second you're doing a local, non-packaged import. Python sees modules as different. The first import is internally cached as mypackage.mymodule and the second one as mymodule only.
A way to solve this is to always use absolute imports. That is, always give your module absolute import paths from the top-level package onwards:
def add(x):
from mypackage import mymodule
mymodule.l.append(x)
print "updated list",mymodule.l
def get():
from mypackage import mymodule
return mymodule.l
Remember that your entry point (the file you run, main.py) also should be outside the package. When you want the entry point code to be inside the package, usually you use a run a small script instead. Example:
runme.py, outside the package:
from mypackage.main import main
main()
And in main.py you add:
def main():
# your code
I find this document by Jp Calderone to be a great tip on how to (not) structure your python project. Following it you won't have issues. Pay attention to the bin folder - it is outside the package. I'll reproduce the entire text here:
Filesystem structure of a Python project
Do:
name the directory something
related to your project. For example,
if your project is named "Twisted",
name the top-level directory for its
source files Twisted. When you do
releases, you should include a version
number suffix: Twisted-2.5.
create a directory Twisted/bin and
put your executables there, if you
have any. Don't give them a .py
extension, even if they are Python
source files. Don't put any code in
them except an import of and call to a
main function defined somewhere else
in your projects.
If your project
is expressable as a single Python
source file, then put it into the
directory and name it something
related to your project. For example,
Twisted/twisted.py. If you need
multiple source files, create a
package instead (Twisted/twisted/,
with an empty
Twisted/twisted/__init__.py) and
place your source files in it. For
example,
Twisted/twisted/internet.py.
put
your unit tests in a sub-package of
your package (note - this means that
the single Python source file option
above was a trick - you always need at
least one other file for your unit
tests). For example,
Twisted/twisted/test/. Of course,
make it a package with
Twisted/twisted/test/__init__.py.
Place tests in files like
Twisted/twisted/test/test_internet.py.
add Twisted/README and Twisted/setup.py to explain and
install your software, respectively,
if you're feeling nice.
Don't:
put your source in a directory
called src or lib. This makes it
hard to run without installing.
put
your tests outside of your Python
package. This makes it hard to run the
tests against an installed version.
create a package that only has a
__init__.py and then put all your
code into __init__.py. Just make a
module instead of a package, it's
simpler.
try to come up with
magical hacks to make Python able to
import your module or package without
having the user add the directory
containing it to their import path
(either via PYTHONPATH or some other
mechanism). You will not correctly
handle all cases and users will get
angry at you when your software
doesn't work in their environment.
I can only replicate this if main.py is the file you are actually running. In that case you will get the current directory of main.py on the sys path. But you apparently also have a system path set so that mypackage can be imported.
Python will in that situation not realize that mymodule and mypackage.mymodule is the same module, and you get this effect. This change illustrates this:
def add(x):
from mypackage import mymodule
print "mypackage.mymodule path", mymodule
mymodule.l.append(x)
print "updated list",mymodule.l
def get():
import mymodule
print "mymodule path", mymodule
return mymodule.l
add(1)
print "lets check",get()
add(1)
print "lets check again",get()
$ export PYTHONPATH=.
$ python mypackage/main.py
mypackage.mymodule path <module 'mypackage.mymodule' from '/tmp/mypackage/mymodule.pyc'>
mymodule path <module 'mymodule' from '/tmp/mypackage/mymodule.pyc'>
But add another mainfile, in the currect directory:
realmain.py:
from mypackage import main
and the result is different:
mypackage.mymodule path <module 'mypackage.mymodule' from '/tmp/mypackage/mymodule.pyc'>
mymodule path <module 'mypackage.mymodule' from '/tmp/mypackage/mymodule.pyc'>
So I suspect that you have your main python file within the package. And in that case the solution is to not do that. :-)