In a big application I am working, several people import same modules differently e.g.
import x
or
from y import x
the side effects of that is x is imported twice and may introduce very subtle bugs, if someone is relying on global attributes
e.g. suppose I have a package mypakcage with three file mymodule.py, main.py and init.py
mymodule.py contents
l = []
class A(object): pass
main.py contents
def add(x):
from mypackage import mymodule
mymodule.l.append(x)
print "updated list",mymodule.l
def get():
import mymodule
return mymodule.l
add(1)
print "lets check",get()
add(1)
print "lets check again",get()
it prints
updated list [1]
lets check []
updated list [1, 1]
lets check again []
because now there are two lists in two different modules, similarly class A is different
To me it looks serious enough because classes itself will be treated differently
e.g. below code prints False
def create():
from mypackage import mymodule
return mymodule.A()
def check(a):
import mymodule
return isinstance(a, mymodule.A)
print check(create())
Question:
Is there any way to avoid this? except enforcing that module should be imported one way onyl. Can't this be handled by python import mechanism, I have seen several bugs related to this in django code and elsewhere too.
Each module namespace is imported only once. Issue is, you're importing them differently. On the first you're importing from the global package, and on the second you're doing a local, non-packaged import. Python sees modules as different. The first import is internally cached as mypackage.mymodule and the second one as mymodule only.
A way to solve this is to always use absolute imports. That is, always give your module absolute import paths from the top-level package onwards:
def add(x):
from mypackage import mymodule
mymodule.l.append(x)
print "updated list",mymodule.l
def get():
from mypackage import mymodule
return mymodule.l
Remember that your entry point (the file you run, main.py) also should be outside the package. When you want the entry point code to be inside the package, usually you use a run a small script instead. Example:
runme.py, outside the package:
from mypackage.main import main
main()
And in main.py you add:
def main():
# your code
I find this document by Jp Calderone to be a great tip on how to (not) structure your python project. Following it you won't have issues. Pay attention to the bin folder - it is outside the package. I'll reproduce the entire text here:
Filesystem structure of a Python project
Do:
name the directory something
related to your project. For example,
if your project is named "Twisted",
name the top-level directory for its
source files Twisted. When you do
releases, you should include a version
number suffix: Twisted-2.5.
create a directory Twisted/bin and
put your executables there, if you
have any. Don't give them a .py
extension, even if they are Python
source files. Don't put any code in
them except an import of and call to a
main function defined somewhere else
in your projects.
If your project
is expressable as a single Python
source file, then put it into the
directory and name it something
related to your project. For example,
Twisted/twisted.py. If you need
multiple source files, create a
package instead (Twisted/twisted/,
with an empty
Twisted/twisted/__init__.py) and
place your source files in it. For
example,
Twisted/twisted/internet.py.
put
your unit tests in a sub-package of
your package (note - this means that
the single Python source file option
above was a trick - you always need at
least one other file for your unit
tests). For example,
Twisted/twisted/test/. Of course,
make it a package with
Twisted/twisted/test/__init__.py.
Place tests in files like
Twisted/twisted/test/test_internet.py.
add Twisted/README and Twisted/setup.py to explain and
install your software, respectively,
if you're feeling nice.
Don't:
put your source in a directory
called src or lib. This makes it
hard to run without installing.
put
your tests outside of your Python
package. This makes it hard to run the
tests against an installed version.
create a package that only has a
__init__.py and then put all your
code into __init__.py. Just make a
module instead of a package, it's
simpler.
try to come up with
magical hacks to make Python able to
import your module or package without
having the user add the directory
containing it to their import path
(either via PYTHONPATH or some other
mechanism). You will not correctly
handle all cases and users will get
angry at you when your software
doesn't work in their environment.
I can only replicate this if main.py is the file you are actually running. In that case you will get the current directory of main.py on the sys path. But you apparently also have a system path set so that mypackage can be imported.
Python will in that situation not realize that mymodule and mypackage.mymodule is the same module, and you get this effect. This change illustrates this:
def add(x):
from mypackage import mymodule
print "mypackage.mymodule path", mymodule
mymodule.l.append(x)
print "updated list",mymodule.l
def get():
import mymodule
print "mymodule path", mymodule
return mymodule.l
add(1)
print "lets check",get()
add(1)
print "lets check again",get()
$ export PYTHONPATH=.
$ python mypackage/main.py
mypackage.mymodule path <module 'mypackage.mymodule' from '/tmp/mypackage/mymodule.pyc'>
mymodule path <module 'mymodule' from '/tmp/mypackage/mymodule.pyc'>
But add another mainfile, in the currect directory:
realmain.py:
from mypackage import main
and the result is different:
mypackage.mymodule path <module 'mypackage.mymodule' from '/tmp/mypackage/mymodule.pyc'>
mymodule path <module 'mypackage.mymodule' from '/tmp/mypackage/mymodule.pyc'>
So I suspect that you have your main python file within the package. And in that case the solution is to not do that. :-)
Related
The project has the same structure as in the picture: I'm trying to import from "mod.py " in "index.py "
from .. import mod
However, it gives the error: "ImportError: attempted relative import with no known parent package" If you use this option:
from pack1 import mod
Then error: "ModuleNotFoundError error: there is no module named 'pack1'"
enter image description here
PROJECT/
pack1/
__init__.py
mod.py
pack2/
__init__.py
index.py
What is the problem?
This is a recurring question on StackOverflow. And much of the confusion (in my opinion) comes from how Python interprets the files and folders it sees is based on where Python is run from. First, some terminology:
module: a file containing Python code.
package: a folder containing files with Python code and other folders.
When you start Python in a directory (folder), it doesn't "know" what the namespace of that directory should be. I.e., if you are working in Z:\path\to_my\project\ when you start Python:
it does NOT consider project to be a package.
any .py files you want to import from will be in their own namespace as modules.
any folders you want to import from will also be in their own namespace as packages.
What about __init__.py? Since version 3.3, Python has implicit namespace packages, which allows importing without needing to create an empty __init__.py file.
Consider #2: if you have two files: first.py and second.py:
path/
to_my/
project/
>>Python is running here<<
first.py
second.py
with these contents:
# first.py
first_var = 'hello'
# second.py
from .first import first_var
second_var = first_var + ' world'
if you try to import like this:
>>> import second
Python basically does the following:
"ok, I see second.py"
"Reading that in as a module, chief!"
"Ok, it wants to import .first
"The . means get the package (folder) that contains first.py"
"Wait, I don't have a parent package for first.py!"
"Better raise an error."
The same rules apply for #3 as well. If we add a few packages to the project like this:
path/
to_my/
project/
>>Python is running here<<
first.py
second.py
pack1/
mod.py
other_mod.py
pack2/
index.py
with the following contents:
# pack1/mod.py
mod_var = 1234
# pack1/other_mod.py
from .mod import mod_var
other_var = mod_var * 10
# pack2/index.py
from ..pack1 import mod
and when you try to import like this:
>>> from pack2 import index.py
The import in pack2/index.py is going to fail for the same reason second.py, Python will work its way up the import chain of dots like this:
"Reading in in index.py as a module."
"Looks like it wants to import mod from ..pack1.
"Ok, . is the pack2 parent package namespace of index.py, found that."
"So, .. is the parent package of pack2."
"But, I don't have a parent package for pack2!"
"Better raise an error."
How do we make it work? Two thing.
First, move where Python is running up one level so that all of the .py files and subfolders are considered to be part of the same package namespace, which allows the file to reference each other using relative references.
path/
to_my/
>>Python is running here now<<
project/
first.py
second.py
pack1/
mod.py
other_mod.py
pack2/
index.py
So now Python sees project as a package namespace, and all of the files within can use relative references up to that level.
This changes how you import when you are in the Python interpreter:
>>> from project.pack2 import index.py
Second, you make explicit references instead of relative references. That can make the import statements really long, but if you have several top-level modules that need to pull from one another, this is how you can do it. This is useful when you are defining your functions in one file and writing your script in another.
# first.py
first_var = 'hello'
# second.py
from first import first_var # we dropped the dot
second_var = first_var + ' world'
I hope this helps clear up some of the confusion about relative imports.
Say I am using Python 3 (and hence absolute imports) and my directory structure looks like this:
> package:
> sub_directory
__init__.py
sub_dir_file.py
sub_dir_file2.py
__init__.py
main_dir_file.py
In the file sub_dir_file.py I wish to import a function from sub_dir_file2.py. The catch is, I want to be able to run sub_dir_file.py with __name__ == '__main__', as well as import it in main_dir_file.py. Hence, if in sub_dir_file.py I use a relative import:
from .sub_dir_file2 import some_function
the module executes perfectly fine when run from main_dir_file.py, but throws an error when executed directly (as the relative import cannot be executed when __name__ == '__main__'. If I however use a normal absolute import, sub_dir_file.py will execute as a main, but cannot be imported from main_dir_file.py.
What would be the most elegant way of solving this problem? One obvious solution seems to be:
if __name__ == '__main__':
from sub_dir_file2 import some_function
else:
from .sub_dir_file2 import some_function
However, it doesn't seem very pythonic.
You should use the relative import syntax from .sub_dir_file2 import some_function or eventually the absolute syntax from package.sub_directory.sub_dir_file2 import some_function.
Then, in order to call one of the package sub module, it is simpler to use the -m option of the python interpreter to execute its content as the __main__ module.
Search sys.path for the named module and execute its contents as the
main module.
Since the argument is a module name, you must not give a file
extension (.py). The module name should be a valid absolute Python
module name, but the implementation may not always enforce this (e.g.
it may allow you to use a name that includes a hyphen).
For example:
> python -m package.main_dir_file
> python -m package.sub_directory.sub_dir_file
I would suggest using a main() function invoked if name is __main__. It's a good habit anyway, as far as I'm aware.
That way you can just call the imported module's main() yourself. It has other benefits, too, like allowing you to test or re-invoke the module without necessarily executing the file every time.
Suppose I have a module foo.py and a package foo/. If I call
import foo
which one will be loaded? How can I specify I want to load the module, or the package?
I believe the package will always get loaded. You can't work around this, as far as I know. So change either the package or the module name. Docs: http://docs.python.org/tutorial/modules.html#the-module-search-path
Actually, it is possible, by manually guiding the import machinery to use a .py file instead of directory. (This code is not well tested, but seems to work). UPDATE 2020: Note that this requires using custom import_module() function instead of normal import statement. However, with modern Python3 and its importlib, it might be possible to make the bare import statement to work the same way too. (Note that this answer shows flexibility which Python offers. It's not an encouragement to use this in your applications. Use this only if you know what you're doing.)
File foo.py
print "foo module loaded"
File foo/__init__.py
print "foo package loaded"
File test1.py
import foo
File test2.py
import os, imp
def import_module(dir, name):
""" load a module (not a package) with a given name
from the specified directory
"""
for description in imp.get_suffixes():
(suffix, mode, type) = description
if not suffix.startswith('.py'): continue
abs_path = os.path.join(dir, name + suffix)
if not os.path.exists(abs_path): continue
fh = open(abs_path)
return imp.load_module(name, fh, abs_path, (description))
import_module('.', 'foo')
Running
$ python test1.py
foo package loaded
$ python test2.py
foo module loaded
Maybe you want to move your classes from foo.py module to __init__.py.
This way you'll be able to import them from the package as well as importing optional subpackages:
File foo/__init__.py:
class Bar(object):
...
File foo/subfoo.py:
class SubBar(object):
...
File mymodule.py:
from foo import Bar
from foo.subfoo import SubBar
I'm using python and virtualenv/pip. I have a module installed via pip called test_utils (it's django-test-utils). Inside one of my django apps, I want to import that module. However I also have another file test_utils.py in the same directory. If I go import test_utils, then it will import this local file.
Is it possible to make python use a non-local / non-relative / global import? I suppose I can just rename my test_utils.py, but I'm curious.
You can switch the search order by changing sys.path:
del sys.path[0]
sys.path.append('')
This puts the current directory after the system search path, so local files won't shadow standard modules.
My problem was even more elaborate:
importing a global/site-packages module from a file with the same name
Working on aero the pm recycler I wanted access to the pip api, in particular pip.commands.search.SearchCommand from my adapter class Pip in source file pip.py.
In this case trying to modify sys.path is useless, I even went as far as wiping sys.path completely and adding the folder .../site-packages/pip...egg/ as the only item in sys.path and no luck.
I would still get:
print pip.__package__
# 'aero.adapters'
I found two options that did eventually work for me, they should work equally well for you:
using __builtin__.__import__() the built-in function
global_pip = __import__('pip.commands.search', {}, {}, ['SearchCommand'], -1)
SearchCommand = global_pip.SearchCommand
Reading the documentation though, suggests using the following method instead.
using importlib.import_module() the __import__ conv wrapper.
The documentation explains that import_module() is a minor subset of functionality from Python 3.1 to help ease transitioning from 2.7 to 3.1
from importlib import import_module
SearchCommand = import_module('pip.commands.search').SearchCommand
Both options get the job done while import_module() definitely feels more Pythonic if you ask me, would you agree?
nJoy!
I was able to force python to import the global one with
from __future__ import absolute_import
at the beginning of the file (this is the default in python 3.0)
You could reset your sys.path:
import sys
first = sys.path[0]
sys.path = sys.path[1:]
import test_utils
sys.path = first + sys.path
The first entry of sys.path is "always" (as in "per default": See python docs) the current directory, so if you remove it you will do a global import.
Since my test_utils was in a django project, I was able to go from ..test_utils import ... to import the global one.
Though, in first place, I would always consider keeping the name of local file not matching with any global module name, an easy workaround, without modifying 'sys.path' can be to include global module in some other file and then import this global module from that file.
Remember, this file must be in some other folder then in the folder where file with name matching with global module is.
For example.
./project/root/workarounds/global_imports.py
import test_utils as tutil
and then in
./project/root/mycode/test_utils.py
from project.root.workarounds.global_imports import tutil
# tutil is global test_utils
# you can also do
from project.root.workarounds.global_imports import test_utils
I have a Python project with following directory structure:
/(some files)
/model/(python files)
/tools/(more python files)
...
So, I have Python files in couple subdirectories and there are some
dependencies between directories as well: tools are used by model, etc. Now
my problem is that I want to make doctests for both models and tools,
and I want be able to run tests from command line like this: ./model/car.py .
I can make this work, but only with messy boilerplate code. I would like
to know what is the correct way, or is there any?
Question: How should I write my imports?
Thanx. Here is an example...
Content of tools/tool.py:
#!/usr/bin/env python
"""
>>> is_four(21)
False
>>> is_four(4)
True
"""
def is_four(val):
return val == 4
if __name__ == '__main__':
import doctest
doctest.testmod()
... and model/car.py:
#!/usr/bin/env python
"""
>>> car = Car()
>>> car.ok()
True
"""
from tools.tool import *
class Car(object):
def __init__(self):
self.tire_count = 4
def ok(self):
return is_four(self.tire_count)
if __name__ == '__main__':
import doctest
doctest.testmod()
By adding following lines in the begin of car.py it works, but doesn't look nice. :(
if __name__ == '__main__':
import sys
import os
sys.path.append(os.path.abspath(os.path.dirname('..')))
What you are trying to do is a relative import. It works fine in Python, but on the module level, not on the file system level. I know, this is confusing.
It means that if you run a script in a subdir, it doesn't see the upper dirs because for the running script, the root of the module is the current dir: there is no upper module.
So what are relative imports for?
Well, module in subdirs car import module in upper dirs as long as they are themself imported from a upperdir.
In your case it means you must run your scripts from "/" so it becomes the root of the module, and the submodules are allowed to use relative import.
A possible solution to your problem is to remove your if __name__ == "__main__" block and create /tests.py:
import doctest
from model import car
from tools import tool
doctest.testmod(car)
doctest.testmod(tool)
Then run in too launch all the tests.
Ultimately you will want to automatize the process, a simple solution is to use unittest so you can create test suites and just add the module names you want to test:
import unittest
import doctest
modules = ("model.car",
"tools.tool")
suite = unittest.TestSuite()
for mod in modules:
suite.addTest(doctest.DocTestSuite(mod))
runner = unittest.TextTestRunner()
runner.run(suite)
Another solution (recommended) is to use a tool such as nose that automates this for you.
easy_install nose
nosetests --with-doctest # done :-)
And by the way, avoid from x import *. This works for quick scripts, but when your program will grow, you really will need to explicitly name what you import. Either import x or from x import y
Use packages. Add an __init__.py file to your working directory and all subfolders then your imports will search the parent directories if it doesn't find the module in the current directory.
See http://www.network-theory.co.uk/docs/pytut/Packages.html
Also this question is a duplicate of:
Import a module from a relative path
Don't frob sys.path in this manner. Instead either use $PYTHONPATH to force the base directory in when invoking python, or use python -m model.car from that directory.