I have a Python project with following directory structure:
/(some files)
/model/(python files)
/tools/(more python files)
...
So, I have Python files in couple subdirectories and there are some
dependencies between directories as well: tools are used by model, etc. Now
my problem is that I want to make doctests for both models and tools,
and I want be able to run tests from command line like this: ./model/car.py .
I can make this work, but only with messy boilerplate code. I would like
to know what is the correct way, or is there any?
Question: How should I write my imports?
Thanx. Here is an example...
Content of tools/tool.py:
#!/usr/bin/env python
"""
>>> is_four(21)
False
>>> is_four(4)
True
"""
def is_four(val):
return val == 4
if __name__ == '__main__':
import doctest
doctest.testmod()
... and model/car.py:
#!/usr/bin/env python
"""
>>> car = Car()
>>> car.ok()
True
"""
from tools.tool import *
class Car(object):
def __init__(self):
self.tire_count = 4
def ok(self):
return is_four(self.tire_count)
if __name__ == '__main__':
import doctest
doctest.testmod()
By adding following lines in the begin of car.py it works, but doesn't look nice. :(
if __name__ == '__main__':
import sys
import os
sys.path.append(os.path.abspath(os.path.dirname('..')))
What you are trying to do is a relative import. It works fine in Python, but on the module level, not on the file system level. I know, this is confusing.
It means that if you run a script in a subdir, it doesn't see the upper dirs because for the running script, the root of the module is the current dir: there is no upper module.
So what are relative imports for?
Well, module in subdirs car import module in upper dirs as long as they are themself imported from a upperdir.
In your case it means you must run your scripts from "/" so it becomes the root of the module, and the submodules are allowed to use relative import.
A possible solution to your problem is to remove your if __name__ == "__main__" block and create /tests.py:
import doctest
from model import car
from tools import tool
doctest.testmod(car)
doctest.testmod(tool)
Then run in too launch all the tests.
Ultimately you will want to automatize the process, a simple solution is to use unittest so you can create test suites and just add the module names you want to test:
import unittest
import doctest
modules = ("model.car",
"tools.tool")
suite = unittest.TestSuite()
for mod in modules:
suite.addTest(doctest.DocTestSuite(mod))
runner = unittest.TextTestRunner()
runner.run(suite)
Another solution (recommended) is to use a tool such as nose that automates this for you.
easy_install nose
nosetests --with-doctest # done :-)
And by the way, avoid from x import *. This works for quick scripts, but when your program will grow, you really will need to explicitly name what you import. Either import x or from x import y
Use packages. Add an __init__.py file to your working directory and all subfolders then your imports will search the parent directories if it doesn't find the module in the current directory.
See http://www.network-theory.co.uk/docs/pytut/Packages.html
Also this question is a duplicate of:
Import a module from a relative path
Don't frob sys.path in this manner. Instead either use $PYTHONPATH to force the base directory in when invoking python, or use python -m model.car from that directory.
Related
Okay, pictured are the two MWE's and their failure messages.
In words, I can't naturally import, from the same subdirectory, into a module that I want to run as main for testing purposes. That's the first screenshot.
The next screenshot shows that I cannot naturally import, from the project root, into a subdirectory module that I am similarly running as main for testing purposes.
Visuals:
What I don't want to do is hack my way through like I've been doing in all my projects with lines such as:
if __name__ == '__main__:
import sys
sys.append('..')
This is inelegant because it doesn't withstand changing the folder structure.
I've also tried putting in blank __init__.py's into the subdirs and also tried putting in those the lines such as:
from .e import E
Nothing easy seems to work. It's really a bummer when I can't handle basic file structures when importing things.
So what is the general fix that solves both of these issues for good and forever?
Text Code:
<proj root>/d.py:
class D:
pass
<proj root>/root.py:
from b.a import A
print("import from project root main")
<proj root>/b/a.py:
from b.c import C
class A(C):
pass
if __name__ == '__main__':
print('import from project subdir')
<proj root>/b/c.py:
class C:
pass
<proj root>/b/e.py:
from d import D
class E(D):
pass
if __name__ == '__main__':
print("e can't import something from root")
First set entry point to a.py and for the other error set entry point to e.py.
Thank you!
For the first error, you create an __init__.py in the subdir b with the contents:
from .c import *
The same does not work for the root directory however. But the most succinct way I've found, without having to do a sys.path.append() is to open up your project properties, and add precisely .. to the Python Path:
In WingWare:
And that solves the second issue. There is a warning when you add simply .. that you should use absolute paths, but I'm okay with that.
If you work without an IDE simply modify the PYTHONPATH environment variable.
Happy Coding!
Say I am using Python 3 (and hence absolute imports) and my directory structure looks like this:
> package:
> sub_directory
__init__.py
sub_dir_file.py
sub_dir_file2.py
__init__.py
main_dir_file.py
In the file sub_dir_file.py I wish to import a function from sub_dir_file2.py. The catch is, I want to be able to run sub_dir_file.py with __name__ == '__main__', as well as import it in main_dir_file.py. Hence, if in sub_dir_file.py I use a relative import:
from .sub_dir_file2 import some_function
the module executes perfectly fine when run from main_dir_file.py, but throws an error when executed directly (as the relative import cannot be executed when __name__ == '__main__'. If I however use a normal absolute import, sub_dir_file.py will execute as a main, but cannot be imported from main_dir_file.py.
What would be the most elegant way of solving this problem? One obvious solution seems to be:
if __name__ == '__main__':
from sub_dir_file2 import some_function
else:
from .sub_dir_file2 import some_function
However, it doesn't seem very pythonic.
You should use the relative import syntax from .sub_dir_file2 import some_function or eventually the absolute syntax from package.sub_directory.sub_dir_file2 import some_function.
Then, in order to call one of the package sub module, it is simpler to use the -m option of the python interpreter to execute its content as the __main__ module.
Search sys.path for the named module and execute its contents as the
main module.
Since the argument is a module name, you must not give a file
extension (.py). The module name should be a valid absolute Python
module name, but the implementation may not always enforce this (e.g.
it may allow you to use a name that includes a hyphen).
For example:
> python -m package.main_dir_file
> python -m package.sub_directory.sub_dir_file
I would suggest using a main() function invoked if name is __main__. It's a good habit anyway, as far as I'm aware.
That way you can just call the imported module's main() yourself. It has other benefits, too, like allowing you to test or re-invoke the module without necessarily executing the file every time.
I have a folder with all my unittests. They all include:
if __name__ == '__main__':
unittest.TextTestRunner(verbosity=2).run(suite())
So to test them I only have to import the test script. I have a test_all script that does all the unittests by importing them one by one. Currently this looks like this:
from pyMS.test import test_baseFunctions
from pyMS.test import test_compareFeatureXMLmzML
from pyMS.test import test_config
from pyMS.test import test_featureFunctions
from pyMS.test import test_fileHandling
from pyMS.test import test_mzmlFunctions
from pyMS.test import test_output
from pyMS.test import test_parseFeatureXML
from pyMS.test import test_rFunctions
from pyMS.test import test_rPlots
[...]
This means that every time I add or remove a new test I need to change the imports. So instead I want to use
from pyMS.test import *
However, this does not run any of the code. I'm curious for the reason why import * does not run the code.
Also, if someone knows a solution (that is note nose) to run all unittests without having to import them one by one would be great.
Thanks
Niek
If you use python 2.7 you can use the command line:
python -m unittest discover
This will automatically find and execute all tests in all subdirectories. For more options, see:
python -m unittest -h
This module has been backported to python 2.3+ and can be downloaded here. If you use the backport there's an included command line script called unit2 or unit2.py (your choice), invoked like this:
unit2.py discover
As for from XXX import *, this actually imports everything in the namespace of the file XXX/__init__.py. Put the following code in __init__.py to automatically load any direct submodules:
import os
all_files = os.listdir(os.path.dirname(__file__))
modules = [x for x in all_files if x.endswith(".py") and not x.startswith("_")]
__all__ = [x.rpartition(".")[0] for x in modules]
A detailed explanation of how this works can be found in the python docs for the __all__ global variable.
The easiest way (without using an external tool) to run all the tests is probably to use TestLoader.discover.
__name__ is only set to "__main__" for the initial python file being read by the interpreter. This allows the module to be imported by other modules without the code after the if __name__ == "__main__": being executed.
Any code that is not protected by the if __name__ == "__main__": will be executed. So you could remove it in each of the files then when you do the import unittest.TextTestRunner(verbosity=2).run(suite()) will be executed.
A better method is to use a method from unittest.TestLoader() to load your tests into a suite and then give that suite to unittest.TextTestRunner. The loader can then be automated without needing to change the imports in the test file. Add a test file to the directory structure and the tests will be executed automatically.
In a big application I am working, several people import same modules differently e.g.
import x
or
from y import x
the side effects of that is x is imported twice and may introduce very subtle bugs, if someone is relying on global attributes
e.g. suppose I have a package mypakcage with three file mymodule.py, main.py and init.py
mymodule.py contents
l = []
class A(object): pass
main.py contents
def add(x):
from mypackage import mymodule
mymodule.l.append(x)
print "updated list",mymodule.l
def get():
import mymodule
return mymodule.l
add(1)
print "lets check",get()
add(1)
print "lets check again",get()
it prints
updated list [1]
lets check []
updated list [1, 1]
lets check again []
because now there are two lists in two different modules, similarly class A is different
To me it looks serious enough because classes itself will be treated differently
e.g. below code prints False
def create():
from mypackage import mymodule
return mymodule.A()
def check(a):
import mymodule
return isinstance(a, mymodule.A)
print check(create())
Question:
Is there any way to avoid this? except enforcing that module should be imported one way onyl. Can't this be handled by python import mechanism, I have seen several bugs related to this in django code and elsewhere too.
Each module namespace is imported only once. Issue is, you're importing them differently. On the first you're importing from the global package, and on the second you're doing a local, non-packaged import. Python sees modules as different. The first import is internally cached as mypackage.mymodule and the second one as mymodule only.
A way to solve this is to always use absolute imports. That is, always give your module absolute import paths from the top-level package onwards:
def add(x):
from mypackage import mymodule
mymodule.l.append(x)
print "updated list",mymodule.l
def get():
from mypackage import mymodule
return mymodule.l
Remember that your entry point (the file you run, main.py) also should be outside the package. When you want the entry point code to be inside the package, usually you use a run a small script instead. Example:
runme.py, outside the package:
from mypackage.main import main
main()
And in main.py you add:
def main():
# your code
I find this document by Jp Calderone to be a great tip on how to (not) structure your python project. Following it you won't have issues. Pay attention to the bin folder - it is outside the package. I'll reproduce the entire text here:
Filesystem structure of a Python project
Do:
name the directory something
related to your project. For example,
if your project is named "Twisted",
name the top-level directory for its
source files Twisted. When you do
releases, you should include a version
number suffix: Twisted-2.5.
create a directory Twisted/bin and
put your executables there, if you
have any. Don't give them a .py
extension, even if they are Python
source files. Don't put any code in
them except an import of and call to a
main function defined somewhere else
in your projects.
If your project
is expressable as a single Python
source file, then put it into the
directory and name it something
related to your project. For example,
Twisted/twisted.py. If you need
multiple source files, create a
package instead (Twisted/twisted/,
with an empty
Twisted/twisted/__init__.py) and
place your source files in it. For
example,
Twisted/twisted/internet.py.
put
your unit tests in a sub-package of
your package (note - this means that
the single Python source file option
above was a trick - you always need at
least one other file for your unit
tests). For example,
Twisted/twisted/test/. Of course,
make it a package with
Twisted/twisted/test/__init__.py.
Place tests in files like
Twisted/twisted/test/test_internet.py.
add Twisted/README and Twisted/setup.py to explain and
install your software, respectively,
if you're feeling nice.
Don't:
put your source in a directory
called src or lib. This makes it
hard to run without installing.
put
your tests outside of your Python
package. This makes it hard to run the
tests against an installed version.
create a package that only has a
__init__.py and then put all your
code into __init__.py. Just make a
module instead of a package, it's
simpler.
try to come up with
magical hacks to make Python able to
import your module or package without
having the user add the directory
containing it to their import path
(either via PYTHONPATH or some other
mechanism). You will not correctly
handle all cases and users will get
angry at you when your software
doesn't work in their environment.
I can only replicate this if main.py is the file you are actually running. In that case you will get the current directory of main.py on the sys path. But you apparently also have a system path set so that mypackage can be imported.
Python will in that situation not realize that mymodule and mypackage.mymodule is the same module, and you get this effect. This change illustrates this:
def add(x):
from mypackage import mymodule
print "mypackage.mymodule path", mymodule
mymodule.l.append(x)
print "updated list",mymodule.l
def get():
import mymodule
print "mymodule path", mymodule
return mymodule.l
add(1)
print "lets check",get()
add(1)
print "lets check again",get()
$ export PYTHONPATH=.
$ python mypackage/main.py
mypackage.mymodule path <module 'mypackage.mymodule' from '/tmp/mypackage/mymodule.pyc'>
mymodule path <module 'mymodule' from '/tmp/mypackage/mymodule.pyc'>
But add another mainfile, in the currect directory:
realmain.py:
from mypackage import main
and the result is different:
mypackage.mymodule path <module 'mypackage.mymodule' from '/tmp/mypackage/mymodule.pyc'>
mymodule path <module 'mypackage.mymodule' from '/tmp/mypackage/mymodule.pyc'>
So I suspect that you have your main python file within the package. And in that case the solution is to not do that. :-)
I don't do tests, but I'd like to start. I have some questions :
Is it ok to use the unittest module for that? From what I understand, the unittest module will run any method starting with test.
if I have a separate directory for the tests ( consider a directory named tests ), how would I import the code I'm testing? Do I need to use the imp module? Here's the directory structure :
src/
tests/
Another good way to start tests with your Python code is use the doctest module, whereby you include tests inside method and class comments. The neat bit is that these serve as code examples, and therefore, partial documentation. Extremely easy to do, too.
It's fine to use unittest. This module will run methods beginning with test for classes which inherit from unittest.TestCase.
There's no need to use the imp module - your test module would just import the code under test as normal. You might need to add the src directory to your path:
import sys
sys.path.append('../src') # OnLinux - use r'..\src' for Windows
This code would be in your test module(s) before any import of your modules.
A better approach is to use the OS environment variable PYTHONPATH.
(Windows) SET PYTHONPATH=path\to\module; python test.py
(Linux) PYTHONPATH=path/to/module; python test.py
An alternative to unittest is nose.
As mentioned by Sebastian P., Mark Pilgrim's Dive Into Python has great chapters on unit testing and test-driven development using Python's unittest module. I used these chapters to get started with testing, myself.
I wrote a blog post describing my approach to importing modules for testing. Note that it solves the shortcoming of Vinay Sajip's approach, which will not work if you call the testing module from anywhere but the directory in which it resides. A reader posted a nice solution in the comments of my blog post as well.
S. Lott hints at a method using PYTHONPATH in Vinay's post; I hope he will expound on it.
I had a bit of problem with the whole seperate directory for tests issue.
The way I solved it was by having a test runner in the source directory with the following code:
import sys, os, re, unittest
# Run all tests in t/
def regressionTest():
path = os.path.abspath(os.path.dirname(sys.argv[0])) + '/t'
files = os.listdir(path)
test = re.compile("^t_.+\.py$", re.IGNORECASE)
files = filter(test.search, files)
filenameToModuleName = lambda f: 't.'+os.path.splitext(f)[0]
moduleNames = map(filenameToModuleName, files)
modules = map(__import__, moduleNames)
modules = map(lambda name: sys.modules[name], moduleNames)
load = unittest.defaultTestLoader.loadTestsFromModule
return unittest.TestSuite(map(load, modules))
suite = regressionTest()
if __name__ == "__main__":
unittest.TextTestRunner(verbosity=2).run(suite)
Then I had a folder named t containing all my tests, named t_<something>.py.
If you need help on getting started with unit testing in Python, I can recommend the official documentation and Dive Into Python among other things.