Develop Reference

What is the proper way to write unit tests on modules (in subdirectories) which use other modules in the parent directory in Python

Tried to describe it as best as I could in the title. I'm quite new to using python for a big project. Usually, I only use it for small scripts. I read through the manual and found out what is the proper way to make modules.
So I have the following file structure set up now:
- main.py
- module1
- __init__.py
- thing1.py
- thing2.py
- module2
- __init__.py
- thingA.py
- thingB.py
My project will run when I start main.py.
I know I can write simple unit tests for every 'thing' by using the if __name__ == '__main__': in every one of them which is what I do.
The file thingB.py needs thing2.py to work. As long as main.py is started, thingB will find thing2. But when I start thingB directly (to make its unit test run) in its own directory, it won't find thing2.
What is the proper approach for this?

You should run everything from your main directory:
python -m module1.thing1
python -m module2.thingA
this way module1 and module2 can import from each other. Unit tests should be run in the same way:
python -m unittest discover module1
python -m unittest discover module2

Module not found running on command line

I have the following project structure:
project/
example/
__init__.py
foo.py
boo.py
meh.py
tests/
example/
test_foo.py
test_boo.py
test_meh.py
As example, I'm importing foo.py in boo.py as import example.foo as f. And I'm running tests with python3 -m pytest -s -v --cov tests on root folder (project). The unit tests are running very smooth but when I try to run a single file as python3 example/boo.py I got an error:
ModuleNotFoundError: No module named 'example'

Modules inside a package shouldn't really be run (some exceptions).
But, you can set a PYTHONPATH before running the module if you really want. For a one off, use e.g.
PYTHONPATH=$(pwd) python3 example/boo.py
An alternative is to use relative imports: from . import foo as f inside boo.py. But that still implies that modules shouldn't really be run.
To elaborate a bit more:
A module should be imported, not run like a script. That is what a module is for. If, for some reason, you really really feel you need to execute module, then 1/ reconsider, 2/ rewrite your module, 3/ wrap a script around that module by calling the necessary function(s) inside the module (and keep the script itself relatively short).
Note that setuptools already has this functionality through entry points.
A simpler alternative is to use a proper
if __name__ == '__main__':
main()
line at the end of your module, where main() calls into your module functionality, then execute the module using the Python -m switch:
python -m mypackage.mymodule
But, again, try and limit this functionality.

It's usually a problem with environment variables. You can force the path using the following, and the import should work under all circumstances:
import sys
sys.path.append("/absolute/module/path")
import local_module

How to import tested modules in tests?

I am new at Python and coming from Java background.
Suppose I am developing a Python project with package hello:
hello_python/
hello/
hello.py
__init__.py
test/
test_hello1.py
test_hello2.py
I believe the project structure is correct.
Suppose hello.py contains function do_hello() I want to use in tests. How to import do_hello in tests test_hello1.py and test_hello2.py ?

You've 2 small issues here. Firstly, you're running your test command from the wrong directory, and secondly you've not quite structured your project right.
Usually when I'm developing a python project I try to keep everything focussed around the project's root, which would be hello_python/ in your case. Python has the current working directory on its load path by default, so if you've got a project like this:
hello_python/
hello/
hello.py
__init__.py
test/
test_hello1.py
test_hello2.py
# hello/hello.py
def do_hello():
return 'hello'
# test/test_hello.py
import unittest2
from hello.hello import do_hello
class HelloTest(unittest2.TestCase):
def test_hello(self):
self.assertEqual(do_hello(), 'hello')
if __name__ == '__main__':
unittest2.main()
Second, test isn't a module right now, since you've missed the __init__.py in that directory. You should have a hierarchy that looks like this:
hello_python/
hello/
hello.py
__init__.py
test/
__init__.py # <= This is what you were missing
test_hello1.py
test_hello2.py
When I try that on my machine, running python -m unittest test.hello_test works fine for me.
You may find that this is still a bit cumbersome. I'd strongly recommend installing nose, which will let you simply invoke nosetests from your project's root to find and execute all your tests automagically - providing you've got correct modules with __init__.pys.

Change cwd before running tests

I have a bunch of unittest test cases in separate directories. There is also a directory which just contains helper scripts for the tests. So my file tree looks like this
test_dir1
test_dir2
test_dir3
helper_scripts
Each python file in test_dir* will have these lines:
import sys
sys.path.append('../helper_scripts')
import helper_script
This all works fine, as long as I run the tests from within their directory. However, I would like to be at the project root and just run:
py.test
and have it traverse all the directories and run each test it finds. The problem is that the tests are being run from the wrong directory, so the sys.path.append doesn't append the helper_scripts directory, it appends the parent of the project root. This makes all the imports fail with an Import Error.
Is there a way to tell py.test to run the test scripts from their directory? ie. change the cwd before executing them? If not, is there another test runner I can use that will?

What I usually do is structure my project like this:
myproject/
setup.py
myproject/
__init__.py
mymodule.py
tests/
__init__.py
test_dir1/
test_mymodule.py
helper_scripts/
__init__.py
helper_script.py
For running tests, I use a virtualenv with myproject installed in development mode using one of the following commands in the myproject root directory:
pip install -e .
python setup.py develop
Now in test_mymodule.py I can just say
from myproject.tests.helper_scripts import helper_script
I can then just run pytest and there's no need to change the working directory in tests at all.
See Pytest's Good Integration Practices for a great summary of pros and cons for different project directory structures.

os.chdir("newdir")
will change your current working directory

I would suggest that you instead configure your environment so that import helper_scripts will work regardless of the current directory. This is the recommended approach.
If you absolutely must though, you can use relative imports instead:
from .. import helper_script

python unittests, do I need to import from the tests/__init__.py?

I have my code organized like this:
base_project_dir/src
/tests
/test1.py
/test2.py
in test1.py and test2.py I have the classes extending unittest.TestCase
according to python API doc, I should be able to run
python -m unittest tests from base dir and run all tests.
but doing so shows that it finds 0 tests. so I added
from test1 import *
from test2 import *
into tests/__init__.py
now the above command works. but when I want to run individual tests, it sources the module init, which forcefully run all tests.
what is the correct way to organize this?
thanks
Yang

Try this in your main project dir:
python -m unittest discover -v
This way you don't change the location for your relative imports. When you do python -m unittest tests it switches to this directory first and then can't import the projects code anymore.