I am new at Python and coming from Java background.
Suppose I am developing a Python project with package hello:
hello_python/
hello/
hello.py
__init__.py
test/
test_hello1.py
test_hello2.py
I believe the project structure is correct.
Suppose hello.py contains function do_hello() I want to use in tests. How to import do_hello in tests test_hello1.py and test_hello2.py ?
You've 2 small issues here. Firstly, you're running your test command from the wrong directory, and secondly you've not quite structured your project right.
Usually when I'm developing a python project I try to keep everything focussed around the project's root, which would be hello_python/ in your case. Python has the current working directory on its load path by default, so if you've got a project like this:
hello_python/
hello/
hello.py
__init__.py
test/
test_hello1.py
test_hello2.py
# hello/hello.py
def do_hello():
return 'hello'
# test/test_hello.py
import unittest2
from hello.hello import do_hello
class HelloTest(unittest2.TestCase):
def test_hello(self):
self.assertEqual(do_hello(), 'hello')
if __name__ == '__main__':
unittest2.main()
Second, test isn't a module right now, since you've missed the __init__.py in that directory. You should have a hierarchy that looks like this:
hello_python/
hello/
hello.py
__init__.py
test/
__init__.py # <= This is what you were missing
test_hello1.py
test_hello2.py
When I try that on my machine, running python -m unittest test.hello_test works fine for me.
You may find that this is still a bit cumbersome. I'd strongly recommend installing nose, which will let you simply invoke nosetests from your project's root to find and execute all your tests automagically - providing you've got correct modules with __init__.pys.
Related
I have made an app with lots of packages and modules. I'm developing these packages (within their directories) and since I'm executing files within the package directory the import paths for modules/packages change.
For instance, the project looks a little bit like this...
├── app.py
└── utils
├── database.py
└── robot
├── __init__.py
├── run.py
├── recognition.py
└── housekeeping.py
If I executed python3 app.py then import utils.database is valid.
But if I was in the package and ran python3 run.py then import utils.database is not valid.
I want to know how to execute functions and import the modules/packages without getting an error.
I have got this working, but I don't think it is right. So far, I've tried using __name__ == "__main__" in every main .py file and using sys.path.append, but I feel like there must be an easier better-looking way.
This is what the inside of run.py looks like, I also had to do the same in database.py, is this the best way?
if __name__ == "__main__":
import sys
sys.path.append('../../')
from utils.robot.recognition import *
from utils.robot.housekeeping import *
import utils.database as db
Set the $PYTHONPATH environment variable to the directory containing utils, then your imports will work.
Never run a module contained within a package directly. It's problematic if the same is run directly and can also be imported from the package, because then there will be two copies of the module in the interpreter, which guarantees confusion.
Instead, use the -m interpreter option:
python -m utils.robot.run
Alternatively, if you never need to import run.py from other modules, you can remove the file from the package entirely and keep it in a separate 'scripts' directory, and then you don't need to use -m. But you still need to set up sys.path, either using $PYTHONPATH as before, or by doing sys.path manipulations. If you choose to do sys.path manipulations, then you should make it work regardless of what the current working directory is:
sys.path.append(os.path.join(os.path.dirname(__file__), '../../'))
I have a python app that I’m attempting to unit test. The code is sat in a src directory the tests in a test directory.
The problem comes when trying to test the class containing my main method. This class sits in the src directory and has references to classes in other modules that all sit in the src directory. In order to execute my app I have the references in this format from <module> import <class>
When I come to test this main class I write a test class and place it in the test directory. Running the test I find I have problems with the imports in the main class. To resolve I have to change the imports to be from from src.<module> import <class> Having done this my tests now pass but the app itself fails.
How can I resolve the issue so that my imports are valid for both my unit tests and normal execution?
It may fail due to the way import works.
I would recommend always using absolute imports. This means, that when creating a package, all imports within that package import all the way from the top level down: from package.module import thing. And more importantly, installing your package - not as a normal Python package, but with a symbolic link allows you to edit your source code while it still is considered an installed package. This way you can position your tests and scripts however you want them and the thing will always work.
Installing your package is done with pip install -e . from root. See a packaging tutorial for an example of packaging if you're not familiar with it.
An example package structure might be:
root/
setup.py
MANIFEST.in
...
package/
__init__.py
app.py
tests.py
module/
__init__.py
component.py
module_test.py
# Or have a tests folder under root
tests/
test_module.py
tests.py
And for example in app.py, and everywhere really:
from package.module.component import Class
from package.module import thing # that was declared in module/__init__.py
So to your case, if I understood it correctly, you'd have:
root/
setup.py
MANIFEST.in
...
src/
__init__.py
app.py
module.py
tests/
test_module.py
tests.py
I have a bunch of unittest test cases in separate directories. There is also a directory which just contains helper scripts for the tests. So my file tree looks like this
test_dir1
test_dir2
test_dir3
helper_scripts
Each python file in test_dir* will have these lines:
import sys
sys.path.append('../helper_scripts')
import helper_script
This all works fine, as long as I run the tests from within their directory. However, I would like to be at the project root and just run:
py.test
and have it traverse all the directories and run each test it finds. The problem is that the tests are being run from the wrong directory, so the sys.path.append doesn't append the helper_scripts directory, it appends the parent of the project root. This makes all the imports fail with an Import Error.
Is there a way to tell py.test to run the test scripts from their directory? ie. change the cwd before executing them? If not, is there another test runner I can use that will?
What I usually do is structure my project like this:
myproject/
setup.py
myproject/
__init__.py
mymodule.py
tests/
__init__.py
test_dir1/
test_mymodule.py
helper_scripts/
__init__.py
helper_script.py
For running tests, I use a virtualenv with myproject installed in development mode using one of the following commands in the myproject root directory:
pip install -e .
python setup.py develop
Now in test_mymodule.py I can just say
from myproject.tests.helper_scripts import helper_script
I can then just run pytest and there's no need to change the working directory in tests at all.
See Pytest's Good Integration Practices for a great summary of pros and cons for different project directory structures.
os.chdir("newdir")
will change your current working directory
I would suggest that you instead configure your environment so that import helper_scripts will work regardless of the current directory. This is the recommended approach.
If you absolutely must though, you can use relative imports instead:
from .. import helper_script
packageName\
__init__.py
src\
__init__.py
someFile.py
classes\
__init__.py
engine1.py
engine2.py
engine.py
tests\
__init__.py
myTests.py
temp\
I'm working on OOP with Python, and I have a few questions.
I understand that __init__.py defines the folder as a module, but what I do not understand is how this benefits me.
How would I run myTests.py if it needs to import a class from the packageName/src/classes folder?
$ python packageName/tests/myTests.py
The above call is how I guess I'd intend to run my tests. If the structure is the case, how can I import the classes?
from ..src.classes.engine1 import *
As #Mike mentioned, The Hitchhiker's Guide to Packaging would be a great place to start.
Now, as far as your tests go: for various Reasons, you can't just run python packageName/tests/myTests.py (specifically, Python will get confused about the package layout — when you run a file with python foo.py, it's expected to be outside a package, so the from ..src … line will fail with Attempted relative import in non-package).
Instead, you need to start your tests from outside the package… Either using a test runner like nose or unittest2, or writing a simple script in the top level directory (ie, above packageName/):
$ ls
packageName/ run_tests
$ cat run_tests.py
from packageName.tests.myTests import main
main()
Then put a main() function in myTests().
However, using a test runner is much simpler. If you've written your tests correctly, you can run them with:
$ ls
packageName/
$ nosetests
..................................
----------------------------------------------------------------------
Ran 34 tests in 1.440s
OK
Relative import not working properly in python2.6.5 getting "ValueError: Attempted relative import in non-package".
I am having all those __init__.py in proper place.
I have seen that error before when running a script that is actually inside a package. To the interpreter, it appears as though the package is not a package.
Try taking the script into another directory, putting your package inside your pythonpath, and import absolutely. Then, relative imports inside your package will work. NOTE: you can STILL not relatively import inside the end script - the easiest thing to do in this case is to make a "wrapper" script, that simply calls some entry point in your package.
You can go even further here by using setuptools to create a setup.py for your package, to make it distributable. Then, as a part of that, entry points would allow you to autogenerate scripts that called your package's code.
EDIT:
From your comment, it appears as though I wasn't quite clear. I'm not 100% sure of your directory structure because your comment above wasn't formatted, but I took it to be like this:
PythonEvent/
main.py
__init__.py
DBConnector/
__init__.py
connector.py
service/
__init__.py
myservice.py
When in myservice.py you have the line from ..DBConnector.connector import DBUpdate, the interpreter tries to import it relatively, UNLESS you are running myservice.py directly. This is what it appears you are doing.
Try making another dummy script outside of PythonEvent/ that is simply as follows:
from PythonEvent.service import myservice
if __name__ == '__main__':
myservice.main() # or whatever the entry point is called in myservice.
Then, set your PYTHONPATH environment variable to point to the parent directory of PythonEvent/ (or move PythonEvent/ to your site-packages).
main.py
setup.py
Main Package/ ->
__init__.py
subpackage_a/ ->
__init__.py
module_a.py
subpackage_b/ ->
__init__.py
module_b.py
i)
1.You run python main.py
2.main.py does: import app.package_a.module_a
3.module_a.py does import app.package_b.module_b
ii)
Alternatively 2 or 3 could use: from app.package_a import module_a
That will work as long as you have app in your PYTHONPATH. main.py could be anywhere then.
So you write a setup.py to copy (install) the whole app package and subpackages to the target system's python folders, and main.py to target system's script folders.
Thanks to https://stackoverflow.com/a/1083169