I have a project hierarchy like below, when I run python src/bot/main I got no error. While if I run python -m src.bot.main I got error. Why?
This is my file hierarchy:
MyProject
└── src
├── __init__.py
├── bot
│ ├── __init__.py
│ ├── main.py
│ └── sib1.py
└── mod
├── __init__.py
└── module1.py
This is the content of main.py:
import sys
if __name__ == "__main__":
# sys.path will print the parent folder.
print(sys.path, end="\n\n")
# my problem is on this line.
import sib1
sib1.test()
The error:
Traceback (most recent call last):
File "/usr/local/Caskroom/miniconda/base/envs/test/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/Caskroom/miniconda/base/envs/test/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/Users/me/Desktop/test_py/src/bot/main.py", line 16, in <module>
import sib1
ModuleNotFoundError: No module named 'sib1'
Some conclusion I've made so far:
Since the output of sys.path in both cases include /Users/me/Desktop/MyProject, the reason should not related to scope?
The output of sys.path of both python -m src.bot.main and python src/bot/main:
(test) ✔ me Desktop/test_py % python -m src.bot.main
['/Users/me/Desktop/test_py', '/usr/local/Caskroom/miniconda/base/envs/test/lib/python39.zip', '/usr/local/Caskroom/miniconda/base/envs/test/lib/python3.9', '/usr/local/Caskroom/miniconda/base/envs/test/lib/python3.9/lib-dynload', '/usr/local/Caskroom/miniconda/base/envs/test/lib/python3.9/site-packages']
I will try my best to clarify each of my confusions in a Q&A form, and organize #Brain's comments and during the way:
Q1: when I run python src/bot/main I got no error.
The sys.path will include the current directory containing the file main.py, i.e. the interpreter will see the file MyProject/src/bot:
import sib1
is (logically) equivalent to:
import /path/to/MyProject/src/bot/sib1.py
Hence, no error.
BUT this is called IMPLICIT-RELATIVE-IMPORT, which should be used only on the main script. See Example.
Q2: While if I run python -m src.bot.main I got error. Why?
Now it's time to quote #Brain's valuable (first) comment:
Using python -m src.bot.main tells Python that src is a top-level package. Everything below src in the directory structure will be considered submodules/subpackages of src. The proper name for sib1 under that organization is src.bot.sib1. No top-level module named sib1 exists as far as Python is concerned.
(emphasis by me)
So:
The -m option will provide some context information(details, I recommend reading the first two high-upvoted answers entirely) for the file you're going to run.
Now every absolute (import-)path that does not begin with src. will be regarded as built-in or third-party libraries installed in your virtual environment. Since I didn't install any package called sib1, you got the error. (Ignore about the inclusion of the current directory in the sys.path. I guess this is for compatibility reason. Remember that we DONT want IMPLICIT-RELATIVE-IMPORT in python3)
You can use relative path BUT you should not go out of the src (since src is the top-level package in the execution) by prepending too many .'s.
For example this will work:
from . import sib1
from ..mod import module1 # The `..` is equivalent to the MyProject/src.
module1.hello()
sib1.test()
Finally, don't test your package by inserting many if __name__ == '__main__'. Do so by professional tools:
If you only need to run submodules for testing purpose, you could consider using more robust testing tools like doctest (lightweight) or unittest (heavyweight).
Related
What I have tried
First let me say, I have read maybe 5 or 6 other stackoverflow questions related to this and many things have been said about this. Here's what I found and what I tried:
"Do sys.path.append(<root dir>) before other imports."
Issue: This works, but it's against PEP standard practice so I'd rather find a solution that does not require modifying sys.path.
"Use the imp library to find your module."
Issue: The solution outlined in that answer is deprecated as importlib is the new imp, and I'm not sure how to use importlib even after looking at the docs.
"Add an empty __init__.py file in your root directory."
Issue: This does not work.
Details of my problem
Here is my file structure:
Project/
├ subdir/
| └ script.py
├ __init__.py
└ module.py
(Note that I left the empty __init__.py file in there just in case it's needed.)
I'm trying to import from module.py to subdir/script.py:
Contents of module.py:
def func():
print('func was called')
Contents of subdir/script.py:
from module import func
func()
However, when I run python3 script.py in terminal (on Mac), I get this error message:
Traceback (most recent call last):
File "script.py", line 1, in <module>
from module import func
ModuleNotFoundError: No module named 'module'
I would appreciate a straightforward solution that does not require modifying sys.path.
Probably the easiest way to solve your problem is to restructure your project:
Project
├── script.py
└── module
├── __init__.py
└── module.py
I solved my issue by setting PYTHONPATH. In my case, it works with this.
PYTHONPATH=<root dir> python3 subdir/script.py
I created a project that runs fine locally, but fails when run from PyPi deploy/install, with ModuleNotFound. The link contains a very small sample, and run / deploy instructions. My original thought was regarding inheritance, but it seems more basic.
Traceback (most recent call last):
File "/Users/val/python/vscode/inheritance/venv/bin/inheritance-run", line 5, in <module>
from inheritance.run import start
ModuleNotFoundError: No module named 'inheritance'
Your setup.py has:
packages=find_packages(),
This will find modules (directories with __init__.py files) in the same directory as your setup.py file, but you don't have any. Instead, you have a base.py file.
If you want to be able to import inheritance, you should move this into an __init__.py file in new directory named inheritance, so you have something like:
.
├── inheritance
│ └── __init__.py
└── setup.py
Then setuptools will find this module and include it as an importable package in your project.
FYI, It's usually preferable to use a src-based layout instead, however. See https://github.com/pypa/sampleproject/ for a full example.
I wonder why my simple example building a class is not properly working:
The tree-structure looks like the following
class_project/
├── class_project
│ ├── __init__.py
│ └── classtheclass.py
└── tests
├── __init__.py
└── test.py
classtheclass.py looks like this:
class ClassTheClass(object):
def __init__(self):
print "yeees"
test.py looks like this:
from class_project.classtheclass import ClassTheClass
i = ClassTheClass()
While init.py are empty
so if i execute on the shell
python test.py
its giving me
Traceback (most recent call last): File "test.py", line 1, in
from class_project.classtheclass import ClassTheClass ImportError: No module named class_project.classtheclass
Whats wrong with that. In Pycharm this even works...!
When you run python test.py, the interpreter will look for Python code in standard library locations (e.g. /usr/local/lib/python2.7/site-packages and friends) and in the tests folder where you invoked the interpreter (this set of locations is known as the "Python Path", and you can view it with: import sys; print sys.path).
None of those locations include class_project.classtheclass.
There are a variety of ways to solve this problem.
You can set the PYTHONPATH environment variable to include class_project:
export PYTHONPATH="/path/to/class_project:$PYTHONPATH" # Note: this has to be the top-level class_project directory, you have two with this name here.
python test.py # This will now work
You could probably also use a relative import, but I'd argue that's working around the problem not solving the problem.
Your file tree structure is wrong, if you are not exporting PYTHONPATH with the new lib path, you have to put the test.py as below structure to make the test.py to access classtheclass from class_project.
class_project/
├── class_project
│ ├── classtheclass.py
│ └── __init__.py
└── test.py
Python search imported module as follow, for example import foo
Firstly, search the built-in module, If foo isn't a built-in module, go to next
Interpreter will search the sys.path, sys.path will include: current directory, PYTHONPATH and installation dependent path. If foo is in sys.path, it will be imported. Otherwise, it will be ImportError.
Here, you can execute at project parent directory or explicit add /path/to/project to sys.path.
I have problems with python nosetests.
When I try to run the command, I get an import error.
I checked that the module is correctly installed on my machine.
In fact, if I run the interpreter from the directory where I run nosetests,
I am able to import the module.
I checked that the problems are to import not only that module, but different ones.
Where could the fix be?
Here is a possible traceback after I run nosetests:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/nose/loader.py", line 418, in loadTestsFromName
addr.filename, addr.module)
File "/Library/Python/2.7/site-packages/nose/importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/Library/Python/2.7/site-packages/nose/importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/Users/user-me/Desktop/bla/tests/blatest1", line 1, in <module>
import a_module as mo
ImportError: No module named a_module
but if I open the python interpreter, I am able to import a_module.
Here is my directory structure:
ROOT
└── package
├── __init__.py
├── package1
└── tests
├── tests1
│ └── package1 -> ../../package1
└── tests2
Your problem has nothing to do with nose itself. nose doesn't perform any magic additional to the Python interpreter when looking up modules and packages.
So if we assume a directory structure like this
ROOT
`-- package
|-- __init__.py
`-- tests
`-- __init__.py
and try to do python -c import package - when does that work, and when fail?
It's simple. It fails anywhere other than if you invoke the command from within ROOT. Nowhere else package is known.
The reason is that Python keeps a list of paths (sys.path) which mark roots for packages and modules. A statement import package will be taken, and iterating through all entries in sys.path will be searched for either a package.py, a package-dir with an __init__.py in it, or some other cases (C-extensions, new style namespace packages)
So where does ROOT get into that list of paths? Simple: the Python interpreter will always add the current working directory to the list of paths.
To sum it up: just call nose from within ROOT.
There are additional ways to add a path to the sys.path, e.g. by using virtualenvs. Then you can import package from everywhere, given you use the venv interpreter.
I can't seem to get the nose testing framework to recognize modules beneath my test script in the file structure. I've set up the simplest example that demonstrates the problem. I'll explain it below.
Here's the the package file structure:
./__init__.py
./foo.py
./tests
./__init__.py
./test_foo.py
foo.py contains:
def dumb_true():
return True
tests/test_foo.py contains:
import foo
def test_foo():
assert foo.dumb_true()
Both init.py files are empty
If I run nosetests -vv in the main directory (where foo.py is), I get:
Failure: ImportError (No module named foo) ... ERROR
======================================================================
ERROR: Failure: ImportError (No module named foo)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python/site-packages/nose-0.11.1-py2.6.egg/nose/loader.py", line 379, in loadTestsFromName
addr.filename, addr.module)
File "/usr/lib/python/site-packages/nose-0.11.1-py2.6.egg/nose/importer.py", line 39, in importFromPath
return self.importFromDir(dir_path, fqname)
File "/usr/lib/python/site-packages/nose-0.11.1-py2.6.egg/nose/importer.py", line 86, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "/home/user/nose_testing/tests/test_foo.py", line 1, in <module>
import foo
ImportError: No module named foo
----------------------------------------------------------------------
Ran 1 test in 0.002s
FAILED (errors=1)
I get the same error when I run from inside the tests/ directory. According to the documentation and an example I found, nose is supposed to add all parent packages to the path as well as the directory from which it is called, but this doesn't seem to be happening in my case.
I'm running Ubuntu 8.04 with Python 2.6.2. I've built and installed nose manually (not with setup_tools) if that matters.
You've got an __init__.py in your top level directory. That makes it a package. If you remove it, your nosetests should work.
If you don't remove it, you'll have to change your import to import dir.foo, where dir is the name of your directory.
Are you in a virtualenv? In my case, nosetests was the one in /usr/bin/nosetests, which was using /usr/bin/python. The packages in the virtualenv definitely won't be in the system path. The following fixed this:
source myvirtualenv/activate
pip install nose
which nosetests
/home/me/myvirtualenv/bin/nosetests
To those of you finding this question later on: I get the import error if I don't have an __init__.py file in my tests directory.
My directory structure was like this:
./tests/
./test_some_random_stuff.py
If I ran nosetests:
nosetests -w tests
It would give the ImportError that everyone else is seeing. If I add a blank __init__.py file it works just fine:
./tests/
./__init__.py
./test_some_random_stuff.py
Another potential problem appears to be hyphens/dashes in the directory tree. I recently fixed a nose ImportError issue by renaming a directory from sub-dir to sub_dir.
Of course if you have a syntax error in the module being imported that will cause this. For me the problem reared its head when I had a backup of a tests file with a path like module/tests.bak.py in the same directory as tests.py. Also, to deal with the init package/module problem in a Django app, you can run the following (in a bash/OSX shell) to make sure you don't have any init.pyc files lying around:
find . -name '*.pyc' -delete
I got this error message because I run the nosetests command from the wrong directory.
Silly, but happens.
I just ran into one more thing that might cause this issue: naming of tests in the form testname.test.py. That extra . confounds nose and leads to it importing things it should not. I suppose it may be obvious that using unconventional test naming conventions will break things, but I thought it might be worth noting.
For example, with the following directory structure, if you want to run nosetests in m1, m2 or m3 to test some functions in n.py, you should use from m2.m3 import n in test.py.
m1
└── m2
├── __init__.py
└── m3
├── __init__.py
├── n.py
└── test
└── test.py
Just to complete the question:
If you're struggling with structure like this:
project
├── m1
├ ├── __init__.py
├ ├── foo1.py
├ └──m2
├ ├── __init__.py
├ └── foo2.py
├
└── test
├── __init__.py
└── test.py
And maybe you want to run test from a path outside the project,
include your project path inside your PYTHONPATH.
export PYTHONPATH=$PYTHONPATH:$HOME/path/to/project
paste it inside your .profile.
If you're under a virtual environment, paste it inside the activate in your venv root