I have read probably all of the posts on here regarding imports and I still cannot figure out what is going on with the imports, I have spent hours trying to get a very simple example working and am literally pulling my hair out.
I am using python 3.7 and pycharm but I am running my code from the commandline, for the unit tests I am using pytest.
My project structure is:
my_message_validator/
__init__.py
module_1/
__init.py__
foo.py
module_2/
__init.py__
bar.py
baz.py
module_3
context.py
test_all.py
module_1.init.py
from module_1 import foo
module_2.init.py
# For some reason pycharm doesnt complain when I use '.' but if I use module_2 it does
from . import bar, baz
If I try to run my code or my tests from the commandline no matter how I move things around I seem to get either ModuleNotFoundError: No module named, when I have managed to get the tests working I still cannot run my code on its own from the commandline.
How can I import module_1 into module_2 and are these actually packages? I am coming from java and find the imports a lot easier to understand, I am finding the python importing very confusing...
Also how can I can then import whatever I need into my test module\package\folders context.py?
Currently the test context look like:
import os
import sys
# Is this needed as it doesnt seem to do anything?
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
from module_1.foo import Foo
from module_2 import bar, baz
In test_all.py I am trying to import from the context file like this:
from .context import bar,baz
from .context import Foo
# Calling in test like
Foo.load_file(file)
bar.method_one()
baz.method_two()
Do I need all the __init.py__ files and what should I be putting in them to make my methods and classes public\exposed? I would like this entire package to be reusable so want to be able to treat it like a jar file in java.
Any help would be much appreciated as it seems everytime I change something I get an error in a different place, python seems so much more complicated than java right now.
First, do not use relative imports (with .), as it is known for causing multiple issues. Always write your imports relative to the root of your project. For example, you did it well for from module_1.foo import Foo. You should also do it in test_all.py and context.py. Moreover, after using relative imports, the __init__.py files can be left empty in your case.
Most likely, the Python interpreter cannot find your modules because the PYTHONPATH environment variable does not contain the root of your project. If you run export PYTHONPATH="YOUR_PROJECT_ROOT_ABSOLUTE_PATH:$PYTHONPATH" before your script, it should run as expected. To make sure this variable is set all the time, you can add the export statement to your shell profile file (e.g. .bashrc or .bash_profile).
After chatting with the author, it turns out there was a fourth issue. It was a name collision like the one in this other question. In his project directory, module_1 was actually called foo like its child foo.py, which confused the interpreter.
Have you tried importing like:
from my_message_validator.module_1.foo import Foo
from my_message_validator.module_2 import bar, baz
I had the same case.
I started the application in this way:
flask run
And every time I got a ModuleNotFoundError error on the website.
When I started the application like this:
python3 -m flask run
the application started without errors :-) .
Related
I have the following directory structure:
base_folder
methods_folder
method_1.py
method_2.py
.
.
.
method_n.py
class_methods.py
top_class.py
class_methods.py imports the other files in the same directory, like this:
from method_1 import method_1
from method_2 import method_2
.
.
.
from method_n import method_n
(obs: these methods files has a method with its own file names inside them)
If I run class_methods.py by myself, no problem. But if I try to run top_class.py, which imports class_methods.py, I get the error no module named method_1
So, when executing top_class.py, it is not seeing the files in methods_folder/. Why?
the correct import inside top_class.py would be from methods_folder.method_n import method_n. This is because you are treating methods_folder as a package. If you are running a version of Python that is before 3.3 you must also unclude __init__.py file inside the methods_folder in order to turn it into a package.
Files only have direct access to things they import. Say we have a.py which imports b.py, and b.py imports c.py. When running within a.py functions in b.py that use c.py, this will work fine, because a has access to b, and b has access to c. This does not mean, however, that the imports chain (as in C++) and you can use functions from c in a. You will get an error, because a can only see the contents of b, which it imported.
So if you want to use all your method_i.py files from within top_class.py, you need to import them directly in the same file.
Edit: You also have some other issues. To import other files in a subfolder, you would need to, inside top_class.py, call import methods_folder.method_i. To import something in the same directory, just use import method_i. Since you have a method of the same name in each file, what you have works fine in class_methods.py You also need to create an empty file called __init__.py in any folder containing python files you intend to import to/from which lets python know it's allowed to look there.
You can create an importable package in python in one of two ways. The first way is what you are doing: you create a file called my_package.py and import it with import my_package. This is commonly used for simpler packages that don't need to be further broken up. For this to work, your .py file has to be on the PYTHONPATH which is an environment variable that tells python where to look for packages. If not defined, there are some default places that python will use to look for packages. One of these default locations is the current working directory, which is why your first set of imports works.
In theory you should be able to run the second piece of code from the same location (python ../top_class.py) and use the same import style, but I assume you are changing directories to run that file. This means your files are no longer in the current working directory and no longer findable by python.
One way to get your code to work using the existing style would be to define PYTHONPATH with the location of your methodX.py files. You will typically add to the python search path like this:
PYTHONPATH=$PYTHONPATH:./methods_folder python top_class.py
This tells python to look in methods_folder, in addition to the standard places, when you try to import something. Playing with the PYTHONPATH gets a little annoying after a while, so I actually prefer the next approach.
A second way to create a package is by creating a folder with an __init__.py file inside. This tells python that you want it to treat that directory as a package. This is the preferred style for more complicated pieces of code that might benefit from organization across multiple files. For your example, you could organize your code in the following way:
base_folder
methods_folder
__init__.py
method_1.py
method_2.py
.
.
.
method_n.py
class_methods.py
top_class.py
And then your import in top_class.py would look like this:
from methods_folder.method1 import method1
from methods_folder.method2 import method2
from methods_folder.method3 import method3
This has the effect of creating a top level methods_folder package with modules method1, method2, etc. Because methods_folder is in the same directory as the one you are running top_class.py from, python picks that up as a package using the default PYTHONPATH and lets you import from within it.
I assume you are running them from their respective directories? Unless they are installed in your Python path (I'm going to assume they are not), then Python will by default look for imports in your current directory. So when you run class_methods.py from its directory, then it can find methods_1.py in order to satisfy from methods_1 import method_1. But when you execute top_class.py, it looks for methods_1.py or methods_1/__init__.py, neither of which it fines from that directory.
Assuming Python 3, you would need to use relative imports in class_methods.py.
# class_methods.py
from .methods_1 import method_1
from .methods_2 import method_2
This will let you run it from top_class.py. Unfortunately, you can't use relative imports when running a script in the same package, so you wouldn't be able to run class_methods.py directly.
Another option in that case is to keep the absolute imports in class_methods.py and add the methods folder to the path in top_class.py.
import os
import sys
sys.path.append(os.path.join(os.path.dirname(__file__), 'methods'))
import class_methods
from methods_1 import method
Of course editing sys.path is fine for small scripts and standalone things. But if this grows past being a script, you'll need to be careful about doing it, and you'll probably just want to come up with another solution. Likely, the best thing is to create a package that you install (you can still do this from your source directory while you are developing) and then you can import the same way from anywhere.
The recommended way of running a python script is using the -m switch from the parent of your root package - so in your case:
$ cd base_folder
$ python -m top_class
Python will automatically add the base_folder to its sys.path and you don't need to do any sys.path/PYTHOPATH hacks that are just this - hacks that bloat the code with boilerplate and will blow when least expected.
Now to run the class_methods the correct way is also
$ cd base_folder
$ python -m methods_folder.class_methods
but then the imports in class_methods should be modified to either absolute:
from methods_folder.method1 import method1
...
or relative:
from .method1 import method1
...
I am having some trouble figuring out how to do relative imports in Python. I am currently working on my first major project so I want to do it right using unit tests. However, I am having trouble with my file structure and relative imports.
Here is my current structure:
App/
__init__.py
src/
__init__.py
person.py
tests/
__init__.py
person_tests.py
What I want to do is be able to import person.py into person_tests.py for the unit tests. I have attempted the following:
from . import person
from .. import person
from .App.src import person
from ..App.src import person
from ..src.person import *
from ..src import person
from .src import person
Every one of the above throws either a syntax error or
ValueError: Attempted relative import in non-package
Can someone please clarify this for me?
Edit: Python version is 2.7.
Edit: I would like to be able to use this with say unittest or nose.
My guess (and I'll delete this if I'm wrong) is that you're trying to use person_tests.py as a top-level script, rather than as a module inside a package, by doing something like this:
$ cd App/tests
$ python person_tests.py
In that case, person_tests does not end up as App.tests.person_tests, but as just __main__ (or, with minor variations, as the top-level person_tests, which has the same basic issues). So, .. does not refer to App, and therefore there is no way to get to person as a relative import.
More generally, nothing on PYTHONPATH, including ., should ever be in the middle of any package directory, or things will get broken.
The right answer is to not do that. Do something like this:
$ python -m App.tests.person_tests
Or write a top-level (outside the package) script that imports the module, and run that top-level script.
Just as abarnert says, you can not excute the script as main for the relative import based on the current name of the script. Quoted from the doc:
Note that both explicit and implicit relative imports are based on the
name of the current module. Since the name of the main module is
always "__main__", modules intended for use as the main module of a
Python application should always use absolute imports.
. represents the current package and .. represents the parent package. So some of your import statement is wrong.
from . import person # wrong for no person module in package tests
from .. import person # wrong for no person module in package app
from .App.src import person # wrong for no app package in package tests
from ..App.src import person # wrong for no app package in package app
from ..src.person import * # right
from ..src import person # right
from .src import person # wrong
You can refer PEP328 for the standard of relative import.
you could append it to sys.path.
import sys
sys.path.append("../src")
import person
Multilevel relative import
I have following folder structure
top\
__init__.py
util\
__init__.py
utiltest.py
foo\
__init__.py
foo.py
bar\
__init__.py
foobar.py
I want to access from foobar.py the module utiltest.py. I tried following relative import, but this doesn't work:
from ...util.utiltest import *
I always get
ValueError: Attempted relative import beyond toplevel package
How to do such a multileve relative import?
I realize this is an old question, but I feel the accepted answer likely misses the main issue with the questioner's code. It's not wrong, strictly speaking, but it gives a suggestion that only coincidentally happens to work around the real issue.
That real issue is that the foobar.py file in top\foo\bar is being run as a script. When a (correct!) relative import is attempted, it fails because the Python interpreter doesn't understand the package structure.
The best fix for this is to run foobar.py not by filename, but instead to use the -m flag to the interpreter to tell it to run the top.foo.bar.foobar module. This way Python will know the main module it's loading is in a package, and it will know exactly where the relative import is referring.
You must import foobar from the parent folder of top:
import top.foo.bar.foobar
This tells Python that top is the top level package. Relative imports are possible only inside a package.
So I am working on a Python project that was here before me in an SVN repo. When I first pulled it, the structure was a bit odd due to the fact that it was similar to:
Proj\
src\
tags\
trunk\
And then everything is inside src\ are the python module files except src\ turns out to just be a logical folder with no overall package inside. There isn't a __init__.py in the project anywhere. So I want to restructure it at least so I can use relative imports through my project. I also want to set it up so it looks more like this.
Proj\
src\
model\
controller\
view\
test\
tags\
trunk\
However, I tried setting this up and no matter what I seem to do, it cannot resolve the relative import the moment I have to traverse packages. I placed a __init__.py file in each level package including one inside the src\ folder with all of them having __all__ defined. However, when I try to make a unit test in my test\ package and do an import saying:
from ..model.foo import Foo
to attempt to import the Foo class from module foo.py located inside of the model package, it doesn't resolve. Just in case it was a problem specifically with unit tests, I also tried this with a module in the controller package that was dependent on a class in the model package and vice versa. None of them worked. How do I resolve this?
Have you added the root folder to your system path?
import sys
sys.path.append(<place the Proj dir here>)
then you could import as follows:
from src.model.somefile import Something
If you don't know the absolute path for Proj, you can always use combinations such as
os.path.dirname(os.getcwd())
I have a Python package with several subpackages.
myproject/
__init__.py
models/
__init__.py
...
controllers/
__init__.py
..
scripts/
__init__.py
myscript.py
Within myproject.scripts.myscript, how can I access myproject.models? I've tried
from myproject import models # No module named myproject
import models # No module named models
from .. import models # Attempted relative import in non-package
I've had to solve this before, but I can never remember how it's supposed to be done. It's just not intuitive to me.
This is the correct version:
from myproject import models
If it fails with ImportError: No module named foo it is because you haven't set PYTHONPATH to include the directory which contains myproject/.
I'm afraid other people will suggest tricks to let you avoid setting PYTHONPATH. I urge you to disregard them. This is why PYTHONPATH exists: to tell Python where to look for code to load. It is robust, reasonably well documented, and portable to many environments. Tricks people play to avoid having to set it are none of these things.
The explicit relative import will work even without PYTHONPATH being set, since it can just walk up the directory hierarchy until it finds the right place, it doesn't need to find the top and then walk down. However, it doesn't work in a script you pass as a command line argument to python (or equivalently, invoke directly with a #!/usr/bin/python line). This is because in both these cases, it becomes the __main__ module of the process. There's nowhere to walk up to from __main__ - it's already at the top! If you invoke the code in your script by importing that module, then it will be fine. That is, compare:
python myproject/scripts/myscript.py
to
python -c 'import myproject.scripts.myscript'
You can take advantage of this by not executing your script module directly, but creating a bin/myscript that does the import and perhaps calls a main function:
import myprojects.scripts.myscript
myprojects.scripts.myscript.main()
Compare to how Twisted's command line scripts are defined: http://twistedmatrix.com/trac/browser/trunk/bin/twistd
Your project is not in your path.
Option A
Install your package so that python can find it via its absolute name from anywhere (using from myproject import models )
Option B
Trickery to add the relative parent to your path
sys.path.append(os.path.abspath('..'))
The former option is recommended.