SUMMARY
I am fairly new to designing full-fledged python projects, and all my Python work earlier has been with Jupyter Notebooks. Now that I am designing some application with Python, I am having considerable difficulty making it 'run'.
I have visited the following sites -
Relative imports in Python
Ultimate answer to relative python imports
python relative import example code does not work
But none of them seem to solve my issue.
PROBLEM
Here's my repo structure -
my_app/
__init__.py
code/
__init__.py
module_1/
some_code_1.py
module_2/
some_code_2.py
module_3/
some_code_3.py
main.py
tests/
__init__.py
module_1/
test_some_code_1.py
module_2/
test_some_code_2.py
module_3/
test_some_code_3.py
resources/
__init__.py
config.json
data.csv
I am primarily using PyCharm and VS Code for development and testing.
The main.py file has the following imports -
from code.module_1.some_code_1 import class_1
from code.module_2.some_code_2 import class_2
from code.module_3.some_code_3 import class_3
In the PyCharm run configuration, I have the working directory set to `User/blah/blah/my_app/
Whenever I run the main.py from PyCharm, it runs perfectly.
But if I run the program from terminal like -
$ python code/main.py
Traceback (most recent call last):
File "code/main.py", line 5, in <module>
from code.module_1.some_code_1 import class_1
ModuleNotFoundError: No module named 'code.module_1.some_code_1'; 'code' is not a package
I get the same error if I run the main.py from VS Code.
Is there a way to make this work for PyCharm as well as terminal?
If I change the imports to -
from module_1.some_code_1 import class_1
from module_2.some_code_2 import class_2
from module_3.some_code_3 import class_3
This works on the terminal but doesn't work in PyCharm. The test cases fail too.
Is there something I am missing, or some configuration that can be done to make all this work seamlessly?
Can someone help me with this?
Thanks!
The problem is when you do python code/main.py it makes your current working directory code/, which makes all of your absolute imports incorrect since Python doesn't see above that directory unless you explicitly change a setting like the PYTHONPATH environment variable.
Your best option is to rename main.py to __main__.py and then use python -m code (although do note that package name clashes with a module in the stdlib).
You also don't need the __init__.py in my_app/ unless you're going to treat that entire directory as a package.
And I would consider using relative imports instead of absolute ones (and I would also advise importing to the module and not the object/class in a module in your import statements to avoid circular import issues). For instance, for the from code.module_1.some_code_1 import class_1 line in code.main, I would make it from .module_1 import some_code_1.
Related
I'm trying to import a class in a different directory to another file, but can't seem to get it to work. I know this question has been asked a lot and I have looked through multiple stackoverflow solutions and at https://docs.python.org/3/tutorial/modules.html#packages
1: Importing files from different folder
2: import python file in another directory failed
I want to try to just use the method containing just __init__.py file instead of doing an import sys
My directory structure is as follows:
django_vue/
__init__.py
devices/
__init__.py
models.py
lib/
__init__.py
my_file.py
I'm trying to import the class Device from /django_vue/devices/models.py to /django_vue/lib/my_file.py by:
from devices.models import Device
However when I do that I still get the error:
from devices.models import Device
ModuleNotFoundError: No module named 'devices'
I'm not sure what I'm dong wrong since I already have the __init__ file in both directories. Any help is appreciated. Also I'm running python 3.6.
This is the folder structure I'm working with.
.
└── django_vue
├── devices
│ └── models.py
└── lib
└── file.py
When you run
$ python file.py
python has no way of knowing what's outside the directory.
python can't go back and then into devices/ just like that.
The easiest way to solve this would be to add the folder devices/ to sys.path. When python imports a module, it searches for the module from sys.path. Adding the path to devices/ would make it available for imports.
Here are my files.
# models.py
Device = 'device'
# file.py
import sys
sys.path.append('..') # adds the parent dir (which is django-vue/) to path
# django-vue dir has devices/ so now this is available for imports
# importing this works now
from devices.models import Device
print(Device)
Output
django_vue/lib$ python3 file.py
device
Think about it your are inside my_file.py and import something called devices.
How can python know where the name devices has come from.
It won't search your entire Drive for that module/package
Relative Import
use a relative import instead. write from ..devices.models import Device. This is telling python to go up one directory to the parent directory and that's where it will find the devices package. Your lib module should now work as a module
If you however run the my_file.py package directly (as in python C:/django_vue/lib/my_file.py)
You will still get an error. Not the same error; the new error will be something like
ImportError: attempted relative import with no known parent package
This is happening because you are actually running my_file.py
If you think about it why would you want to run my_file.py by itself when it is clearly a part of a package. Maybe you are just testing to see if the import works when you use your package. The problem with this is that it makes it seem like your packages relative imports don't work even though this actually works.
Create a main.py in django_vue and write from lib import my_file. This will run your my_file.py and you will notice there is no error.
What's happening here
Have you heard of __package__?
if you put print(str(__package__)) in your my_file.py and run my_file.py directly you will see that it prints None.
However if you run main.py (that you just created) you will see that when It reaches my_file.py, __package__ will actually be defined to something.
Ahhh... you see now it all makes sense; The error you originally got said something about no known parent package. If __package__ is undefined that means there is no relative parent package because the file was obviously run directly instead of as part of a package.
Consider Absolute imports
you also might want to consider using absolute imports because if you are working on the package you might change it directory structure while developing. so you need to keep changing the import references on the affected files.
Although you can find IDE's with python extensions that automatically to this as you change your directory. I believe VS Code does this automatically.
Replace the __init__ files with __main__.
I have read probably all of the posts on here regarding imports and I still cannot figure out what is going on with the imports, I have spent hours trying to get a very simple example working and am literally pulling my hair out.
I am using python 3.7 and pycharm but I am running my code from the commandline, for the unit tests I am using pytest.
My project structure is:
my_message_validator/
__init__.py
module_1/
__init.py__
foo.py
module_2/
__init.py__
bar.py
baz.py
module_3
context.py
test_all.py
module_1.init.py
from module_1 import foo
module_2.init.py
# For some reason pycharm doesnt complain when I use '.' but if I use module_2 it does
from . import bar, baz
If I try to run my code or my tests from the commandline no matter how I move things around I seem to get either ModuleNotFoundError: No module named, when I have managed to get the tests working I still cannot run my code on its own from the commandline.
How can I import module_1 into module_2 and are these actually packages? I am coming from java and find the imports a lot easier to understand, I am finding the python importing very confusing...
Also how can I can then import whatever I need into my test module\package\folders context.py?
Currently the test context look like:
import os
import sys
# Is this needed as it doesnt seem to do anything?
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
from module_1.foo import Foo
from module_2 import bar, baz
In test_all.py I am trying to import from the context file like this:
from .context import bar,baz
from .context import Foo
# Calling in test like
Foo.load_file(file)
bar.method_one()
baz.method_two()
Do I need all the __init.py__ files and what should I be putting in them to make my methods and classes public\exposed? I would like this entire package to be reusable so want to be able to treat it like a jar file in java.
Any help would be much appreciated as it seems everytime I change something I get an error in a different place, python seems so much more complicated than java right now.
First, do not use relative imports (with .), as it is known for causing multiple issues. Always write your imports relative to the root of your project. For example, you did it well for from module_1.foo import Foo. You should also do it in test_all.py and context.py. Moreover, after using relative imports, the __init__.py files can be left empty in your case.
Most likely, the Python interpreter cannot find your modules because the PYTHONPATH environment variable does not contain the root of your project. If you run export PYTHONPATH="YOUR_PROJECT_ROOT_ABSOLUTE_PATH:$PYTHONPATH" before your script, it should run as expected. To make sure this variable is set all the time, you can add the export statement to your shell profile file (e.g. .bashrc or .bash_profile).
After chatting with the author, it turns out there was a fourth issue. It was a name collision like the one in this other question. In his project directory, module_1 was actually called foo like its child foo.py, which confused the interpreter.
Have you tried importing like:
from my_message_validator.module_1.foo import Foo
from my_message_validator.module_2 import bar, baz
I had the same case.
I started the application in this way:
flask run
And every time I got a ModuleNotFoundError error on the website.
When I started the application like this:
python3 -m flask run
the application started without errors :-) .
I have the common problem of the "module not found" error when trying to import a file within a folder in my project directory as a package. I've tried several solutions from Stackoverflow answers, but none are working for me. Here's what's going on, and what I've tried:
I'm working in a conda environment devenv on a Flask project, using PyCharm, and have a project directory like this:
/some/path/project_root/
migrations/
static/
templates/
reporting/
__init__.py
code.py
tests.py
Inside the tests.py file there are import statements to import code.py as a module:
from .code import my_function
However, when I run (devenv) me#comp:project_root$ > python reporting/tests.py
I get the error: ModuleNotFoundError: No module named '__main__.code'; '__main__' is not a package
I tried appending the project directory path to $PYTHONPATH, and echo $PYTHONPATH returns /some/path/project_root/
What do I need to configure to get this to work properly? Also, whatever settings I need to change, can I make those settings specific to the development environment I'm using?
Change from .code import my_function to from code import my_function. The top level of a package is defined by the highest folder with an __init__.py file. So the top level of your project is the reporting folder and code.py does not need to be a relative import. Best to either avoid relative imports or get an editor like PyCharm that will take care of it for you!
I can't make this work..
My structure is:
program_name/
__init__.py
setup.py
src/
__init__.py
Process/
__init__.py
thefile.py
tests/
__init__.py
thetest.py
thetest.py:
from ..src.Process.thefile.py import sth
Running: pytest ./tests/thetest.py from program_name gives :
ValueError: attempted relative import beyond top-level package
I tried also other approaches but i am receiving various errors.
But I would expect for the above to work.
ValueError: Attempted relative import in non-package
States that you're trying to use relative import in the module, which are to be used for packages i.e. to make it a package add __init__.py and call the thetest.py from some file outside the package.
Directly running thetest.py from interpreter won't work.
Relative imports require that the module which uses them is being
imported itself either as package module.
Suggestion 1:
The current tests directory has a __init__.py file but that doesn't allow you to run it as a module (via shell) - to make your current (relative) import work, you need to import it in an external (to package) file/module - let's create a main.py (can name it anything you like):
main.py
program_name/
__init__.py
setup.py
src/
__init__.py
Process/
__init__.py
thefile.py
tests/
__init__.py
thetest.py
src/Process/thefile.py:
s = 'Hello world'
tests/thetest.py:
from ..src.Process.thefile import s
print s
main.py:
from program_name.tests.thetest import s
Executing main.py:
[nahmed#localhost ~]$ python main.py
Hello world
Suggestion 2:
Execute the file just above root dir i.e. one level up the program_name/ , in the following fashion:
[nahmed#localhost ~]$ python -m program_name.tests.thetest
Hell World
P.S. relative imports are for packages, not modules.
Just solved a similar problem with a lot of googling.
Here's two solutions without changing the existing file structor:
1
The way to import module from parent folder from ..src.Process.thefile.py import sth is called "relative import".
It's only supported when launching as a package from the top-level package. In your case, that is launching command line from the directory which contains program_name/ and type (for win environment)
python -m program_name.tests.thetest
or simply (useful for many pytest files):
python -m pytest
2
Otherwise -- when trying to run a script alone or from a non top-level package --
you could manually add directory to the PYTHONPATH at run time.
import sys
from os import path
sys.path.append(path.dirname(path.dirname(path.abspath(__file__))))
from src.Process.thefile import s
Try the first one first see if it's compatiable with the pytest framework. Otherwise the second one should always solve the problem.
Reference (How to fix "Attempted relative import in non-package" even with __init__.py)
When importing a file, Python only searches the current directory, the directory that the entry-point script is running from.
you can use sys.path to include different locations
import sys
sys.path.insert(0, '/path/to/application/app/folder')
import thefile
Given the directory structure:
program/
setup.py
ilm/
__init__.py
app/
__init__.py
bin/
script.py
Note: the setup.py is not a typical setup.py, rather it is a custom-made setup uniquely for py2app.
program/ilm/app/__init__.py is non-empty: it contains a main() function, which instantiates a class in the same file. My question: In program/ilm/bin/script.py, if I want to import and execute the main() function in program/ilm/app/__init__.py, what are the valid ways of achieving this? The reason I ask is that script.py is doing so thus:
import ilm.app as app
if __name__ == '__main__':
app.main()
Based on my (admittedly limited) understanding of packaging and importing, this shouldn't work, since we have not explicitly told script.py where to look for project/ilm/app/__init__.py using ... And indeed, I get:
MacBook-Pro-de-Pyderman:program Pyderman$ python ./bin/script.py
Traceback (most recent call last):
File "./bin/script.py", line 5, in <module>
import ilm.app as app
ImportError: No module named ilm.app
In contrast, when the Python interpreter is started in /project, import ilm.app as app works fine.
This is apparently fully-functional production code which I should not have to change to get running.
Is the import statement valid, given the directory structure, and if so, what am I missing?
If not, what is the recommended way of getting the import to work? Add the path using sys.path.append() above the import statement? Or use .. notation in the import statement to explicitly point to program to pick up project/ilm/app/__init__.py? Is the fact that it is an __init__.py I am trying to import significant?
Two things. You need to make sure the iml directory is in the python path. Either make sure you are running python from the right directory or add the right path to sys.path list. And you need to make sure that both iml and app directory both have
__init__.py
file, since python needs to interpret the whole thing as a hierarchy of modules rather than just dirs. Then you should be able to do
from iml import app
The obvious conclusion would seem to be that the iml directory has an __init__.py inside it, but why that would happen in your production setup is hard to say. Have you checked in the production environment whether this is the case?
Assuming that the production environment is importing the package at iml/app (which you can check by examining app.__file__) then the program will indeed execute the main function from the __init__.py file - but __init__.py might easily be importing it from sonewhere else rather than defining it locally.
The Python package needs proper setup.py which defines the package structure. Furthermore bin/ scripts should be defined as console_scripts entry points.
This all must be installed in a proper Python environment, not just any folder.