Python 3.8
MacOS
I am trying to import different modules inside a package and I am getting ModuleNotFoundError or ImportError when trying to use relative imports.
I have a folder called "practice_dir" with the following structure:
(base) or ƒ(~/practice_dir) >> tree
.
├── __init__.py
├── dir_1
│ ├── __init__.py
│ ├── file_1.py
│ └── subdir
│ ├── __init__.py
│ └── subfile.py
├── dir_2
│ ├── __init__.py
│ └── file_2.py
└── parent_file.py
It was mentioned online that once I have an init file on every folder containing modules than the absolute imports would start from the parent package
And relative imports would be using . or .. notation
Here's some of the examples I tried
In file_1:
from dir_2 import file_2
ModuleNotFoundError
from .dir_2 import file_2
ImportError: attempted relative import with no known parent package
It's the same if I do the same thing in dir_2, and dir_1/subfolder
The init.py files are empty right now, I tried to include imports to the files but that didn't work.
I've been looking at some questions posted online and on youtube, there seems to be various ways and one guy even mentioned that you don't need the init files if you have python 3.3+
Been battling hard with this, any help would be very much appreciated!
You can add the file to the python path at runtime:
import sys
sys.path.insert(0, 'path/to/file')
import file
Related
I am creating a Python package and want to make the example files in the example folder runnable with something like python example_file.py. How do I do it? My example_file.py and __init__.py look like below:
# example_file.py
from demandforecast import DemandForecast # module coded in demandforecast.py
if __name__ == '__main__':
# Example code here
# __init__.py
import os, sys; sys.path.append(os.path.dirname(os.path.realpath(__file__)))
When I navigate to the examples directory and run python example_file.py I get the following error:
Exception has occurred: ModuleNotFoundError
No module named 'demandforecast'
This can be solved by adding using the old sys.path.insert() trick mentioned in Importing files from different folder, but I would rather not include the absolute path because this will vary from user to user.
Here is my directory structure:
├───demandforecast
│ demandforecast.py
│ __init__.py
│
└───examples
example_file.py
I have tried a few posts here, such as Relative imports in Python 3, but without luck. Adding sys.path.append('../') to the example file before the demandforecast import also did not work.
I think your import is simply incorrect, and should be:
from demandforecast.demandforecast import DemandForecast
Since demandforecast is both the package name (the directory is definitely a package, with the __init__.py file) and a module name within that package. (I assume DemandForecast is a class within the demandforecast module.)
from src.pro.demandforecast.demandforecast import foo
if __name__ == '__main__':
foo()
output:
foo
create a folder src and inside of it create a package pro and now inside pro create two packages demandforecast and examples.
Directory structure:
/src$ tree
.
└── pro
├── demandforecast
│ ├── demandforecast.py
│ ├── __init__.py
│ └── __pycache__
│ ├── demandforecast.cpython-38.pyc
│ └── __init__.cpython-38.pyc
├── examples
│ ├── example_file.py
│ └── __init__.py
├── __init__.py
└── __pycache__
└── __init__.cpython-38.pyc
IDE:
This must have been asked before, but I cannot for the life of me find it.
I have a project structure something like this
ROOT
├── README.md
├── modules
│ ├── __init__.py
│ ├── mod1.py
│ └── mod2.py
├── tests
│ ├── test_mod1.py
│ └── test_mod2.py
├── notebooks
│ ├── nb1.ipynb
│ ├── nb2.ipynb
│ └── sub_dir
│ ├── sub_nb.ipynb
│ ├── generate_py
│ └── py_files
│ └── sub_nb.py
├── definitions
└── main.py
So from main.py I am able to import anything from definitions, or any module from ROOT/modules.
What I want is to be able to import from these from anywhere within the notebooks directory tree. I know I could do this using:
import sys
sys.path.append("..")
But the notebooks directory tree has many layers, and I don't want my code to start looking like this:
import sys
sys.path.append("../../../../")
What's more, the file generate_py is a bash script, that converts the jupyter notebooks (.ipynb) to .py files and stashes the .py files into the ./py_files subdir.
With the above method i end up having to manually edit every file to put an extra ../ into the sys.append(). This is annoying.
If I run files from within pycharm all works well (I'm guessing it updates your PYTHONPATH to include the project root when you create main.py?)
To further complicate, this project is run on several machines, shared via git. So absolute references are out.
But running from terminal, or from within jupyter it cant gind modules, or definitions without going through the sys.append() process. And I feel there must be a better way.
So what is the best practice here?
I have been researching how to build the folder structure for a custom python package. There were several attempts, but none of them seemed to be applicable in general. In particular, the usage (or not usage) of the \__init__.py file(s).
I have a package that consists of several sub-packages, each being responsible to parse Files of a certain kind. Therefore I currently adopted this structure:
Parsers/
├── __init__.py
|
├── ExternalPackages
│ ├── __init__.py
│ ├── package1
│ └── package2
|
├── FileType1_Parsers/
│ ├── __init__.py
│ ├── parsers1.py
│ └── containers1.py
│
└── FileType2_Parsers/
├── __init__.py
├── parsers2.py
└── containers2.py
But it seems not very pythonic, that when I import his package and I want to use a certain class of a module I have to type something like
from Parsers.FileType1_Parsers.parsers1 import example_class
Is there any convention on how to structure such packages or any rules on how to avoid such long import lines?
You can add the following line to Parsers/__init__.py
from .FileType1_Parsers.parsers1 import example_class
Then you can import example_class by
from Parsers import example_class
This is a common practice in large package.
You can modify sys.path at run-time so that it contains a directory for each module you'll be using. For example, for package1 issue the following statements:
>>> sys.path.append(r"[package directory path]\\Parsers\\FileType1_Parsers\\package1")
You can do this for any other modules in the package as well. Now, you can just use this command:
from package1 import example_class
Hope this helps!
Yes, I know this is a recurrent question but I still couldn't find a convincing answer. I even read at https://chrisyeh96.github.io/2017/08/08/definitive-guide-python-imports.html but could not find out how to solve the problem:
I'm running python 3.6 project that includes jupyter (ipython) notebooks. I want the notebook to import a custom local helpers.py package that I will probably use also later in other sources.
The project structure is similar to:
my_project/
│
├── my_project/
│ ├── notebooks/
│ └── a_notebook.ipynb
│ ├── __init__.py # suppose to make package `my_project` importable
│ └── helpers.py
│
├── tests/
│ └── helpers_tests.py
│
├── .gitignore
├── LICENSE
├── README.md
├── requirements.txt
└── setup.py
When importing helpers in the notebook I get the error:
----> 4 import helpers
ModuleNotFoundError: No module named 'helpers'
I also tried from my_project import helpers and I get the same error ModuleNotFoundError: No module named 'my_project'
I finally (and temporarily) used the usual trick:
import sys
sys.path.append('..')
import helpers
But it looks awful and I'm still looking for a better solution
One can tell python where to look for modules via sys.path. I have a project structure like this:
project/
│
├── src/
│ └── my_module/
│ ├── __init__.py
│ └── helpers.py
├── notebooks/
│ └── a_notebook.ipynb
...
I was able to load the module like so:
import sys
sys.path.append('../src/')
from my_module import helpers
One should be able load the module from wherever they have it.
Here I could find several solutions. Some of them are similar to the ones answered before:
https://mg.readthedocs.io/importing-local-python-modules-from-jupyter-notebooks/index.html
If you move the notebooks directory out one level, and then explicitly import your module from the package, that should do it. So your directory would look like this:
my_project/
│
├── my_project/
│ ├── __init__.py
│ └── helpers.py
├── notebooks/
│ └── a_notebook.ipynb
...
and then your import statement within the notebook would be:
from my_project import helpers.
Try the following line:
from my_project.helpers import what_you_need
This line should also work:
import my_project.helpers
I think you need a __init__.py module in the notebooks/ directory. I haven't really used Jupyter notebooks before so I could be wrong. You may also need to try changing your import statement to:
import .. helpers
to indicate that the import statement is for a local package that is located in the parent directory of the Jupyter notebook.
This worked for me.
import sys
MODULE_FULL_PATH = '/<home>/<username>/my_project/my_project'
sys.path.insert(1, MODULE_FULL_PATH)
from my_module import helpers
If you are on a Unix/Linux system another elegant solution may be creating a "soft link" to the module file helpers.py that you would like to use. Change to the notebooks directory and create the link to the module file this way:
cd notebooks; ln -fs ../my_project/helpers.py .
This "soft link" is essentially a pointer (a shortcut) to the original target file. With the link in place you will be import your module file as usual:
import helpers
I've a project which uses git submodules. In my python file I want to use functions from another python file in the submodule project.
In order to work I had to add the init.py file to all subfolders in the path. My folder tree is the following:
myproj
├── gitmodules
│ ├── __init__.py
│ ├── __init__.pyc
│ └── mygitsubmodule
│ ├── __init__.py
│ ├── __init__.pyc
│ └── file.py
└── myfile.py
Is there any way to make it work without touching mygitsubmodule ?
Thanks
you can add to sys.path in the file you want to be able to access the module, something like:
import sys
sys.path.append("/home/me/myproj/gitmodules")
import mygitsubmodule
This example is adding a path as a raw string to make it clear what's happening. You should really use the more sophisticated, system independent methods described below to determine and assemble the path.
Also, I have found it better, when I used this method, to use sys.path.insert(1, .. as some functionality seems to rely of sys.path[0] being the starting directory of the program.
I am used to avoiding modifying sys.path.
The problem is, when using git submodule, submodule is a project directory, not a Python package. There is a "gap" between your module and that package, so you can't import.
Suppose you have created a submodule named foo_project, and there is a foo package inside.
.
├── foo_project
│ ├── README.rst
│ └── foo
│ └── __init__.py
└── main.py
My solution will be creating a soft link to expose that package to your module:
ln -s foo_project/foo foo
.
├── foo_project
│ ├── README.rst
│ └── foo
│ └── __init__.py
├── foo -> foo_project/foo
└── main.py
Now you can import foo in the main.py.
For reference,
from submodulefolder.file import func_name
or
import submodulefolder.file as lib_name
where file excludes the extension of file.py, seems to work in relative terms without modifying the subfolder / git submodule with a init.py since python 3.3+,
as shown here.
Tested on py3.8.5 linux native and py3.7.8 anaconda Windows, both in Spyder's Ipython-console, as well as natively on linux via terminal.