Yes, I know this is a recurrent question but I still couldn't find a convincing answer. I even read at https://chrisyeh96.github.io/2017/08/08/definitive-guide-python-imports.html but could not find out how to solve the problem:
I'm running python 3.6 project that includes jupyter (ipython) notebooks. I want the notebook to import a custom local helpers.py package that I will probably use also later in other sources.
The project structure is similar to:
my_project/
│
├── my_project/
│ ├── notebooks/
│ └── a_notebook.ipynb
│ ├── __init__.py # suppose to make package `my_project` importable
│ └── helpers.py
│
├── tests/
│ └── helpers_tests.py
│
├── .gitignore
├── LICENSE
├── README.md
├── requirements.txt
└── setup.py
When importing helpers in the notebook I get the error:
----> 4 import helpers
ModuleNotFoundError: No module named 'helpers'
I also tried from my_project import helpers and I get the same error ModuleNotFoundError: No module named 'my_project'
I finally (and temporarily) used the usual trick:
import sys
sys.path.append('..')
import helpers
But it looks awful and I'm still looking for a better solution
One can tell python where to look for modules via sys.path. I have a project structure like this:
project/
│
├── src/
│ └── my_module/
│ ├── __init__.py
│ └── helpers.py
├── notebooks/
│ └── a_notebook.ipynb
...
I was able to load the module like so:
import sys
sys.path.append('../src/')
from my_module import helpers
One should be able load the module from wherever they have it.
Here I could find several solutions. Some of them are similar to the ones answered before:
https://mg.readthedocs.io/importing-local-python-modules-from-jupyter-notebooks/index.html
If you move the notebooks directory out one level, and then explicitly import your module from the package, that should do it. So your directory would look like this:
my_project/
│
├── my_project/
│ ├── __init__.py
│ └── helpers.py
├── notebooks/
│ └── a_notebook.ipynb
...
and then your import statement within the notebook would be:
from my_project import helpers.
Try the following line:
from my_project.helpers import what_you_need
This line should also work:
import my_project.helpers
I think you need a __init__.py module in the notebooks/ directory. I haven't really used Jupyter notebooks before so I could be wrong. You may also need to try changing your import statement to:
import .. helpers
to indicate that the import statement is for a local package that is located in the parent directory of the Jupyter notebook.
This worked for me.
import sys
MODULE_FULL_PATH = '/<home>/<username>/my_project/my_project'
sys.path.insert(1, MODULE_FULL_PATH)
from my_module import helpers
If you are on a Unix/Linux system another elegant solution may be creating a "soft link" to the module file helpers.py that you would like to use. Change to the notebooks directory and create the link to the module file this way:
cd notebooks; ln -fs ../my_project/helpers.py .
This "soft link" is essentially a pointer (a shortcut) to the original target file. With the link in place you will be import your module file as usual:
import helpers
Related
I have (ended up with) my packages being setup in the following way. There are two packages with the same name, but one of them is inside package_a.
├── package_a/
│ ├── __init__.py
│ └── package_aa/
│ ├── __init__.py
│ └── module_a.py
│ └── module_c.py
└── package_aa/
├── __init__.py
└── module_b.py
How do I import both module_a.py and module_b.py in my code? The following
from package_a.package_aa import module_a; from package_aa import module_b;
fails with the error message
ImportError: cannot import name 'module_b' from 'package_aa' (.\package_a\package_aa\__init__.py)
All __init__.py files contain a single line
import os, sys; sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)))
Renaming is unfortunately not an option as I am not the owner of these packages.
EDIT : Swapping works. Thanks. I have a follow up question : Suppose in module_c.py, modula_a.py is imported like this from package_aa import module_a. Then the following fail with similar error, but it is not as apparent (to the user) as in the original question
from package_aa import module_b; from package_a import module_c
How can the owner of package_a modify it so that its users don't run into errors like this? I am starting to think, the way __init__.py is written might not be right. Although, I have seen answers elsewhere which suggests adding sys.path.append or sys.path.insert in this fashion.
My guess would be to swap them like this:
from package_aa import module_b
from package_a.package_aa import module_a
This way, package_aa is not referenced in the path.
Python 3.8
MacOS
I am trying to import different modules inside a package and I am getting ModuleNotFoundError or ImportError when trying to use relative imports.
I have a folder called "practice_dir" with the following structure:
(base) or ƒ(~/practice_dir) >> tree
.
├── __init__.py
├── dir_1
│ ├── __init__.py
│ ├── file_1.py
│ └── subdir
│ ├── __init__.py
│ └── subfile.py
├── dir_2
│ ├── __init__.py
│ └── file_2.py
└── parent_file.py
It was mentioned online that once I have an init file on every folder containing modules than the absolute imports would start from the parent package
And relative imports would be using . or .. notation
Here's some of the examples I tried
In file_1:
from dir_2 import file_2
ModuleNotFoundError
from .dir_2 import file_2
ImportError: attempted relative import with no known parent package
It's the same if I do the same thing in dir_2, and dir_1/subfolder
The init.py files are empty right now, I tried to include imports to the files but that didn't work.
I've been looking at some questions posted online and on youtube, there seems to be various ways and one guy even mentioned that you don't need the init files if you have python 3.3+
Been battling hard with this, any help would be very much appreciated!
You can add the file to the python path at runtime:
import sys
sys.path.insert(0, 'path/to/file')
import file
I copied python project B to my project A as a model. But B's include path is just based on B. So how can I call B's function from A. For example,
.
└── mypackage A
├── subpackage_1
│ ├── test11.py
│ └── test12.py
├── subpackage_2
│ ├── test21.py
│ └── test22.py
└── subpackage B
├── test31.py
└── test32.py
test31.py may include test32.py by
import test32
But from A's prospective, I should include it by
import B.test32
In fact, B is more complex than this example. How can I refactor it?
first make module using __init__.py in B
then try this:
from B.test32 import 'function Name'
and then call it.
As #Wàlid Bachri said, you should start placing __init__.py files, but it is a bit more complex than than.
You must put those __init__.py files, in each directory. So it would Look something like this.
.
└── mypackage A
├── __init__.py # here under A
├── subpackage_1
| ├── __init__.py
│ ├── test11.py
│ └── test12.py
├── subpackage_2
| ├── __init__.py # under package 2
│ ├── test21.py
│ └── test22.py
└── subpackage B
├── __init__.py # under package B
├── test31.py
└── test32.py
In each of the __init__.py files, you need to import all the files in the same directory. So for example in subpackage B you need to do this.
#/subpackage B/__init__.py
from . import test31
from . import test32
in mypackage A, where there are no files, just directories, you also do the same thing (from . import subpackage_2 etc).
If I suppose that mypackage A is the "main" package (as can be seen by your diargam), so it is not a submodule, to run any file you will need to execute the following.
First cd to the parent directory of mypackage A and then
# Suppose you want to execute /subpackage_1/test11
python -m mypackage_A.subpackage_1.test11 # WARNING mypackage A should have no whitespace
You may get a RuntimeWarning about your sys.modules being modified, but I can assure you, from experience, that you can safely ignore it. This is how python modules for pip are normally done, to ensure that pushing to production is easy.
EDIT : Also note that in all of your files, you should switch to package imports for relative imports, so you must use the dot syntax.
from . import test32 # this will import .test32
if you were to instead to import test32 from test31, the interpeter would try to search for a global package named test32 and not look in the same directory.
Simply add an empty python file called __init__.py to your B project to make Python treat this project as a normal python module.
I have been researching how to build the folder structure for a custom python package. There were several attempts, but none of them seemed to be applicable in general. In particular, the usage (or not usage) of the \__init__.py file(s).
I have a package that consists of several sub-packages, each being responsible to parse Files of a certain kind. Therefore I currently adopted this structure:
Parsers/
├── __init__.py
|
├── ExternalPackages
│ ├── __init__.py
│ ├── package1
│ └── package2
|
├── FileType1_Parsers/
│ ├── __init__.py
│ ├── parsers1.py
│ └── containers1.py
│
└── FileType2_Parsers/
├── __init__.py
├── parsers2.py
└── containers2.py
But it seems not very pythonic, that when I import his package and I want to use a certain class of a module I have to type something like
from Parsers.FileType1_Parsers.parsers1 import example_class
Is there any convention on how to structure such packages or any rules on how to avoid such long import lines?
You can add the following line to Parsers/__init__.py
from .FileType1_Parsers.parsers1 import example_class
Then you can import example_class by
from Parsers import example_class
This is a common practice in large package.
You can modify sys.path at run-time so that it contains a directory for each module you'll be using. For example, for package1 issue the following statements:
>>> sys.path.append(r"[package directory path]\\Parsers\\FileType1_Parsers\\package1")
You can do this for any other modules in the package as well. Now, you can just use this command:
from package1 import example_class
Hope this helps!
I've a project which uses git submodules. In my python file I want to use functions from another python file in the submodule project.
In order to work I had to add the init.py file to all subfolders in the path. My folder tree is the following:
myproj
├── gitmodules
│ ├── __init__.py
│ ├── __init__.pyc
│ └── mygitsubmodule
│ ├── __init__.py
│ ├── __init__.pyc
│ └── file.py
└── myfile.py
Is there any way to make it work without touching mygitsubmodule ?
Thanks
you can add to sys.path in the file you want to be able to access the module, something like:
import sys
sys.path.append("/home/me/myproj/gitmodules")
import mygitsubmodule
This example is adding a path as a raw string to make it clear what's happening. You should really use the more sophisticated, system independent methods described below to determine and assemble the path.
Also, I have found it better, when I used this method, to use sys.path.insert(1, .. as some functionality seems to rely of sys.path[0] being the starting directory of the program.
I am used to avoiding modifying sys.path.
The problem is, when using git submodule, submodule is a project directory, not a Python package. There is a "gap" between your module and that package, so you can't import.
Suppose you have created a submodule named foo_project, and there is a foo package inside.
.
├── foo_project
│ ├── README.rst
│ └── foo
│ └── __init__.py
└── main.py
My solution will be creating a soft link to expose that package to your module:
ln -s foo_project/foo foo
.
├── foo_project
│ ├── README.rst
│ └── foo
│ └── __init__.py
├── foo -> foo_project/foo
└── main.py
Now you can import foo in the main.py.
For reference,
from submodulefolder.file import func_name
or
import submodulefolder.file as lib_name
where file excludes the extension of file.py, seems to work in relative terms without modifying the subfolder / git submodule with a init.py since python 3.3+,
as shown here.
Tested on py3.8.5 linux native and py3.7.8 anaconda Windows, both in Spyder's Ipython-console, as well as natively on linux via terminal.