This must have been asked before, but I cannot for the life of me find it.
I have a project structure something like this
ROOT
├── README.md
├── modules
│ ├── __init__.py
│ ├── mod1.py
│ └── mod2.py
├── tests
│ ├── test_mod1.py
│ └── test_mod2.py
├── notebooks
│ ├── nb1.ipynb
│ ├── nb2.ipynb
│ └── sub_dir
│ ├── sub_nb.ipynb
│ ├── generate_py
│ └── py_files
│ └── sub_nb.py
├── definitions
└── main.py
So from main.py I am able to import anything from definitions, or any module from ROOT/modules.
What I want is to be able to import from these from anywhere within the notebooks directory tree. I know I could do this using:
import sys
sys.path.append("..")
But the notebooks directory tree has many layers, and I don't want my code to start looking like this:
import sys
sys.path.append("../../../../")
What's more, the file generate_py is a bash script, that converts the jupyter notebooks (.ipynb) to .py files and stashes the .py files into the ./py_files subdir.
With the above method i end up having to manually edit every file to put an extra ../ into the sys.append(). This is annoying.
If I run files from within pycharm all works well (I'm guessing it updates your PYTHONPATH to include the project root when you create main.py?)
To further complicate, this project is run on several machines, shared via git. So absolute references are out.
But running from terminal, or from within jupyter it cant gind modules, or definitions without going through the sys.append() process. And I feel there must be a better way.
So what is the best practice here?
Related
I have a python project with hierarchy kind of like this: (but with more folders under src)
ip-rep/
│
├── ip_rep/
│ ├── __init__.py
│ ├── run.py
│ ├── src/
│ ├── __init__.py
│ ├── cli_tool/
│ ├── __init__.py
│ ├---- cli_loader.py
├── tests/
│ ├── test_cli_loader
│ ├── test_cli_loader.py
│
├── dockefile
├── bash_start.sh
now in the dockerfile I have something like this (simplified):
RUN mkdir app
WORKDIR /app
ADD ip_rep /app/ip_rep
ADD load_ips.sh /app/
RUN chmod +x /app/load_ips.sh
ENTRYPOINT ["/bin/bash", "load_ips.sh"]
and this is the bash file (again simplified):
python ip_rep/run.py
and this is an example of an import in run.py file:
from ip_rep.src.cli_tool.cli_loader import CliTool
When I try to run it I get the error:
Traceback (most recent call last):
File "ip_rep/run.py", line 9, in <module>
from ip_rep.src.cli_tool.cli_loader import CliTool
ModuleNotFoundError: No module named 'ip_rep'
I can't really understand what I did wrong.
I can't remove the ip_rep in run.py and do only src. file because then the tests stop working.
What am I missing?
Use this structure instead
ip-rep/
│
├── ip_rep/
│ ├── __init__.py
│ ├── cli_tool/
│ ├── __init__.py
│ ├---- cli_loader.py
├── scripts/
│ ├── run.py
│
├── tests/
│ ├── test_cli_loader
│ ├── test_cli_loader.py
│
├── dockerfile
├── bash_start.sh
(You can ignore the ${SOMEDIR}/ part, but it's needed as example below.)
That moves run.py into a separate directory at the same level as the ip_rep package directory, but not inside it; similar to the tests.
It also removes the unnecessary src/ directory, as commented by Lenormju.
Your run.py file should now have the import line:
from ip_rep.cli_tool.cli_loader import CliTool
and your bash_start.sh should (at least) have something like
PYTHONPATH=app/ip_rep
python app/ip_rep/scripts/run.py
It is not unlikely that you don't need to set PYTHONPATH, if your bash script is already in the app/ip_rep directory: Python will automatically put the current directory on its search path. The script could also then be relative, and the whole thing becomes
python scripts/run.py
The first variant is just more explicit, and allows for putting the bash script elsewhere.
Naturally, the best part is to install the package and script(s) properly, so that they are on the system PATH and PYTHONPATH. That would require something like pip install . to be run, with a valid setup.py or pyproject.toml file for the project. With that, the bash script becomes simply run.py (just that one line) or even run, depending on how you configured things. Of course, "run" is a generic and bad name for a script, but that should be what you could be working towards.
I have a problem where this is my project structure:
.
├── Resources/
├── src/
│ ├── __init__.py
│ ├── main.py
│ ├── utils/
│ │ ├── __init__.py
│ │ ├── util.py
│ │ └── otherUtil.py
│ └── calculations/
│ ├── __init__.py
│ └── financials.py
└── tests/
├── __init__.py
└── test.py
My problem is that I can't reach the classes from the src/ folder from the tests, although the code in src/ can reach the Resources folder, through the first shown method.
I have tried:
To append the home library path this way:
Here I used the from src import util after these lines, I even tried from .src import util.
Then this way:
Here I used the from src import util after these lines, I even tried from .src import util.
Than without the sys.path.append() with no use.
I have tried every combination I know, but for no use, and I don't want to install them as individual packages. Does someone have an idea, witch will solve my problem?
Clarification edit:
I don't want to put the tests in the source folder, i want to keep them separate.
You can use this code found here:
# test.py
import sys
# insert at 1, 0 is the script path (or '' in REPL)
sys.path.insert(1, '/path/to/utils/')
import utils
Python 3.8
MacOS
I am trying to import different modules inside a package and I am getting ModuleNotFoundError or ImportError when trying to use relative imports.
I have a folder called "practice_dir" with the following structure:
(base) or ƒ(~/practice_dir) >> tree
.
├── __init__.py
├── dir_1
│ ├── __init__.py
│ ├── file_1.py
│ └── subdir
│ ├── __init__.py
│ └── subfile.py
├── dir_2
│ ├── __init__.py
│ └── file_2.py
└── parent_file.py
It was mentioned online that once I have an init file on every folder containing modules than the absolute imports would start from the parent package
And relative imports would be using . or .. notation
Here's some of the examples I tried
In file_1:
from dir_2 import file_2
ModuleNotFoundError
from .dir_2 import file_2
ImportError: attempted relative import with no known parent package
It's the same if I do the same thing in dir_2, and dir_1/subfolder
The init.py files are empty right now, I tried to include imports to the files but that didn't work.
I've been looking at some questions posted online and on youtube, there seems to be various ways and one guy even mentioned that you don't need the init files if you have python 3.3+
Been battling hard with this, any help would be very much appreciated!
You can add the file to the python path at runtime:
import sys
sys.path.insert(0, 'path/to/file')
import file
I copied python project B to my project A as a model. But B's include path is just based on B. So how can I call B's function from A. For example,
.
└── mypackage A
├── subpackage_1
│ ├── test11.py
│ └── test12.py
├── subpackage_2
│ ├── test21.py
│ └── test22.py
└── subpackage B
├── test31.py
└── test32.py
test31.py may include test32.py by
import test32
But from A's prospective, I should include it by
import B.test32
In fact, B is more complex than this example. How can I refactor it?
first make module using __init__.py in B
then try this:
from B.test32 import 'function Name'
and then call it.
As #Wàlid Bachri said, you should start placing __init__.py files, but it is a bit more complex than than.
You must put those __init__.py files, in each directory. So it would Look something like this.
.
└── mypackage A
├── __init__.py # here under A
├── subpackage_1
| ├── __init__.py
│ ├── test11.py
│ └── test12.py
├── subpackage_2
| ├── __init__.py # under package 2
│ ├── test21.py
│ └── test22.py
└── subpackage B
├── __init__.py # under package B
├── test31.py
└── test32.py
In each of the __init__.py files, you need to import all the files in the same directory. So for example in subpackage B you need to do this.
#/subpackage B/__init__.py
from . import test31
from . import test32
in mypackage A, where there are no files, just directories, you also do the same thing (from . import subpackage_2 etc).
If I suppose that mypackage A is the "main" package (as can be seen by your diargam), so it is not a submodule, to run any file you will need to execute the following.
First cd to the parent directory of mypackage A and then
# Suppose you want to execute /subpackage_1/test11
python -m mypackage_A.subpackage_1.test11 # WARNING mypackage A should have no whitespace
You may get a RuntimeWarning about your sys.modules being modified, but I can assure you, from experience, that you can safely ignore it. This is how python modules for pip are normally done, to ensure that pushing to production is easy.
EDIT : Also note that in all of your files, you should switch to package imports for relative imports, so you must use the dot syntax.
from . import test32 # this will import .test32
if you were to instead to import test32 from test31, the interpeter would try to search for a global package named test32 and not look in the same directory.
Simply add an empty python file called __init__.py to your B project to make Python treat this project as a normal python module.
I have been researching how to build the folder structure for a custom python package. There were several attempts, but none of them seemed to be applicable in general. In particular, the usage (or not usage) of the \__init__.py file(s).
I have a package that consists of several sub-packages, each being responsible to parse Files of a certain kind. Therefore I currently adopted this structure:
Parsers/
├── __init__.py
|
├── ExternalPackages
│ ├── __init__.py
│ ├── package1
│ └── package2
|
├── FileType1_Parsers/
│ ├── __init__.py
│ ├── parsers1.py
│ └── containers1.py
│
└── FileType2_Parsers/
├── __init__.py
├── parsers2.py
└── containers2.py
But it seems not very pythonic, that when I import his package and I want to use a certain class of a module I have to type something like
from Parsers.FileType1_Parsers.parsers1 import example_class
Is there any convention on how to structure such packages or any rules on how to avoid such long import lines?
You can add the following line to Parsers/__init__.py
from .FileType1_Parsers.parsers1 import example_class
Then you can import example_class by
from Parsers import example_class
This is a common practice in large package.
You can modify sys.path at run-time so that it contains a directory for each module you'll be using. For example, for package1 issue the following statements:
>>> sys.path.append(r"[package directory path]\\Parsers\\FileType1_Parsers\\package1")
You can do this for any other modules in the package as well. Now, you can just use this command:
from package1 import example_class
Hope this helps!