Importing modules from parent folder - python
I am running Python 2.5.
This is my folder tree:
ptdraft/
nib.py
simulations/
life/
life.py
(I also have __init__.py in each folder, omitted here for readability)
How do I import the nib module from inside the life module? I am hoping it is possible to do without tinkering with sys.path.
Note: The main module being run is in the ptdraft folder.
You could use relative imports (python >= 2.5):
from ... import nib
(What’s New in Python 2.5) PEP 328: Absolute and Relative Imports
EDIT: added another dot '.' to go up two packages
I posted a similar answer also to the question regarding imports from sibling packages. You can see it here.
Solution without sys.path hacks
Summary
Wrap the code into one folder (e.g. packaged_stuff)
Create a setup.py script where you use setuptools.setup().
Pip install the package in editable state with pip install -e <myproject_folder>
Import using from packaged_stuff.modulename import function_name
Setup
I assume the same folder structure as in the question
.
└── ptdraft
├── __init__.py
├── nib.py
└── simulations
├── __init__.py
└── life
├── __init__.py
└── life.py
I call the . the root folder, and in my case it is located in C:\tmp\test_imports.
Steps
Add a setup.py to the root folder
--
The contents of the setup.py can be simply
from setuptools import setup, find_packages
setup(name='myproject', version='1.0', packages=find_packages())
Basically "any" setup.py would work. This is just a minimal working example.
Use a virtual environment
If you are familiar with virtual environments, activate one, and skip to the next step. Usage of virtual environments are not absolutely required, but they will really help you out in the long run (when you have more than 1 project ongoing..). The most basic steps are (run in the root folder)
Create virtual env
python -m venv venv
Activate virtual env
. venv/bin/activate (Linux) or ./venv/Scripts/activate (Win)
Deactivate virtual env
deactivate (Linux)
To learn more about this, just Google out "python virtualenv tutorial" or similar. You probably never need any other commands than creating, activating and deactivating.
Once you have made and activated a virtual environment, your console should give the name of the virtual environment in parenthesis
PS C:\tmp\test_imports> python -m venv venv
PS C:\tmp\test_imports> .\venv\Scripts\activate
(venv) PS C:\tmp\test_imports>
pip install your project in editable state
Install your top level package myproject using pip. The trick is to use the -e flag when doing the install. This way it is installed in an editable state, and all the edits made to the .py files will be automatically included in the installed package.
In the root directory, run
pip install -e . (note the dot, it stands for "current directory")
You can also see that it is installed by using pip freeze
(venv) PS C:\tmp\test_imports> pip install -e .
Obtaining file:///C:/tmp/test_imports
Installing collected packages: myproject
Running setup.py develop for myproject
Successfully installed myproject
(venv) PS C:\tmp\test_imports> pip freeze
myproject==1.0
Import by prepending mainfolder to every import
In this example, the mainfolder would be ptdraft. This has the advantage that you will not run into name collisions with other module names (from python standard library or 3rd party modules).
Example Usage
nib.py
def function_from_nib():
print('I am the return value from function_from_nib!')
life.py
from ptdraft.nib import function_from_nib
if __name__ == '__main__':
function_from_nib()
Running life.py
(venv) PS C:\tmp\test_imports> python .\ptdraft\simulations\life\life.py
I am the return value from function_from_nib!
Relative imports (as in from .. import mymodule) only work in a package.
To import 'mymodule' that is in the parent directory of your current module:
import os
import sys
import inspect
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parentdir = os.path.dirname(currentdir)
sys.path.insert(0, parentdir)
import mymodule
edit: the __file__ attribute is not always given. Instead of using os.path.abspath(__file__) I now suggested using the inspect module to retrieve the filename (and path) of the current file
It seems that the problem is not related to the module being in a parent directory or anything like that.
You need to add the directory that contains ptdraft to PYTHONPATH
You said that import nib worked with you, that probably means that you added ptdraft itself (not its parent) to PYTHONPATH.
You can use OS depending path in "module search path" which is listed in sys.path .
So you can easily add parent directory like following
import sys
sys.path.insert(0,'..')
If you want to add parent-parent directory,
sys.path.insert(0,'../..')
This works both in python 2 and 3.
Don't know much about python 2.
In python 3, the parent folder can be added as follows:
import sys
sys.path.append('..')
...and then one is able to import modules from it
If adding your module folder to the PYTHONPATH didn't work, You can modify the sys.path list in your program where the Python interpreter searches for the modules to import, the python documentation says:
When a module named spam is imported, the interpreter first searches for a built-in module with that name. If not found, it then searches for a file named spam.py in a list of directories given by the variable sys.path. sys.path is initialized from these locations:
the directory containing the input script (or the current directory).
PYTHONPATH (a list of directory names, with the same syntax as the shell variable PATH).
the installation-dependent default.
After initialization, Python programs can modify sys.path. The directory containing the script being run is placed at the beginning of the search path, ahead of the standard library path. This means that scripts in that directory will be loaded instead of modules of the same name in the library directory. This is an error unless the replacement is intended.
Knowing this, you can do the following in your program:
import sys
# Add the ptdraft folder path to the sys.path list
sys.path.append('/path/to/ptdraft/')
# Now you can import your module
from ptdraft import nib
# Or just
import ptdraft
Here is an answer that's simple so you can see how it works, small and cross-platform.
It only uses built-in modules (os, sys and inspect) so should work
on any operating system (OS) because Python is designed for that.
Shorter code for answer - fewer lines and variables
from inspect import getsourcefile
import os.path as path, sys
current_dir = path.dirname(path.abspath(getsourcefile(lambda:0)))
sys.path.insert(0, current_dir[:current_dir.rfind(path.sep)])
import my_module # Replace "my_module" here with the module name.
sys.path.pop(0)
For less lines than this, replace the second line with import os.path as path, sys, inspect,
add inspect. at the start of getsourcefile (line 3) and remove the first line.
- however this imports all of the module so could need more time, memory and resources.
The code for my answer (longer version)
from inspect import getsourcefile
import os.path
import sys
current_path = os.path.abspath(getsourcefile(lambda:0))
current_dir = os.path.dirname(current_path)
parent_dir = current_dir[:current_dir.rfind(os.path.sep)]
sys.path.insert(0, parent_dir)
import my_module # Replace "my_module" here with the module name.
It uses an example from a Stack Overflow answer How do I get the path of the current
executed file in Python? to find the source (filename) of running code with a built-in tool.
from inspect import getsourcefile
from os.path import abspath
Next, wherever you want to find the source file from you just use:
abspath(getsourcefile(lambda:0))
My code adds a file path to sys.path, the python path list
because this allows Python to import modules from that folder.
After importing a module in the code, it's a good idea to run sys.path.pop(0) on a new line
when that added folder has a module with the same name as another module that is imported
later in the program. You need to remove the list item added before the import, not other paths.
If your program doesn't import other modules, it's safe to not delete the file path because
after a program ends (or restarting the Python shell), any edits made to sys.path disappear.
Notes about a filename variable
My answer doesn't use the __file__ variable to get the file path/filename of running
code because users here have often described it as unreliable. You shouldn't use it
for importing modules from parent folder in programs used by other people.
Some examples where it doesn't work (quote from this Stack Overflow question):
• it can't be found on some platforms • it sometimes isn't the full file path
py2exe doesn't have a __file__ attribute, but there is a workaround
When you run from IDLE with execute() there is no __file__ attribute
OS X 10.6 where I get NameError: global name '__file__' is not defined
Here is more generic solution that includes the parent directory into sys.path (works for me):
import os.path, sys
sys.path.append(os.path.join(os.path.dirname(os.path.realpath(__file__)), os.pardir))
The pathlib library (included with >= Python 3.4) makes it very concise and intuitive to append the path of the parent directory to the PYTHONPATH:
import sys
from pathlib import Path
sys.path.append(str(Path('.').absolute().parent))
In a Jupyter Notebook (opened with Jupyter LAB or Jupyter Notebook)
As long as you're working in a Jupyter Notebook, this short solution might be useful:
%cd ..
import nib
It works even without an __init__.py file.
I tested it with Anaconda3 on Linux and Windows 7.
I found the following way works for importing a package from the script's parent directory. In the example, I would like to import functions in env.py from app.db package.
.
└── my_application
└── alembic
└── env.py
└── app
├── __init__.py
└── db
import os
import sys
currentdir = os.path.dirname(os.path.realpath(__file__))
parentdir = os.path.dirname(currentdir)
sys.path.append(parentdir)
Above mentioned solutions are also fine. Another solution to this problem is
If you want to import anything from top level directory. Then,
from ...module_name import *
Also, if you want to import any module from the parent directory. Then,
from ..module_name import *
Also, if you want to import any module from the parent directory. Then,
from ...module_name.another_module import *
This way you can import any particular method if you want to.
Two line simplest solution
import os, sys
sys.path.insert(0, os.getcwd())
If parent is your working directory and you want to call another child modules from child scripts.
You can import all child modules from parent directory in any scripts and execute it as
python child_module1/child_script.py
For me the shortest and my favorite oneliner for accessing to the parent directory is:
sys.path.append(os.path.dirname(os.getcwd()))
or:
sys.path.insert(1, os.path.dirname(os.getcwd()))
os.getcwd() returns the name of the current working directory, os.path.dirname(directory_name) returns the directory name for the passed one.
Actually, in my opinion Python project architecture should be done the way where no one module from child directory will use any module from the parent directory. If something like this happens it is worth to rethink about the project tree.
Another way is to add parent directory to PYTHONPATH system environment variable.
Though the original author is probably no longer looking for a solution, but for completeness, there one simple solution. It's to run life.py as a module like this:
cd ptdraft
python -m simulations.life.life
This way you can import anything from nib.py as ptdraft directory is in the path.
I think you can try this in that specific example, but in python 3.6.3
import sys
sys.path.append('../')
same sort of style as the past answer - but in fewer lines :P
import os,sys
parentdir = os.path.dirname(__file__)
sys.path.insert(0,parentdir)
file returns the location you are working in
In a Linux system, you can create a soft link from the "life" folder to the nib.py file. Then, you can simply import it like:
import nib
I have a solution specifically for git-repositories.
First I used sys.path.append('..') and similar solutions. This causes especially problems if you are importing files which are themselves importing files with sys.path.append('..').
I then decided to always append the root directory of the git repository. In one line it would look like this:
sys.path.append(git.Repo('.', search_parent_directories=True).working_tree_dir)
Or in more details like this:
import os
import sys
import git
def get_main_git_root(path):
main_repo_root_dir = git.Repo(path, search_parent_directories=True).working_tree_dir
return main_repo_root_dir
main_repo_root_dir = get_main_git_root('.')
sys.path.append(main_repo_root_dir)
For the original question: Based on what the root directory of the repository is, the import would be
import ptdraft.nib
or
import nib
Our folder structure:
/myproject
project_using_ptdraft/
main.py
ptdraft/
__init__.py
nib.py
simulations/
__init__.py
life/
__init__.py
life.py
The way I understand this is to have a package-centric view.
The package root is ptdraft, since it's the top most level that contains __init__.py
All the files within the package can use absolute paths (that are relative to package root) for imports, for example
in life.py, we have simply:
import ptdraft.nib
However, to run life.py for package dev/testing purposes, instead of python life.py, we need to use:
cd /myproject
python -m ptdraft.simulations.life.life
Note that we didn't need to fiddle with any path at all at this point.
Further confusion is when we complete the ptdraft package, and we want to use it in a driver script, which is necessarily outside of the ptdraft package folder, aka project_using_ptdraft/main.py, we would need to fiddle with paths:
import sys
sys.path.append("/myproject") # folder that contains ptdraft
import ptdraft
import ptdraft.simulations
and use python main.py to run the script without problem.
Helpful links:
https://tenthousandmeters.com/blog/python-behind-the-scenes-11-how-the-python-import-system-works/ (see how __init__.py can be used)
https://chrisyeh96.github.io/2017/08/08/definitive-guide-python-imports.html#running-package-initialization-code
https://stackoverflow.com/a/50392363/2202107
https://stackoverflow.com/a/27876800/2202107
Work with libraries.
Make a library called nib, install it using setup.py, let it reside in site-packages and your problems are solved.
You don't have to stuff everything you make in a single package. Break it up to pieces.
I had a problem where I had to import a Flask application, that had an import that also needed to import files in separate folders. This is partially using Remi's answer, but suppose we had a repository that looks like this:
.
└── service
└── misc
└── categories.csv
└── test
└── app_test.py
app.py
pipeline.py
Then before importing the app object from the app.py file, we change the directory one level up, so when we import the app (which imports the pipeline.py), we can also read in miscellaneous files like a csv file.
import os,sys,inspect
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
parentdir = os.path.dirname(currentdir)
sys.path.insert(0,parentdir)
os.chdir('../')
from app import app
After having imported the Flask app, you can use os.chdir('./test') so that your working directory is not changed.
It's seems to me that you don't really need to import the parent module. Let's imagine that in nib.py you have func1() and data1, you need to use in life.py
nib.py
import simulations.life.life as life
def func1():
pass
data1 = {}
life.share(func1, data1)
life.py
func1 = data1 = None
def share(*args):
global func1, data1
func1, data1 = args
And now you have the access to func1 and data in life.py. Of course you have to be careful to populate them in life.py before you try to use them,
I made this library to do this.
https://github.com/fx-kirin/add_parent_path
# Just add parent path
add_parent_path(1)
# Append to syspath and delete when the exist of with statement.
with add_parent_path(1):
# Import modules in the parent path
pass
This is the simplest solution that works for me:
from ptdraft import nib
After removing some sys path hacks, I thought it might be valuable to add
My preferred solution.
Note: this is a frame challenge - it's not necessary to do in-code.
Assuming a tree,
project
└── pkg
└── test.py
Where test.py contains
import sys, json; print(json.dumps(sys.path, indent=2))
Executing using the path only includes the package directory
python pkg/test.py
[
"/project/pkg",
...
]
But using the module argument includes the project directory
python -m pkg.test
[
"/project",
...
]
Now, all imports can be absolute, from the project directory. No further skullduggery required.
Although it is against all rules, I still want to mention this possibility:
You can first copy the file from the parent directory to the child directory. Next import it and subsequently remove the copied file:
for example in life.py:
import os
import shutil
shutil.copy('../nib.py', '.')
import nib
os.remove('nib.py')
# now you can use it just fine:
nib.foo()
Of course there might arise several problems when nibs tries to import/read other files with relative imports/paths.
This works for me to import things from a higher folder.
import os
os.chdir('..')
Related
ModuleNotFoundError: No module named when getting a module in a parent folder [duplicate]
I am running Python 2.5. This is my folder tree: ptdraft/ nib.py simulations/ life/ life.py (I also have __init__.py in each folder, omitted here for readability) How do I import the nib module from inside the life module? I am hoping it is possible to do without tinkering with sys.path. Note: The main module being run is in the ptdraft folder.
You could use relative imports (python >= 2.5): from ... import nib (What’s New in Python 2.5) PEP 328: Absolute and Relative Imports EDIT: added another dot '.' to go up two packages
I posted a similar answer also to the question regarding imports from sibling packages. You can see it here. Solution without sys.path hacks Summary Wrap the code into one folder (e.g. packaged_stuff) Create a setup.py script where you use setuptools.setup(). Pip install the package in editable state with pip install -e <myproject_folder> Import using from packaged_stuff.modulename import function_name Setup I assume the same folder structure as in the question . └── ptdraft ├── __init__.py ├── nib.py └── simulations ├── __init__.py └── life ├── __init__.py └── life.py I call the . the root folder, and in my case it is located in C:\tmp\test_imports. Steps Add a setup.py to the root folder -- The contents of the setup.py can be simply from setuptools import setup, find_packages setup(name='myproject', version='1.0', packages=find_packages()) Basically "any" setup.py would work. This is just a minimal working example. Use a virtual environment If you are familiar with virtual environments, activate one, and skip to the next step. Usage of virtual environments are not absolutely required, but they will really help you out in the long run (when you have more than 1 project ongoing..). The most basic steps are (run in the root folder) Create virtual env python -m venv venv Activate virtual env . venv/bin/activate (Linux) or ./venv/Scripts/activate (Win) Deactivate virtual env deactivate (Linux) To learn more about this, just Google out "python virtualenv tutorial" or similar. You probably never need any other commands than creating, activating and deactivating. Once you have made and activated a virtual environment, your console should give the name of the virtual environment in parenthesis PS C:\tmp\test_imports> python -m venv venv PS C:\tmp\test_imports> .\venv\Scripts\activate (venv) PS C:\tmp\test_imports> pip install your project in editable state Install your top level package myproject using pip. The trick is to use the -e flag when doing the install. This way it is installed in an editable state, and all the edits made to the .py files will be automatically included in the installed package. In the root directory, run pip install -e . (note the dot, it stands for "current directory") You can also see that it is installed by using pip freeze (venv) PS C:\tmp\test_imports> pip install -e . Obtaining file:///C:/tmp/test_imports Installing collected packages: myproject Running setup.py develop for myproject Successfully installed myproject (venv) PS C:\tmp\test_imports> pip freeze myproject==1.0 Import by prepending mainfolder to every import In this example, the mainfolder would be ptdraft. This has the advantage that you will not run into name collisions with other module names (from python standard library or 3rd party modules). Example Usage nib.py def function_from_nib(): print('I am the return value from function_from_nib!') life.py from ptdraft.nib import function_from_nib if __name__ == '__main__': function_from_nib() Running life.py (venv) PS C:\tmp\test_imports> python .\ptdraft\simulations\life\life.py I am the return value from function_from_nib!
Relative imports (as in from .. import mymodule) only work in a package. To import 'mymodule' that is in the parent directory of your current module: import os import sys import inspect currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe()))) parentdir = os.path.dirname(currentdir) sys.path.insert(0, parentdir) import mymodule edit: the __file__ attribute is not always given. Instead of using os.path.abspath(__file__) I now suggested using the inspect module to retrieve the filename (and path) of the current file
It seems that the problem is not related to the module being in a parent directory or anything like that. You need to add the directory that contains ptdraft to PYTHONPATH You said that import nib worked with you, that probably means that you added ptdraft itself (not its parent) to PYTHONPATH.
You can use OS depending path in "module search path" which is listed in sys.path . So you can easily add parent directory like following import sys sys.path.insert(0,'..') If you want to add parent-parent directory, sys.path.insert(0,'../..') This works both in python 2 and 3.
Don't know much about python 2. In python 3, the parent folder can be added as follows: import sys sys.path.append('..') ...and then one is able to import modules from it
If adding your module folder to the PYTHONPATH didn't work, You can modify the sys.path list in your program where the Python interpreter searches for the modules to import, the python documentation says: When a module named spam is imported, the interpreter first searches for a built-in module with that name. If not found, it then searches for a file named spam.py in a list of directories given by the variable sys.path. sys.path is initialized from these locations: the directory containing the input script (or the current directory). PYTHONPATH (a list of directory names, with the same syntax as the shell variable PATH). the installation-dependent default. After initialization, Python programs can modify sys.path. The directory containing the script being run is placed at the beginning of the search path, ahead of the standard library path. This means that scripts in that directory will be loaded instead of modules of the same name in the library directory. This is an error unless the replacement is intended. Knowing this, you can do the following in your program: import sys # Add the ptdraft folder path to the sys.path list sys.path.append('/path/to/ptdraft/') # Now you can import your module from ptdraft import nib # Or just import ptdraft
Here is an answer that's simple so you can see how it works, small and cross-platform. It only uses built-in modules (os, sys and inspect) so should work on any operating system (OS) because Python is designed for that. Shorter code for answer - fewer lines and variables from inspect import getsourcefile import os.path as path, sys current_dir = path.dirname(path.abspath(getsourcefile(lambda:0))) sys.path.insert(0, current_dir[:current_dir.rfind(path.sep)]) import my_module # Replace "my_module" here with the module name. sys.path.pop(0) For less lines than this, replace the second line with import os.path as path, sys, inspect, add inspect. at the start of getsourcefile (line 3) and remove the first line. - however this imports all of the module so could need more time, memory and resources. The code for my answer (longer version) from inspect import getsourcefile import os.path import sys current_path = os.path.abspath(getsourcefile(lambda:0)) current_dir = os.path.dirname(current_path) parent_dir = current_dir[:current_dir.rfind(os.path.sep)] sys.path.insert(0, parent_dir) import my_module # Replace "my_module" here with the module name. It uses an example from a Stack Overflow answer How do I get the path of the current executed file in Python? to find the source (filename) of running code with a built-in tool. from inspect import getsourcefile from os.path import abspath Next, wherever you want to find the source file from you just use: abspath(getsourcefile(lambda:0)) My code adds a file path to sys.path, the python path list because this allows Python to import modules from that folder. After importing a module in the code, it's a good idea to run sys.path.pop(0) on a new line when that added folder has a module with the same name as another module that is imported later in the program. You need to remove the list item added before the import, not other paths. If your program doesn't import other modules, it's safe to not delete the file path because after a program ends (or restarting the Python shell), any edits made to sys.path disappear. Notes about a filename variable My answer doesn't use the __file__ variable to get the file path/filename of running code because users here have often described it as unreliable. You shouldn't use it for importing modules from parent folder in programs used by other people. Some examples where it doesn't work (quote from this Stack Overflow question): • it can't be found on some platforms • it sometimes isn't the full file path py2exe doesn't have a __file__ attribute, but there is a workaround When you run from IDLE with execute() there is no __file__ attribute OS X 10.6 where I get NameError: global name '__file__' is not defined
Here is more generic solution that includes the parent directory into sys.path (works for me): import os.path, sys sys.path.append(os.path.join(os.path.dirname(os.path.realpath(__file__)), os.pardir))
The pathlib library (included with >= Python 3.4) makes it very concise and intuitive to append the path of the parent directory to the PYTHONPATH: import sys from pathlib import Path sys.path.append(str(Path('.').absolute().parent))
In a Jupyter Notebook (opened with Jupyter LAB or Jupyter Notebook) As long as you're working in a Jupyter Notebook, this short solution might be useful: %cd .. import nib It works even without an __init__.py file. I tested it with Anaconda3 on Linux and Windows 7.
I found the following way works for importing a package from the script's parent directory. In the example, I would like to import functions in env.py from app.db package. . └── my_application └── alembic └── env.py └── app ├── __init__.py └── db import os import sys currentdir = os.path.dirname(os.path.realpath(__file__)) parentdir = os.path.dirname(currentdir) sys.path.append(parentdir)
Above mentioned solutions are also fine. Another solution to this problem is If you want to import anything from top level directory. Then, from ...module_name import * Also, if you want to import any module from the parent directory. Then, from ..module_name import * Also, if you want to import any module from the parent directory. Then, from ...module_name.another_module import * This way you can import any particular method if you want to.
Two line simplest solution import os, sys sys.path.insert(0, os.getcwd()) If parent is your working directory and you want to call another child modules from child scripts. You can import all child modules from parent directory in any scripts and execute it as python child_module1/child_script.py
For me the shortest and my favorite oneliner for accessing to the parent directory is: sys.path.append(os.path.dirname(os.getcwd())) or: sys.path.insert(1, os.path.dirname(os.getcwd())) os.getcwd() returns the name of the current working directory, os.path.dirname(directory_name) returns the directory name for the passed one. Actually, in my opinion Python project architecture should be done the way where no one module from child directory will use any module from the parent directory. If something like this happens it is worth to rethink about the project tree. Another way is to add parent directory to PYTHONPATH system environment variable.
Though the original author is probably no longer looking for a solution, but for completeness, there one simple solution. It's to run life.py as a module like this: cd ptdraft python -m simulations.life.life This way you can import anything from nib.py as ptdraft directory is in the path.
I think you can try this in that specific example, but in python 3.6.3
import sys sys.path.append('../')
same sort of style as the past answer - but in fewer lines :P import os,sys parentdir = os.path.dirname(__file__) sys.path.insert(0,parentdir) file returns the location you are working in
In a Linux system, you can create a soft link from the "life" folder to the nib.py file. Then, you can simply import it like: import nib
I have a solution specifically for git-repositories. First I used sys.path.append('..') and similar solutions. This causes especially problems if you are importing files which are themselves importing files with sys.path.append('..'). I then decided to always append the root directory of the git repository. In one line it would look like this: sys.path.append(git.Repo('.', search_parent_directories=True).working_tree_dir) Or in more details like this: import os import sys import git def get_main_git_root(path): main_repo_root_dir = git.Repo(path, search_parent_directories=True).working_tree_dir return main_repo_root_dir main_repo_root_dir = get_main_git_root('.') sys.path.append(main_repo_root_dir) For the original question: Based on what the root directory of the repository is, the import would be import ptdraft.nib or import nib
Our folder structure: /myproject project_using_ptdraft/ main.py ptdraft/ __init__.py nib.py simulations/ __init__.py life/ __init__.py life.py The way I understand this is to have a package-centric view. The package root is ptdraft, since it's the top most level that contains __init__.py All the files within the package can use absolute paths (that are relative to package root) for imports, for example in life.py, we have simply: import ptdraft.nib However, to run life.py for package dev/testing purposes, instead of python life.py, we need to use: cd /myproject python -m ptdraft.simulations.life.life Note that we didn't need to fiddle with any path at all at this point. Further confusion is when we complete the ptdraft package, and we want to use it in a driver script, which is necessarily outside of the ptdraft package folder, aka project_using_ptdraft/main.py, we would need to fiddle with paths: import sys sys.path.append("/myproject") # folder that contains ptdraft import ptdraft import ptdraft.simulations and use python main.py to run the script without problem. Helpful links: https://tenthousandmeters.com/blog/python-behind-the-scenes-11-how-the-python-import-system-works/ (see how __init__.py can be used) https://chrisyeh96.github.io/2017/08/08/definitive-guide-python-imports.html#running-package-initialization-code https://stackoverflow.com/a/50392363/2202107 https://stackoverflow.com/a/27876800/2202107
Work with libraries. Make a library called nib, install it using setup.py, let it reside in site-packages and your problems are solved. You don't have to stuff everything you make in a single package. Break it up to pieces.
I had a problem where I had to import a Flask application, that had an import that also needed to import files in separate folders. This is partially using Remi's answer, but suppose we had a repository that looks like this: . └── service └── misc └── categories.csv └── test └── app_test.py app.py pipeline.py Then before importing the app object from the app.py file, we change the directory one level up, so when we import the app (which imports the pipeline.py), we can also read in miscellaneous files like a csv file. import os,sys,inspect currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe()))) parentdir = os.path.dirname(currentdir) sys.path.insert(0,parentdir) os.chdir('../') from app import app After having imported the Flask app, you can use os.chdir('./test') so that your working directory is not changed.
It's seems to me that you don't really need to import the parent module. Let's imagine that in nib.py you have func1() and data1, you need to use in life.py nib.py import simulations.life.life as life def func1(): pass data1 = {} life.share(func1, data1) life.py func1 = data1 = None def share(*args): global func1, data1 func1, data1 = args And now you have the access to func1 and data in life.py. Of course you have to be careful to populate them in life.py before you try to use them,
I made this library to do this. https://github.com/fx-kirin/add_parent_path # Just add parent path add_parent_path(1) # Append to syspath and delete when the exist of with statement. with add_parent_path(1): # Import modules in the parent path pass
This is the simplest solution that works for me: from ptdraft import nib
After removing some sys path hacks, I thought it might be valuable to add My preferred solution. Note: this is a frame challenge - it's not necessary to do in-code. Assuming a tree, project └── pkg └── test.py Where test.py contains import sys, json; print(json.dumps(sys.path, indent=2)) Executing using the path only includes the package directory python pkg/test.py [ "/project/pkg", ... ] But using the module argument includes the project directory python -m pkg.test [ "/project", ... ] Now, all imports can be absolute, from the project directory. No further skullduggery required.
Although it is against all rules, I still want to mention this possibility: You can first copy the file from the parent directory to the child directory. Next import it and subsequently remove the copied file: for example in life.py: import os import shutil shutil.copy('../nib.py', '.') import nib os.remove('nib.py') # now you can use it just fine: nib.foo() Of course there might arise several problems when nibs tries to import/read other files with relative imports/paths.
This works for me to import things from a higher folder. import os os.chdir('..')
Basic Python import mechanics
I have the following directory tree: project/ A/ __init__.py foo.py TestA/ __init__.py testFoo.py the content of testFoo is: import unittest from A import foo from the project directory I run python testA/testFoo.py I get a ModuleNotFoundError No module named A I have two question: how to improt and run A.foo from TestA.testFoo and why is it so difficult to grasp the import logic in Python? Isn't there any debug trick to solve this kind of issues rapidly, I'm sorry I have to bother you with such basics questions?
When your are executing a file an environment variable called python path is generated, python import work with this variable to find your file to import, this path is generated with the path of the file you are executing and it will search in the current directory and sub directories containing an __init__.py file, if you want to import from a directory on the same level you need to modify your python path or change the architecture of your project so the file executed is always on top level. you can include path to your python path like this : import sys sys.path.insert(0, "/path/to/file.py") You can read more on import system : https://docs.python.org/3/reference/import.html The best way in my opinion is to not touch the python path and include your test directoy into the directory where tested files are: project/ A/ __init__.py foo.py TestA/ __init__.py testFoo.py Then run the python -m unittest command into your A or project directory, it will search into your current and sub directories for test and execute it. More on unittest here : https://docs.python.org/3/library/unittest.html
Add the folder project/testA to the system pythonpath first: import sys sys.path.insert(0, "/path/to/pythonfile") and try the import again.
Can you try this ? Create an empty file __init__.py in subdirectory TestA. And add at the begin of main code from __future__ import absolute_import Then import as below : import A.foo as testfoo
The recommended way in py3 may be like below echo $pwd $ /home/user/project python -m testA.testFoo The way of execute module python -m in python is a good way to replace relative references。
You definitely cannot find A because python need look from sys.path, PYTHONPATH to find the module. And python will automatically add current top level script to sys.path not currently directory to sys.path. So if you add print(sys.path) in testFoo.py, you will see it only add project/TestA to the sys.path. Another word, the project did not be included in sys.path, then how python can find the module A? So you had to add the project folder to sys.path by yourself, and, this just needed in top script, something like follows: import unittest import sys import os file_path = os.path.abspath(os.path.dirname(__file__)).replace('\\', '/') lib_path = os.path.abspath(os.path.join(file_path, '..')).replace('\\', '/') sys.path.append(lib_path)
Do I need to add my project directory to the system path in every script to import a function from another directory?
I'm trying to keep a data science project well-organized so I've created a directory inside my src directory called utils that contains a file called helpers.py, which contains some helper functions that will be used in many scripts. What is the best practice for how I should import func_name from src/utils/helpers.py into a file in a totally different directory, such as src/processing/clean_data.py? I see answers to this question, and I've implemented a solution that works, but this feels ugly: sys.path.append(os.path.dirname(os.path.dirname(os.path.dirname(os.path.realpath(__file__)))))) Am I doing this right? Do I need to add this to every script that wants to import func_name, like train_model.py? My current project folder structure: myproject /notebooks notebook.ipynb /src /processing clean_data.py /utils helpers.py /models train_model.py __init__.py Example files: # clean_data.py import os import sys sys.path.append(os.path.dirname(os.path.dirname(os.path.dirname(os.path.realpath(__file__)))))) from src.utils.helpers import func_name func_name() # helpers.py def func_name(): print('I'm a helper function.')
The correct way to do it is to use __init__.py, setup.py and the setuptools Python package: myPackage/ myPackage/ __init__.py setup.py This link has all the steps.
First of all, let me describe you the differences between a Python module & a Python package so that both of us are on the same page. ✌ A module is a single .py file (or files) that are imported under one import and used. ✔ import aModuleName # Here 'aModuleName' is just a regular .py file. Whereas, a package is a collection of modules in directories that give a package hierarchy. A package contains a distinct __init__.py file. ✔ from aPackageName import aModuleName # Here 'aPackageName` is a folder with a `__init__.py` file # and 'aModuleName', which is just a regular .py file. Therefore, when we have a project directory named proj-dir of the following structure ⤵ proj-dir --|--__init__.py --package1 --|--__init__.py --|--module1.py --package2 --|--__init__.py --|--module2.py 🔎 Notice that I've also added an empty __init__.py into the proj-dir itself which makes it a package too. 👍 Now, if you want to import any python object from module2 of package2 into module1 of package1, then the import statement in the file module1.py would be from package2.module2 import object2 # if you were to import the entire module2 then, from package2 import module2 I hope this simple explanation clarifies your doubts on Python imports' mechanism and solves the problem. If not then do comment here. 😊
First of all let me clarify you that importing an entire module, if you are going to use a part of it, then is not a good idea. Instead of that you can use from to import specific function under a library/package. By doing this, you make your program efficient in terms of memory and performance. To know more refer these: 'import module' or 'from module import' difference between import and from Net let us look into the solution. Before starting off with the solution, let me clarify you the use of __init__.py file. It just tells the python interpreter that the *.py files present there are importable which means they are modules and are/maybe a part of a package. So, If you have N no of sub directories you have to put __init__.py file in all those sub directories such that they can also be imported. Inside __init__.py file you can also add some additional information like which path should be included, default functions,variables,scope,..etc. To know about these just google about __init__.py file or take some python library and go through the same __init__.py file to know about it. (Here lies the solution) More Info: modules Be pythonic So as stated by #Sushant Chaudhary your project structure should be like proj-dir --|--__init__.py --package1 --|--__init__.py --|--module1.py --package2 --|--__init__.py --|--module2.py So now, If I put __init__.py file under my directory like above, Will it be importable and work fine? yes and no. Yes : If you are importing the modules within that project/package directory. for example in your case you are importing package1.module1 in pakage2.module2 as from package1 import module1. Here you have to import the base dir inside the sub modules, Why? the project will run fine if you are running the module from the same place. i.e: inside package2 as python module2.py, But will throw ModuleNotFoundError If you run the module from some other directory. i.e: any other path except under package2 for example under proj-dir as python package2/module2.py. This is what happening in your case. You are running the module from project-dir. So How to fix this? 1- You have to append basedir path to system path in module2.py as from sys import path dir_path = "/absolute/path/to/proj-dir" sys.path.insert(0, dir_path) So that module2 will be able to find package1 (and module1 inside it). 2- You have to add all the sub module paths in __init__.py file under proj-dir. For example: #__init__.py under lxml # this is a package def get_include(): """ Returns a list of header include paths (for lxml itself, libxml2 and libxslt) needed to compile C code against lxml if it was built with statically linked libraries. """ import os lxml_path = __path__[0] include_path = os.path.join(lxml_path, 'includes') includes = [include_path, lxml_path] for name in os.listdir(include_path): path = os.path.join(include_path, name) if os.path.isdir(path): includes.append(path) return includes This is the __init__.py file of lxml (a python library for parsing html,xml data). You can refer any __init__.py file under any python libraries having sub modules.ex (os,sys). Here I've mentioned lxml because I thought it will be easy for you to understand. You can even check __init__.py file under other libraries/packages. Each will have it's own way of defining the path for submodules. No If you are trying to import modules outside the directory. Then you have to export the module path such that other modules can find them into environment variables. This can be done directly by appending absolute path of the base dir to PYTHONPATH or to PATH. To know more: PATH variables in OS PYTHONPATH variable So to solve your problem, include the paths to all the sub modules in __init__.py file under proj-dir and add the /absolute/path/to/proj-dir either to PYTHONPATH or PATH. Hope the answer explains you about usage of __init__.py and solves your problem.
On Linux, you can just add the path to the parent folder of your src directory to ~/.local/lib/python3.6/site-packages/my_modules.pth. See Using .pth files. You can then import modules in src from anywhere on your system. NB1: Replace python3.6 by any version of Python you want to use. NB2: If you use Python2.7 (don't know for other versions), you will need to create __init__.py (empty) files in src/ and src/utils. NB3: Any name.pth file is ok for my_modules.pth.
Yes, you can only import code from installed packages or from files in you working directory or subdirectories.
the way I see it, your problem would be solved if you would have your module or package installed, like an yother package one installs and then imports (numpy, xml, json etc.) I also have a package I constantly use in all my projects, ulitilies, and I know it's a pain with the importing. here is a description on how to How to package a python application to make it pip-installable: https://marthall.github.io/blog/how-to-package-a-python-app/
Navigate to your python installation folder Navigate to lib Navigate to site-packages Make a new file called any_thing_you_want.pth Type .../src/utils/helpers.py inside that file with your favorite text editor Note: the ellipsis before scr/utils/helpers.py will look something like: C:/Users/blahblahblah/python_folders/scr... <- YOU DO NEED THIS! This is a cheap way out but it keeps code clean, and is the least complicated. The downside is, for every folder your modules are in, example.pth will need them. Upside: works with Windows all the way up to Windows 10
trying to make paths work - attempted relative import beyond top-level package
I can't make this work.. My structure is: program_name/ __init__.py setup.py src/ __init__.py Process/ __init__.py thefile.py tests/ __init__.py thetest.py thetest.py: from ..src.Process.thefile.py import sth Running: pytest ./tests/thetest.py from program_name gives : ValueError: attempted relative import beyond top-level package I tried also other approaches but i am receiving various errors. But I would expect for the above to work.
ValueError: Attempted relative import in non-package States that you're trying to use relative import in the module, which are to be used for packages i.e. to make it a package add __init__.py and call the thetest.py from some file outside the package. Directly running thetest.py from interpreter won't work. Relative imports require that the module which uses them is being imported itself either as package module. Suggestion 1: The current tests directory has a __init__.py file but that doesn't allow you to run it as a module (via shell) - to make your current (relative) import work, you need to import it in an external (to package) file/module - let's create a main.py (can name it anything you like): main.py program_name/ __init__.py setup.py src/ __init__.py Process/ __init__.py thefile.py tests/ __init__.py thetest.py src/Process/thefile.py: s = 'Hello world' tests/thetest.py: from ..src.Process.thefile import s print s main.py: from program_name.tests.thetest import s Executing main.py: [nahmed#localhost ~]$ python main.py Hello world Suggestion 2: Execute the file just above root dir i.e. one level up the program_name/ , in the following fashion: [nahmed#localhost ~]$ python -m program_name.tests.thetest Hell World P.S. relative imports are for packages, not modules.
Just solved a similar problem with a lot of googling. Here's two solutions without changing the existing file structor: 1 The way to import module from parent folder from ..src.Process.thefile.py import sth is called "relative import". It's only supported when launching as a package from the top-level package. In your case, that is launching command line from the directory which contains program_name/ and type (for win environment) python -m program_name.tests.thetest or simply (useful for many pytest files): python -m pytest 2 Otherwise -- when trying to run a script alone or from a non top-level package -- you could manually add directory to the PYTHONPATH at run time. import sys from os import path sys.path.append(path.dirname(path.dirname(path.abspath(__file__)))) from src.Process.thefile import s Try the first one first see if it's compatiable with the pytest framework. Otherwise the second one should always solve the problem. Reference (How to fix "Attempted relative import in non-package" even with __init__.py)
When importing a file, Python only searches the current directory, the directory that the entry-point script is running from. you can use sys.path to include different locations import sys sys.path.insert(0, '/path/to/application/app/folder') import thefile
Import python package from local directory into interpreter
I'm developing/testing a package in my local directory. I want to import it in the interpreter (v2.5), but sys.path does not include the current directory. Right now I type in sys.path.insert(0,'.'). Is there a better way? Also, from . import mypackage fails with this error: ValueError: Attempted relative import in non-package
You can use relative imports only from in a module that was in turn imported as part of a package -- your script or interactive interpreter wasn't, so of course from . import (which means "import from the same package I got imported from") doesn't work. import mypackage will be fine once you ensure the parent directory of mypackage is in sys.path (how you managed to get your current directory away from sys.path I don't know -- do you have something strange in site.py, or...?) To get your current directory back into sys.path there is in fact no better way than putting it there.
Keep it simple: try: from . import mymodule # "myapp" case except: import mymodule # "__main__" case
See the documentation for sys.path: http://docs.python.org/library/sys.html#sys.path To quote: If the script directory is not available (e.g. if the interpreter is invoked interactively or if the script is read from standard input), path[0] is the empty string, which directs Python to search modules in the current directory first. So, there's no need to monkey with sys.path if you're starting the python interpreter from the directory containing your module. Also, to import your package, just do: import mypackage Since the directory containing the package is already in sys.path, it should work fine.
If you want to run an unmodified python script so it imports libraries from a specific local directory you can set the PYTHONPATH environment variable - e.g. in bash: export PYTHONPATH=/home/user/my_libs python myscript.py If you just want it to import from the current working directory use the . notation: export PYTHONPATH=. python myscript.py
Inside a package if there is setup.py, then better to install it pip install -e .
A simple way to make it work is to run your script from the parent directory using python's -m flag, e.g. python -m packagename.scriptname. Obviously in this situation you need an __init__.py file to turn your directory into a package.
Using sys.path should include current directory already. Try: import . or: from . import sth however it may be not a good practice, so why not just use: import mypackage
A bit late to the party, but this is what worked for me: >>> import sys >>> sys.path.insert(0, '') Apparently, if there is an empty string, Python knows that it should look in the current directory. I did not have the empty string in sys.path, which caused this error.
Speaking for python3.. I wanted to use an improved version of a library that's installed in my environment. There are some extra print statements it makes to show that it and not the original lib is being used. I placed the lib's folder next to the python script. Ran the script.. it ran with the local lib with the modifications. Removed the folder and ran it again - this time it ran with the installed lib. So, solution was simple : place the lib's folder (with same name as in your import statement) in your project folder. That does the job, at least at my end. This is on a standard Linux Mint 20.04 system, with a python 3.8 virutal environment activated (so "(py3.8)" appears in my terminal when I'm in the virtual env)
You can import package_name if the package is a module: this needs you have init.py under the package and things that you want to use are needed to import in the init.py Or if you want to import class under the package, you can use from package_name import class_name