Develop Reference

Do I need to add my project directory to the system path in every script to import a function from another directory?

I'm trying to keep a data science project well-organized so I've created a directory inside my src directory called utils that contains a file called helpers.py, which contains some helper functions that will be used in many scripts. What is the best practice for how I should import func_name from src/utils/helpers.py into a file in a totally different directory, such as src/processing/clean_data.py?
I see answers to this question, and I've implemented a solution that works, but this feels ugly:
sys.path.append(os.path.dirname(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))))
Am I doing this right? Do I need to add this to every script that wants to import func_name, like train_model.py?
My current project folder structure:
myproject
/notebooks
notebook.ipynb
/src
/processing
clean_data.py
/utils
helpers.py
/models
train_model.py
__init__.py
Example files:
# clean_data.py
import os
import sys
sys.path.append(os.path.dirname(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))))
from src.utils.helpers import func_name
func_name()
# helpers.py
def func_name():
print('I'm a helper function.')

The correct way to do it is to use __init__.py, setup.py and the setuptools Python package:
myPackage/
myPackage/
__init__.py
setup.py
This link has all the steps.

First of all, let me describe you the differences between a Python module & a Python package so that both of us are on the same page. ✌
A module is a single .py file (or files) that are imported under one import and used. ✔
import aModuleName
# Here 'aModuleName' is just a regular .py file.
Whereas, a package is a collection of modules in directories that give a package hierarchy. A package contains a distinct __init__.py file. ✔
from aPackageName import aModuleName
# Here 'aPackageName` is a folder with a `__init__.py` file
# and 'aModuleName', which is just a regular .py file.
Therefore, when we have a project directory named proj-dir of the following structure ⤵
proj-dir
--|--__init__.py
--package1
--|--__init__.py
--|--module1.py
--package2
--|--__init__.py
--|--module2.py
🔎 Notice that I've also added an empty __init__.py into the proj-dir itself which makes it a package too.
👍 Now, if you want to import any python object from module2 of package2 into module1 of package1, then the import statement in the file module1.py would be
from package2.module2 import object2
# if you were to import the entire module2 then,
from package2 import module2
I hope this simple explanation clarifies your doubts on Python imports' mechanism and solves the problem. If not then do comment here. 😊

First of all let me clarify you that importing an entire module, if you are going to use a part of it, then is not a good idea. Instead of that you can use from to import specific function under a library/package. By doing this, you make your program efficient in terms of memory and performance.
To know more refer these:
'import module' or 'from module import'
difference between import and from
Net let us look into the solution.
Before starting off with the solution, let me clarify you the use of __init__.py file. It just tells the python interpreter that the *.py files present there are importable which means they are modules and are/maybe a part of a package.
So, If you have N no of sub directories you have to put __init__.py file in all those sub directories such that they can also be imported. Inside __init__.py file you can also add some additional information like which path should be included, default functions,variables,scope,..etc. To know about these just google about __init__.py file or take some python library and go through the same __init__.py file to know about it. (Here lies the solution)
More Info:
modules
Be pythonic
So as stated by #Sushant Chaudhary your project structure should be like
proj-dir
--|--__init__.py
--package1
--|--__init__.py
--|--module1.py
--package2
--|--__init__.py
--|--module2.py
So now, If I put __init__.py file under my directory like above, Will
it be importable and work fine?
yes and no.
Yes :
If you are importing the modules within that project/package directory.
for example in your case
you are importing package1.module1 in pakage2.module2 as from package1 import module1.
Here you have to import the base dir inside the sub modules, Why? the project will run fine if you are running the module from the same place. i.e: inside package2 as python module2.py, But will throw ModuleNotFoundError If you run the module from some other directory. i.e: any other path except under package2 for example under proj-dir as python package2/module2.py. This is what happening in your case. You are running the module from project-dir.
So How to fix this?
1- You have to append basedir path to system path in module2.py as
from sys import path
dir_path = "/absolute/path/to/proj-dir"
sys.path.insert(0, dir_path)
So that module2 will be able to find package1 (and module1 inside it).
2- You have to add all the sub module paths in __init__.py file under proj-dir.
For example:
#__init__.py under lxml
# this is a package
def get_include():
"""
Returns a list of header include paths (for lxml itself, libxml2
and libxslt) needed to compile C code against lxml if it was built
with statically linked libraries.
"""
import os
lxml_path = __path__[0]
include_path = os.path.join(lxml_path, 'includes')
includes = [include_path, lxml_path]
for name in os.listdir(include_path):
path = os.path.join(include_path, name)
if os.path.isdir(path):
includes.append(path)
return includes
This is the __init__.py file of lxml (a python library for parsing html,xml data). You can refer any __init__.py file under any python libraries having sub modules.ex (os,sys). Here I've mentioned lxml because I thought it will be easy for you to understand. You can even check __init__.py file under other libraries/packages. Each will have it's own way of defining the path for submodules.
No
If you are trying to import modules outside the directory. Then you have to export the module path such that other modules can find them into environment variables. This can be done directly by appending absolute path of the base dir to PYTHONPATH or to PATH.
To know more:
PATH variables in OS
PYTHONPATH variable
So to solve your problem, include the paths to all the sub modules in __init__.py file under proj-dir and add the /absolute/path/to/proj-dir either to PYTHONPATH or PATH.
Hope the answer explains you about usage of __init__.py and solves your problem.

On Linux, you can just add the path to the parent folder of your src directory to ~/.local/lib/python3.6/site-packages/my_modules.pth. See
Using .pth files. You can then import modules in src from anywhere on your system.
NB1: Replace python3.6 by any version of Python you want to use.
NB2: If you use Python2.7 (don't know for other versions), you will need to create __init__.py (empty) files in src/ and src/utils.
NB3: Any name.pth file is ok for my_modules.pth.

Yes, you can only import code from installed packages or from files in you working directory or subdirectories.

the way I see it, your problem would be solved if you would have your module or package installed, like an yother package one installs and then imports (numpy, xml, json etc.)
I also have a package I constantly use in all my projects, ulitilies, and I know it's a pain with the importing.
here is a description on how to How to package a python application to make it pip-installable:
https://marthall.github.io/blog/how-to-package-a-python-app/

Navigate to your python installation folder
Navigate to lib
Navigate to site-packages
Make a new file called any_thing_you_want.pth
Type .../src/utils/helpers.py inside that file with your favorite text editor
Note: the ellipsis before scr/utils/helpers.py will look something like: C:/Users/blahblahblah/python_folders/scr... <- YOU DO NEED THIS!
This is a cheap way out but it keeps code clean, and is the least complicated. The downside is, for every folder your modules are in, example.pth will need them. Upside: works with Windows all the way up to Windows 10

Python importing errors and correct importing guidance

My structure is like so
/App
__init__.py
-/dirA
-__init__.py
-a.py
-/dirB
-__init__.py
-b.py
/samy_app
__init__.py
settings.py
urls.py
manage.py
Inside of b.py I want to import a.py
So inside of b.py I have
from App.dirA import a
However the error I'm receiving is "ImportError: No module named 'App'"
I've also tried relative importing with .. but thats not working either.
I would also like to know if I shouldn't put the files in seperate directories like that. It was really for organization. Or should I move them out of the app and place the folders in the same directory as manage.py?
Would really love some help with this. Been trying all day.
sys.path output
['', '/root/django-dev', '/usr/lib/python35.zip', '/usr/lib/python3.5', '/usr/lib/python3.5/plat-x86_64-linux-gnu', '/usr/lib/python3.5/lib-dynload', '/usr/lib/python3.5/site-packages', '/usr/local/lib/python3.5/dist-packages', '/usr/lib/python3/dist-packages']
Thank you everyone for all of the help. Is there maybe something I didnt do when I set up my server? I read the docs and it seems like this import should work without appending any paths...

I believe when I've done this, I usually had to do it like this:
from dirA.a import THING_IM_IMPORTING_FROM_A
where the root of my application would be the App directory.

You are mentioning manage.py, are you using django perhaps ?
Anyway if you are running a script from dirB you will get an error but if you have structure like this:
main.py
App/
__init__.py
dirA/
__init__.py
a.py
dirB
__init__.py
b.py
and in main.py you have
from App.dirB import b
and in b.py you have
from App.dirA import a or from ..dirA import a
you should be ok.
In any other case you should probably add ../ ( or absolute path of it ) to your PYTHONPATH
import sys
sys.path.append("../")
but I think it's not a very good idea.

Parent of the App directory should be in PYTHONPATH (or sys.path). If App directory resides in /home/dorian/mega_project, then PYTHONPATH should contain /home/dorian/mega_project.

Edit: I misunderstood the question!
Can you verify /App is a nested directory of a directory on your module search path?
In an interactive session, import sys, and type sys.path. The first entry is the CWD or the directory of the top-level program file. If you don't see a directory there leading up to /App, you'll need to add it, as Python searches this list left-to-right when you're importing a package to find the directory that is its parent. You can either edit the PYTHONPATH environment variable, or create a .pth text file and simply type in the directory leading up to it (i.e., /home/user/Documents/code') and put that in your installation directory for Python, or your site-packages folder.
Once that's done, when you check sys.path you'll see, for example, '/home/user/Documents/code'. And /Apps should be within the code folder in this example.

What is causing Import Error: "no module named modules"?

I am working on my first open source project.
While getting everything setup for the project, I followed the README.md step by step.
Now, I run into a problem. When I try to run a test and src scripts, I get the following error,
ImportError: No module named modules
Now, below is the file structure.
../
/modules
__init__.py
/src
lyrics.py
/tests
test_lyrics.py
lyrics.py import statements
import modules
def test_lyrics():
assert('lyrics' == modules.process_query('paradise lyrics')[0])
This is where the error "Import Error: modules not found".
Yes, all the requirements on the README were met.
If you want to take a look at the project, check it out on github.

It could be that your module's directory is not being read by your PYTHONPATH. To check this go to your home directory and look for a .bashrc or some kind of .profile file. You may have to use ls -a to see these hidden files. If you don't see your module's address listed, add this to the file:
export PYTHONPATH="${PYTHONPATH}:/my/module/path"
Make sure the address points to the directory with your highest-level __init__.py file as python uses it to read the directory as a python package.

Use python -m site to conveniently view the
effect of the $PYTHONPATH env var on sys.path.
Code authors will often assume that current directory
is in your path:
$ export PYTHONPATH=.
If you intend to use . current directory,
then pay careful attention to which one's in use
at the time the script starts running.
You may want to view this value: os.getcwd()

I am maybe a bit late to the party but I can answer and maybe it helps.
1)Check once more your paths.
2)Check that you run with >./myProj arguments more arguments, and not
myProj arguments more arguments. Hope it helps even a bit

You could try using a relative import
from ..modules import process_query
def test_lyrics():
assert('lyrics' == process_query('paradise lyrics')[0])

Relative Import in Python

I am having the following directory structure and I'm new to python. In folder bin I have a single file "script.py" in which I want to import "module.py" from code package. I have seen many solutions on stackoverflow to this problem that involve modifying sys-path or giving full pathname. Is there any way I could import these with an import statement that works relatively, instead of specifying full path?
project/
bin/
script.py
code/
module.py
__init__.py

To make sure that bin/script.py can be run without configuring environment, add this preambule to script.py before from code import module line (
from twisted/bin/_preambule.py):
# insert `code/__init__.py` parent directory into `sys.path`
import sys, os
path = os.path.abspath(sys.argv[0]) # path to bin/script.py
while os.path.dirname(path) != path: # until top-most directory
if os.path.exists(os.path.join(path, 'code', '__init__.py')):
sys.path.insert(0, path)
break
path = os.path.dirname(path)
The while loop is to support running it from bin/some-other-directory/.../script.py

While correct, I think that the "dynamically add my current directory to the path" is a dirty, dirty hack.
Add a (possibly blank) __init__.py to project. Then add a .pth file containing the path to project to your sites-packages directory. Then:
from project.code import module
This has two advantages, in my opinion:
1) If you refactor, you just need to change the from project.code line and avoid messing with anything else.
2) There will be nothing special about your code -- it will behave exactly like any other package that you've installed through PyPi.
It may seem messy to add your project directory to your PYTHONPATH but I think it's a much, much cleaner solution than any of the alternatives. Personally, I've added the parent directory that all of my python code lives in to my PYTHONPATH with a .pth file, so I can deal with all of the code I write just like 3rd party libraries.
I don't think that there is any issue with cluttering up your PYTHONPATH, since only folders with an __init__.py will be importable.

Path exists in sys.path but still gives module not found error in Python

So, I print the sys.path in the python terminal inside the working python project directory, and find the directory there. I've also put in a __init__.py in the directory, too.
But when I do a import directoryname, I get a module not found error.
Where can I possibly be going wrong?
Ok, I found the problem. It is giving a no such directory error when I check the $PYTHONPATH.
This is my PythonPath
export PYTHONPATH=/usr/lib/python2.7/dist-packages:/home/python/softwares/orade

I assume that you are trying to import the orade module. PYTHONPATH represents the path to the directory that contains python modules, not the collection of paths to the modules.
So you should put the parent directory of your module in the PYTHONPATH. If the path to your module is /home/python/softwares/orade, you should put /home/python/softwares in your PYTHONPATH:
export PYTHONPATH=/usr/lib/python2.7/dist-packages:/home/python/softwares
And then you should be able to do:
>>> import orade
If the orade directory contains a __init__.py file.