First of all, there are a bunch of solutions on stackoverflow regarding this but from the ones I tried none of them is working. I am working on a remote machine (linux). I am prototyping within the dir-2/module_2.py file using an ipython interpreter. Also I am trying to avoid using absolute paths as the absolute path in this remote machine is long and ugly, and I want my code to run on other machines upon download.
My directory structure is as follows:
/project-dir/
-/dir-1/
-/__ init__.py
-/module_1.py
-/dir-2/
-/__ init__.py
-/module_2.py
-/module_3.py
Now I want to import module_1 from module_2. However the solution mentioned in this stackoverflow post: link of using
sys.path.append('../..')
import module_2
Does not work. I get the error: ModuleNotFoundError: No module named 'module_1'
Moreover, within the ipython interpreter things like import .module_3 within module_2 throws error:
import .module_3
^ SyntaxError: invalid syntax
Isn't the dot operator supposed to work within the same directory as well. Overall I am quite confused by the importing mechanism. Any help with the initial problem is greatly appreciated! Thanks a lot!
Why it didn't work?
If you run the module1.py file and you want to import module2 then you need something like
sys.path.append("../dir-2")
If you use sys.path.append("../..") then the folder you added to the path is the folder containing project-dirand there is notmodule2.py` file inside it.
The syntax import .module_3 is for relative imports. if you tried to execute module2.py and it contains import .module_3 it does not work because you are using module2.py as a script. To use relative imports you need to treat both module2.py and module_3.py as modules. That is, some other file imports module2 and module2 import something from module3 using this syntax.
Suggestion on how you can proceed
One possible solution that solves both problems is property organizing the project and (optionally, ut a good idea) packaging your library (that is, make your code "installable"). Then, once your library is installed (in the virtual environment you are working) you don't need hacky sys.path solutions. You will be able to import your library from any folder.
Furthermore, don't treat your modules as scripts (don't run your modules). Use a separate python file as your "executable" (or entry point) and import everything you need from there. With this, relative imports in your module*.py files will work correctly and you don't get confused.
A possible directory structure could be
/project-dir/
- apps/
- main.py
- yourlib/
-/__ init__.py
-/dir-1/
-/__ init__.py
-/module_1.py
-/dir-2/
-/__ init__.py
-/module_2.py
-/module_3.py
Notice that the the yourlib folder as well as subfolders contain an __init__.py file. With this structure, you only run main.py (the name does not need to be main.py).
Case 1: You don't want to package your library
If you don't want to package your library, then you can add sys.path.append("../") in main.py to add "the project-dir/ folder to the path. With that your yourlib library will be "importable" in main.py. You can do something like from yourlib import module_2 and it will work correctly (and module_2 can use relative imports). Alternatively, you can also directly put main.py in the project-dir/ folder and you don't need to change sys.path at all, since project-dir/ will be the "working directory" in that case.
Note that you can also have a tests folder inside project-dir and to run a test file you can do the same as you did to run main.py.
Case 2: You want to package your library
The previous solution already solves your problems, but going the extra mile adds some benefits, such as dependency management and no need to change sys.path no matter where you are. There are several options to package your library and I will show one option using poetry due to its simplicity.
After installing poetry, you can run the command below in a terminal to create a new project
poetry new mylib
This creates the following folder structure
mylib/
- README.rst
- mylib/
- __init__.py
- pyproject.toml
- tests
You can then add the apps folder if you want, as well as subfolders inside mylib/ (each with a __init__.py file).
The pyproject.toml file specifies the dependencies and project metadata. You can edit it by hand and/or use poetry to add new dependencies, such as
poetry add pandas
poetry add --dev mypy
to add pandas as a dependency and mypy as a development dependency, for instance. After that, you can run
poetry build
to create a virtual environment and install your library in it. You can activate the virtual environment with poetry shell and you will be able to import your library from anywhere. Note that you can change your library files without the need to run poetry build again.
At last, if you want to publish your library in PyPi for everyone to see you can use
poetry publish --username your_pypi_username --password _passowrd_
TL; DR
Use an organized project structure with a clear place for the scripts you execute. Particularly, it is better if the script you execute is outside the folder with your modules. Also, don't run a module as a script (otherwise you can't use relative imports).
I am trying to understand how python packages/modules work in virtualenv and how I can structure it for distribution .. I have getting this error: ModuleNotFoundError: No module named lib. I have a very simple project
Project/lib
- __init__.py
- calc.py
Project/tests
- __init__.py
- test_calc.py
I need to be able to access calc.py from the tests and it gives me this error when I import. I understand I should add the Project dir into the sys.path (I have also read the python docs). If I were to distribute this project what do I need to do to make sure the paths are added and the packages are ready to be consumed ?
do I need a Makefile or setup.py. There is so many things I am seeing and its a bit confusing. What is the best approach for production?
You are correct in saying that you should add the directory to the sys.path variable. Now, all you need to do is, depending on how your code is set up, import the main folder (lib), or import the calc file from lib
I have a suite of tests and it's in the project structured like this:
source/
app/
tests/
What would be the best way to make sure that the tests import everything in the app? Currently I'm putting the following three lines at the top of my tests initialization, but it seems like others have found out a more elegant way to do this but I'm not sure how:
import sys, os
sys.path.append(os.path.abspath(os.path.join(__file__, os.pardir, os.pardir)))
from app.api import *
Maybe setup.py is what you want?
You could package your app as a python module.
Here is the story. Firstly create a setup.py in your project, check here. And then install your project using develop mode.
python setup.py develop
After that, you could use the module without adding the directory into sys.path, as this module is installed under your python path.
from app.api import *
With the develop option, you could debug this module.
I'm having a hard time understanding how module importing works in Python (I've never done it in any other language before either).
Let's say I have:
myapp/__init__.py
myapp/myapp/myapp.py
myapp/myapp/SomeObject.py
myapp/tests/TestCase.py
Now I'm trying to get something like this:
myapp.py
===================
from myapp import SomeObject
# stuff ...
TestCase.py
===================
from myapp import SomeObject
# some tests on SomeObject
However, I'm definitely doing something wrong as Python can't see that myapp is a module:
ImportError: No module named myapp
In your particular case it looks like you're trying to import SomeObject from the myapp.py and TestCase.py scripts. From myapp.py, do
import SomeObject
since it is in the same folder. For TestCase.py, do
from ..myapp import SomeObject
However, this will work only if you are importing TestCase from the package. If you want to directly run python TestCase.py, you would have to mess with your path. This can be done within Python:
import sys
sys.path.append("..")
from myapp import SomeObject
though that is generally not recommended.
In general, if you want other people to use your Python package, you should use distutils to create a setup script. That way, anyone can install your package easily using a command like python setup.py install and it will be available everywhere on their machine. If you're serious about the package, you could even add it to the Python Package Index, PyPI.
The function import looks for files into your PYTHONPATH env. variable and your local directory. So you can either put all your files in the same directory, or export the path typing into a terminal::
export PYTHONPATH="$PYTHONPATH:/path_to_myapp/myapp/myapp/"
exporting path is a good way. Another way is to add a .pth to your site-packages location.
On my mac my python keeps site-packages in /Library/Python shown below
/Library/Python/2.7/site-packages
I created a file called awesome.pth at /Library/Python/2.7/site-packages/awesome.pth and in the file put the following path that references my awesome modules
/opt/awesome/custom_python_modules
You can try
from myapp.myapp import SomeObject
because your project name is the same as the myapp.py which makes it search the project document first
You need to have
__init__.py
in all the folders that have code you need to interact with.
You also need to specify the top folder name of your project in every import even if the file you tried to import is at the same level.
In your first myapp directory ,u can add a setup.py file and add two python code in setup.py
from setuptools import setup
setup(name='myapp')
in your first myapp directory in commandline , use pip install -e . to install the package
pip install on Windows 10 defaults to installing in 'Program Files/PythonXX/Lib/site-packages' which is a directory that requires administrative privileges. So I fixed my issue by running pip install as Administrator (you have to open command prompt as administrator even if you are logged in with an admin account). Also, it is safer to call pip from python.
e.g.
python -m pip install <package-name>
instead of
pip install <package-name>
In my case it was Windows vs Python surprise, despite Windows filenames are not case sensitive, Python import is. So if you have Stuff.py file you need to import this name as-is.
let's say i write a module
import os
my_home_dir=os.environ['HOME'] // in windows 'HOMEPATH'
file_abs_path=os.path.join(my_home_dir,"my_module.py")
with open(file_abs_path,"w") as f:
f.write("print('I am loaded successfully')")
import importlib
importlib.util.find_spec('my_module') ==> cannot find
we have to tell python where to look for the module. we have to add our path to the sys.path
import sys
sys.path.append(file_abs_path)
now importlib.util.find_spec('my_module') returns:
ModuleSpec(name='my_module', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7fa40143e8e0>, origin='/Users/name/my_module.py')
we created our module, we informed python its path, now we should be able to import it
import my_module
//I am loaded successfully
This worked for me:
from .myapp import SomeObject
The . signifies that it will search any local modules from the parent module.
Short Answer:
python -m ParentPackage.Submodule
Executing the required file via module flag worked for me. Lets say we got a typical directory structure as below:
my_project:
| Core
->myScript.py
| Utils
->helpers.py
configs.py
Now if you want to run a file inside a directory, that has imports from other modules, all you need to do is like below:
python -m Core.myscript
PS: You gotta use dot notation to refer the submodules(Files/scripts you want to execute). Also I used python3.9+. So I didnt require neither any init.py nor any sys path append statements.
Hope that helps! Happy Coding!
If you use Anaconda you can do:
conda develop /Path/To/Your/Modules
from the Shell and it will write your path into a conda.pth file into the standard directory for 3rd party modules (site-packages in my case).
If you are using the IPython Console, make sure your IDE (e.g., spyder) is pointing to the right working directory (i.e., your project folder)
Besides the suggested solutions like the accepted answer, I had the same problem in Pycharm, and I didn't want to modify imports like the relative addressing suggested above.
I finally found out that if I mark my src/ (root directory of my python codes) as the source in Interpreter settings, the issue will be resolved.
I have some problems in structuring my python project. Currently it is a bunch of files in the same folder. I have tried to structure it like
proj/
__init__.py
foo.py
...
bar/
__init__.py
foobar.py
...
tests/
foo_test.py
foobar_test.py
...
The problem is that I'm not able, from inner directories, to import modules from the outer directories. This is particularly annoying with tests.
I have read PEP 328 about relative imports and PEP 366 about relative imports from the main module. But both these methods require the base package to be in my PYTHONPATH. Indeed I obtain the following error
ValueError: Attempted relative import in non-package.
So I added the following boilerplate code on top of the test files
import os, sys
sys.path.append(os.path.join(os.getcwd(), os.path.pardir))
Still I get the same error. What is the correct way to
structure a package, complete with tests, and
add the base directory to the path to allow imports?
EDIT As requested in the comment, I add an example import that fails (in the file foo_test.py)
import os, sys
sys.path.append(os.path.join(os.getcwd(), os.path.pardir))
from ..foo import Foo
When you use the -m switch to run code, the current directory is added to sys.path. So the easiest way to run your tests is from the parent directory of proj, using the command:
python -m proj.tests.foo_test
To make that work, you will need to include an __init__.py file in your tests directory so that the tests are correctly recognised as part of the package.
I like to import modules using the full proj.NAME package prefix whenever possible. This is the approach the Google Python styleguide recommends.
One option to allow you to keep your package structure, use full package paths, and still move forward with development would be to use a virtualenv and put your project in develop mode. Your project's setup.py will need to use setuptools instead of distutils, to get the develop command.
This will let you avoid the sys.path.append stuff above:
% virtualenv ~/virt
% . ~/virt/bin/activate
(virt)~% cd ~/myproject
(virt)~/myproject% python setup.py develop
(virt)~/myproject% python tests/foo_test.py
Where foo_test.py uses:
from proj.foo import Foo
Now when you run python from within your virtualenv your PYTHONPATH will point to all of the packages in your project. You can create a shorter shell alias to enter your virtualenv without having to type . ~/virt/bin/activate every time.