Before I even start I want to say - I know, that there are like bazillion questions similar to this, but I couldn't find the answer to my problem. I have a dir structure like this:
.
├── project
│ ├── A
│ │ ├── __init__.py
│ │ └── somelib.py
│ ├── B
│ ├── C
│ │ └── C
│ │ ├── foo.py
│ │ └── __init__.py
│ └── __init__.py
└── run.sh
run.sh:
python3 project/C/C/foo.py
foo.py:
from project.A.somelib import somefunc
VS Code actually gets the intellisense in foo.py - it tells me what funcitons/variables I can import from somelib. But when I run run.sh, I get this error message:
from project.A.somelib import somefunc
ModuleNotFoundError: No module named 'project'
Is there a way to solve this while preserving this directory structure?
adding project/__init__.py changed nothing
the sys.path in foo.py looks like this:
['/home/dabljues/projects/project/project/C/C', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/lib/python3.7/site-packages']
restrictions:
I can't modify neither sys.path in the files nor PYTHONPATH before running the scripts
I can't pip-install anything
I don't have sudo-access
I can't create a virtualenv, since the scripts are supposed to be downloadable and quickly executable
IDEs like VSCode or Pycharm make their own assumptions about a project, and will usually correctly link modules even if the interpreter that will ultimately run the code can't.
The reason why project.A.somelib can't be found is visible in your sys.path output, which gives you the places where python will search for modules. Since '/home/dabljues/projects/project/project' is not included, there is no way for python to resolve it during runtime.
A quick hack
You can just add the path manually to sys.path, either in the source file by running import sys; sys.insert(0, '/home/dabljues/projects/project/project/') in foo.py before any other imports happen, or by running export PYTHONPATH="${PYTHONPATH}:/home/dabljues/projects/project/project/" in your shell before run.sh.
Installing the project
Since it looks like you're developing a library, you might as well use the mechanisms python offers to make libraries shareable and thereby fixing any import issues. Add a minimal setup.py to the project root (i.e. /home/dabljues/projects/project/project/setup.py):
from setuptools import setup, find_packages
setup(
name='project',
version='0.1.0',
packages=find_packages('project'),
)
And install your project in editable mode:
$ python3 -m pip install -e .
This will put a link in your python3 executable's site-packages that points to the project root, which makes it accessible whenever you run anything with python3.
Tests
I included print(__name__) at the top of all python files to get some output.
running run.sh without installing the package:
$ sh run.sh
Traceback (most recent call last):
File "project/C/C/foo.py", line 1, in <module>
from project.A.somelib import somefunc
ModuleNotFoundError: No module named 'project'
after installing it
$ sh run.sh
__main__
project.A.somelib
As you can see, project.C.C.foo is executed as a script, yet it finds all imports that start with project because project is installed.
Run python in package mode helps.
1) Add __init__.py for every path:
.
├── project
│ ├── A
│ │ ├── __init__.py
│ │ └── somelib.py
│ ├── B
│ ├── C
│ │ ├── __init__.py
│ │ └── C
│ │ ├── foo.py
│ │ └── __init__.py
│ └── __init__.py
└── run.sh
2) Import module with relative path in foo.py:
from ...A.somelib import somefunc
3) Run python in package mode:
python -m project.C.C.foo
It works for me.
I'm unable to reproduce this given your code (assuming run.sh just starts a script).
Are you sure it's not a case of e.g. circular imports?
$ mkdir -p project/A project/C/C
$ cat > project/C/C/foo.py
print('moof!')
$ cat > project/A/somelib.py
print('beef!')
$ cat > script.py
import project.A.somelib
import project.C.C.foo
$ tree
.
├── project
│ ├── A
│ │ └── somelib.py
│ └── C
│ └── C
│ └── foo.py
└── script.py
$ python3 script.py
beef!
moof!
Change your run.sh script to run foo as a module. python3 -m proj.C.C.foo
Related
I am trying to better understand importing modules. I read about how to do this from here https://stackoverflow.com/a/14132912/14179793 and I can get it to work using solution 1. There is an additional variable that I need to figure out though.
This is a dummy project I am testing with:
.
├── a_package
│ ├── __init__.py
│ └── lib_a.py
├── b_package
│ ├── __init__.py
│ └── test_script.py
├── main.py
└── src
└── src_lib
└── src_lib.py
With this setup I can do:
python -m b_package.test_script
this is lib a function
This is src_lib_function.
test_script.py:
from a_package.lib_a import lib_a_function
from src.src_lib.src_lib import src_lib_function
if __name__ == '__main__':
lib_a_function()
src_lib_function()
pass
The goal is to make b_package/test_script.py executable without using python test_script ie ./test_script
However, adding the shebang at the top #!/usr/bin/env python causes an import error:
$ ./b_package/test_script.py
Traceback (most recent call last):
File "./b_package/test_script.py", line 2, in <module>
from a_package.lib_a import lib_a_function
ModuleNotFoundError: No module named 'a_package'
I assume it is because python is not loading it as a module based off the above mentioned question but I am not sure how to resolve this.
I ended up using setuptools as suggested by kosciej16 to achieve the desired results.
New project structure:
.
├── a_package
│ ├── __init__.py
│ └── lib_a.py
├── b_package
│ ├── __init__.py
│ └── test_script.py
├── main.py
├── pyproject.toml
├── setup.cfg
└── src
├── __init__.py
└── src_lib
├── __init__.py
└── src_lib.py
setup.cfg:
[metadata]
name = cli_test
version = 0.0.1
[options]
packages = find:
[options.entry_points]
console_scripts =
test_script = b_package.test_script:main
This allows the user to clone the repo and run pip install . from the top level then they can execute the script by just typing test_script
Based on this solution, I would like to know how to do the opposite.
For example how to import a function from scriptA.py into test.py that is in the directory mylib ?
Also, is it possible to put a __init__.py file in the root of the project directory?
.
├── mylib
│ ├── fun1.py
│ ├── fun2.py
│ ├── __init__.py
│ └── test.py
├── scriptA.py
├── scriptB.py
└── script_test.py
There are two aspects to this. How you import and which module you actually execute.
For the import, put this into test.py:
from ..scriptA import function_name
Then, assuming you're executing test.py, run it from the top level directory using python -m mylib.test
I think that ought to do it.
EDIT: tho now I'm doubting myself slightly on the -m execution. You might have to run it from one level further up like this python -m base_dir_name.mylib.test
I have the following structure:-
Test
├── package1
│ └── script1.py
│ └── __init__.py
├── package2
└── script2.py
└── __init__.py
I want to import script1.py from script2.py and have tried:-
from package1.script1 import *
I am running the script python3 script2.py from a linux terminal but this does not seem to work and I get ModuleNotFoundError: No module named 'package1'. How do I fix this?
Ideally you should run a python script from the top level package (which imports and/or runs code that starts the whole application) like the following:
Test
├── package1
│ └── script1.py
│ └── __init__.py
├── package2
| └── script2.py
| └── __init__.py
|___ run.py
Then run python3 run.py
When you import from package1.script1 import *, you are saying to Python to search the packages from the path the main script was run. Thus it assumes you are searching package1 from package2. If you place package1 inside package2 and run python3 script2.py, it will also work.
Test
├── package2
├── package1
│ └── script1.py
│ └── __init__.py
└── script2.py
└── __init__.py
If you still want to run from script2 with the same directory structure then you have to either be in the Test directory and run python3 package2/script2.py or modify the python path.
Which directory are you in ?
In order to make python to recognize package1,
1 ) You have go to package1's directory via terminal,
cd {package1directory}
or
2 ) You have to append system path to package1's directory as shown below
import sys
sys.path.append('/home/user/package1dir')
3 ) Define PYTHONPATH as system variable. Indicate package1directory
I am trying to run from the directory folder:
$ python subdirectoryTwo/file.py command (Python 2.7).
Folders structure:
-directory
-subdirectoryOne
__init.py__
config.py
-subdirectoryTwo
__init.py__
file.py
My file.py has:
from subdirectoryOne.config import config
However I am getting an error:
file.pyImportError: No module named subdirectoryOne.config`
(I guess it still looks in the directory folder)
There few thins you need to change.
(public)landpacks-MacBook-Pro:qx frank$ tree
.
├── __init__.py
├── __init__.pyc
├── a
│ ├── __init__.py
│ ├── __init__.pyc
│ ├── config.py
│ └── config.pyc
└── b
├── __init__.py
└── test.py
Create a __init__.py with with your subdirectoryOne and subdirectoryTwo like here I used a and b. and then add few codes at the begin of your file.py. And I name it as test.py here. the code is:
import sys
sys.path.append("..")
from project.a.config import myconf
print(myconf)
You can see I import it by project.a.config instead of a.config. because you run your code under the project.
UPDATE
My a/config.py just simple with:
(public)landpacks-MacBook-Pro:qx frank$ cat a/config.py
myconf='127.0.0.1'
One of the solutions (not the optimum one) is to set PYTHONPATH to your directory:
$ export PYTHONPATH='/absolute/path/to/directory'
I've a project which uses git submodules. In my python file I want to use functions from another python file in the submodule project.
In order to work I had to add the init.py file to all subfolders in the path. My folder tree is the following:
myproj
├── gitmodules
│ ├── __init__.py
│ ├── __init__.pyc
│ └── mygitsubmodule
│ ├── __init__.py
│ ├── __init__.pyc
│ └── file.py
└── myfile.py
Is there any way to make it work without touching mygitsubmodule ?
Thanks
you can add to sys.path in the file you want to be able to access the module, something like:
import sys
sys.path.append("/home/me/myproj/gitmodules")
import mygitsubmodule
This example is adding a path as a raw string to make it clear what's happening. You should really use the more sophisticated, system independent methods described below to determine and assemble the path.
Also, I have found it better, when I used this method, to use sys.path.insert(1, .. as some functionality seems to rely of sys.path[0] being the starting directory of the program.
I am used to avoiding modifying sys.path.
The problem is, when using git submodule, submodule is a project directory, not a Python package. There is a "gap" between your module and that package, so you can't import.
Suppose you have created a submodule named foo_project, and there is a foo package inside.
.
├── foo_project
│ ├── README.rst
│ └── foo
│ └── __init__.py
└── main.py
My solution will be creating a soft link to expose that package to your module:
ln -s foo_project/foo foo
.
├── foo_project
│ ├── README.rst
│ └── foo
│ └── __init__.py
├── foo -> foo_project/foo
└── main.py
Now you can import foo in the main.py.
For reference,
from submodulefolder.file import func_name
or
import submodulefolder.file as lib_name
where file excludes the extension of file.py, seems to work in relative terms without modifying the subfolder / git submodule with a init.py since python 3.3+,
as shown here.
Tested on py3.8.5 linux native and py3.7.8 anaconda Windows, both in Spyder's Ipython-console, as well as natively on linux via terminal.