My folder structure is as follows
./fff
├── __init__.py
├── fg
│ ├── __init__.py
│ └── settings
│ ├── __init__.py
│ └── settings.py
└── obng
└── test.py
I want to import the settings.py inside fg/settings as a module into the test.py
I have added the line
from ..fg.settings import settings
But when I run it, it gives me the following error
Traceback (most recent call last):
File "/mnt/d/Repos/fff/obng/test.py", line 1, in
from ..fg.settings import settings
ImportError: attempted relative import with no known parent package
This style of relative importing is supported as per https://docs.python.org/3/reference/import.html#package-relative-imports
What am I doing wrong here?
It is a matter of how you run your project - you should run from the parent directory of the top-level package as in
$ cd ../fff
$ python -m fff.obng.test # note no py
Then relative imports will be resolved correctly. It is an antipattern running a script directly from its folder
Normally you can't use relative imports when you run your python module as main module like python filename.py but there is a hack using __package__ to achieve this. Remember __package__ is how python resolves relative imports:
1- Create a file called __init__.py in your root directory - fff. ( I can see that you have it, I mentioned for completeness)
2- Put this code on top of your test.py module:
if __name__ == '__main__' and not __package__:
import sys
sys.path.insert(0, <path to parent directory of root directory - fff>)
__package__ = 'fff.obng'
Note: sys.path is where python searches for modules to import them.
3- Now place your relative import statement after the code above (inside the if statement, because we don't wanna mess when your test.py is being imported) :
from ..fg.settings import settings
Now you can call you test.py, it will run without problem. I don't recommend using these hacks but showing the flexibility of the language and doing exactly what you wanna do in some cases is beneficial.
Other good solutions: Absolute import I think is easier and cleaner than this. In addition take a look at #Mr_and_Mrs_D's answer another good solution would be to run your module with -m command-line flag.
Relative imports are based on the name of the current module. When running
python fff/obng/test.py
the name of test.py will be __main__ and the import will not work.
What will work is having another script called "test.py" outside the fff module that imports the fff.obng.test
fff_top
├── fff
│ ├── fg
│ │ ├── __init__.py
│ │ └── settings
│ │ ├── __init__.py
│ │ └── settings.py
│ ├── __init__.py
│ └── obng
│ ├── __init__.py
│ └── test.py
└── test.py
with fff_top/test.py:
import fff.obng.test
Then, running the "external" test.py should be ok:
python fft_top/test.py
Alternatively, I would recommend dropping relative imports entirely. One way to do this is using a virtual environment for every package you write, using for example the venv library:
python -m venv venv
Then, add a setup.py in the root folder with the content:
from setuptools import setup, find_packages
setup(name="fff", packages=find_packages())
and change the imports in obng/test.py:
from fff.fg.settings import settings
Finally, activate your virtual environment:
source venv/bin/activate
and install your package in editable mode:
pip install -e .
Then, after you have completed all the steps above:
python fff/obng/test.py
should work.
In Linux, you could create a symbolic link:
$ ln -s ../folder1 mymodules
$ python
>>> import mymodules.myfancymodule as fancy
Related
I am not finding the way to properly code so that both pylint and the execution of the code (within VSCode or from the command line) would work.
There are some similar questions but none seems to apply to my project structure with a src directory under which there will be multiple packages. Here's the simplified project structure:
.
├── README.md
├── src
│ ├── rssita
│ │ ├── __init__.py
│ │ ├── feeds.py
│ │ ├── rssita.py
│ │ └── termcolors.py
│ └── zanotherpackage
│ ├── __init__.py
│ └── anothermodule.py
└── tests
├── __init__.py
└── test_feeds.py
From what I understand rssita is one of my my packages (because of the init.py file) with some modules under it amongst which rssita.py file contains the following imports:
from feeds import RSS_FEEDS
from termcolors import PC
The rssita.py code as shown above runs well from both within VSCode and from command line python ( python src/rssita/rssita.py ) from the project root, but at the same time pylint (both from within VSCode and from the command line (pylint src or pylint src/rssita)) flags the two imports as not found.
If I modify the code as follows:
from rssita.feeds import RSS_FEEDS
from rssita.termcolors import PC
pylint will then be happy but the code will not run anymore since it would not find the imports.
What's the cleanest fix for this?
As far as I'm concerned pylinty is right, your setup / PYTHONPATH is screwed up: in Python 3, all imports are absolute by default, so
from feeds import RSS_FEEDS
from termcolors import PC
should look for top-level packages called feeds and termcolors which I don't think exist.
python src/rssita/rssita.py
That really ain't the correct invocation, it's going to setup a really weird PYTHONPATH in order to run a random script.
The correct imports should be package-relative:
from .feeds import RSS_FEEDS
from .termcolors import PC
Furthermore if you intend to run a package, that should be either a runnable package using __main__:
python -m rssita
or you should run the sub-package as a module:
python -m rssita.rssita
Because you're using an src-package, you'll either need to create a pyproject.toml so you can use an editable install, or you'll have to PYTHONPATH=src before you run the command. This ensures the packages are visible at the top-level of the PYTHONPATH, and thus correctly importable. Though I'm not a specialist in the interaction of src layouts & runnable packages, so there may be better solutions.
I have a project structure as follows:
python-test » tree
.
├── asdf
│ ├── __init__.py
│ ├── mycode2.py
│ ├── mycode.py
│ └── scripts
│ └── __init__.py
├── __init__.py
└── scripts
├── __init__.py
└── mymod.py
asdf/mycode.py contains:
import scripts.mymod
and scripts/mymod.py contains:
print('hello world')
All the __init__.pys are empty.
I would like to import scripts.mymod.
I am running asdf/mycode.py which fails because it is looking inside asdf/scripts instead of the root scripts folder for mymod.
> PYTHONPATH=: python asdf/mycode.py
Traceback (most recent call last):
File "asdf/mycode.py", line 1, in <module>
import scripts.mymod
ModuleNotFoundError: No module named 'scripts.mymod'
As a workaround, I can manually modify the path, which I have done in asdf/mycode2.py.
import sys
sys.path = [p for p in sys.path if 'asdf' not in p]
import scripts.mymod
This works correctly:
> PYTHONPATH=: python asdf/mycode2.py
hello world
Is there a way to import scripts.mymod in asdf/mycode.py without modifying the sys.path manually, and without changing the PYTHONPATH which has to be set to :?
TL;DR
Python imports are weird and cumbersome and you are probably best off either creating an entrypoint in the top of the directory or using a tool to handle the path manipulations.
Python Imports
Python has two different ways of handling imports, the first is the standard: import numpy as np, which is resolved against the environment PYTHONPATH in order until it finds the requested module. Included in that path is the directory of the file that is currently being executed, which you see in your example in that you are needing to manually remove that from the sys.path.
The other way that python handles imports is via relative paths, which always lead with . like from . import a or import .a or from .. import b. Unfortunately, relative import only work if the file that they are in is not being directly run (This happens if you have two files within a module where one of them imports an object from another, but both of them are intended to be imported by and external script). This is because python uses the builtin name global to resolve the relative path, which if the file is being directly run from a shell gets overridden as "__main__".
Consider a file structure:
.
├── a.py
└── b.py
where a.py is
import b
and b.py is
print(__name__)
If you run:
python3 b.py # prints "__main__"
python3 a.py # prints "b"
So, if you want to be able to import the scripts/mymod.py from asdf/mycode2.py, you could put change the import to:
from .. import scripts.mymod
But, if you do that, you will not be able to run the asdf/mycode2.py file directly, you will need to create a third file somewhere else that imports asdf/mycode2.py and run that third script (these files are called entrypoints). For example:
python-test » tree
.
├── asdf
│ ├── __init__.py
│ ├── mycode2.py
│ ├── mycode.py
│ └── scripts
│ └── __init__.py
├── __init__.py
├── entrypoint.py
└── scripts
├── __init__.py
└── mymod.py
where entrypoint.py is
import asdf.mycode2
An Alternative Approach
The alternative approach would be to develop a tool which handles manipulating python's sys.path to allow relative imports even when the current file is being run a "__main__". Full disclosure, I am currently working on such a tool based off nodejs's require function, called repyrer, which is pip-installable as
pip3 install repyrer
It allows you to do this kind of thing:
from repyrer import repyre
mymod = repyre('../scripts/mymod')
which will work even if you run the file directly.
Hope that helps!
Before I even start I want to say - I know, that there are like bazillion questions similar to this, but I couldn't find the answer to my problem. I have a dir structure like this:
.
├── project
│ ├── A
│ │ ├── __init__.py
│ │ └── somelib.py
│ ├── B
│ ├── C
│ │ └── C
│ │ ├── foo.py
│ │ └── __init__.py
│ └── __init__.py
└── run.sh
run.sh:
python3 project/C/C/foo.py
foo.py:
from project.A.somelib import somefunc
VS Code actually gets the intellisense in foo.py - it tells me what funcitons/variables I can import from somelib. But when I run run.sh, I get this error message:
from project.A.somelib import somefunc
ModuleNotFoundError: No module named 'project'
Is there a way to solve this while preserving this directory structure?
adding project/__init__.py changed nothing
the sys.path in foo.py looks like this:
['/home/dabljues/projects/project/project/C/C', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/lib/python3.7/site-packages']
restrictions:
I can't modify neither sys.path in the files nor PYTHONPATH before running the scripts
I can't pip-install anything
I don't have sudo-access
I can't create a virtualenv, since the scripts are supposed to be downloadable and quickly executable
IDEs like VSCode or Pycharm make their own assumptions about a project, and will usually correctly link modules even if the interpreter that will ultimately run the code can't.
The reason why project.A.somelib can't be found is visible in your sys.path output, which gives you the places where python will search for modules. Since '/home/dabljues/projects/project/project' is not included, there is no way for python to resolve it during runtime.
A quick hack
You can just add the path manually to sys.path, either in the source file by running import sys; sys.insert(0, '/home/dabljues/projects/project/project/') in foo.py before any other imports happen, or by running export PYTHONPATH="${PYTHONPATH}:/home/dabljues/projects/project/project/" in your shell before run.sh.
Installing the project
Since it looks like you're developing a library, you might as well use the mechanisms python offers to make libraries shareable and thereby fixing any import issues. Add a minimal setup.py to the project root (i.e. /home/dabljues/projects/project/project/setup.py):
from setuptools import setup, find_packages
setup(
name='project',
version='0.1.0',
packages=find_packages('project'),
)
And install your project in editable mode:
$ python3 -m pip install -e .
This will put a link in your python3 executable's site-packages that points to the project root, which makes it accessible whenever you run anything with python3.
Tests
I included print(__name__) at the top of all python files to get some output.
running run.sh without installing the package:
$ sh run.sh
Traceback (most recent call last):
File "project/C/C/foo.py", line 1, in <module>
from project.A.somelib import somefunc
ModuleNotFoundError: No module named 'project'
after installing it
$ sh run.sh
__main__
project.A.somelib
As you can see, project.C.C.foo is executed as a script, yet it finds all imports that start with project because project is installed.
Run python in package mode helps.
1) Add __init__.py for every path:
.
├── project
│ ├── A
│ │ ├── __init__.py
│ │ └── somelib.py
│ ├── B
│ ├── C
│ │ ├── __init__.py
│ │ └── C
│ │ ├── foo.py
│ │ └── __init__.py
│ └── __init__.py
└── run.sh
2) Import module with relative path in foo.py:
from ...A.somelib import somefunc
3) Run python in package mode:
python -m project.C.C.foo
It works for me.
I'm unable to reproduce this given your code (assuming run.sh just starts a script).
Are you sure it's not a case of e.g. circular imports?
$ mkdir -p project/A project/C/C
$ cat > project/C/C/foo.py
print('moof!')
$ cat > project/A/somelib.py
print('beef!')
$ cat > script.py
import project.A.somelib
import project.C.C.foo
$ tree
.
├── project
│ ├── A
│ │ └── somelib.py
│ └── C
│ └── C
│ └── foo.py
└── script.py
$ python3 script.py
beef!
moof!
Change your run.sh script to run foo as a module. python3 -m proj.C.C.foo
I've got a python package written locally with a structure like
package
├── __init__.py
├── __main__.py
├── tests
│ ├── __init__.py
│ └── package_tests.py
└── package
├── __init__.py
├── package.py
This works great when run using python -m package in a Python3 virtualenv from the root of the project (parent dir of the first package dir in that tree)
However when run in a Python2.7 virtualenv, I get an ImportError in the __main__.py script as it tries to import the functions from package.py
__main__.py:
import sys
from package.package.package import foo, bar
def main(args):
f = foo(args)
bar(f)
if __name__ == "__main__":
main(sys.argv[1:])
The error:
ImportError: No module named package
What do I need to change to make it compatible with both?
(Obviously the package isn't actually called package)
Despite looking for an explanation for so long, immediately after posting this question I found a solution.
After looking over the changes to imports between python 2 and 3 I found that I only needed to use relative imports.
So the import line in my __main__.py became
from .package.package import foo, bar
I run my Python script with
python3.7 -m opencryptobot.START -lvl 20
opencryptobot is a folder here and START is a module. At some point I'd like to restart the script with this code:
os.execl(sys.executable, sys.executable, *sys.argv)
I use this code because it worked perfectly for my other scripts that I don't execute with the -m argument. So the above code will be executed at some point and then I get this error:
ModuleNotFoundError: No module named 'opencryptobot'
Which sounds correct since opencryptobot is just a folder and not a module. I played around with various versions of os.exec to see if I’m able to restart it but nothing really works.
So my question is, how can I restart my script if I run it the way I do?
This is a minimal version of my folder structure:
.
├── LICENSE
├── Pipfile
├── Pipfile.lock
├── README.md
├── opencryptobot
│ ├── START.py
│ ├── config.py
│ ├── constants.py
│ ├── database.py
│ ├── emoji.py
│ ├── plugin.py
│ ├── plugins
│ │ ├── about.py
│ │ ├── admin.py
│ │ └── alltimehigh.py
│ ├── ratelimit.py
│ ├── telegrambot.py
│ └── utils.py
└── start.sh
I execute START from the root folder.
sys.argv does not start with -m opencryptobot.START. The module name is removed from the sys.argv list altogether while resolving the opencryptobot.START module filename (sys.argv is set to ['-m', '-lvl', '20'] until there is a filename) and then '-m' is replaced with the full filename of the module.
From the -m switch documentation:
If this option is given, the first element of sys.argv will be the full path to the module file (while the module file is being located, the first element will be set to "-m").
In effect, Python is simulating running a script, as if you ran python /path/to/opencrytobot/START.py ...; only the __package__ context is kept so you can still use package-relative imports such as from . import ....
So to recreate your original command-line, use '-m', __spec__.name, *sys.argv[1:] as the argument list beyond sys.executable:
os.execl(sys.executable, sys.executable, '-m', __spec__.name, *sys.argv[1:])
You can't use __name__, unfortunately, as that has been set to '__main__' when using -m. However, the __spec__ object set on your module does know what the full qualified name is for the current module, so we can re-use that here.
Note that even though there is no __init__.py file in opencryptobot, by using -m you are telling Python that opencryptobot is really an implicit namespace package. If that wasn't the intention here, then don't use -m to load your script.
Demo:
$ ls -1 opencryptobot/
START.py
$ cat opencryptobot/START.py
import sys
import os
if __name__ == '__main__':
print('Running as the __main__ script')
print('sys.argv:', sys.argv)
print('__spec__.name:', __spec__.name)
if 'restarted' not in sys.argv:
print('Respawning:')
# add an extra command-line option to stop respawning a second time
os.execl(sys.executable, sys.executable, '-m', __spec__.name, *sys.argv[1:], 'restarted')
$ python3.7 -m opencryptobot.START -lvl 20
Running as the __main__ script
sys.argv: ['/.../opencryptobot/START.py', '-lvl', '20']
__spec__.name: opencryptobot.START
Respawning:
Running as the __main__ script
sys.argv: ['/.../opencryptobot/START.py', '-lvl', '20', 'restarted']
__spec__.name: opencryptobot.START