Develop Reference

Python 3.8.x Intra and Extra Imports

When defining multiple python 3 packages in a project, I cannot determine how to configure the environment (using VS Code) such that I can run a file in any package with intra-package imports without breaking extra-package imports.
Example with absolute paths:
src/
foo/
a.py
b.py
goo/
c.py
# b.py contents
from foo.a import *
# c.py contents
from foo.b import *
where PYTHONPATH="${workspaceFolder}\src", so it should be able to see the foo and goo directories.
I can run c.py, but running b.py gives a ModuleNotFoundError: "No module named 'foo'".
Modifying b.py to use relative paths:
# modified b.py contents
from a import *
Then allows me to run b.py, but attempting to then run c.py gives a ModuleNotFoundError: "No module named 'b'".
I have also added __init__.py files to foo/ and goo/ directories, but the errors still occurs.
I did set the cwd in the launch.json file to be the ${workspaceFolder} directory, if that is relevant.

A solution I was trying to avoid, but does work in-case there are no better solutions, is putting each individual package-dependency onto the PYTHONPATH environment variable.
E.g., "${workspaceFolder}\src;${workspaceFolder}\foo" when running c.py.
But this is not an easily scalable solution and is also something that needs to be tracked across project changes, so I do not think it is a particularly good or pythonic solution.

In fact, you can find it didn't work if you try to print the current path. It just add ".." to the path instead of the parent directory.
import sys
print(sys.path)
sys.path.append('..')
print(sys.path)
So, what we have to do is add the path of module we used to "sys" in path.
Here we have three ways:
Put the written .py file into the directory that has been added to the system environment variable;
Create a new .pth file under \Python\Python310\Lib\site-packages.
Write the path of the module in this new .pth file , one path per line.
Using the pythonpath environment variable as upstairs said.
Then we can use this two ways to import :
sys. path. append ('a').
sys. path. insert (0, 'a')

how to properly import a module which need another module to run？

I met an import error called no module named XX.
my project file is organized as follows:
-------A.py
|
B-----__init__.py wrote: from .C import C
|
C.py
|​
D -----__init__.py wrote: from .E import E
|
E.py
In A.py, I need to import class C from C.py. but class C needs to use class E(in E.py)to run
In A.py, I wrote import B.C
In C.py, I wrote import D.E
when I run the test in A.py, it gives the error: No module named 'D'
But if I test C.py, there is no problem at all.
Can anyone tell me why and how to fix it?

When you try to run A.py, it fails because the Python interpreter (the program that runs your Python script) cannot find the D package.
When you run C.py, however, the interpreter does find the D package. This is because the script you are running (C.py) is located in the same directory/folder as the D package.
Detailed explanation found in the Python Docs:
When a module named spam is imported, the interpreter first searches for a built-in module with that name. If not found, it then searches for a file named spam.py in a list of directories given by the variable sys.path. sys.path is initialized from these locations:
The directory containing the input script (or the current directory when no file is specified).
PYTHONPATH (a list of directory names, with the same syntax as the shell variable PATH).
The installation-dependent default (by convention including a site-packages directory, handled by the site module).
A quick fix would be to use a relative import in C.py:
from .D import E

Why importing in the imported modules works differently?

If I have
a.py
b.py
In b.py I can import a
But if I have
c.py
m
a.py
b.py
and in c.py do import m.b, suddenly in b.py I get ModuleNotFoundError: No module named 'a'
What's the problem? I don't see how the second case is any different from the first
So... the modules are searched in the directory of the module that was started initially. I just don't understand the reasoning.
I'm not asking how to fix the problem. But rather asking why there's a problem in the first place...
(tested on Python 3.8.8)

It's based what file you run python.
If you run with python c.py
file c.py use:
from m import a, b
file b.py also use:
from m import a
If you go python b.py you just need import a to use in b.py
so depend what file you run it's will be the parent for relative import.

Let's start with why your first one works.
Assuming you have a file structure such that:
c.py
m - |
|
__init__.py
a.py
b.py
Let's say your current directory is within the m folder and you run the command.
python b.py
If within your b.py you place something like:
import a
import sys
print(sys.path)
it will work just fine. But let's take a look at the output from print(sys.path).
On windows, this is will look something like ['C:\\path\\to\\myprogram\\m', '<path to python installation>', etc.]
The important thing is that the m folder is on your sys.path and that is how import a gets resolved.
However if we go up one level and run:
python c.py
you should immediately notice that m is no longer on your sys.path and instead is replaced by 'C:\\path\\to\\myprogram'.
That is why it fails. Python automatically includes your current working directory in sys.path and changing out of it means it no longer knows where to look for the import.
This is an example of an absolute import. You can manipulate where python looks for these imports by modifying sys.path to include the path of the file you want to import.
sys.path.append('C:\\path\\to\\myprogram\\m')
But there's a better way to do it. If m is a package or sub-package (includes an __init__.py) you can use a relative import.
from . import a
However, there is a small caveat to this.
You can only use relative imports when the file using them is being run as a module or package.
So running them directly as a top level script
python b.py
will produce
ImportError: attempted relative import with no known parent package
Python luckily has the ability to run a script as a module built in.
if you cd one level up so as to encapsulate your m package, you can run
python -m m.b
and have it run just fine.
Additionally, since b.py is being treated as a module in c.py because of the import, the relative import will work in this case as well.
python c.py

Resolving module conflict in python [duplicate]

Okay, the scenario is very simple. I have this file structure:
.
├── interface.py
├── pkg
│   ├── __init__.py
│   ├── mod1.py
│   ├── mod2.py
Now, these are my conditions:
mod2 needs to import mod1.
both interface.py and mod2 needs to be run independently as a main script. If you want, think interface as the actual program and mod2 as an internal tester of the package.
So, in Python 2 I would simply do import mod1 inside mod2.py and both python2 mod2.py and python2 interface.py would work as expected.
However, and this is the part I less understand, using Python 3.5.2, if I do import mod1; then I can do python3 mod2.py, but python3 interface.py throws: ImportError: No module named 'mod1' :(
So, apparently, python 3 proposes to use import pkg.mod1 to avoid collisions against built-in modules. Ok, If I use that I can do python3 interface.py; but then I can't python3 mod2.py because: ImportError: No module named 'pkg'
Similarly, If I use relative import:
from . import mod1 then python3 interface.py works; but mod2.py says SystemError: Parent module '' not loaded, cannot perform relative import :( :(
The only "solution", I've found is to go up one folder and do python -m pkg.mod2 and then it works. But do we have to be adding the package prefix pkg to every import to other modules within that package? Even more, to run any scripts inside the package, do I have to remember to go one folder up and use the -m switch? That's the only way to go??
I'm confused. This scenario was pretty straightforward with python 2, but looks awkward in python 3.
UPDATE: I have upload those files with the (referred as "solution" above) working source code here: https://gitlab.com/Akronix/test_python3_packages. Note that I still don't like it, and looks much uglier than the python2 solution.
Related SO questions I've already read:
Python -- import the package in a module that is inside the same package
How to do relative imports in Python?
Absolute import module in same package
Related links:
https://docs.python.org/3.5/tutorial/modules.html
https://www.python.org/dev/peps/pep-0328/
https://www.python.org/dev/peps/pep-0366/

TLDR:
Run your code with python -m pkg.mod2.
Import your code with from . import mod1.
The only "solution", I've found is to go up one folder and do python -m pkg.mod2 and then it works.
Using the -m switch is indeed the "only" solution - it was already the only solution before. The old behaviour simply only ever worked out of sheer luck; it could be broken without even modifying your code.
Going "one folder up" merely adds your package to the search path. Installing your package or modifying the search path works as well. See below for details.
But do we have to be adding the package prefix pkg to every import to other modules within that package?
You must have a reference to your package - otherwise it is ambiguous which module you want. The package reference can be either absolute or relative.
A relative import is usually what you want. It saves writing pkg explicitly, making it easier to refactor and move modules.
# inside mod1.py
# import mod2 - this is wrong! It can pull in an arbitrary mod2 module
# these are correct, they uniquely identify the module
import pkg.mod2
from pkg import mod2
from . import mod2
from .mod2 import foo # if pkg.mod2.foo exists
Note that you can always use <import> as <name> to bind your import to a different name. For example, import pkg.mod2 as mod2 lets you work with just the module name.
Even more, to run any scripts inside the package, do I have to remember to go one folder up and use the -m switch? That's the only way to go??
If your package is properly installed, you can use the -m switch from anywhere. For example, you can always use python3 -m json.tool.
echo '{"json":"obj"}' | python -m json.tool
If your package is not installed (yet), you can set PYTHONPATH to its base directory. This includes your package in the search path, and allows the -m switch to find it properly.
If you are in the executable's directory, you can execute export PYTHONPATH="$(pwd)/.." to quickly mount the package for import.
I'm confused. This scenario was pretty straightforward with python 2, but looks awkward in python 3.
This scenario was basically broken in python 2. While it was straightforward in many cases, it was difficult or outright impossible to fix in any other cases.
The new behaviour is more awkward in the straightforward case, but robust and reliable in any case.

I had similar problem.
I solved it adding
import sys
sys.path.insert(0,".package_name")
into the __init__.py file in the package folder.

Shouldn't the imports be absolute by default in python27?

Imagine the directory structure:
/
a/
__init__.py
b.py
c.py
c.py
File /a/b.py looks like:
import c
should_be_absolute = c
All the other files (including __init__) are empty.
When running a test script (using python 2.7):
import a.b
print a.b.should_be_absolute
with PYTHONPATH=/ from an empty directory (so nothing is added to PYTHONPATH from current directory) I get
<module 'a.c' from '/a/c.py'>
where according to PEP 328 and the statement import <> is always absolute I would expect:
<module 'c' from '/c.py'>
The output is as expected when I remove the /a/c.py file.
What am I missing? And if this is the correct behavior - how to import the c module from b (instead of a.c)?
Update:
According to python dev mailing list it appears to be a bug in the documentation. The imports are not absolute by default in python27.

you need to add from __future__ import absolute_import or use importlib.import_module('c') on Python 2.7
It is default on Python 3.
There was a bug in Python: __future__.py and its documentation claim absolute imports became mandatory in 2.7, but they didn't.

If you are only adding / to your PYTHONPATH, then the search order could still be looking for c in the current directory. It would be a lot better if you placed everything under a root package, and referred to it absolutely:
/myPackage
a/
__init__.py
b.py
c.py
__init__.py
c.py
And a PYTHONPATH like: export PYTHONPATH=/:$PYTHONPATH
So in your a.c you would do and of these:
from myPackage import c
from myPackage.c import Foo
import myPackage.c
This way, it is always relative to your package.

"Absolute" doesn't mean the one that you think it does; instead, it means that the "usual" package resolving procedure takes place: first, it looks in the directory of the package, then in all the elements of sys.path; which includes the elements from PYTHONPATH.
If you really want to, you can use tools like the imp module, but I'd recommend against it for something like this. Because in general, you shouldn't ever have to create a module with the same name as one in the standard Python distribution.