Develop Reference

sys.path.append which main.py will be imported

If I have a project with two files main.py and main2.py. In main2.py I do import sys; sys.path.append(path), and I do import main, which main.py will be imported? The one in my project or the one in path (the question poses itself only if there are two main.py obviously)?
Can I force Python to import a specific one?

The one that is in your project.
According to python docs, the import order is:
Built-in python modules. You can see the list in the variable sys.modules
The sys.path entries
The installation-dependent default locations
Your added directory to sys.path, so it depends on the order of entries in sys.path.
You can check the order of imports by running this code:
import sys
print(sys.path)
As you will observe, the first entry of sys.path is your module's directory ("project directory") and the last is the one you appended.
You can force python to use the outer directory by adding it in the beginning of your sys.path:
sys.path.insert(0,path)
Although I would recommend against having same names for the modules as it is often hard to manage.
Another thing to keep in mind is that you can't import relative paths to your sys.path. Check this answer for more information on how to work around that: https://stackoverflow.com/a/35259170/6599912

When two paths contain the same module name, using sys.path.append(path) isn't going to change the priority of the imports, since append puts path at the end of the python path list
I'd use
sys.path.insert(0,path)
so modules from path come first
Example: my module foo.py imports main and there's one main.py in the same directory and one main.py in the sub directory
foo.py
main.py
sub/main.py
in foo.py if I do:
import sys
sys.path.append("sub")
import main
print(main.__file__)
I get the main.py file path of the current directory because sys.path always starts by the main script directory.
So I could remove that directory from sys.path but it's a bad idea because I could need other modules from there.
Now if I use insert instead:
import sys
sys.path.insert(0,"sub")
import main
print(main.__file__)
I get sub/main.py as sub is the first directory python looks into when searching modules.

Python 3.8.x Intra and Extra Imports

When defining multiple python 3 packages in a project, I cannot determine how to configure the environment (using VS Code) such that I can run a file in any package with intra-package imports without breaking extra-package imports.
Example with absolute paths:
src/
foo/
a.py
b.py
goo/
c.py
# b.py contents
from foo.a import *
# c.py contents
from foo.b import *
where PYTHONPATH="${workspaceFolder}\src", so it should be able to see the foo and goo directories.
I can run c.py, but running b.py gives a ModuleNotFoundError: "No module named 'foo'".
Modifying b.py to use relative paths:
# modified b.py contents
from a import *
Then allows me to run b.py, but attempting to then run c.py gives a ModuleNotFoundError: "No module named 'b'".
I have also added __init__.py files to foo/ and goo/ directories, but the errors still occurs.
I did set the cwd in the launch.json file to be the ${workspaceFolder} directory, if that is relevant.

A solution I was trying to avoid, but does work in-case there are no better solutions, is putting each individual package-dependency onto the PYTHONPATH environment variable.
E.g., "${workspaceFolder}\src;${workspaceFolder}\foo" when running c.py.
But this is not an easily scalable solution and is also something that needs to be tracked across project changes, so I do not think it is a particularly good or pythonic solution.

In fact, you can find it didn't work if you try to print the current path. It just add ".." to the path instead of the parent directory.
import sys
print(sys.path)
sys.path.append('..')
print(sys.path)
So, what we have to do is add the path of module we used to "sys" in path.
Here we have three ways:
Put the written .py file into the directory that has been added to the system environment variable;
Create a new .pth file under \Python\Python310\Lib\site-packages.
Write the path of the module in this new .pth file , one path per line.
Using the pythonpath environment variable as upstairs said.
Then we can use this two ways to import :
sys. path. append ('a').
sys. path. insert (0, 'a')

Basic Python import mechanics

I have the following directory tree:
project/
A/
__init__.py
foo.py
TestA/
__init__.py
testFoo.py
the content of testFoo is:
import unittest
from A import foo
from the project directory I run python testA/testFoo.py
I get a ModuleNotFoundError No module named A
I have two question: how to improt and run A.foo from TestA.testFoo and why is it so difficult to grasp the import logic in Python? Isn't there any debug trick to solve this kind of issues rapidly, I'm sorry I have to bother you with such basics questions?

When your are executing a file an environment variable called python path is generated, python import work with this variable to find your file to import, this path is generated with the path of the file you are executing and it will search in the current directory and sub directories containing an __init__.py file, if you want to import from a directory on the same level you need to modify your python path or change the architecture of your project so the file executed is always on top level.
you can include path to your python path like this :
import sys
sys.path.insert(0, "/path/to/file.py")
You can read more on import system : https://docs.python.org/3/reference/import.html
The best way in my opinion is to not touch the python path and include your test directoy into the directory where tested files are:
project/
A/
__init__.py
foo.py
TestA/
__init__.py
testFoo.py
Then run the python -m unittest command into your A or project directory, it will search into your current and sub directories for test and execute it.
More on unittest here : https://docs.python.org/3/library/unittest.html

Add the folder project/testA to the system pythonpath first:
import sys
sys.path.insert(0, "/path/to/pythonfile")
and try the import again.

Can you try this ?
Create an empty file __init__.py in subdirectory TestA. And add at the begin of main code
from __future__ import absolute_import
Then import as below :
import A.foo as testfoo

The recommended way in py3 may be like below
echo $pwd
$ /home/user/project
python -m testA.testFoo
The way of execute module python -m in python is a good way to replace relative references。

You definitely cannot find A because python need look from sys.path, PYTHONPATH to find the module.
And python will automatically add current top level script to sys.path not currently directory to sys.path. So if you add print(sys.path) in testFoo.py, you will see it only add project/TestA to the sys.path.
Another word, the project did not be included in sys.path, then how python can find the module A?
So you had to add the project folder to sys.path by yourself, and, this just needed in top script, something like follows:
import unittest
import sys
import os
file_path = os.path.abspath(os.path.dirname(__file__)).replace('\\', '/')
lib_path = os.path.abspath(os.path.join(file_path, '..')).replace('\\', '/')
sys.path.append(lib_path)

Do I need to add my project directory to the system path in every script to import a function from another directory?

I'm trying to keep a data science project well-organized so I've created a directory inside my src directory called utils that contains a file called helpers.py, which contains some helper functions that will be used in many scripts. What is the best practice for how I should import func_name from src/utils/helpers.py into a file in a totally different directory, such as src/processing/clean_data.py?
I see answers to this question, and I've implemented a solution that works, but this feels ugly:
sys.path.append(os.path.dirname(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))))
Am I doing this right? Do I need to add this to every script that wants to import func_name, like train_model.py?
My current project folder structure:
myproject
/notebooks
notebook.ipynb
/src
/processing
clean_data.py
/utils
helpers.py
/models
train_model.py
__init__.py
Example files:
# clean_data.py
import os
import sys
sys.path.append(os.path.dirname(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))))
from src.utils.helpers import func_name
func_name()
# helpers.py
def func_name():
print('I'm a helper function.')

The correct way to do it is to use __init__.py, setup.py and the setuptools Python package:
myPackage/
myPackage/
__init__.py
setup.py
This link has all the steps.

First of all, let me describe you the differences between a Python module & a Python package so that both of us are on the same page. ✌
A module is a single .py file (or files) that are imported under one import and used. ✔
import aModuleName
# Here 'aModuleName' is just a regular .py file.
Whereas, a package is a collection of modules in directories that give a package hierarchy. A package contains a distinct __init__.py file. ✔
from aPackageName import aModuleName
# Here 'aPackageName` is a folder with a `__init__.py` file
# and 'aModuleName', which is just a regular .py file.
Therefore, when we have a project directory named proj-dir of the following structure ⤵
proj-dir
--|--__init__.py
--package1
--|--__init__.py
--|--module1.py
--package2
--|--__init__.py
--|--module2.py
🔎 Notice that I've also added an empty __init__.py into the proj-dir itself which makes it a package too.
👍 Now, if you want to import any python object from module2 of package2 into module1 of package1, then the import statement in the file module1.py would be
from package2.module2 import object2
# if you were to import the entire module2 then,
from package2 import module2
I hope this simple explanation clarifies your doubts on Python imports' mechanism and solves the problem. If not then do comment here. 😊

First of all let me clarify you that importing an entire module, if you are going to use a part of it, then is not a good idea. Instead of that you can use from to import specific function under a library/package. By doing this, you make your program efficient in terms of memory and performance.
To know more refer these:
'import module' or 'from module import'
difference between import and from
Net let us look into the solution.
Before starting off with the solution, let me clarify you the use of __init__.py file. It just tells the python interpreter that the *.py files present there are importable which means they are modules and are/maybe a part of a package.
So, If you have N no of sub directories you have to put __init__.py file in all those sub directories such that they can also be imported. Inside __init__.py file you can also add some additional information like which path should be included, default functions,variables,scope,..etc. To know about these just google about __init__.py file or take some python library and go through the same __init__.py file to know about it. (Here lies the solution)
More Info:
modules
Be pythonic
So as stated by #Sushant Chaudhary your project structure should be like
proj-dir
--|--__init__.py
--package1
--|--__init__.py
--|--module1.py
--package2
--|--__init__.py
--|--module2.py
So now, If I put __init__.py file under my directory like above, Will
it be importable and work fine?
yes and no.
Yes :
If you are importing the modules within that project/package directory.
for example in your case
you are importing package1.module1 in pakage2.module2 as from package1 import module1.
Here you have to import the base dir inside the sub modules, Why? the project will run fine if you are running the module from the same place. i.e: inside package2 as python module2.py, But will throw ModuleNotFoundError If you run the module from some other directory. i.e: any other path except under package2 for example under proj-dir as python package2/module2.py. This is what happening in your case. You are running the module from project-dir.
So How to fix this?
1- You have to append basedir path to system path in module2.py as
from sys import path
dir_path = "/absolute/path/to/proj-dir"
sys.path.insert(0, dir_path)
So that module2 will be able to find package1 (and module1 inside it).
2- You have to add all the sub module paths in __init__.py file under proj-dir.
For example:
#__init__.py under lxml
# this is a package
def get_include():
"""
Returns a list of header include paths (for lxml itself, libxml2
and libxslt) needed to compile C code against lxml if it was built
with statically linked libraries.
"""
import os
lxml_path = __path__[0]
include_path = os.path.join(lxml_path, 'includes')
includes = [include_path, lxml_path]
for name in os.listdir(include_path):
path = os.path.join(include_path, name)
if os.path.isdir(path):
includes.append(path)
return includes
This is the __init__.py file of lxml (a python library for parsing html,xml data). You can refer any __init__.py file under any python libraries having sub modules.ex (os,sys). Here I've mentioned lxml because I thought it will be easy for you to understand. You can even check __init__.py file under other libraries/packages. Each will have it's own way of defining the path for submodules.
No
If you are trying to import modules outside the directory. Then you have to export the module path such that other modules can find them into environment variables. This can be done directly by appending absolute path of the base dir to PYTHONPATH or to PATH.
To know more:
PATH variables in OS
PYTHONPATH variable
So to solve your problem, include the paths to all the sub modules in __init__.py file under proj-dir and add the /absolute/path/to/proj-dir either to PYTHONPATH or PATH.
Hope the answer explains you about usage of __init__.py and solves your problem.

On Linux, you can just add the path to the parent folder of your src directory to ~/.local/lib/python3.6/site-packages/my_modules.pth. See
Using .pth files. You can then import modules in src from anywhere on your system.
NB1: Replace python3.6 by any version of Python you want to use.
NB2: If you use Python2.7 (don't know for other versions), you will need to create __init__.py (empty) files in src/ and src/utils.
NB3: Any name.pth file is ok for my_modules.pth.

Yes, you can only import code from installed packages or from files in you working directory or subdirectories.

the way I see it, your problem would be solved if you would have your module or package installed, like an yother package one installs and then imports (numpy, xml, json etc.)
I also have a package I constantly use in all my projects, ulitilies, and I know it's a pain with the importing.
here is a description on how to How to package a python application to make it pip-installable:
https://marthall.github.io/blog/how-to-package-a-python-app/

Navigate to your python installation folder
Navigate to lib
Navigate to site-packages
Make a new file called any_thing_you_want.pth
Type .../src/utils/helpers.py inside that file with your favorite text editor
Note: the ellipsis before scr/utils/helpers.py will look something like: C:/Users/blahblahblah/python_folders/scr... <- YOU DO NEED THIS!
This is a cheap way out but it keeps code clean, and is the least complicated. The downside is, for every folder your modules are in, example.pth will need them. Upside: works with Windows all the way up to Windows 10

Importing correctly with pytest

I just got set up to use pytest with Python 2.6. It has worked well so far with the exception of handling "import" statements: I can't seem to get pytest to respond to imports in the same way that my program does.
My directory structure is as follows:
src/
main.py
util.py
test/
test_util.py
geom/
vector.py
region.py
test/
test_vector.py
test_region.py
To run, I call python main.py from src/.
In main.py, I import both vector and region with
from geom.region import Region
from geom.vector import Vector
In vector.py, I import region with
from geom.region import Region
These all work fine when I run the code in a standard run. However, when I call "py.test" from src/, it consistently exits with import errors.
Some Problems and My Solution Attempts
My first problem was that, when running "test/test_foo.py", py.test could not "import foo.py" directly. I solved this by using the "imp" tool. In "test_util.py":
import imp
util = imp.load_source("util", "util.py")
This works great for many files. It also seems to imply that when pytest is running "path/test/test_foo.py" to test "path/foo.py", it is based in the directory "path".
However, this fails for "test_vector.py". Pytest can find and import the vector module, but it cannot locate any of vector's imports. The following imports (from "vector.py") both fail when using pytest:
from geom.region import *
from region import *
These both give errors of the form
ImportError: No module named [geom.region / region]
I don't know what to do next to solve this problem; my understanding of imports in Python is limited.
What is the proper way to handle imports when using pytest?
Edit: Extremely Hacky Solution
In vector.py, I changed the import statement from
from geom.region import Region
to simply
from region import Region
This makes the import relative to the directory of "vector.py".
Next, in "test/test_vector.py", I add the directory of "vector.py" to the path as follows:
import sys, os
sys.path.append(os.path.realpath(os.path.dirname(__file__)+"/.."))
This enables Python to find "../region.py" from "geom/test/test_vector.py".
This works, but it seems extremely problematic because I am adding a ton of new directories to the path. What I'm looking for is either
1) An import strategy that is compatible with pytest, or
2) An option in pytest that makes it compatible with my import strategy
So I am leaving this question open for answers of these kinds.

The issue here is that Pytest walks the filesystem to discover files that contain tests, but then needs to generate a module name that will cause import to load that file. (Remember, files are not modules.)
Pytest comes up with this test package name by finding the first directory at or above the level of the file that does not include an __init__.py file and declaring that the "basedir" for the module tree containing a module generated from this file. It then adds the basedir to sys.path and imports using the module name that will find that file relative to the basedir.
There are some implications of this of which you should beware:
The basepath may not match your intended basepath in which case the module will have a name that doesn't match what you would normally use. E.g., what you think of as geom.test.test_vector will actually be named just test_vector during the Pytest run because it found no __init__.py in src/geom/test/ and so added that directory to sys.path.
You may run into module naming collisions if two files in different directories have the same name. For example, lacking __init__.py files anywhere, adding geom/test/test_util.py will conflict with test/test_util.py because both are loaded as import test_util.py, with both test/ and geom/test/ in the path.
The system you're using here, without explicit __init__.py modules, is having Python create implicit namespace packages for your directories. (A package is a module with submodules.) Ideally we'd configure Pytest with a path from which it would also generate this, but it doesn't seem to know how to do that.
The easiest solution here is simply to add empty __init__.py files to all of the subdirectories under src/; this will cause Pytest to import everything using package/module names that start with directory names under src/.
The question How do I Pytest a project using PEP 420 namespace packages? discusses other solutions to this.

import looks in the following directories to find a module:
The home directory of the program. This is the directory of your root script. When you are running pytest your home directory is where it is installed (/usr/local/bin probably). No matter that you are running it from your src directory because the location of your pytest determines your home directory. That is the reason why it doesn't find the modules.
PYTHONPATH. This is an environment variable. You can set it from the command line of your operating system. In Linux/Unix systems you can do this by executing: 'export PYTHONPATH=/your/custom/path' If you wanted Python to find your modules from the test directory you should include the src path in this variable.
The standard libraries directory. This is the directory where all your libraries are installed.
There is a less common option using a pth file.
sys.path is the result of combining the home directory, PYTHONPATH and the standard libraries directory. What you are doing, modifying sys.path is correct. It is something I do regularly. You could try using PYTHONPATH if you don't like messing with sys.path

If you include an __init__.py file inside your tests directory, then when the program is looking to set a home directory it will walk 'upwards' until it finds one that does not contain an init file. In this case src/.
From here you can import by saying :
from geom.region import *
you must also make sure that you have an init file in any other subdirectories, such as the other nested test directory

I was wondering what to do about this problem too. After reading this post, and playing around a bit, I figured out an elegant solution. I created a file called "test_setup.py" and put the following code in it:
import sys, os
sys.path.append(os.path.dirname(os.path.abspath(__file__)))
I put this file in the top-level directory (such as src). When pytest is run from the top-level directory, it will run all test files including this one since the file is prefixed with "test". There are no tests in the file, but it is still run since it begins with "test".
The code will append the current directory name of the test_setup.py file to the system path within the test environment. This will be done only once, so there are not a bunch of things added to the path.
Then, from within any test function, you can import modules relative to that top-level folder (such as import geom.region) and it knows where to find it since the src directory was added to the path.
If you want to run a single test file (such as test_util.py) instead of all the files, you would use:
pytest test_setup.py test\test_util.py
This runs both the test_setup and test_util code so that the test_setup code can still be used.

Are so late to answer that question but usining python 3.9 or 3.10 u just need to add __init__.py folder in tests folders.
When u add this file python interprets this folders as a module.
Wold be like this
src/
main.py
util.py
test/
__init__.py
test_util.py
geom/
vector.py
region.py
test/
__init__.py
test_vector.py
test_region.py
so u just run pytest.
Sorry my poor english

Not the best solution, but maybe the fastest one:
cd path/python_folder
python -m pytest python_file.py