Develop Reference

Importing modules from different folders

I am following the pytest "Get Started" guide, and I just can't make it work. It seems to be something very elementary, but I just cant find it. The problem resides in importing modules from other folders, and, although I am following the official documentation, I cant make it work
Following PyPa and the official documentation, I have assembled the following file structure:
(lpthw) ex47_pypa> tree
.
├── dist
│ ├── example_package_vcmota-0.0.1-py3-none-any.whl
│ └── example_package_vcmota-0.0.1.tar.gz
├── LICENSE
├── pyproject.toml
├── README.md
├── src
│ └── example_package_VCMota
│ ├── example.py
│ └── __init__.py
└── tests
└── test_example.py
4 directories, 8 files
(lpthw) ex47_pypa> cat src/example_package_VCMota/example.py
def add_one(number):
return number+1
(lpthw) ex47_pypa> cat tests/test_example.py
# import example
# import example_package_VCMota
# from src/example_package_VCMota import example
# from src import example_package_VCMota/example
# import src
# import importlib
# from src.example_package_VCMota.example import add_one
from ..src.example import add_one
def test_dif():
assert add_one(2) == 9
(lpthw) ex47_pypa>
That seems to me as an ok implementation of everything in pytest pages, except that it does not work:
(lpthw) ex47_pypa> pytest
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.10.8, pytest-7.1.3, pluggy-1.0.0
rootdir: /archive/MyBooks/LearnPython3TheHardWay/mypython/projects/ex47_pypa
collected 0 items / 1 error
====================================================================================================== ERRORS ======================================================================================================
______________________________________________________________________________________ ERROR collecting tests/test_example.py ______________________________________________________________________________________
ImportError while importing test module '/archive/MyBooks/LearnPython3TheHardWay/mypython/projects/ex47_pypa/tests/test_example.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.10/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_example.py:8: in <module>
from example_package_VCMota.example import add_one
E ModuleNotFoundError: No module named 'example_package_VCMota'
============================================================================================= short test summary info ==============================================================================================
ERROR tests/test_example.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================================================================= 1 error in 0.05s =================================================================================================
(lpthw) ex47_pypa>
The error is due to how I call the function add_one in example.p, and, as you may see in the test_example.py, I have tried every imaginable way of calling this function.
The only thing I know for sure is that the error has nothing to do with pytest: I have assembled another file structure:
teste> tree
.
├── src
│ ├── example.py
│ └── __init__.py
└── tests
└── test_example.py
2 directories, 3 files
teste> cd tests/
tests> cat test_example.py
# import example
# import example_package_VCMota
# from src/example_package_VCMota import example
# from src import example_package_VCMota/example
# import src
# import importlib
# from src.example_package_VCMota.example import add_one
from ..src.example import add_one
def test_dif():
assert add_one(2) == 9
tests>
but it still does not work:
tests>python test_example.py
Traceback (most recent call last):
File "/archive/MyBooks/LearnPython3TheHardWay/mypython/projects/teste/tests/test_example.py", line 10, in <module>
from ..src.example import add_one
ImportError: attempted relative import with no known parent package
tests>
I am aware that there are multiple other ways of importing modules from other folders, as stated in this 11 years old post from stackoverflow itself, but the dot one is surely the simplest way, and, since it is clearly stated in the official documentation, it should work, but I cant make it work.
Thank you all for your attention.
EDIT:
Following Raphael's comments, I have started to suspect of errors/problems in my python installs, and therefore assembled the following file structure inside a virtual machine running Ubuntu LTS 20:
├── source
│ ├── example.py
│ └── init.py
└── tests
└── test_example.py
where I have tried multiple import commands in test_example.py, such as:
from .source.example import add_one
from ...source.example import add_one
and so on, and, again, none have worked. Since I am running Gentoo in my main machine, I suspect that this test indicates that whatever is the issue, it is certainly not with my python installs.

It's a common issue in Python that the tests for a package cannot find the package itself.
The main reason for the issues is your working directory. You cannot just cd into the tests folder and run the tests. In fact, you need to be one level above your project folder, so you need to be in ex47_pypa/.. and then run python -m ex47_pypa.tests.test_example.
This will give Python enough directory hierarchy to actually resolve the relative imports in your tests, so the from ..src.example can be resolved.
I've created an experimental, new import library: ultraimport
It gives you more control over your imports and lets you do file system based imports.
In your test_example.py your could then write:
import ultraimport
add_one = ultraimport('__dir__/../src/example_package_VCMota/example.py', 'add_one')
This will always work, no matter how you run your code or what's your current working directory or what's in your sys.path.

import another subpackage from a subpackage doesn't show docstirng in Pycharm

To make it simpler to understand, here is a minimal reproducible example.
Structure:
C:.
├───.idea
│ │ PyCharm stuff
└───src
│ main.py
│
├───folder
│ │ script.py
│ │ __init__.py
│
└───folder2
│ script2.py
│ __init__.py
I launch PyCharm in src, that's something I want.
My main.py code is the following:
from folder.script import my_function_script
if __name__ == '__main__':
my_function_script()
This works very well. Autocomplete works and docstrings is available when hovering over the function names.
This is the module script.py from the package folder:
from folder2.script2 import my_function_script_2
def my_function_script() -> None:
"""
This is a docstring
:return: None
"""
my_function_script_2()
Here is the problem. Because the IDE thinks I want to import the package folder2 which is in folder nothing works (autocomplete, docstring, etc ...). But that's not the case since I know I only call script.py from main.py (which is not in the package folder). Hence why, I need to write from folder2.script2 - otherwise if I call script.py from main.py, python won't find folder2.script2.
My question is then:
How do make it so that PyCharm understands that when I import stuff in the package folder, I do it by "being in main.py", therefore displaying the docstrings, checking the type of the variables if type hinting has been done, etc ...

After days of research, I found the answer.
Solution 1 :
Do from src.folder2.script2 instead of from folder2.script2 because I'm opening the project from the parent directory of src.
Solution 2 :
Set the source directory of the project as src as it currently is the parent directory of src.

How do I import a Python package from the top level (from FastAPI docs)

I am working through the "Bigger Applications - Multiple Files" section of the FastAPI docs. This describes a package structure with a top-level directory called app, a main.py and __init__.py in that directory, and then two subdirectories named routers and internal each with their own __init__.py and source files.
In main.py (again, from the top level, this is app/main.py), they describe adding the line: from app.routers import items
However, this does not work for me, even in the simplest example:
├── app
│ ├── __init__.py
│ ├── main.py
│ └── foo
│ │ ├── __init__.py
│ │ ├── bar.py
Both __init__.py files are empty. main.py is one line:
from app.foo import bar
foo/bar.py is:
def baz():
print("hi")
That's it. As simple as it gets, while still following the model described in the FastAPI docs. However it fails:
Traceback (most recent call last):
File "/path/to/my/src/app/main.py", line 1, in <module>
from app.foo import bar
Similarly, they describe using relative imports like from .internal import admin. This too does not work.
What gives? I can find similar patterns in many projects and documents, but I can't figure out what I am missing in making this work. Yes, I could put main.py a level up and everything else in a subdirectory. But that's adding clutter, and based on the docs it seems like this is something that should work (I realize this is not a FastAPI-specific problem, it's just that their docs have a clear example). Similarly, I have found answers that talk about changing PYTHONPATH, but I have seen no evidence that other projects using this pattern do that. There must be something I am missing.
I am using Python 3.9.5, but I've tried it with 3.6.x also.

Differance between $ pytest and $ python -m pytest tests [duplicate]

This question already has answers here:
PATH issue with pytest 'ImportError: No module named YadaYadaYada'
(25 answers)
Closed 2 years ago.
I would like to understand what is the differance between running:
$ pytest
and
$ python -m pytest tests
I am worikng in a virtual enviroment on a Mac.
The first solution gives me an error :
ModuleNotFoundError: No module named 'app'
The second solution works fine.
Why does the second solution work and the the first one not?
How can I make it work with just $ pytest?
My folder structure:
├── app.py
├── requirements.txt
├── templates
├── tests
│ ├── conftest.py
│ ├── functional
│ │ ├── __init__.py
│ │ └── test_s.py
│ └── unit
│ ├── __init__.py
│ └── test_app.py
└── venv
In test_app.py, this line fails:
from app import app

python -m pytest adds the current directory to sys.path, i.e. lets you import modules from there.
This is a hint that there is something - let's call it - not optimal with your directory structure.
Why this may not be a good thing, you can read in this fantastic blog post
https://blog.ganssle.io/articles/2019/08/test-as-installed.html
I also wrote a short blog post about this topic:
https://jugmac00.github.io/til/what-is-the-difference-between-invoking-pytest-and-python-m-pytest/

The difference is comming from the Python algorithm of searching modules, but also the way how Pytest discovers tests.
1. Python module searching
It is done according to Python implementation as following:
search in builtin module
search in sys.path list:
first 'current dir' is added to sys.path
second PYTHONPATH content is is added to sys.path
then in the end all Python installation directories are added to sys.path
It can be quite complicated. I see this nicely explained "in extended mode" here:
Where is Python's sys.path initialized from?
2. Pytest discovery mechanism
So
from app import app
when you are importing 'app' like above it will be searched in diffirent way in both cases. In the other words 'current dir' in sys.path is set diffirently for both cases. You can check this by printing sys.path list.
It is not issue, but the way how python/pytest works.
3. Example of workaround
You can workaround this by for example by something like below (I am putting this to separate pathmagic.py file and importing later pathmagic in test_xyz )
import pathmagic # noqa Pep8 uncheck
pathmagic.py
import sys
import os
sys.path.append(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))
sys.path.append(os.path.dirname((os.path.dirname(os.path.dirname(os.path.realpath(__file__))))))

The following module won't import

I've got a PyCharm "project" in Python, which is to say that I have a folder that is a conglomeration of all sorts of experimental python files, convenience methods/classes, and Jupyter notebooks from following along with online classes.
I've actually just written something that I'm proud of and would like to re-use. I'm finding it difficult to import. I have looked at and attempted implementing the answers to the following questions to no avail:
Can't import my own modules in Python
How to import my own modules in python 3.6?
How to import my own modules the elegant way?
Project structure:
learning_project
|
├───.idea
│ ├───dictionaries
│ └───inspectionProfiles
|
├───decision_trees
├───linear_algebra
├───neural_networks
| ├───based_sequential.py <---------------------------- # Module location #
│ ├───cross-entropy-gradient-descent
│ └───learning pytorch
| ├─── class_notebook.ipynb <---------------------- # Want to import for use here #
| └───Cat_Dog_data
|
└───venv
├───Include
├───Lib
│ └───site-packages
└───Scripts
I have tried the following:
import based_sequential
from based_sequential import ClassName
import based_sequential.ClassName
import neural_networks
from neural_networks import based_sequential
import neural_networks.based_sequential
from neural_networks.based_sequential import ClassName
All result in the error No module named '<pick your poison>'
Question 1: Obviously, what am I missing?
Question 2: Is my organization here part of the problem? I'm starting to suspect that it is.
I also suspect I have some work to do learning the administrative aspects of writing code that is bigger than just one .py file.

I hope you're returning some value in the function/module you're trying to import. If not, check that.
Otherwise, just use sys.path from sys module and direct it towards the file you want to import.
>>> import sys
>>> sys.path
['',
'C:\\Python33\\Lib\\idlelib',
'C:\\Windows\\system32\\python33.zip',
'C:\\Python33\\DLLs',
'C:\\Python33\\lib',
'C:\\Python33',
'C:\\Python33\\lib\\site-packages']

I suggest you check on relative imports.
https://realpython.com/absolute-vs-relative-python-imports/
The way you can solve this problem is shown below:
FILE STRUCTURE I'LL BE USING:
└── project
├── package1
│ ├── module1.py
│ └── module2.py
└── package2
├── __init__.py
├── module3.py
├── module4.py
└── subpackage1
└── module5.py
To import from module 2 while in module 1 use:
from .module2 import function1
In the case of importing the module3 from module2 use:
from ..package2 import module3
IN YOUR OWN CASE, IT SHOULD BE:
from ..based_sequential import whatever