Develop Reference

Problems with relative imports and pytest

I am having an issue whereby relative imports from a .py file within a module are not recognised as such by pytest.
Consider the following directory structure:
.
├── import_pytest_issue
│ ├── __init__.py
│ ├── main.py
│ ├── user.py
│ └── utils.py
└── tests
├── __init__.py
└── test_main.py
main.py, user.py, and utils.py are as follows"
# ./import_pytest_issue/main.py
from user import User
if __name__ == "__main__":
user = User("Fred")
print(f"The user is called {user.name}, and their favourite number is {user.favourite_number}")
# ----------------------------------------------------------------------
# ./import_pytest_issue/user.py
from utils import random_number
class User:
favourite_number = random_number()
def __init__(self, name: str) -> None:
self.name = name
# ----------------------------------------------------------------------
# ./import_pytest_issue/utils.py
from random import choice
def random_number():
return choice(range(1, 11))
...and my test_main.py file is like this:
# ./tests/test_main.py
from import_pytest_issue.user import User
def test_user():
user = User("Larry")
assert user.name == "Larry"
assert user.favourite_number in range(1, 11)
main.py runs without issue:
> python import_pytest_issue/main.py
# The user is called Fred, and their favourite number is 1
...but running pytest I get a ModuleNotFoundError:
> pytest
# ======================================================================== test session starts ========================================================================
# platform linux -- Python 3.10.6, pytest-7.2.1, pluggy-1.0.0
# collected 0 items / 1 error
# ============================================================================== ERRORS ===============================================================================
# ________________________________________________________________ ERROR collecting tests/test_main.py ________________________________________________________________
# ImportError while importing test module 'import-pytest-issue/tests/test_main.py'.
# Hint: make sure your test modules/packages have valid Python names.
# Traceback:
# /usr/lib/python3.10/importlib/__init__.py:126: in import_module
# return _bootstrap._gcd_import(name[level:], package, level)
# tests/test_main.py:1: in <module>
# from import_pytest_issue.user import User
# import_pytest_issue/user.py:1: in <module>
# from utils import random_number
# E ModuleNotFoundError: No module named 'utils'
# ====================================================================== short test summary info ======================================================================
# ERROR tests/test_main.py
# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# ========================================================================= 1 error in 0.05s ==========================================================================
FWIW this also happens if I run `python -m pytest`.
Any ideas?

Pytest not able to run test where script A importing another script B in the same folder level as A and giving me ModuleNotFoundError

I am trying to run the unit test using pytest in this project, here main_0.py is importing s3 file.
I am getting ModuleNotFoundError: no module named 's3'
Project Folder Structure
some_project
└───src
├───main
│ └───lambda_function
│ └───some
│ main_0.py
│ s3.py
│
└───test
└───unittest
└───lambda_function
└───some
test_main_0.py
test_s3.py
main_0.py
from s3 import PrintS3
def lambda_handler():
obj = PrintS3()
res = obj.print_txt()
return res
s3.py
class PrintS3:
def __init__(self) -> None:
self.txt = "Hello"
def print_txt(self):
print(self.txt)
return self.txt
test_main_0.py
import unittest
class TestSomeMain(unittest.TestCase):
def test_main_0(self):
from src.main.lambda_function.some.main_0 import lambda_handler
res = lambda_handler()
assert res == "Hello"
test_s3.py is empty.
I also tried adding an empty __init__.py file in both the dir but still the same error
Project Folder Structure after adding __init__.py file
some_project
└───src
├───main
│ └───lambda_function
│ └───some
│ main_0.py
│ s3.py
│ __init__.py
│
└───test
└───unittest
└───lambda_function
└───some
test_main_0.py
test_s3.py
__init__.py
the command I am using to run pytest:
python -m pytest ./src/test
and I am inside some_project folder and also using main_0.py instead of main.py because to not get confused with main folder
Edit 2:
I am to run the test case successfully by adding sys.path in the test_main_0.py file but it is breaking linting and hinting in the code editor (vscode) it didn't broke the linting and hinting, both import statement works but is there any better way.
new test_main_0.py:
import unittest
import os
import sys
sys.path.append(os.path.abspath("./src/main/lambda_function/some/"))
class TestSomeMain(unittest.TestCase):
def test_main_0(self):
from src.main.lambda_function.some.main_0 import lambda_handler # this works
from main_0 import lambda_handler # this also works but break linting and hinting in the code editor
res = lambda_handler()
assert res == "Hello"

could you please try
import os
import sys
sys.path.append(os.path.join(os.path.dirname(__file__), ".."))
from some.s3 import PrintS3
def lambda_handler():
obj = PrintS3()
res = obj.print_txt()
return res

I found a somewhat working solution.
added setUp() and tearDown() methods in the class for inserting and removing path in sys.path
path in sys.path is the location of the directory where the main_0.py and s3.py is located
import unittest
import os
import sys
class TestSomeMain(unittest.TestCase):
def setUp(self) -> None:
sys.path.insert(0, os.path.abspath("./src/main/lambda_function/some/"))
def tearDown(self) -> None:
sys.path.remove(os.path.abspath("./src/main/lambda_function/some/"))
def test_main_0(self):
from src.main.lambda_function.some.main_0 import lambda_handler
res = lambda_handler()
assert res == "Hello"
also update the test command in the terminal:
python -m pytest ./src/test/unittest/lambda_function/some --cov ./src/main/lambda_function/some --cov-report html

Why does the "package" has a "package" in the Doxygen tree?

I generate the documentation of my Python code from the docstrings via Doxygen (1.9.1) in addition with doxypypy (git version from today).
My project is called Project and the packages name in it (which should be imported) is mypackage. When I look into the tree sidebar of the generated html it looks like this:
└── Project
   └── Packages
      └── Packages
└──mypackages
The two packages are linking to the same target: ../html/namespaces.html.
The files and folders are structured like this
Project
├── LICENSE
├── README.md
├── docs
└── ...
└── src
├── mypackage
│   ├── a.py
│   ├── __init__.py
│   └── _mypackage.py
├── setup.cfg
└── setup.py
The Doxyfile is located in Project/docs, doxygen is run in there and use ../src/mypackage as INPUT directory.
More details
__init__.py
__version__ = '0.0.1a'
from ._mypackage import *
_mypackage.py
# -*- coding: utf-8 -*-
"""Example __init__.py short.
Now some longer with multiple lines. Here
comes the scond line.
"""
def foo(bar):
"""
This is foo() in mypackage.
Args:
bar (str): A paramenter.
Returns:
(int): Fixed seven.
"""
print(bar)
return 7
a.py
# -*- coding: utf-8 -*-
"""This is mypackage.a
"""
import mypackage
def bar(bar):
"""
This is the function named bar.
The function calls `mypackage.foo()` and returns an 'a'.
Paramters:
bar (str): Just a parameter.
Returns:
str: Just an 'a'.
"""
mypackage.foo(bar)
return('a')
Some (maybe) related Doxyfile settings
BRIEF_MEMBER_DESC = YES
REPEAT_BRIEF = YES
ALWAYS_DETAILED_SEC = NO
FULL_PATH_NAMES = YES
JAVADOC_AUTOBRIEF = NO
PYTHON_DOCSTRING = YES
OPTIMIZE_OUTPUT_FOR_C = NO
OPTIMIZE_OUTPUT_JAVA = YES
OPTIMIZE_FOR_FORTRAN = NO
OPTIMIZE_OUTPUT_VHDL = NO
OPTIMIZE_OUTPUT_SLICE = NO
MARKDOWN_SUPPORT = YES
EXTRACT_ALL = YES
EXTRACT_PRIVATE = YES
EXTRACT_PACKAGE = YES
EXTRACT_STATIC = YES
EXTRACT_LOCAL_CLASSES = YES
EXTRACT_LOCAL_METHODS = YES
EXTRACT_ANON_NSPACES = NO
RESOLVE_UNNAMED_PARAMS = YES
HIDE_UNDOC_MEMBERS = NO
HIDE_UNDOC_CLASSES = NO
HIDE_FRIEND_COMPOUNDS = NO
HIDE_IN_BODY_DOCS = NO
INPUT = ../src/mypackage
FILE_PATTERNS =
RECURSIVE = YES
FILTER_PATTERNS = *.py=./py_filter
GENERATE_HTML = YES
GENERATE_TREEVIEW = YES
I opened an Issue about that.

How do I pass in a self argument to python cProfile

I am trying to use cProfiling with python.
My python project has the following directory structure:
my-project
├── src
│ ├── lib
│ └── app
│ └── data
│ └── car_sim.py
│
│
│
│
├── ptests
│ ├── src
│ └── lib
│ └── app
│ └── data
│ └── cprofile_test.py
I have a function inside car_sim.py that I want to cprofile and it is called "sim_text". It contains a function called:
#car_sim.py
import os
class RootSimulator:
def sim_text(self, text):
return text
I use the following code inside cprofile_test.py:
#cprofile_test.py
import cProfile
import pstats
import io
import src.lib.app.data.car_sim as car_sim_functions
pr = cProfile.Profile()
pr.enable()
text = 'my blabla sentence' #i can pass in this text below i guess...
#how do i pass to the below????!!
my_result = car_sim_functions.RootSimulator.sim_text()
pr.disable()
s = io.StringIO()
ps = pstats.Stats(pr, stream=s).sort_stats('tottime')
ps.print_stats()
with open('test.txt', 'w+') as f:
f.write(s.getvalue())
Now... when I run it using the command
python -m cProfile ptests/src/lib/app/data/cprofile_test.py
I get the following error:
TypeError: sim_text() missing 2 required positional arguments: 'self' and 'text'
My question is... It expects 2 args, so how do I pass in the "self" arg. For the 2nd arg, "text" I can pass in a value no problem.

class RootSimulator:
def sim_text(self, text):
return text
Defines an instance method on instances of RootSimulator. You are trying to call sim_text from the class itself. You need to create an instance:
simulator = car_sim_functions.RootSimulator()
my_result = simulator.sim_text()
If sim_text() does not actually need to be attached to an instance of the simulator, perhaps you don't need a class at all (just make it a plain function), or you could make it a static method:
class RootSimulator:
#staticmethod
def sim_text(text):
return text
Note that it doesn't need self anymore.

How to get relative path for logging.ini and log file in python over multiple OS

I have created one wrapper logger class which actually wraps the functionality of logging module. I have created it so that every class in my application using logging doesn't need to add logging.config.fileConfig() and logging.getLogger() at the top every-time.
My wrapper class looks like this -
class MyFileLogger():
# The config file for logging formatter.
logging.config.fileConfig('logging.ini', disable_existing_loggers=False)
def __init__(self, name):
self.logger=logging.getLogger(name)
def log_debug(self, message):
self.logger.debug(message)
logging.ini file -
[loggers]
keys=root
[handlers]
keys=fileHandler
[formatters]
keys=myFormatter
[logger_root]
level=NOTSET
handlers=fileHandler
[handler_fileHandler]
class=handlers.TimedRotatingFileHandler
level=DEBUG
formatter=myFormatter
args=('testlog.log','midnight',)
[formatter_myFormatter]
format=%(asctime)s |%(name)-12s |%(levelname)-8s |%(message)s
datefmt=
class=logging.Formatter
MyFileLogger.py and logging.ini file resides on the same path. My intention to create testlog.log is also in same path as logging.ini file.
Implementation of the MyFileLogger.py would be like -
imp.py -
fileLogger = MyFileLogger(__name__)
message = "hey!"
fileLogger.log_debug(message)
Now this imp.py doesn't reside on the same path as MyFileLogger.py and logging.ini. That's why when I run it, this gives me below error cause the python is running at imp.py path.
File "C:\Python34\lib\logging\config.py", line 76, in fileConfig
formatters = _create_formatters(cp)
File "C:\Python34\lib\logging\config.py", line 109, in _create_formatters
flist = cp["formatters"]["keys"]
File "C:\Python34\lib\configparser.py", line 937, in __getitem__
raise KeyError(key)
KeyError: 'formatters'
I know if I just give an absolute path to logging.ini and testlog.log it will work. But I don't want to go there, since this application can be ran in Windows or in Linux. So there would be path problem.
Now even if I don't care about the Windows-Linux thing and only concentrate on the Linux. There is another problem. So if we assume the current directory structure as this ("ProjectDir/src" is added in PYTHONPATH and assuming there is __init__.py in every module folders)-
ProjectDir
├── src
│ └── sales
│ ├── common
│ │ ├── logger
│ │ │ ├── logging.ini
│ │ │ ├── MyFileLogger.py
│ │ │ └── testlog.log
│ │ └── implmnts
│ │ └── imp.py
│ └── unitTest
│ └── test_MyFileLogger.py
└── bar.txt
Let's assume:
imp.py -
fileLogger = MyFileLogger(__name__)
message = "From imp.py"
fileLogger.log_debug(message)
test_MyFileLogger.py -
fileLogger = MyFileLogger(__name__)
message = "from test_MyFileLogger.py"
fileLogger.log_debug(message)
Then if I run imp.py I need to change the logging.config.fileConfig('logging.ini', disable_existing_loggers=False) to logging.config.fileConfig('../logger/logging.ini', disable_existing_loggers=False)
After that it will run but when I try to run test_MyFileLogger.py I need to change again to logging.config.fileConfig('../common/logger/logging.ini', disable_existing_loggers=False)
So my code is dependent upon which file I am running which I want to get rid of. Is there any way I can dynamically get the path of logging.ini and testlog.log independent of which file(from other path) is invoking this? I don't want to hardcore the absolute path, since this needs to run in both Windows and Linux, any solution?