So I'm working on a rather big Python 3 project where I'm writing unit tests for some of the files using the unittest library. In one unit test, the tested file imports a function from another python package whose __init__ file itself imports from the tested file. This leads to an ImportError during the unit test.
It is desired for the __init__.py to import from the tested file periphery\foo.py, so I would like to know if there is a possibility to make the unit test work without removing the import from __init__.py
Since the project contains a lot of files, I created a minimal example that illustrates the structure and in which the error can be reproduced. The project structure looks like this
├───core
│ bar.py
│ __init__.py
│
├───periphery
│ foo.py
│ __init__.py
│
└───tests
│ __init__.py
│
├───core
│ __init__.py
│
└───periphery
test_foo.py
__init__.py
The init file in core core/__init__.py contains the code
# --- periphery ---
from periphery.foo import Foo
while periphery/foo.py, which is the file to be tested, looks like
from core.bar import Bar
def Foo():
bar = Bar()
return bar
Finally, the unit test has the following structure
from unittest import TestCase
from periphery.foo import Foo
class Test(TestCase):
def test_foo(self):
""" Test that Foo() returns "bar" """
self.assertEqual(Foo(), "bar")
Running the unit test yields the following error:
ImportError: cannot import name 'Foo' from partially initialized module 'periphery.foo' (most likely due to a circular import)
Related
I am trying to run the unit test using pytest in this project, here main_0.py is importing s3 file.
I am getting ModuleNotFoundError: no module named 's3'
Project Folder Structure
some_project
└───src
├───main
│ └───lambda_function
│ └───some
│ main_0.py
│ s3.py
│
└───test
└───unittest
└───lambda_function
└───some
test_main_0.py
test_s3.py
main_0.py
from s3 import PrintS3
def lambda_handler():
obj = PrintS3()
res = obj.print_txt()
return res
s3.py
class PrintS3:
def __init__(self) -> None:
self.txt = "Hello"
def print_txt(self):
print(self.txt)
return self.txt
test_main_0.py
import unittest
class TestSomeMain(unittest.TestCase):
def test_main_0(self):
from src.main.lambda_function.some.main_0 import lambda_handler
res = lambda_handler()
assert res == "Hello"
test_s3.py is empty.
I also tried adding an empty __init__.py file in both the dir but still the same error
Project Folder Structure after adding __init__.py file
some_project
└───src
├───main
│ └───lambda_function
│ └───some
│ main_0.py
│ s3.py
│ __init__.py
│
└───test
└───unittest
└───lambda_function
└───some
test_main_0.py
test_s3.py
__init__.py
the command I am using to run pytest:
python -m pytest ./src/test
and I am inside some_project folder and also using main_0.py instead of main.py because to not get confused with main folder
Edit 2:
I am to run the test case successfully by adding sys.path in the test_main_0.py file but it is breaking linting and hinting in the code editor (vscode) it didn't broke the linting and hinting, both import statement works but is there any better way.
new test_main_0.py:
import unittest
import os
import sys
sys.path.append(os.path.abspath("./src/main/lambda_function/some/"))
class TestSomeMain(unittest.TestCase):
def test_main_0(self):
from src.main.lambda_function.some.main_0 import lambda_handler # this works
from main_0 import lambda_handler # this also works but break linting and hinting in the code editor
res = lambda_handler()
assert res == "Hello"
could you please try
import os
import sys
sys.path.append(os.path.join(os.path.dirname(__file__), ".."))
from some.s3 import PrintS3
def lambda_handler():
obj = PrintS3()
res = obj.print_txt()
return res
I found a somewhat working solution.
added setUp() and tearDown() methods in the class for inserting and removing path in sys.path
path in sys.path is the location of the directory where the main_0.py and s3.py is located
import unittest
import os
import sys
class TestSomeMain(unittest.TestCase):
def setUp(self) -> None:
sys.path.insert(0, os.path.abspath("./src/main/lambda_function/some/"))
def tearDown(self) -> None:
sys.path.remove(os.path.abspath("./src/main/lambda_function/some/"))
def test_main_0(self):
from src.main.lambda_function.some.main_0 import lambda_handler
res = lambda_handler()
assert res == "Hello"
also update the test command in the terminal:
python -m pytest ./src/test/unittest/lambda_function/some --cov ./src/main/lambda_function/some --cov-report html
I have faced a rather famous issue while importing my python modules inside the project.
This code is written to replicate the existing situation:
multiply_func.py:
def multiplier(num_1, num_2):
return num_1 * num_2
power_func.py:
from math_tools import multiplier
def pow(num_1, num_2):
result = num_1
for _ in range(num_2 - 1):
result = multiplier(num_1, result)
return result
The project structure:
project/
│ main.py
│
└─── tools/
│ __init__.py
│ power_func.py
│
└─── math_tools/
│ __init__.py
│ multiply_func.py
I've added these lines to __init__ files to make the importing easier:
__init__.py (math_tools):
from .multiply_func import multiplier
__init__.py (tools):
from .power_func import pow
from .math_tools.multiply_func import multiplier
Here is my main file.
main.py:
from tools import pow
print(pow(2, 3))
Whenever I run it, there is this error:
>>> ModuleNotFoundError: No module named 'math_tools'
I tried manipulating the sys.path, but I had no luck eliminating this puzzling issue. I'd appreciate your kind help. Thank you in advance!
You messed it up in the "power_func.py" file.
You have to use . before math_tools to refer to the current directory module.
Update "power_func.py" as bellow, it works perfectly.
from .math_tools import multiplier
My Project looks like this:
├─outer_module
│ │ __init__.py
│ │
│ └─inner_module
│ a.py
│ b.py
├─test.py
__init__.py:
from outer_module.inner_module import a
from outer_module.inner_module import b
a.py:
instance_a = 1
b.py:
instance_b = 1
print("instance_b created!")
test.py:
from outer_module.inner_module import a
I want to shorten import path in test.py, i.e. use from outer_module import a. That is not unusual when I turn my project into a release module. But using __init__.py, it will automatically invoke b.py and print instance_b created!. Seperating a.py and b.py from inner_module is not recommended because they are functionally similar. Other .py file may invoke b.py so b.py must appear in __init__.py
Could anyone give some advice?
In your __init__.py file try importing the a and b files individually and adding the two imports to your __all__ variable.
from outer_module.inner_module import a
from outer_module.inner_module import b
__all__ = [
'a',
'b',
]
Now you can import a and b directly from outer_module.
from outermodule import a
import outermodule.b as b
I have a dir which has such structure:
├── aaa.py
├── src
│ └── subsrc
│ ├── else.py
│ └── util.py (there is a "foo" function")
└── tests
├── __init__.py
└── unittests
├── __init__.py
└── test_aaa.py
so "aaa.py", "tests" dir and "src" dir are in project root. and in "test_aaa.py", I use mock to mock function in "util.py":
from src.subsrc.util import foo
import pytest
from unittest import mock
#mock.patch("src.subsrc.util.foo")
def test_foo(mock):
mock.return_value = 111
and then I run python3.7 -m pytest inside "unittests" dir, it worked. This makes sense to me since pytest will find the first dir without __init__.py and then add it to PATH(in this case project root dir will be added) so it could find "src.subsrc.util.foo".
But then I made a small change to "test_aaa.py", in its "mock.patch", I added "aaa" at the beginning:
from src.subsrc.util import foo
import pytest
from unittest import mock
#mock.patch("aaa.src.subsrc.util.foo")
def test_foo(mock):
mock.return_value = 111
it still worked, "aaa.py" is an executable, in "aaa.py":
#!python3.7
from src.subsrc.else import other
if __name__ = "__main__":
# ...
pass
I am very confused why #mock.patch("aaa.src.subsrc.util.foo") also worked, is Python so smart that it could ignore 'aaa' then go "src.subsrc.." to find what it needs? Thanks!
update:
I suspect if because "aaa.py"'s name is special so I changed it to different names, but it still worked. Like I change it to "bbb.py", then in mock.patch, "aaa.src..." does not work but "bbb.src..." still worked. So I am sure "mock.patch" find this executable first.
update:
I guess it could be related to how "mock.patch()" works?
Your example seems to be a bit too stripped-down, but I'll try to expand it in order to explain. When reading about mocks in Python, you will often encounter the phrase "mock it where it's used", which isn't really helpful if you are new to the topic (but here's an excellent article on this concept).
In your test_aaa.py you will probably want to test some functionality of your aaa.py module, which may call some function from src/subsrc/util.py. After importing your foo() function in the aaa.py module, that's the exact location where you should point #mock.patch to: #mock.patch("aaa.foo"). By doing this, your mock will have access to all invocations of foo() in the functions you are about to test, namely aaa.do_something(). I've expanded your example as follows:
# aaa.py
from src.subsrc.util import foo
def do_something():
return foo()
if __name__ == "__main__":
value = do_something()
print(f"value is {value}")
# src/subsrc/util.py
def foo():
return 222
# tests/unittests/test_aaa.py
from unittest import mock
from aaa import do_something
#mock.patch("aaa.foo")
def test_foo(foo_mocked):
foo_mocked.return_value = 111
value = do_something()
assert value == 111
When executing this like python aaa.py, I get the output as expected (value is 222) while the test passes with its assert value == 111.
In your example, #mock.patch("src.subsrc.util.foo") obviously worked, but probably didn't do what you intended. From your example code, I cannot see how #mock.patch("aaa.src.subsrc.util.foo") shouldn't have returned a ModuleNotFoundError.
Have a module in a subdirectory and when I try to import it, I get a NameError: namefoois not defined. When I put the class code directly into the __main__.py file it works fine. __init__.py files are empty.
I've tried the following all with other errors:
MyProject/
├── __init__.py
├── __main__.py
├── foo/
│ ├── bar.py
│ ├── __init__.py
bar.py
class Bar:
def __init__(self):
print( 'am here' )
pass
__main__.py
from MyProject import foo
#from MyProject import bar # errors with cannot import bar from MyProject
#from foo import bar # errors with No module named foo
if __name__ == '__main__':
w = Bar()
Is there perhaps a better way to organise this?
The Bar class is in the file bar.py, so I think you'd need to do
from MyProject.foo import bar
w = bar.Bar()
or
from MyProject.foo.bar import Bar
w = Bar()
You didn't share your foo/__init__.py, but you could fix the situation by adding something like this to it:
from .bar import Bar
That adds Bar to the namespace of foo and causes Python to find Bar when you just import foo.
However, you would probably do well to look at a few standard examples for writing a package. For one, you probably shouldn't name your project MyProject, as that name signals it's a class (with TitleCase). Naming it 'project' further confuses the issue, as it appears you're writing a package, so my_package sounds about right.
If you dont know from what direcrory file will be runed then use . to show where package is
from .foo.bar import Bar
or
from .foo import bar
w = bar.Bar()
or
import .foo
w = foo.bar.Bar()
. before package name means that package lockated in same directory with current file