Develop Reference

Mock python k8s client

Here is my snippets:
folder structure
utils/
├── __init__.py
├── k8s_client.py
├── simple.py
└── tests
├── k8s_client_test.py
k8s_client.py
import os
from kubernetes import client, config
config.load_kube_config(os.getenv('KUBECONFIG')) if os.getenv(
'KUBECONFIG'
) else config.load_incluster_config()
class K8sClient:
def __init__(self) -> None:
self.k8s_api_client = client.CustomObjectsApi()
def get_crs(self) -> list:
custom_resources = self.k8s_api_client.list_cluster_custom_object(
group=os.environ['GROUP'],
version=os.environ['VERSION'],
plural=os.environ['PLURAL'],
)
# list with all custom resources of certain type
return custom_resources["items"]
sample.py
from utils.k8s_client import K8sClient
def initialize_config():
return K8sClient().get_crs()
and finally I am trying to unittest initialize_config function. Here is my test code:
from utils.simple import initialize_config
from mock import patch
#patch('utils.k8s_client.K8sClient.get_crs')
#patch('utils.k8s_client.config.load_incluster_config')
def test_sample(kube_config_mock, k8s_client_mock):
custom_resources = initialize_config()
assert k8s_client_mock.assert_called_once()
This is what I get for error:
kubernetes.config.config_exception.ConfigException: Service host/port is not set.
This is because don't export kubeconfig at all, but I don't want to do it and I don't wish to make a real calls to the k8s cluster to get the real custom resources from there. I just want to fake these calls and kubeconfig exporting and in my case to see if k8s_client_mock is called and if I handle with that I can give return value to k8s_client_mock to assert its length and so on. But the main idea is How to fake this k8s client and to manage it as I want.

Python — Init class reflectively

I am creating a commands system in Python. I have module vkcommands that has a class that processes commands from chat (this is a chat-bot), and inside it, I also have class VKCommand with attributes like name, usage, min_rank, etc. Then I have module vkcmds with submodules that implement these commands:
...
vkcommands.py
vkcmds
|- __init__.py # empty
|- add_group.py
|- another_cmd.py
|- ...
Implementations of commands (e.g. add_group) look like this:
import ranks
import vkcommands
from vkcommands import VKCommand
class AddGroup(VKCommand):
def __init__(self, kristy):
VKCommand.__init__(self, kristy,
label='create',
# ... (other attributes)
min_rank=ranks.Rank.USER)
def execute(self, chat, peer, sender, args=None, attachments=None):
# implementation (called from vkcommands.py)
When a user sends a message in the chat, the command manager analyzes it and looks through the registered commands list to see if this is an ordinary message or a bot command. Currently I register all commands in the commands list manually like this:
class VKCommandsManager:
def __init__(self, kristy):
from vkcmds import (
add_group,
next_class
)
self.kristy = kristy
self.commands = (
add_group.AddGroup(kristy),
next_class.NextClass(kristy)
)
Now I would like all commands to be registered automatically using reflections instead. In Java, I'd iterate over all classes in my commands package, reflectively getConstructor of each, call it to retrieve the VKCommand object, and add it to the commands list.
How can I do so in Python? Again, what I need is to:
iterate over all submodules in module (folder) vkcmds/;
for each submodule, check if there is some class X that extends VKCommand inside;
if (2) is true, then call the constructor of that class with one argument (it is guaranteed that the constructor for all commands only has one argument of a known type (my bot's main class));
add the object (? extends VKCommand) constructed in (3) to the commands list that I can iterate over later.

With this file structure:
- Project
├─ commands
| ├─ base.py
| ├─ baz.py
| └─ foo_bar.py
|
└─ main.py
And the following inside the commands directory files:
base.py
class VKCommand:
""" We will inherit from this class if we want to include the class in commands. """
baz.py
from commands.base import VKCommand
class Baz(VKCommand):
pass
def baz():
""" Random function we do not want to retrieve.
foo_bar.py
from .base import VKCommand
class Foo(VKCommand):
""" We only want to retrieve this command. """
pass
class Bar:
""" We want to ignore this class. """
pass
def fizz():
""" Random function we do not want to retrieve. """
We can retrieve the class instances and names directly using the following code:
main.py
"""
Dynamically load all commands located in submodules.
This file is assumed to be at most 1 level higher than the
specified folder.
"""
import pyclbr
import glob
import os
def filter_class(classes):
inherit_from = 'VKCommand'
classes = {name: info for name, info in classes.items() if inherit_from in info.super}
return classes
# Locate all submodules and classes that it contains without importing it.
folder = 'commands' # `vkcmds`.
submodules = dict()
absolute_search_path = os.path.join(os.path.dirname(__file__), folder, '*.py')
for path in glob.glob(absolute_search_path):
submodule_name = os.path.basename(path)[:-3]
all_classes = pyclbr.readmodule(f"commands.{submodule_name}")
command_classes = filter_class(all_classes)
if command_classes:
submodules[submodule_name] = command_classes
# import the class and store an instance of the class into the command list
class_instances = dict()
for submodule_name, class_names in submodules.items():
module = __import__(f"{folder}.{submodule_name}")
submodule = getattr(module, submodule_name)
for class_name in class_names:
class_instance = getattr(submodule, class_name)
class_instances[class_name] = class_instance
print(class_instances)
Explanation
The solution is twofold. It first locates all submodules that have a class which inherit from VKCommand and are located in the folder 'commands`. This leads to the following output containing the module and the class that have to be imported and instantiated respectively:
{'baz': {'Baz': <pyclbr.Class object at 0x000002BF886357F0>}, 'foo_bar': {'Foo': <pyclbr.Class object at 0x000002BF88660668>}}
The second part of the code imports the correct module and class name at run time. The variable class_instance contains the class name and a reference to the class which can be used to instantiate it. The final output will be:
{'Baz': <class 'commands.baz.Baz'>, 'Foo': <class 'commands.foo_bar.Foo'>}
Important notes:
The code only works when importing modules that are 1 dictionary deeper. If you want to use it recursively, you will have to locate the relative path difference and update the pyclbr.readmodule and __import__ with the correct (full) relative import path.
Only the modules that contain a class which inherit from VKCommand get loaded. All other modules are not imported, and have to be imported manually.

I believe that you can make an array of all the commands you have in your folder and then go over them and instantiate the objects.
in __init__.py
all_commands = [AddGroup, AnotherCmd, ...]
instantiate them like this:
objects = [Cmd(arg1, arg2, ...) for Cmd in all_commands]
Edit:
you could also retrieve the class names with the method that you said you had of getting all class names in the folder.

How to import a class from another file in python?

Im new to python and have looked at various stack overflow posts. i feel like this should work but it doesnt. How do you import a class from another file in python?
This folder structure
src/example/ClassExample
src/test/ClassExampleTest
I have this class
class ClassExample:
def __init__(self):
pass
def helloWorld(self):
return "Hello World!"
I have this test class
import unittest
from example import ClassExample
class ClassExampleTest(unittest.TestCase):
def test_HelloWorld(self):
hello = ClassExample()
self.assertEqual("Hello World!", hello.helloWorld())
if __name__ == '__main__':
unittest.main()
When the unit test runs the object is None:
AttributeError: 'NoneType' object has no attribute 'helloWorld'
What's wrong with this? How do you import a class in python?

If you're using Python 3, then imports are absolute by default. This means that import example will look for an absolute package named example, somewhere in the module search path.
So instead, you probably want a relative import. This is useful when you want to import a module that is relative the module doing the importing. In this case:
from ..example.ClassExample import ClassExample
I'm assuming that your folders are Python packages, meaning that they contain __init__.py files. So your directory structure would look like this.
src
|-- __init__.py
|-- example
| |-- __init__.py
| |-- ClassExample.py
|-- test
| |-- __init__.py
| |-- ClassExampleTest.py

Mock.patch in python unittest could work for two paths

I have a dir which has such structure:
├── aaa.py
├── src
│   └── subsrc
│   ├── else.py
│   └── util.py (there is a "foo" function")
└── tests
├── __init__.py
└── unittests
├── __init__.py
└── test_aaa.py
so "aaa.py", "tests" dir and "src" dir are in project root. and in "test_aaa.py", I use mock to mock function in "util.py":
from src.subsrc.util import foo
import pytest
from unittest import mock
#mock.patch("src.subsrc.util.foo")
def test_foo(mock):
mock.return_value = 111
and then I run python3.7 -m pytest inside "unittests" dir, it worked. This makes sense to me since pytest will find the first dir without __init__.py and then add it to PATH(in this case project root dir will be added) so it could find "src.subsrc.util.foo".
But then I made a small change to "test_aaa.py", in its "mock.patch", I added "aaa" at the beginning:
from src.subsrc.util import foo
import pytest
from unittest import mock
#mock.patch("aaa.src.subsrc.util.foo")
def test_foo(mock):
mock.return_value = 111
it still worked, "aaa.py" is an executable, in "aaa.py":
#!python3.7
from src.subsrc.else import other
if __name__ = "__main__":
# ...
pass
I am very confused why #mock.patch("aaa.src.subsrc.util.foo") also worked, is Python so smart that it could ignore 'aaa' then go "src.subsrc.." to find what it needs? Thanks!
update:
I suspect if because "aaa.py"'s name is special so I changed it to different names, but it still worked. Like I change it to "bbb.py", then in mock.patch, "aaa.src..." does not work but "bbb.src..." still worked. So I am sure "mock.patch" find this executable first.
update:
I guess it could be related to how "mock.patch()" works?

Your example seems to be a bit too stripped-down, but I'll try to expand it in order to explain. When reading about mocks in Python, you will often encounter the phrase "mock it where it's used", which isn't really helpful if you are new to the topic (but here's an excellent article on this concept).
In your test_aaa.py you will probably want to test some functionality of your aaa.py module, which may call some function from src/subsrc/util.py. After importing your foo() function in the aaa.py module, that's the exact location where you should point #mock.patch to: #mock.patch("aaa.foo"). By doing this, your mock will have access to all invocations of foo() in the functions you are about to test, namely aaa.do_something(). I've expanded your example as follows:
# aaa.py
from src.subsrc.util import foo
def do_something():
return foo()
if __name__ == "__main__":
value = do_something()
print(f"value is {value}")
# src/subsrc/util.py
def foo():
return 222
# tests/unittests/test_aaa.py
from unittest import mock
from aaa import do_something
#mock.patch("aaa.foo")
def test_foo(foo_mocked):
foo_mocked.return_value = 111
value = do_something()
assert value == 111
When executing this like python aaa.py, I get the output as expected (value is 222) while the test passes with its assert value == 111.
In your example, #mock.patch("src.subsrc.util.foo") obviously worked, but probably didn't do what you intended. From your example code, I cannot see how #mock.patch("aaa.src.subsrc.util.foo") shouldn't have returned a ModuleNotFoundError.

Importing class that require other imports

I have this structure:
myApp
|---> module
| |---> __init__.py
| |---> constants.py
| |---> job.py
|---> processor.py
job.py has a Job class that imports some constants in constants.py
import constants
class Job:
def __init__(self, id):
self.id = id
self.status = constants.JOB_WAITING
.
.
.
Then in my processor.py I'm trying to use the Job class.
from module1 import job
j = job.Job(123)
print(j.id)
I'm met with the exception "ModuleNotFoundError: No module named 'constants'" from just the first line "from module1 import job"
A naive solution of adding "from module1 import constants" before that doesn't help. Nor would that be desirable because from the perspective of processor.py it just care about importing job and not worry about importing whatever else job needs.
Is the problem due to when I import job, it then looks for the import constants in the wrong path? Not sure how I fix it if that's the case.

Since you are in the same package, try
from . import constants