Hi
I have some code on my CI failing (local runs do not fail).
The problem is that class instance fails isinstance() check.
Code:
File: main.py
class MyController(SuperController):
# Overrides default definition of get_variables_context()
from my_options import get_variables_context
File: my_options.py
...
def get_variables_context(self: SuperController, **kwargs):
from main import MyController
self: MyController
print(f"type(self) is {type(self)} (it {'IS' if (isinstance(self, MyController)) else 'IS NOT'} a subclass of MyController)")
_super = super(MyController, self).get_variables_context(**kwargs) or dict()
_super.update(result)
return _super
Got output and error:
type(self) is <class '__main__.SomeController'> (it IS NOT a subclass of SomeController
Traceback (most recent call last):
File "main.py", line 24, in <module>
SomeController.main(**params)
File "/builds/RND/my/rcv-nginx/tests/nginx_tests/flow.py", line 391, in main
_tests_suite, _, _ = self.prepare()
File "/builds/RND/my/rcv-nginx/tests/nginx_tests/flow.py", line 359, in prepare
context['variables_context'] = self.get_variables_context(**context)
File "/builds/RND/my/tests/integration/my_options.py", line 81, in get_variables_context
_super = super(SomeController, self).get_variables_context(**kwargs) or dict()
TypeError: super(type, obj): obj must be an instance or subtype of type
I've found the solution while investigating the root cause.
In the local run, ...
I actually call the python unittest which then calls main.py which then creates a class MyController which then calls my_options.py, and the class is added to the loaded module 'main'.
Then, MyController.get_variables_context asks for the module 'main', which is already loaded, then for the class MyController in that module, so the same type instance is returned and type check succeeds.
In the CI run, ...
I call directly main.py with the argument "test" (which should create a controller and run all tests from it via unittest), so the class MyController is created inside module __main__. MyController.get_variables_context still asks for the MyController class in main.py, but the module 'main' is not loaded here, so python loads it, creates new class MyController, and then returns it.
So, basically the answer is ...
to move MyController from main.py to the other file, i.e. controller.py
Related
I have three files that all contain classes with the same names but slightly different definitions. Some methods in these classes are identical across all three files, so I abstracted them out to another file, utils.py, where they are defined within a "template" version of the original class. The problem is that these methods invoke functions and modules that exist in the original files but not this new one.
My original approach was to use multiple class inheritance, which would initialize the template class within the scope of the parent class, allowing access to all the functions and modules it requires. However, I was instructed to avoid multiple class inheritance and to simply import the utils file.
Importing does not apply the same scoping logic as mentioned above with inheritance. So here arises my problem. I have created a small example to show what I mean. I am using a module called datajoint. You don't need to know much about it except that a schema is basically a table or collection of tables in a database.
schemas.py
import datajoint as dj
from datetime import datetime
import utils
dj.conn()
schema = dj.Schema('adib_example1')
schema.drop()
schema = dj.Schema('adib_example1')
def test_print():
print("test")
#schema
class Subject(dj.Lookup):
definition = """
subject_id: int
"""
contents = [dict(subject_id=1)]
#schema
class Session(dj.Computed):
definition = """
-> Subject
time: varchar(30)
"""
def make(self, key):
utils.SessionTemplate.make(self,key)
Session.populate() # invokes Session's make(), passing Subject's primary key
Approach 1
Import scoping not working like inheritance
utils.py
class SessionTemplate():
#staticmethod
def make(table, key):
test_print() # parent function usage example
table.time = f"{datetime.now()}" # parent module usage example
new_entry = dict(**key, time=table.time)
table.insert1(new_entry)
error
Traceback (most recent call last):
File "/home/.anaconda/imported_make/schemas.py", line 30, in <module>
Session.populate() # invokes Session's make(), passing Subject's primary key
File "/opt/conda/lib/python3.9/site-packages/datajoint/autopopulate.py", line 153, in populate
make(dict(key))
File "/home/.anaconda/imported_make/schemas.py", line 28, in make
utils.SessionTemplate.make(self,key)
File "/home/.anaconda/imported_make/utils.py", line 5, in make
test_print() # parent function usage example
NameError: name 'test_print' is not defined
Approach 2
Importing schemas.py into utils.py works, but requires including schemas. before every imported function and module, which is not practical in my case.
utils.py
import schemas
class SessionTemplate():
#staticmethod
def make(table, key):
schemas.test_print() # parent function usage example
table.time = f"{schemas.datetime.now()}" # parent module usage example
new_entry = dict(**key, time=table.time)
table.insert1(new_entry)
Approach 3
Import using * to avoid having to add schemas. before each parent function/module somehow does not provide access to the parents modules and functions.
from schemas import *
class SessionTemplate():
#staticmethod
def make(table, key):
test_print() # parent function usage example
table.time = f"{datetime.now()}" # parent module usage example
new_entry = dict(**key, time=table.time)
table.insert1(new_entry)
error
Traceback (most recent call last):
File "/home/.anaconda/imported_make/run.py", line 1, in <module>
import schemas
File "/home/.anaconda/imported_make/schemas.py", line 30, in <module>
Session.populate() # invokes Session's make(), passing Subject's primary key
File "/opt/conda/lib/python3.9/site-packages/datajoint/autopopulate.py", line 153, in populate
make(dict(key))
File "/home/.anaconda/imported_make/schemas.py", line 28, in make
utils.SessionTemplate().make(self,key)
File "/home/.anaconda/imported_make/utils.py", line 7, in make
test_print() # parent function usage example
NameError: name 'test_print' is not defined
I know import * is bad practice, but it would have been fine in this instance if it worked, and I'm not sure why it doesn't.
boss.py
class tasks():
def job1(input):
// do something
return output
def job2(input):
// do something
return output
worker.py
import boss.tasks
from boss.tasks import job1, job2
input_value = "xyz"
output1 = boss.tasks().job1(input_value)
output2 = boss.tasks().job2(input_value)
I'm going crazy trying to perform simple editing while creating a Python class constructor. I can create the simple class and constructor variable but once I try to change the names of the variable I get a KeyError.
The class is below:
class Collect:
def __init__(self, **kwargs):
self.foo = kwargs["foo"]
And the script where I instantiate the class and print out its attributes is below:
import mapper as m
payload = m.Collect(foo="Hello")
print(payload.foo)
Now this works just fine, but if I change "foo" to "bar" I get a KeyError Like below:
class Collect:
def __init__(self, **kwargs):
self.bar = kwargs["bar"]
and then running:
import mapper as m
payload = m.Collect(bar="Hello")
print(payload.bar)
will throw the following error:
Traceback (most recent call last): File "<stdin>", line 1, in
<module> File
"/path/to/mapper.py",
line 8, in __init__
self.bar = kwargs["bar"] KeyError: 'foo'
And the print function will throw the error below:
Traceback (most recent call last): File "<stdin>", line 1, in
<module> AttributeError: 'Collect' object has no attribute 'bar'
The weird thing is that if I hit save and then close VSCode and reopen, the new class with bar will work just fine. And, also even if I don't close VSCode the class will instantiate and the print statement will run just fine when I switch to Debug mode and run it. But when I try to highlight it and run a selection it throws that error.
How can I just test if the code works using the run selection operations and not relying on running in debug mode with breakpoints or having to close and reopen VSCode?
The solution was to reload my Python session. The VSCode reload extension makes this easy.
I have an uninitialzed global variable in my module that is properly initialized during application startup. For typechecking I use the syntax var: Type without a value from python >= 3.6 so I do not have to complicate the typechecking for the default value (which would be None).
Now I want to unittest another function that uses this global variable but I get an error from unittest.mock.patch. Here is a simplified version of what I am doing:
File mod.py:
global_var: bool
#global_var: bool = False
def init(val: bool) -> None:
global global_var
global_var = val
def do_stuff() -> str:
return "a" if global_var else "b"
File test.py:
import unittest.mock, mod
class MockUninitializedGlobal(unittest.TestCase):
def test_it(self):
with unittest.mock.patch("mod.global_var", True):
actual = mod.do_stuff()
expected = "a"
self.assertEqual(expected, actual)
I run it with python3 -m unittest test.py.
The exception is
Traceback (most recent call last):
File "/home/luc/soquestion/test.py", line 4, in test_it
with mock.patch("mod.global_var", True):
File "/nix/store/vs4vj1yzqj1bkcqkf3b6sxm6jfy1gb4j-python3-3.7.7/lib/python3.7/unittest/mock.py", line 1323, in __enter__
original, local = self.get_original()
File "/nix/store/vs4vj1yzqj1bkcqkf3b6sxm6jfy1gb4j-python3-3.7.7/lib/python3.7/unittest/mock.py", line 1297, in get_original
"%s does not have the attribute %r" % (target, name)
AttributeError: <module 'mod' from '/home/luc/soquestion/mod.py'> does not have the attribute 'global_var'
If I comment line 1 and uncomment line 2 in mod.py the test passes.
Is there any way to make the test pass without definig a default value for the global variable in my application code?
Edit: As noted in the comments the actual variable is a parsed config file which I do not want to load during tests. It is used in a command line application and therefore the variable is always set after command line parsing is done. The real code is here and here. In the actual test I am not trying to mock the global variable as a bool directly but some attribute on the config object.
I used the setUp and tearDown methods on the unittest.TestCase class to solve this.
With my edit above about it actually being a config object the code looks like this:
class Config:
def __init__(self, filename: str):
"Load config file and set self.stuff"
self.stuff = _code_to_parse(filename)
config: Config
def do_stuff() -> str:
return "a" if config.stuff else "b"
And the test creates a mocked config object before the tests in order to mock some attribute on the config object during the test. In the teardown I also delete the object again so that nothing will spill to the next test case.
The actual mocking of the attributes I need is then done in a with block in the actual test as it is (a) cleaner and (b) I can just mock what I need in contrast to mocking all attributes on the config object in the setUp method.
import unittest, unittest.mock, mod
class MockUninitializedGlobal(unittest.TestCase):
def setUp(self):
mod.config = unittest.mock.Mock(spec=mod.Config)
def tearDown(self):
del mod.config
def test_it(self):
with unittest.mock.patch("mod.config.stuff", True):
actual = mod.do_stuff()
expected = "a"
self.assertEqual(expected, actual)
The code below is used as part of a SimpleXMLRPCServer to receive commands from a Java client I've written. The Java client will just call this execute function and pass in a function name from the CameraAssembler class.
from nsCamera.CameraAssembler import CameraAssembler
class MyFunctions:
ca = None
def initialize(self):
# Create Camera object
self.ca = CameraAssembler(commname=COMM, boardname=BOARD, sensorname=SENSOR, verbose=True)
return True
def execute(self, code):
func = getattr(self.ca,code)
output = func()
return output
myfuncs = MyFunctions()
myfuncs.initialize()
output = myfuncs.execute('arm()')
print(output)
Output:
Traceback (most recent call last):
File "pyTestServer.py", line 31, in <module>
output = myfuncs.execute("arm()")
File "pyTestServer.py", line 21, in execute
func = getattr(MyFunctions.ca,code)
AttributeError: CameraAssembler instance has no attribute 'arm()'
Your parentheses are in the wrong place. The attribute is not called arm(), it's called arm; you need to call the result of getting that attribute.
output = myfuncs.execute('arm')()
(Note, this code isn't particularly idiomatic. In particular, I can't see why you're setting ca as a class attribute, rather than an instance one. Also, initialisation usually goes in an __init__ method, which is called automatically on instantiation.)
I am attempting to use multiprocessing to call derived class member function defined in a different module. There seem to be several questions dealing with calling class methods from the same module, but none from different modules. For example, if I have the following structure:
main.py
multi/
__init__.py (empty)
base.py
derived.py
main.py
from multi.derived import derived
from multi.base import base
if __name__ == '__main__':
base().multiFunction()
derived().multiFunction()
base.py
import multiprocessing;
# The following two functions wrap calling a class method
def wrapPoolMapArgs(classInstance, functionName, argumentLists):
className = classInstance.__class__.__name__
return zip([className] * len(argumentLists), [functionName] * len(argumentLists), [classInstance] * len(argumentLists), argumentLists)
def executeWrappedPoolMap(args, **kwargs):
classType = eval(args[0])
funcType = getattr(classType, args[1])
funcType(args[2], args[3:], **kwargs)
class base:
def multiFunction(self):
mppool = multiprocessing.Pool()
mppool.map(executeWrappedPoolMap, wrapPoolMapArgs(self, 'method', range(3)))
def method(self,args):
print "base.method: " + args.__str__()
derived.py
from base import base
class derived(base):
def method(self,args):
print "derived.method: " + args.__str__()
Output
base.method: (0,)
base.method: (1,)
base.method: (2,)
Traceback (most recent call last):
File "e:\temp\main.py", line 6, in <module>
derived().multiFunction()
File "e:\temp\multi\base.py", line 15, in multiFunction
mppool.map(executeWrappedPoolMap, wrapPoolMapArgs(self, 'method', range(3)))
File "C:\Program Files\Python27\lib\multiprocessing\pool.py", line 251, in map
return self.map_async(func, iterable, chunksize).get()
File "C:\Program Files\Python27\lib\multiprocessing\pool.py", line 567, in get
raise self._value
NameError: name 'derived' is not defined
I have tried fully qualifying the class name in the wrapPoolMethodArgs method, but that just gives the same error, saying multi is not defined.
Is there someway to achieve this, or must I restructure to have all classes in the same package if I want to use multiprocessing with inheritance?
This is almost certainly caused by the ridiculous eval based approach to dynamically invoking specific code.
In executeWrappedPoolMap (in base.py), you convert a str name of a class to the class itself with classType = eval(args[0]). But eval is executed in the scope of executeWrappedPoolMap, which is in base.py, and can't find derived (because the name doesn't exist in base.py).
Stop passing the name, and pass the class object itself, passing classInstance.__class__ instead of classInstance.__class__.__name__; multiprocessing will pickle it for you, and you can use it directly on the other end, instead of using eval (which is nearly always wrong; it's code smell of the strongest sort).
BTW, the reason the traceback isn't super helpful is that the exception is raised in the worker, caught, pickle-ed, and sent back to the main process and re-raise-ed. The traceback you see is from that re-raise, not where the NameError actually occurred (which was in the eval line).