Develop Reference

Class instance fails isinstance check

Hi
I have some code on my CI failing (local runs do not fail).
The problem is that class instance fails isinstance() check.
Code:
File: main.py
class MyController(SuperController):
# Overrides default definition of get_variables_context()
from my_options import get_variables_context
File: my_options.py
...
def get_variables_context(self: SuperController, **kwargs):
from main import MyController
self: MyController
print(f"type(self) is {type(self)} (it {'IS' if (isinstance(self, MyController)) else 'IS NOT'} a subclass of MyController)")
_super = super(MyController, self).get_variables_context(**kwargs) or dict()
_super.update(result)
return _super
Got output and error:
type(self) is <class '__main__.SomeController'> (it IS NOT a subclass of SomeController
Traceback (most recent call last):
File "main.py", line 24, in <module>
SomeController.main(**params)
File "/builds/RND/my/rcv-nginx/tests/nginx_tests/flow.py", line 391, in main
_tests_suite, _, _ = self.prepare()
File "/builds/RND/my/rcv-nginx/tests/nginx_tests/flow.py", line 359, in prepare
context['variables_context'] = self.get_variables_context(**context)
File "/builds/RND/my/tests/integration/my_options.py", line 81, in get_variables_context
_super = super(SomeController, self).get_variables_context(**kwargs) or dict()
TypeError: super(type, obj): obj must be an instance or subtype of type

I've found the solution while investigating the root cause.
In the local run, ...
I actually call the python unittest which then calls main.py which then creates a class MyController which then calls my_options.py, and the class is added to the loaded module 'main'.
Then, MyController.get_variables_context asks for the module 'main', which is already loaded, then for the class MyController in that module, so the same type instance is returned and type check succeeds.
In the CI run, ...
I call directly main.py with the argument "test" (which should create a controller and run all tests from it via unittest), so the class MyController is created inside module __main__. MyController.get_variables_context still asks for the MyController class in main.py, but the module 'main' is not loaded here, so python loads it, creates new class MyController, and then returns it.
So, basically the answer is ...
to move MyController from main.py to the other file, i.e. controller.py

python default configuration reuse of variables

class DefaultConfig(object):
class S3(object):
DATA_ROOT = 's3://%(bucket_name)s/NAS'
DATA_LOCATION = '{}/%(instrument_id)s/%(run_id)s'.format(DefaultConfig.S3.DATA_ROOT)
The code above gives me the following error.
File "./s3Utils.py", line 5, in <module>
from InfraConfig import InfraConfig as IC
File "/opt/src/datasource/src/main/python/util/InfraConfig.py", line 4, in <module>
class DefaultConfig(object):
File "/opt/src/datasource/src/main/python/util/InfraConfig.py", line 6, in DefaultConfig
class S3(object):
File "/opt/src/datasource/src/main/python/util/InfraConfig.py", line 14, in S3
DATA_LOCATION = '{}/%(instrument_id)s/%(run_id)s'.format(DefaultConfig.S3.DATA_ROOT)
NameError: name 'DefaultConfig' is not defined
Why is it unable to find DefaultConfig.S3.DATA_ROOT
Also, this is my attempt at writing structured configuration with reuse of values of DefaultConfig. Is there a way to do this without writing a yml file?

It is unable to find the DefaultConfing because DefaultConfig is not defined at the moment S3 is created.
Remember that class are objects. Because there are objects, that means they need to be instantiate. Python instantiate a class at the end of its definition, and therefore register it in the globals. Because the class definition is not finished, you can't use the DefaultConfig name.

You should use it without any prefixes:
class DefaultConfig(object):
class S3(object):
DATA_ROOT = 's3://%(bucket_name)s/NAS'
DATA_LOCATION = '{}/%(instrument_id)s/%(run_id)s'.format(DATA_ROOT)
print DefaultConfig.S3.DATA_LOCATION
returns:
> s3://%(bucket_name)s/NAS/%(instrument_id)s/%(run_id)s

importing module causes TypeError: module.__init__() takes at most 2 arguments (3 given)

Please don't mark as duplicate, other similar questions did not solve my issue.
This is my setup
/main.py
/actions/ListitAction.py
/actions/ViewAction.py
Main.py:
from actions import ListitAction, ViewAction
ListitAction.py:
class ListitAction(object):
def __init__(self):
#some init behavior
def build_uri():
return "test.uri"
ViewAction.py
from actions import ListitAction
class ViewAction(ListitAction):
def __init__(self, view_id):
ListitAction.__init__(self)
self.view_id = view_id
def build_uri():
return "test"
Running:
$ python3 main.py
The only error message I receive is:
Traceback (most recent call last):
File "/home/jlevac/workspace/project/listit.py", line 11, in <module>
from actions import ListitAction, ViewAction, CommentsAction
File "/home/jlevac/workspace/project/actions/ViewAction.py", line 3, in <module>
class ViewAction(ListitAction):
TypeError: module.__init__() takes at most 2 arguments (3 given)
Even if I try for the python3 console, I received the same error message:
$python3
from actions import ViewAction
I am new to Python, but not new to programming. I'm assuming that my error messages have to do with the import statements, but based on the message I can't really figure out what it means.

Your imports are wrong, so you're trying to inherit from the modules themselves, not the classes (of the same name) defined inside them.
from actions import ListitAction
in ViewAction.py should be:
from actions.ListitAction import ListitAction
and similarly, all other uses should switch to explicit imports of from actions.XXX import XXX (thanks to the repetitive names), e.g. from actions import ListitAction, ViewAction must become two imports:
from actions.ListitAction import ListitAction
from actions.ViewAction import ViewAction
because the classes being imported come from different modules under the actions package.

You are passing self when you don't need to, that's all.
Edit: see MSeifert's comment answer since I don't want to steal content.

If your file is in the root directory of the project then you can directly write the file name and import.
For example, if filename is Parent1.py and class name is Parent, then you would write
from Parent1 import Parent
However, if your file Parent1.py is under any folder for example:
DemoFolder -> Parent1.py- > Parent
(Folder). (File). (Class name)
Then you would have to write:
from Test.Parent1 import Parent

Creating classes and instance of variables
class Student:
# Creating a Class Variables
perc_Raise = 1.05
# Creating a constructor or a special method to initialize values
def __init__(self,firstName,lastName,marks):
self.firstName = firstName
self.lastName = lastName
self.email = firstName + "." + lastName +"#northalley.com"
self.marks = marks
def fullName(self):
return '{} {}'.format(self.firstName,self.lastName)
def apply_raise(self):
self.marks = int(self.marks * self.perc_Raise)
Creating a two instance variable for a Student class
std_1 = Student('Mahesh','Gatta',62)
std_2 = Student('Saran','D',63)
print(std_1.fullName())
print(std_1.marks)
std_1.apply_raise()
print(std_1.marks)
print(std_1.email)
print(std_1.__dict__)
print(std_2.fullName())
print(std_2.marks)
std_2.apply_raise()
print(std_2.marks)
print(std_2.email)
print(std_2.__dict__)
print(Student.__dict__)
Inheritance
class Dumb(Student):
perc_Raise = 1.10
def __init__(self,firstName,lastName,marks,prog_lang):
super().__init__(firstName,lastName,marks)
self.prog_lang = prog_lang
std_1 = Dumb('Suresh','B',51,'Python')
print(std_1.fullName())
print(std_1.marks)
std_1.apply_raise()
print(std_1.marks)
print(std_1.prog_lang)

multiprocessing and modules

I am attempting to use multiprocessing to call derived class member function defined in a different module. There seem to be several questions dealing with calling class methods from the same module, but none from different modules. For example, if I have the following structure:
main.py
multi/
__init__.py (empty)
base.py
derived.py
main.py
from multi.derived import derived
from multi.base import base
if __name__ == '__main__':
base().multiFunction()
derived().multiFunction()
base.py
import multiprocessing;
# The following two functions wrap calling a class method
def wrapPoolMapArgs(classInstance, functionName, argumentLists):
className = classInstance.__class__.__name__
return zip([className] * len(argumentLists), [functionName] * len(argumentLists), [classInstance] * len(argumentLists), argumentLists)
def executeWrappedPoolMap(args, **kwargs):
classType = eval(args[0])
funcType = getattr(classType, args[1])
funcType(args[2], args[3:], **kwargs)
class base:
def multiFunction(self):
mppool = multiprocessing.Pool()
mppool.map(executeWrappedPoolMap, wrapPoolMapArgs(self, 'method', range(3)))
def method(self,args):
print "base.method: " + args.__str__()
derived.py
from base import base
class derived(base):
def method(self,args):
print "derived.method: " + args.__str__()
Output
base.method: (0,)
base.method: (1,)
base.method: (2,)
Traceback (most recent call last):
File "e:\temp\main.py", line 6, in <module>
derived().multiFunction()
File "e:\temp\multi\base.py", line 15, in multiFunction
mppool.map(executeWrappedPoolMap, wrapPoolMapArgs(self, 'method', range(3)))
File "C:\Program Files\Python27\lib\multiprocessing\pool.py", line 251, in map
return self.map_async(func, iterable, chunksize).get()
File "C:\Program Files\Python27\lib\multiprocessing\pool.py", line 567, in get
raise self._value
NameError: name 'derived' is not defined
I have tried fully qualifying the class name in the wrapPoolMethodArgs method, but that just gives the same error, saying multi is not defined.
Is there someway to achieve this, or must I restructure to have all classes in the same package if I want to use multiprocessing with inheritance?

This is almost certainly caused by the ridiculous eval based approach to dynamically invoking specific code.
In executeWrappedPoolMap (in base.py), you convert a str name of a class to the class itself with classType = eval(args[0]). But eval is executed in the scope of executeWrappedPoolMap, which is in base.py, and can't find derived (because the name doesn't exist in base.py).
Stop passing the name, and pass the class object itself, passing classInstance.__class__ instead of classInstance.__class__.__name__; multiprocessing will pickle it for you, and you can use it directly on the other end, instead of using eval (which is nearly always wrong; it's code smell of the strongest sort).
BTW, the reason the traceback isn't super helpful is that the exception is raised in the worker, caught, pickle-ed, and sent back to the main process and re-raise-ed. The traceback you see is from that re-raise, not where the NameError actually occurred (which was in the eval line).

Python Help: Accessing static member variable from another class

I'll do my best to describe the issue I am having. I am building a Python program that is built on multiple classes and uses the unittest framework. In a nutshell, the Main.py file has a "ValidateDriver" class that defines a "driver" variable as an ElementTree type. If I point this directly to the XML file I need to parse, (i.e. driver = ElementTree.parse(rC:\test.xml)) then I can access it from another class. However, in reality I don't have the actual XML file that is passed in from the command-line until you get to the Main function in the ValidateDriver class. So under the ValidateDriver class driver would really be driver = ElementTree and then in the main function I would reassign that variable to ValidateDriver.driver = ElementTree.parse(args.driver). However, this is the crux. When I go to the other class and try to call ValidateDriver.driver I don't have the "findall" method/attribute available. Again, the only way it will work is to do something like: ElementTree.parse(rC:\test.xml)). If I did this in C# it would work, but I am new to Python and this is kicking my butt. Any help/suggestions is appreciated. I've included the code for both classes.
Main Function:
import sys
import argparse
import xml.etree.ElementTree as ElementTree
import unittest
import Tests.TestManufacturer
class ValidateDriver:
driver = ElementTree
def main(argv):
parser = argparse.ArgumentParser(description='Validation.')
parser.add_argument('-d', '--driver', help='Path and file name xml file', required=True)
parser.add_argument('-v', '--verbosity',
help='Verbosity for test output. 1 for terse, 2 for verbose. Default is verbose',
default=2, type=int)
#args = parser.parse_args()
args = r'C:\test.c4i'
#print ("Validate Driver: %s" % args.driver)
#print ("Verbosity Level: %s" % args.verbosity)
ValidateDriver.driver = ElementTree.parse(r'C:\test.c4i')
loader = unittest.TestLoader()
suite = loader.loadTestsFromModule(Tests.TestManufacturer)
runner = unittest.TextTestRunner(verbosity=2) # TODO Remove this...
# TODO Uncomment this...
runner = unittest.TextTestRunner(verbosity=args.verbosity)
result = runner.run(suite)
if __name__ == "__main__":
main(sys.argv[1:])
Other Class, Test Manufacturer:
import unittest
import Main
manufacturer = ['']
class Tests(unittest.TestCase):
# Test to see if Manufacturer exists.
def test_manufacturer_exists(self):
for m in Main.ValidateDriver.driver.findall('./manufacturer'):
print m.text
Producing the following error:
C:\Python27\python.exe C:\Users\test\PycharmProjects\Validator\Main.py
Traceback (most recent call last):
File "C:\Users\test\PycharmProjects\Validator\Main.py", line 22, in <module>
class ValidateDriver:
File "C:\Users\test\PycharmProjects\Validator\Main.py", line 65, in ValidateDriver
main(sys.argv[1:])
File "C:\Users\test\PycharmProjects\Validator\Main.py", line 36, in main
ValidateDriver.driver = ElementTree.parse(r'C:\test.c4i')
NameError: global name 'ValidateDriver' is not defined
Process finished with exit code 1

The main problem seems to be that your main script is wrapped in a class. There's really no reason for this, and is quite confusing.
if __name__ == "__main__":
main_object = ValidateDriver()
main_object.main(sys.argv[1:])
This should go outside the class definition

This has nothing to do with "findall" being available. The issue is that the class itself hasn't yet been completely declared at the time you try to access it. In python, the file is read top to bottom. For example, this is not allowed:
if __name__ == "__main__":
f()
def f():
...
The call to f must happen at the bottom of the file after it is declared.
What you're doing with ValidateDriver is similar, because the class isn't defined until the statements directly in its body are executed (this is different from functions, whose bodies of course aren't executed until they are called). You call main(sys.argv[1:]) inside the class body, which in turn tries to access ValidateDriver.driver, which doesn't exist yet.
Preferably, the main function, as well as the code which calls it, should be outside the class. As far as I can tell, the class doesn't need to exist at all (this isn't C# or Java -- you can put code directly at the module level without a class container). If you insist on putting it in a class as a static method, it must be defined as a class method:
#classmethod
def main(cls, argv):
...
which can then be called (outside the class definition) like:
ValidateDriver.main(sys.argv[1:])
But I stress that this is non-standard and should not be necessary.