Namespace for class declaration in file executed at runtime - python

I wrote a program to allow users to give me a class name and the file that defines that class like so:
input = raw_input("Input the new classname with a space and then the file name defining the class")
class_name, file_name = input.split(" ")
I then execute the file defining the class:
execfile(file_name)
and try to create an instance of the newly defined class:
d[k] = eval(new_class)(**d[k])
However this creates the following error:
d[i] = eval(new_class)(**d[i])
File "<string>", line 1, in <module>
NameError: name 'MeasureDay' is not defined
This is after I fed in the following class definition that was executed:
class MeasureDay:
print("defining measureday")
def __init__(self, **kargs):
self.day = kwargs.get('day', -1)
self.measurement_value = kwargs('measurement_value', -1)
def __str__(self):
return "here is a custom defined string for MeasureDay"
I know the class definition was executed because I see the logged statement "defining measureday" as desired.
What am I doing wrong? I am guessing this has to do with some namespace convention. Do I need to define a namespace when I define classes like so, and if so how would I do it. Otherwise, what is the problem?
Thanks for any suggestions.

eval and execfile are to be avoided, and are unnecessary here. importlib.import_module can import the module file from a string, and you can then use getattr to get the class for instantiation.
class_name, file_name = input.split(" ")
mod = import_module(file_name)
cls = getattr(mod, class_name)
d[k] = cls(**d[k])

Related

Python3.x self as parameter in a method call to an other class

I am trying to call a method on an other class and give the called class a reference of the current class along with some other parameters. But somehow it takes the self given as a parameter as the self of the called class.
Let me show you:
import os, sys
from wsPart import wsPart
class thermo(wsPart):
functional = False ## see line 8
file = '/sys/bus/w1/devices/28-00000833e8ff/w1_slave'
def __init__(self, name, logger):
super().__init__(name, logger)
functional = True
def read(self):
fileobject = open(self.file)
filecontent = fileobject.read()
fileobject.close()
self.logger.writeLog(self,"Completed Meassurement") ##Problem on this line
return filecontent
So I call the class logger and the method writeLog on it. Giving the Parameters message and a reference of the class thermo (self).
import datetime
from wsPart import wsPart
class logger():
logfile = "/var/log/wheaterstation.log"
name = "Logger"
def writeLog(self, sender, message):
conn = open(self.logfile, "w")
now = str(datetime.datetime.now().isoformat())
conn.write("[" + now + "]" + " (" + sender.getName() + "): " + message + "\n") ##Problem on this line
conn.close()
As you can see I put the parameters self because its a method that belongs to a class, the sender should be the reference to the class thermo that was passed as self in the thermo class. Lastly there is the message which was passed in the thermo class as well.
But this just gives me the error:
Traceback (most recent call last):
File "scrLib/wsControl.py", line 61, in <module>
controller = controller()
File "scrLib/wsControl.py", line 22, in __init__
self.thermo = thermo("Thermometer", logger)
File "/home/joco/git/wheaterstation/scrLib/thermo.py", line 10, in __init__
super().__init__(name, logger)
File "/home/joco/git/wheaterstation/scrLib/wsPart.py", line 8, in __init__
self.logger.writeLog(self, "created")
TypeError: writeLog() missing 1 required positional argument: 'message'
So it seems that the self parameter which was passed in the thermo class is interpeted as the self of the class logger which gets it all mixed up.
Can you guys help me here?
The full code + additional comments can be viewed Here
Edit:
Both the logger and the thermo class get initialized in the file wsPart.py:
class controller():
name = ""
logger = None
thermo = None
dbConnector = None
def __init__(self):
##THis created the controller and all the other objects
self.name = "Controller"
##Create Objects
self.logger = logger()
self.logger.writeLog(self,"logger created") ##This line Works
self.thermo = thermo("Thermometer", logger)
self.dbConnector = dbConnector("DBConnector",logger)
yes, bad idea to call the instance and the class name the same. Here:
self.logger = logger()
self.logger.writeLog(self,"logger created") ##This line Works
self.thermo = thermo("Thermometer", logger)
self.dbConnector = dbConnector("DBConnector",logger)
You're passing the class itself to your constructors. So the methods are seen as static/expect one more parameter. You need to change the 2 last lines to pass the instance you just created:
self.thermo = thermo("Thermometer", self.logger)
self.dbConnector = dbConnector("DBConnector", self.logger)
more importantly, you need to use different names for classes and instances of the same objects to avoid that confusion (python convention for class names is starting each word with upper case (camelcase) ex: Logger. Other languages don't use that convention, but python is a lot about conventions).
With a different name you'd have gotten a NameError exception and you would have fixed the error yourself.
Aside: don't "initialize" members like this in the class definition:
name = ""
logger = None
thermo = None
dbConnector = None
those are creating class members, not instance members. Remove those, and let __init__ create instance members like you're currently doing. __init__ is called no matter what, and those lines above just add to the confusion (except for some corner cases, only constants should be declared that way)
Totally unrelated but code in comments is unreadable so I post this as an answer:
this does not work as you seem to expect:
class Whatever():
functional = False ## see line 8
def __init__(self, name, logger):
super().__init__(name, logger)
functional = True
Python has no "implied this" sor here in __init__ you're not creating an instance attribute but a local variable. You want self.functional = True
Make sure you close files
def read(self):
fileobject = open(self.file)
filecontent = fileobject.read()
fileobject.close()
If anything wrong happens between open() and fileobject.close(), the file is not garanteed to be properly closed. You want eiher a try/finally block ie
f = open(path)
try:
do_something_with(f)
finally:
f.close()
or much better a with block:
with open(path) as f:
do_something_with(f)
which will ensure the file is closed whatever happens.
write mode truncates the file
def writeLog(self, sender, message):
conn = open(self.logfile, "w")
now = str(datetime.datetime.now().isoformat())
conn.write("[" + now + "]" + " (" + sender.getName() + "): " + message + "\n") ##Problem on this line
conn.close()
as documented, opening a file in write mode truncates the file. You probably want the "append" mode instead here.
Don't reinvent the squared wheel when there's a round one already
Logging is not as trivial as just writing to a file (concurrency issues, need to send the log message to some other destination, logging levels etc), and even if you don't need more (at least the moment) your solution is quite inefficient (opening a file is expensive).
Python has a very comprehensive logging package in it's standard lib. I wholefully agree that it requires a bit of learning to configure and use properly but that's still a huge win compared to the time you'll spend trying to make a naive half-backed custom implementation works properly on production, AND this is a knowledge that you will need for just any serious project anyway.

python default configuration reuse of variables

class DefaultConfig(object):
class S3(object):
DATA_ROOT = 's3://%(bucket_name)s/NAS'
DATA_LOCATION = '{}/%(instrument_id)s/%(run_id)s'.format(DefaultConfig.S3.DATA_ROOT)
The code above gives me the following error.
File "./s3Utils.py", line 5, in <module>
from InfraConfig import InfraConfig as IC
File "/opt/src/datasource/src/main/python/util/InfraConfig.py", line 4, in <module>
class DefaultConfig(object):
File "/opt/src/datasource/src/main/python/util/InfraConfig.py", line 6, in DefaultConfig
class S3(object):
File "/opt/src/datasource/src/main/python/util/InfraConfig.py", line 14, in S3
DATA_LOCATION = '{}/%(instrument_id)s/%(run_id)s'.format(DefaultConfig.S3.DATA_ROOT)
NameError: name 'DefaultConfig' is not defined
Why is it unable to find DefaultConfig.S3.DATA_ROOT
Also, this is my attempt at writing structured configuration with reuse of values of DefaultConfig. Is there a way to do this without writing a yml file?
It is unable to find the DefaultConfing because DefaultConfig is not defined at the moment S3 is created.
Remember that class are objects. Because there are objects, that means they need to be instantiate. Python instantiate a class at the end of its definition, and therefore register it in the globals. Because the class definition is not finished, you can't use the DefaultConfig name.
You should use it without any prefixes:
class DefaultConfig(object):
class S3(object):
DATA_ROOT = 's3://%(bucket_name)s/NAS'
DATA_LOCATION = '{}/%(instrument_id)s/%(run_id)s'.format(DATA_ROOT)
print DefaultConfig.S3.DATA_LOCATION
returns:
> s3://%(bucket_name)s/NAS/%(instrument_id)s/%(run_id)s

importing module causes TypeError: module.__init__() takes at most 2 arguments (3 given)

Please don't mark as duplicate, other similar questions did not solve my issue.
This is my setup
/main.py
/actions/ListitAction.py
/actions/ViewAction.py
Main.py:
from actions import ListitAction, ViewAction
ListitAction.py:
class ListitAction(object):
def __init__(self):
#some init behavior
def build_uri():
return "test.uri"
ViewAction.py
from actions import ListitAction
class ViewAction(ListitAction):
def __init__(self, view_id):
ListitAction.__init__(self)
self.view_id = view_id
def build_uri():
return "test"
Running:
$ python3 main.py
The only error message I receive is:
Traceback (most recent call last):
File "/home/jlevac/workspace/project/listit.py", line 11, in <module>
from actions import ListitAction, ViewAction, CommentsAction
File "/home/jlevac/workspace/project/actions/ViewAction.py", line 3, in <module>
class ViewAction(ListitAction):
TypeError: module.__init__() takes at most 2 arguments (3 given)
Even if I try for the python3 console, I received the same error message:
$python3
from actions import ViewAction
I am new to Python, but not new to programming. I'm assuming that my error messages have to do with the import statements, but based on the message I can't really figure out what it means.
Your imports are wrong, so you're trying to inherit from the modules themselves, not the classes (of the same name) defined inside them.
from actions import ListitAction
in ViewAction.py should be:
from actions.ListitAction import ListitAction
and similarly, all other uses should switch to explicit imports of from actions.XXX import XXX (thanks to the repetitive names), e.g. from actions import ListitAction, ViewAction must become two imports:
from actions.ListitAction import ListitAction
from actions.ViewAction import ViewAction
because the classes being imported come from different modules under the actions package.
You are passing self when you don't need to, that's all.
Edit: see MSeifert's comment answer since I don't want to steal content.
If your file is in the root directory of the project then you can directly write the file name and import.
For example, if filename is Parent1.py and class name is Parent, then you would write
from Parent1 import Parent
However, if your file Parent1.py is under any folder for example:
DemoFolder -> Parent1.py- > Parent
(Folder). (File). (Class name)
Then you would have to write:
from Test.Parent1 import Parent
Creating classes and instance of variables
class Student:
# Creating a Class Variables
perc_Raise = 1.05
# Creating a constructor or a special method to initialize values
def __init__(self,firstName,lastName,marks):
self.firstName = firstName
self.lastName = lastName
self.email = firstName + "." + lastName +"#northalley.com"
self.marks = marks
def fullName(self):
return '{} {}'.format(self.firstName,self.lastName)
def apply_raise(self):
self.marks = int(self.marks * self.perc_Raise)
Creating a two instance variable for a Student class
std_1 = Student('Mahesh','Gatta',62)
std_2 = Student('Saran','D',63)
print(std_1.fullName())
print(std_1.marks)
std_1.apply_raise()
print(std_1.marks)
print(std_1.email)
print(std_1.__dict__)
print(std_2.fullName())
print(std_2.marks)
std_2.apply_raise()
print(std_2.marks)
print(std_2.email)
print(std_2.__dict__)
print(Student.__dict__)
Inheritance
class Dumb(Student):
perc_Raise = 1.10
def __init__(self,firstName,lastName,marks,prog_lang):
super().__init__(firstName,lastName,marks)
self.prog_lang = prog_lang
std_1 = Dumb('Suresh','B',51,'Python')
print(std_1.fullName())
print(std_1.marks)
std_1.apply_raise()
print(std_1.marks)
print(std_1.prog_lang)

multiprocessing and modules

I am attempting to use multiprocessing to call derived class member function defined in a different module. There seem to be several questions dealing with calling class methods from the same module, but none from different modules. For example, if I have the following structure:
main.py
multi/
__init__.py (empty)
base.py
derived.py
main.py
from multi.derived import derived
from multi.base import base
if __name__ == '__main__':
base().multiFunction()
derived().multiFunction()
base.py
import multiprocessing;
# The following two functions wrap calling a class method
def wrapPoolMapArgs(classInstance, functionName, argumentLists):
className = classInstance.__class__.__name__
return zip([className] * len(argumentLists), [functionName] * len(argumentLists), [classInstance] * len(argumentLists), argumentLists)
def executeWrappedPoolMap(args, **kwargs):
classType = eval(args[0])
funcType = getattr(classType, args[1])
funcType(args[2], args[3:], **kwargs)
class base:
def multiFunction(self):
mppool = multiprocessing.Pool()
mppool.map(executeWrappedPoolMap, wrapPoolMapArgs(self, 'method', range(3)))
def method(self,args):
print "base.method: " + args.__str__()
derived.py
from base import base
class derived(base):
def method(self,args):
print "derived.method: " + args.__str__()
Output
base.method: (0,)
base.method: (1,)
base.method: (2,)
Traceback (most recent call last):
File "e:\temp\main.py", line 6, in <module>
derived().multiFunction()
File "e:\temp\multi\base.py", line 15, in multiFunction
mppool.map(executeWrappedPoolMap, wrapPoolMapArgs(self, 'method', range(3)))
File "C:\Program Files\Python27\lib\multiprocessing\pool.py", line 251, in map
return self.map_async(func, iterable, chunksize).get()
File "C:\Program Files\Python27\lib\multiprocessing\pool.py", line 567, in get
raise self._value
NameError: name 'derived' is not defined
I have tried fully qualifying the class name in the wrapPoolMethodArgs method, but that just gives the same error, saying multi is not defined.
Is there someway to achieve this, or must I restructure to have all classes in the same package if I want to use multiprocessing with inheritance?
This is almost certainly caused by the ridiculous eval based approach to dynamically invoking specific code.
In executeWrappedPoolMap (in base.py), you convert a str name of a class to the class itself with classType = eval(args[0]). But eval is executed in the scope of executeWrappedPoolMap, which is in base.py, and can't find derived (because the name doesn't exist in base.py).
Stop passing the name, and pass the class object itself, passing classInstance.__class__ instead of classInstance.__class__.__name__; multiprocessing will pickle it for you, and you can use it directly on the other end, instead of using eval (which is nearly always wrong; it's code smell of the strongest sort).
BTW, the reason the traceback isn't super helpful is that the exception is raised in the worker, caught, pickle-ed, and sent back to the main process and re-raise-ed. The traceback you see is from that re-raise, not where the NameError actually occurred (which was in the eval line).

python, instance dynamic module.class from given path package and call dynamic method

package_path = '/home/foo/bar/'
module_class = 'hello.World'
method = 'something'
first, i want to import the hello module what it is inside of the package_path (/home/foo/bar/)
then, i must instantiate the class World at this module
and finally i should run the something() method on the new world_instance_object
any idea for do that?
Thank you so much!
To avoid having to manipulate sys.path and to keep all the extra modules you import from cluttering up sys.modules, I'd suggest using execfile() for this. Something like this (untested code):
import os.path
def load_and_call(package_path, module_class, method_name):
module_name, class_name = module_class.split(".")
module_globals = {}
execfile(os.path.join(package_path, module_name + ".py"), module_globals)
return getattr(module_globals[class_name](), method_name, lambda: None)()
You could add some code to cache the parsed objects using a dictionary, if a file will be used more than once (again, untested):
def load_and_call(package_path, module_class, method_name, _module_cache={}):
module_name, class_name = module_class.split(".")
py_path = os.path.join(package_path, module_name + ".py")
module_globals = _module_cache.setdefault(py_path, {})
if not module_globals:
execfile(py_path, module_globals)
return getattr(module_globals[class_name](), method_name, lambda: None)()

Categories

Resources