Instantiate a class from an abstract syntax tree - python

I have a .py file that contains the definition of a class. I cannot change that file.
I have parsed the contents of that file and gotten an abstract syntax tree (ast) for it, more specifically I've got an instance of ast.ClassDef().
How can I instantiate that class from the ast.ClassDef()?
Here's a contrived demo showing what I want to do:
import ast
src = """
class MyClass():
def __init__():
self.a = 1
self.b = 2
self.thesum = self.a + self.b
"""
p = ast.parse(src)
class_ = [node for node in ast.walk(p) if isinstance(node, ast.ClassDef)][0]
class_ #I want to instantiate the class that is referenced by class_
Googling suggests that maybe compile() could help but I couldn't get that to work as it seems to expect executable code rather than a class definition. In other words, this didn't do very much:
exec(compile(class_.body, "fakemodule", 'exec'))
it throws an error:
TypeError: expected a readable buffer object
So, how can I instantiate that class?

Related

Python class attribute 'is not defined' when referenced by another class attribute

Using the following, I am able to successfully create a parser and add my arguments to self._parser through the __init()__ method.
class Parser:
_parser_params = {
'description': 'Generate a version number from the version configuration file.',
'allow_abbrev': False
}
_parser = argparse.ArgumentParser(**_parser_params)
Now I wish to split the arguments into groups so I have updated my module, adding some classes to represent the argument groups (in reality there are several subclasses of the ArgumentGroup class), and updating the Parser class.
class ArgumentGroup:
_title = None
_description = None
def __init__(self, parser) -> ArgumentParser:
parser.add_argument_group(*self._get_args())
def _get_args(self) -> list:
return [self._title, self._description]
class ArgumentGroup_BranchType(ArgumentGroup):
_title = 'branch type arguments'
class Parser:
_parser_params = {
'description': 'Generate a version number from the version configuration file.',
'allow_abbrev': False
}
_parser = argparse.ArgumentParser(**_parser_params)
_argument_groups = [cls(_parser) for cls in ArgumentGroup.__subclasses__()]
However, I'm now seeing an error.
Traceback (most recent call last):
...
File "version2/args.py", line 62, in <listcomp>
_argument_groups = [cls(_parser) for cls in ArgumentGroup.__subclasses__()]
NameError: name '_parser' is not defined
What I don't understand is why _parser_params do exist when they are referred by another class attribute, but _parser seemingly does not exist in the same scenario? How can I refactor my code to add the parser groups as required?
This comes from the confluence of two quirks of Python:
class statements do not create a new local scope
List comprehensions do create a new local scope.
As a result, the name _parser is in a local scope whose closest enclosing scope is the global scope, so it cannot refer to the about-to-be class attribute.
A simple workaround would be to replace the list comprehension with a regular for loop.
_argument_groups = []
for cls in ArgumentGroup.__subclasses()__:
_argument_groups.append(cls(_parser))
(A better solution would probably be to stop using class attributes where instance attributes make more sense.)

how to inherit __init__ attributes from parent class to __init__ in the child class?

I'm trying to inherit the attributes from parent class:
class Human:
def __init__(self,name,date_of_birth,gender,nationality):
self.name = name
self.date_of_birth = date_of_birth
self.gender = gender
self.nationality = nationality
To the child class Student:
class Student(Human):
def __init__(self,university,std_id,first_year_of_registration,study_program):
super(Student,self).__init__(name,date_of_birth,gender,nationality)
self.university = university
self.std_id = std_id
self.first_year_of_registration = first_year_of_registration
self.study_program = study_program
def select(c):
pass
def drop(c):
pass
def __str__(self):
return 'Student ID {self.std_id},and student name is {self.name} and his study program is {self.study_program}'.format(self=self)
So i can create an object using both (parent,student) attributes like So:
s = Student('mark','2000-1-1','M','country','school',2017159,2017,'SE')
then i can:
print(s)
and invoke __str__ function like so:
Student ID 2017159,and student name is mark and his study program is SE
but whenever i create the object it give me error:
Traceback (most recent call last):
Python Shell, prompt 7, line 1
builtins.TypeError: __init__() takes 5 positional arguments but 9 were
given
i'm new to python. So i searched the internet and tried to do like some of the inheritance examples, like so:
Human.__init__()
but nothing worked, also i checked many answers here but none of them solved my problem.
hope anyone can help and point out what is wrong here.
def __init__(self,university,std_id,first_year_of_registration,study_program)
but
Student('mark','2000-1-1','M','country','school',2017159,2017,'SE')
?
Well, you see, Python is not a mind-reader. You need to tell it what you want.
Right now Python wants all and only university,std_id,first_year_of_registration,study_program, nothing more, nothing less.
This means, you need to tell it to get other elements:
class Student(Human):
def __init__(self,name,date_of_birth,gender,nationality, university,std_id,first_year_of_registration,study_program):
...
or by saying "and get some other elements". But remember! Those other elements must be at the end of the constructor!
class Student(Human):
def __init__(self,university,std_id,first_year_of_registration,study_program, *args, **kwargs):
super(Student,self).__init__(*args, **kwargs)
...
It basically catches all positional arguments (args) and keyword arguments (kwargs) and passes them further.
As I said, it only takes argument from the end, so you need to do:
Student('school',2017159,2017,'SE', 'mark','2000-1-1','M','country')
The way interpreter sees it, it will take 1st argument and put it in university, then 2nd in std_id... up to study_program, and then put the rest in the list args.

When using Python classes as program configuration structures (which includes inherited class attributes), a good way to save/restore?

Let's say I have a (simplified) class as below. I am using it for a program configuration (hyperparameters).
# config.py
class Config(object): # default configuration
GPU_COUNT = 1
IMAGES_PER_GPU = 2
MAP = {1:2, 2:3}
def display(self):
pass
# experiment1.py
from config import Config as Default
class Config(Default): # some over-written configuration
GPU_COUNT = 2
NAME='2'
# run.py
from experiment1 import Config
cfg = Config()
...
cfg.NAME = 'ABC' # possible runtime over-writing
# Now I would like to save `cfg` at this moment
I'd like to save this configuration and restore later. The member functions must be out of concern when restoring.
1. When I tried pickle:
import pickle
with open('cfg.pk', 'rb') as f: cfg = pickle.load(f)
##--> AttributeError: Can't get attribute 'Config' on <module '__main__'>
I saw a solution using class_def of Config, but I wish I can restore the configuration without knowing the class definition (eg, export to dict and save as JSON)
2. I tried to convert class to dict (so that I can export as JSON)
cfg.__dict__ # {'NAME': 'ABC'}
vars(cfg) # {'NAME': 'ABC'}
In both cases, it was difficult to access attributes. Is it possible?
The question's title is "how to convert python class to dict", but I suspect you are really just looking for an easy way to represent (hyper)parameters.
By far the easiest solution is to not use classes for this. I've seen it happen on some machine learning tutorials, but I consider it a pretty ugly hack. It breaks some semantics about classes vs objects, and the difficulty pickling is a result from that. How about you use a simple class like this one:
class Params(dict):
__getattr__ = dict.__getitem__
__setattr__ = dict.__setitem__
__delattr__ = dict.__delitem__
def __getstate__(self):
return self
def __setstate__(self, state):
self.update(state)
def copy(self, **extra_params):
return Params(**self, **extra_params)
It can do everything the class approach can. Predefined configs are then just objects you should copy before editing, as follows:
config = Params(
GPU_COUNT = 2,
NAME='2',
)
other_config = config.copy()
other_config.GPU_COUNT = 4
Or alternatively in one step:
other_config = config.copy(
GPU_COUNT = 4
)
Works fine with pickle (although you will need to have the Params class somewhere in your source), and you could also easily write load and save methods for the Params class if you want to use JSON.
In short, do not use a class for something that really is just an object.
Thankfully, #evertheylen's answer was great to me. However, the code returns error when p.__class__ = Params, so I slightly changed as below. I think it works in the same way.
class Params(dict):
__getattr__ = dict.__getitem__
__setattr__ = dict.__setitem__
__delattr__ = dict.__delitem__
def __getstate__(self):
return self
def __setstate__(self, state):
self.update(state)
def copy(self, **extra_params):
lhs = Params()
lhs.update(self)
lhs.update(extra_params)
return lhs
and you can do
config = Params(
GPU_COUNT = 2,
NAME='2',
)
other_config = config.copy()
other_config.GPU_COUNT = 4

Python: Static method inside private inner Enum class

I'm having trouble implementing an inner private enum class called: "LineCode" inside a class named Parser.
LineCode: Private Enum class that defines 6 types of general possible line of codes. I use Enum instantiation to send a Regex Pattern and compile it in the constructor, __init__, and then holds the Regex Matcher as a class variable.
Parser: Parses a programming language, irrelevant what language. Parser is using LineCode to identify the lines and proceed accordingly.
Problem: I can't access the enum members of __LineCode from a Static method.
I wish to have a Static method inside __LineCode, "matchLineCode(line)" that receives a string from the Parser, it then iterates over the Enum members in the following logic:
If Match is found: Return the enum
If no more enums left: Return None
It doesn't seem trivial, I can't access the enum members to do this.
Attempts: I tried iterating over the enums using:
__LineCode.__members__.values()
Parser.__lineCode.__members__.values()
Both failed since it can't find __lineCode.
Ideally: LineCode class must be private, and not be visible to any other class importing the Parser. Parser must use the static method that LineCode class provides to return the Enum. I am willing to accept any solution that solves this issue or one that mimics this behavior.
I omitted some of the irrelevant Parser methods to improve readability.
Code:
class Parser:
class __LineCode(Enum):
STATEMENT = ("^\s*(.*);\s*$")
CODE_BLOCK = ("^\s*(.*)\s*\{\s*$")
CODE_BLOCK_END = ("^\s*(.*)\s*\}\s*$")
COMMENT_LINE = ("^\s*//\s*(.*)$")
COMMENT_BLOCK = ("^\s*(?:/\*\*)\s*(.*)\s*$")
COMMENT_BLOCK_END = ("^\s*(.*)\s*(?:\*/)\s*$")
BLANK_LINE = ("^\s*$")
def __init__(self, pattern):
self.__matcher = re.compile(pattern)
#property
def matches(self, line):
return self.__matcher.match(line)
#property
def lastMatch(self):
try:
return self.__matcher.groups(1)
except:
return None
#staticmethod
def matchLineCode(line):
for lineType in **???**:
if lineType.matches(line):
return lineType
return None
def __init__(self, source=None):
self.__hasNext = False
self.__instream = None
if source:
self.__instream = open(source)
def advance(self):
self.__hasNext = False
while not self.__hasNext:
line = self.__instream.readline()
if line == "": # If EOF
self.__closeFile()
return
lineCode = self.__LineCode.matchLineCode(line)
if lineCode is self.__LineCode.STATEMENT:
pass
elif lineCode is self.__LineCode.CODE_BLOCK:
pass
elif lineCode is self.__LineCode.CODE_BLOCK_END:
pass
elif lineCode is self.__LineCode.COMMENT_LINE:
pass
elif lineCode is self.__LineCode.COMMENT_BLOCK:
pass
elif lineCode is self.__LineCode.COMMENT_BLOCK:
pass
elif lineCode is self.__LineCode.BLANK_LINE:
pass
else:
pass # TODO Invalid file.
I already implemented it in Java, I want to reconstruct the same thing in Python:
private enum LineCode {
STATEMENT("^(.*)" + Syntax.EOL + "\\s*$"), // statement line
CODE_BLOCK("^(.*)" + Syntax.CODE_BLOCK + "\\s*$"), // code block open line
CODE_BLOCK_END("^\\s*" + Syntax.CODE_BLOCK_END + "\\s*$"), // code block close line
COMMENT_LINE("^\\s*" + Syntax.COMMENT + "(.*+)$"), // comment line
BLANK_LINE("\\s*+$"); // blank line
private final static int CONTENT_GROUP = 1;
private Pattern pattern;
private Matcher matcher;
private LineCode(String regex) {
pattern = Pattern.compile(regex);
}
boolean matches(String line) {
matcher = pattern.matcher(line);
return matcher.matches();
}
String lastMatch() {
try {
return matcher.group(CONTENT_GROUP);
} catch (IndexOutOfBoundsException e) {
return matcher.group();
}
}
static LineCode matchLineCode(String line) throws UnparsableLineException {
for (LineCode lineType : LineCode.values())
if (lineType.matches(line)) return lineType;
throw new UnparsableLineException(line);
}
Thanks.
You could change the staticmethod to a classmethod, that way the first argument passed to matchLineCode would be the __lineCode class and you would be able to iterate over it
Edit
I've decided to add a more detailed explanation as to why the matchLineCode using the #staticmethod decorator was unable to see the __lineCode class. First I recommend you read some questions posted on SO that talk about the difference between static and class methods. The main difference is that the classmethod is aware of the Class where the method is defined, while the staticmethod is not. This does not mean that you are unable to see the __lineCode class from the staticmethod, it just means that you will have to do some more work to do so.
The way in which you organized your code, the class __lineCode is a class attribute of class Parser. In python, methods are always public, there are no private or protected class members as in Java. However, the double underscore at the beginning of a class attribute's name (or an instance's attribute's name) mean that the name will be mangled with the class name. This means that any function defined outside of the class Parser could access the __lineCode class as
Parser._Parser__lineCode
This means that using the #staticmethod decorator you could iterate over the __lineCode by doing
#staticmethod
def matchLineCode(line):
for lineType in Parser._Parser__lineCode:
if lineType.matches(line):
return lineType
return None
However, it is much more readable and, in my opinion, understandable to use the #classmethod decorator to allow the function to be aware of the __lineCode class.

Cannot call xml.dom.minidom.parse inside class

I am unable to call xml.dom.minidom.parse() within my class
As a sheer example,
class XmlReader:
def __init__(self, xml):
self.xml = xml
DOMTree = xml.dom.minidom.parse("test.xml")
xmlReader = XmlReader("test.xml")
Throws
File "handler2.py", line 10, in ?
xmlReader = XmlReader("test.xml")
File "handler2.py", line 8, in __init__
DOMTree = xml.dom.minidom.parse("test.xml")
AttributeError: 'str' object has no attribute 'dom'
However outside I am able to call xml.dom.minidom.parse just fine.
What do I need to change in order to be able to call the function within my XmlReader class?
Inside your constructor, xml refers to the parameter xml instead of the module xml. This is called shadowing. Choose a different name for one of them.
import xml as xml_module
or
from xml.dom import minidom
or
def __init__(self, xml_data):

Categories

Resources