Question: Is there a way to make a function object in python using strings?
Info: I'm working on a project which I store data in a sqlite3 server backend. nothing to crazy about that. a DAL class is very commonly done through code generation because the code is so incredibly mundane. But that gave me an idea. In python when a attribute is not found, if you define the function __getattr__ it will call that before it errors. so the way I figure it, through a parser and a logic tree I could dynamically generate the code I need on its first call, then save the function object as a local attrib. for example:
DAL.getAll()
#getAll() not found, call __getattr__
DAL.__getattr__(self,attrib)#in this case attrib = getAll
##parser logic magic takes place here and I end up with a string for a new function
##convert string to function
DAL.getAll = newFunc
return newFunc
I've tried the compile function, but exec, and eval are far from satisfactory in terms of being able to accomplish this kind of feat. I need something that will allow multiple lines of function. Is there another way to do this besides those to that doesn't involve writing the it to disk? Again I'm trying to make a function object dynamically.
P.S.: Yes, I know this has horrible security and stability problems. yes, I know this is a horribly in-efficient way of doing this. do I care? no. this is a proof of concept. "Can python do this? Can it dynamically create a function object?" is what I want to know, not some superior alternative. (though feel free to tack on superior alternatives after you've answered the question at hand)
The following puts the symbols that you define in your string in the dictionary d:
d = {}
exec "def f(x): return x" in d
Now d['f'] is a function object. If you want to use variables from your program in the code in your string, you can send this via d:
d = {'a':7}
exec "def f(x): return x + a" in d
Now d['f'] is a function object that is dynamically bound to d['a']. When you change d['a'], you change the output of d['f']().
can't you do something like this?
>>> def func_builder(name):
... def f():
... # multiline code here, using name, and using the logic you have
... return name
... return f
...
>>> func_builder("ciao")()
'ciao'
basically, assemble a real function instead of assembling a string and then trying to compile that into a function.
If it is simply proof on concept then eval and exec are fine, you can also do this with pickle strings, yaml strings and anything else you decide to write a constructor for.
Related
I have a string stored in a database stands for a class instance creation for example module1.CustomHandler(filename="abc.csv", mode="rb"), where CustomHandler is a class defined in module1.
I would like to evaluate this string to create a class instance for a one time use. Right now I am using something like this
statement = r'module1.CustomHandler(filename="abc.csv", mode="rb")' # actually read from db
exec(f'from parent.module import {statement.split(".")[0]}')
func_or_instance = eval(statement) # this is what I need
Only knowledgable developers can insert such records into database so I am not worried about eval some unwanted codes. But I've read several posts saying eval is unsafe and there is always a better way. Is there a way I can achieve this without using eval?
You might want to take a look at the ast Python module, which stands for abstract syntax trees. It's mainly used when you need to process the grammar of the programming language, work with code in string format, and so much more functions available in the official documentation.
In this case eval() function looks like the best solution, clear and readable, but safe only under certain conditions. For example, if you try to evaluate a code that contains a class not implemented in the code, it will throw an exception. That's the main reason why eval is sometimes unsafe.
I am trying to make an interpreter of code written in some language in python and currently stuck on interpreting functions. It seems there is a way of creating classes dynamically with something like MyClass = type("MyClass", (object, ), dict()) but I can't find a way of creating functions. I have an idea of direct line to line translation of code in python code and execution but that's not really what I want to do. So is there a way to create functions dynamically or the best I can get is something like:
foo_code = compile('def foo(): return "bar"', "<string>", "exec")
foo_func = FunctionType(foo_code.co_consts[0], globals(), "foo")
with need of translation?
Just found out the answer. The best way to dynamically create the function is to not create it at all. It kind of conflicts with my question but still solves the problem. The idea is to put a structure of a function as key in a dictionary and a line of the beginning of the function as a value. So now it is possible by structure of the function find it, execute and then return to normal execution by saving the last line of Main in advance.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Python dynamic function creation with custom names
I have written a little script to determine whether what I wanted to do is possible. It is.
My goal is to dynamically (at runtime) create functions (or methods) which have names based on a list of arbitrary size (size of the list = number of functions dynamically created).
All the functions do the same (for now), they just print their arguments.
The following code does exactly what I want, BUT, it is not clean and very brute-force. I'm trying to figure out if there is a better way to do this.
class Binder:
def __init__(self, test_cases):
""""
test_cases: a list of function/method names.
length of test_case = number of methods created.
"""
for test_case in test_cases:
#construct a code string for creating a new method using "exec"
func_str = "def "
func_str += test_case
func_str += "(*args):"
func_str += "\n\t"
func_str += "for arg in args:"
func_str += "\n\t\t"
func_str += "print arg"
func_str += "\n"
"""
For example, func_str for test_cases[0]= "func1" is simply:
def func1(*args):
for arg in args:
print arg
"""
#use exec to define the function
exec(func_str)
#add the function as a method to this class
# for test_cases[0] = "func1", this is: self.func1 = func1
set_self = "self." + test_case + " = " + test_case
exec(set_self)
if __name__ == '__main__':
#this list holds the names of the new functions to be created
test_cases = ["func1", "func2", "func3", "func4"]
b = Binder(test_cases)
#simply call each function as the instant's attributes
b.func1(1)
b.func2(1, 3, 5)
b.func4(10)
Output is:
1
1
3
5
10
As expected.
update the content of the function would not simply be a for loop printing the args, it would do something more meaningful. I get the exact result I want from the piece of code above, just wondering if there is a better way of doing it.
update I'm tying two ends of a much bigger module. One end determines what the test cases are and among other things, populates a list of the test cases' names. The other end is the functions themselves, which must have 1:1 mapping with the name of the test case. So I have the name of the test cases, I know what I want to do with each test case, I just need to create the functions that have the name of the test cases. Since the name of the test cases are determined at runtime, the function creation based on those test cases must be at runtime as well. The number of test cases is also determined at runtime.
Is there a better way to do this??
Any and all suggestions welcome.
Thanks in advance.
Mahdi
In Python this is the most reasonable approach for generic metaprogramming.
If you need just some constants in the code then however a closure may do the trick... for example:
def multiplier(k):
"Returns a function that multiplies the argument by k"
def f(x):
return x*k
return f
For arbitrary code generation Python has an ast module and you can in theory both inspect or create functions using that approach. However in Python code is hard to represent and manipulate that way so normally the approach is to just do everything at runtime instead of compiling specific functions. When you really can get an advantage you can use eval (to get a lambda) or exec. The ast module is used mostly just for inspection.
Writing code that generates code is what is called metaprogramming and Python is not very friendly to this approach.
You may have heard that C++ claims support for metaprogramming, but indeed it's only template-based metaprogramming where you can substitute types or constants into a fixed structure. You can play some tricks using "advanced metaprogramming" like the SFINAE rule but they're nothing more than trickery.
Python doesn't need templates because the language is not statically typed and you have closures (C++ is statically typed and there are no closures), but Python doesn't help with a general metaprogramming approach (i.e. writing general code that generates or manipulates code).
If you're interested in metaprogramming then the language of choice is probably Common Lisp. Writing code that writes code is not something special for Lisp and while it's of course more difficult to write a macro (a function that writes code) than a regular run-time function still with Lisp the difficulties are mostly essential (the problem is indeed harder) and not artificial because of language limitations.
There's an old joke among lispers that goes more or less like "I'm writing code that writes code that writes code that writes code that people pays me for". Metaprogramming is indeed just the first step, after that you have meta-meta-programming (writing code that writes code generators) and so on. Of course things gets harder and harder (but once again because the problem is harder, not because of arbitrarily imposed limitations...).
First of all, this is probably a really bad idea. To see why, read the comments and the other answers.
But to answer your question, this should work (although it's a hack and might have strange side effects):
from types import MethodType
def myfunc(self, *args):
for arg in args:
print arg
class Binder(object):
def __init__(self, test_cases):
for test_case in test_cases:
method = MethodType(myfunc, self, Binder)
setattr(self, test_case, method)
In this case, the reference to the function is hardcoded, but you can of course pass it as an argument too.
This is the output:
>>> b = Binder(['a', 'b'])
>>> b.a(1, 2, 3)
1
2
3
>>> b.b('a', 20, None)
a
20
None
I have used occasionally (lambda x:<code>)(<some input>) in python, to preserve my namespace's (within the global namespace or elsewhere) cleanliness. One issue with the lambda solution is that it is a very limiting construct in terms of what it may contain.
Note: This is a habit from javascript programming
Is this a recommended way of preserving namespace? If so, is there a better way to implement a self-executing function?
Regarding the second half of the question
is there a better way to implement a self-executing function?
The standard way (<function-expression>)() is not possible in Python, because there is no way to put a multi-line block into a bracket without breaking Python's fundamental syntax. Nonetheless, Python do recognize the need for using function definitions as expressions and provide decorators (PEP318) as an alternative. PEP318 has an extensive discussion on this issue, in case someone would like to read more.
With decorators, it would be like
evalfn = lambda f: f()
#evalfn
def _():
print('I execute immediately')
Although vastly different syntatically, we shall see that it really is the same: the function definition is anonimous and used as an expression.
Using decorator for self-excuting functions is a bit of overkill, compared to the let-call-del method shown below. However, it may worth a try if there are many self-execution functions, a self-executing function is getting too long, or you simply don't bother naming these self-executing functions.
def f():
print('I execute immediately')
f()
del f
For a function A that will be called only in a specific function B, you can define A in B, by which I think the namespace will not be polluted. e.g.,
Instead of :
def a_fn():
//do something
def b_fn():
//do something
def c_fn():
b_fn()
a_fn()
You can:
def c_fn():
def a_fn():
//do something
def b_fn():
//do something
b_fn()
a_fn()
Though I'm not sure if its the pythonic way, I usually do like this.
You don't do it. It's a good in JavaScript, but in Python, you haven either lightweight syntax nor a need for it. If you need a function scope, define a function and call it. But very often you don't need one. You may need to pull code apart into multiple function to make it more understandable, but then a name for it helps anyway, and it may be useful in more than one place.
Also, don't worry about adding some more names to a namespace. Python, unlike JavaScript, has proper namespaces, so a helper you define at module scope is not visible in other files by default (i.e. unless imported).
Suppose I need to create my own small DSL that would use Python to describe a certain data structure. E.g. I'd like to be able to write something like
f(x) = some_stuff(a,b,c)
and have Python, instead of complaining about undeclared identifiers or attempting to invoke the function some_stuff, convert it to a literal expression for my further convenience.
It is possible to get a reasonable approximation to this by creating a class with properly redefined __getattr__ and __setattr__ methods and use it as follows:
e = Expression()
e.f[e.x] = e.some_stuff(e.a, e.b, e.c)
It would be cool though, if it were possible to get rid of the annoying "e." prefixes and maybe even avoid the use of []. So I was wondering, is it possible to somehow temporarily "redefine" global name lookups and assignments? On a related note, maybe there are good packages for easily achieving such "quoting" functionality for Python expressions?
I'm not sure it's a good idea, but I thought I'd give it a try. To summarize:
class PermissiveDict(dict):
default = None
def __getitem__(self, item):
try:
return dict.__getitem__(self, item)
except KeyError:
return self.default
def exec_with_default(code, default=None):
ns = PermissiveDict()
ns.default = default
exec code in ns
return ns
You might want to take a look at the ast or parser modules included with Python to parse, access and transform the abstract syntax tree (or parse tree, respectively) of the input code. As far as I know, the Sage mathematical system, written in Python, has a similar sort of precompiler.
In response to Wai's comment, here's one fun solution that I've found. First of all, to explain once more what it does, suppose that you have the following code:
definitions = Structure()
definitions.add_definition('f[x]', 'x*2')
definitions.add_definition('f[z]', 'some_function(z)')
definitions.add_definition('g.i', 'some_object[i].method(param=value)')
where adding definitions implies parsing the left hand sides and the right hand sides and doing other ugly stuff. Now one (not necessarily good, but certainly fun) approach here would allow to write the above code as follows:
#my_dsl
def definitions():
f[x] = x*2
f[z] = some_function(z)
g.i = some_object[i].method(param=value)
and have Python do most of the parsing under the hood.
The idea is based on the simple exec <code> in <environment> statement, mentioned by Ian, with one hackish addition. Namely, the bytecode of the function must be slightly tweaked and all local variable access operations (LOAD_FAST) switched to variable access from the environment (LOAD_NAME).
It is easier shown than explained: http://fouryears.eu/wp-content/uploads/pydsl/
There are various tricks you may want to do to make it practical. For example, in the code presented at the link above you can't use builtin functions and language constructions like for loops and if statements within a #my_dsl function. You can make those work, however, by adding more behaviour to the Env class.
Update. Here is a slightly more verbose explanation of the same thing.