Develop Reference

Using compile() with an expression with a function inside of it

Let's say I have something like this:
expr="
def something():
return True
something()"
somevar=compile(expr, "","eval")
This throws a syntax error on "def". I've also tried wrapping the compile() with an eval(), no use. I basically want the result of the function call. Is this possible with eval or compile? Is there any library that would achieve this? I want somevar to be False.
Thanks in advance

According to the docs:
The mode argument specifies what kind of code must be compiled; it can be 'exec' if source consists of a sequence of statements, 'eval' if it consists of a single expression, or 'single' if it consists of a single interactive statement
So there's a couple of things here:
You should use multi-line strings in Python (three """)
You shouldn't use "eval" but "exec"
The returned object is not executed. You have to explicitly call it once created.
With all that in mind:
expr = """
def something():
print('hi')
return "done"
something()
"""
somevar = compile(expr, "", "exec")
exec(somevar)
This does print "hi". However, that code itself doesn't return anything. You run the function something() which yes: returns "done" but after running something()? Where would you return the return?
One option, as explained in this other SO thread would be assigning the return value to what would become a sort of "global" variable:
expr = """
def something():
return "done"
retval = something()
"""
somevar = compile(expr, "", "exec")
exec(somevar)
print(retval)
I can't help but saying that this sort of convoluted coding, that runs code stored in strings is probably a bad idea. It's insecure, hard to track, debug, refactor... So if this is going to be used for something besides learning purposes... Well... I would invite you to try and think if there's another way of achieving what you want.

how do exec and eval add __builtins__ to a given environment?

I'm trying to understand how eval and exec treat the environment (globals and locals) that they are given, so I made a class "logdict" which behaves like a dict but logs most methods (__new__ is excluded):
from functools import wraps
class LogDict(dict):
logs = {}
def _make_wrapper(name):
#wraps(getattr(dict, name))
def wrapper(self, *args, **kwargs):
LogDict.logs.setdefault(id(self), []).append({
'name': name,
'args': tuple(map(repr, args)),
'kwargs': dict((key, repr(kwargs[key])) for key in kwargs)
})
return getattr(super(), name)(*args, **kwargs)
return wrapper
for attr in dir(dict):
if callable(getattr(dict, attr)) and attr not in {'__new__',}:
locals()[attr] = _make_wrapper(attr)
def logrepr(self):
return ''.join(
"{fun}({rargs}{optsep}{rkwargs})\n".format(
fun = logitem['name'],
rargs = ', '.join(logitem['args']),
optsep = ', ' if len(logitem['kwargs'])>0 else '',
rkwargs = ', '.join('{} = {}'\
.format(key, logitem['kwargs'][key]) for key in logitem['kwargs'])
)
for logitem in LogDict.logs[id(self)])
as an example, this code:
d = LogDict()
d['1'] = 3
d['1'] += .5
print('1' in d)
print('log:')
print(d.logrepr())
produces this output:
True
log:
__init__()
__setitem__('1', 3)
__getitem__('1')
__setitem__('1', 3.5)
__contains__('1')
__getattribute__('logrepr')
I tried feeding this to exec in order to understand how it was being used, but I can't see it accessing the dictionary beyond what makes sense:
print('\tTesting exec(smth, logdict):')
d = LogDict()
exec('print("this line is inside the exec statement")', d)
print('the log is:')
print(d.logrepr(), end='')
print('the env now contains:')
print(d)
Testing exec(smth, logdict):
this line is inside the exec statement
the log is:
__init__()
__getitem__('print')
__getattribute__('logrepr')
the env now contains:
[a dictionary containing __builtins__]
so the exec function didn't call any of the methods I'm logging except __getitem__ to see if 'print' was in it (__getattribute__ is called later when I print the log); how did it set the key '__builtins__' (or check that it wasn't already defined)? Am I just missing the method it's using, or is it doing something more low-level?

The exec function uses low-level dictionary functions in the Python C API to insert the __builtins__ module into the global namespace dictionary. You can see the call in the CPython source code.
Because the call is to low level dict API, it doesn't look in your class to find your overridden __setitem__ method, it just directly writes into the underlying dictionary storage. The exec function requires that the global namespace passed in to it is a dict (or a dict subclass, but not some other mapping type), so this is always safe, at least in terms of not crashing the interpreter. But it does bypass your logging.
Unfortunately, I don't see any way to get logging added so that you can see __builtins__ get added to the global namespace. That probably means your attempt to directly observe exec's behavior is doomed. But perhaps reading the C source code is a suitable alternative, if you're just trying to understand what it does. One of the perks of using an open source programming language is that you can just go look up how the interpreter is programmed when you have questions like this. It does require reading C, rather than just Python, but the builtin_exec_impl function is straight forward enough (the actual code execution happens elsewhere and is surely much more complicated).

How can I dynamically define a function from user input?

I'm trying to dynamically generate a function from user input. The user gives a function as a string input, gives the differents parameters, and I want to write a function that will change this string into a usable function. My idea so far was to use the exec function to create a lambda function constructed with a string like so : exec("f = lambda {}:{}".format(', '.join(['x'] + parameter_list), function_string)). This would give, for example, a string like "f = lambda x, a:x+a" to the exec function.
This technique works fine if I define the scope of exec to globals(), but I'd like my function to be local so that I can do something like this:
def define_function(function_string, parameter_list):
exec("f = lambda {}:{}".format(', '.join(['x'] + parameter_list), function_string))
return f
or like this :
def define_function(function_string, parameter_list):
exec("return lambda {}:{}".format(', '.join(['x'] + parameter_list), function_string))
However, in the first case, it gives an error along the lines of "f is not defined". In the second case, meanwhile, it says that "return can only be used in a function". I understand why the second error occurs, but I wonder why my first try didn't work. Using globals() makes it work, but I'd rather not use it.
If you have any suggestions for other methods for achieving the same result, I'm interested in those as well.
EDIT
Using eval also raises an error :
line 9, in define_function
eval("f = lambda {}:{}".format(', '.join(['x'] + parameter_list), function_string))
File "<string>", line 1
f = lambda x, a:x+a
^
SyntaxError: invalid syntax

Solution 1: use eval():
def get_func(param_list, result_str):
return eval("lambda {}: {}".format(', '.join(['x'] + param_list), result_str))
Solution 2: if you really want to use exec(), do something like this:
def get_func(param_list, result_str):
exec("f = lambda {}: {}".format(', '.join(['x'] + param_list), result_str))
return locals()['f']
Some good info on the differences between exec and eval here: What's the difference between eval, exec, and compile?
I can see what you're trying to do in your question, by the way, and it's worth noting that the documentation pages for exec() and locals() both explicitly warn that this won't work. Ditto, if you type help(locals) into the interactive interpreter, you get this message:
Help on built-in function locals in module builtins:
locals()
Return a dictionary containing the current scope's local variables.
NOTE: Whether or not updates to this dictionary will affect name lookups in
the local scope and vice-versa is *implementation dependent* and not
covered by any backwards compatibility guarantees.

How to write code in python (from a vb net guy)

I am used to vb.net. I am learning python 3.6/7, as I understand it code writing is in either:
functions that return a result
functions that return a None
In a modules (no functions), ie run the module.py
In a class ie run the class.ID.function (ref (1) and (2))
In .Net you can run code in a function and a sub routine, so python has no sub routines, just functions returning None.
Is this correct?

Every function in python return some value, if you don't specify the return value then it will implicitly return None. So there is no distinction like sub or function in python.

Like other programming languages, in Python:
To let a function return a value, you have to use the return statement.
But if you don't specify the return value then it will simply return Nothing.

According to Wikipedia, In computer programming, a subroutine is a sequence of program instructions that performs a specific task, packaged as a unit. This unit can then be used in programs wherever that particular task should be performed.
So, in python a function can return any kind of value, objects or even None. A function returns None in two cases :
You write return None or return at the end of the function.
def some_func():
#do anything and return None
#you can write "return" instead "return None"
return None
some_var = some_func()
#There is no output to the above line of code
print(some_var)
#There is no output to the above line of code
print(type(some_var))
#Output: NoneType
You don't return anything at all and every function by default returns a None if nothing specified. For example:
def some_func():
print ("I print and return nothing")
some_var = some_func()
#Output : I print and return nothing
print(some_var)
#There is no output to the above line of code
print(type(some_var))
#Output: NoneType
Hope it helps.

Function executes either the key is there or not in dictionary.get in python 2.7

I am working with dictionaries in Python and I am searching for a value using:
my_dictionary_object.get("key")
As we all know if the key is missing in dictionary, it will return None object.
So to save extra line and to make my code more interesting I am trying:
def my_function():
'''do some calculations'''
return missing_value_from_dictionary
Now here is the fun part; when I do
my_dictionary_object.get("key", my_function())
it executes the function whether the key is missing or not so I thought let's remove the brackets and I did:
my_dictionary_object.get("key", my_function)
and (just in case)
my_dictionary_object.get("key", lambda: my_function())
but the one without lambda didn't execute (because it was never called) same happened with the one with lambda.
TL;DR
My question is why is the function get executed if the key is present in the dictionary?
Is it something that I am doing wrong or something I am missing here?

In: my_dictionary_object.get("key", my_function()) the execution looks something like:
Evaluate the first argument ("key")
Evaluate the second argument which is an expression: my_function(). So let's call the function, and use the return value in its place. Very much like in: a = my_function(), python would call the function and put the returned value in its place.
Call the mydictionary_object.get(..) with the above two evaluated arguments.
In other words, dictionary.get("key", default) will simply return the 2nd argument if the key doesn't exist. If it is a lambda, lambda is returned. A lambda is an object. Note that in .get("key", my_function()), my_function() is not technically the second argument. The resultant value returned as a result of execution of that function is the second argument -- which explains where you're wrong, hopefully.
What you are looking for is actually captured in another container called defaultdict.
What you'd do is:
from collections import defaultdict
my_dictionary_object = defaultdict(my_function) # my_function should not take any argument.
my_dictionary_object["non-existing-key"] # Magic happens, see below.
What happens is, if the key (=x) doesn't exist, my_function is called without any arguments and the dictionary is updated with the value returned by the function against the key (=x). Something like:
if "key" not in dictionary:
dictionary["key"] = my_function()
return dictionary["key"]