The Python C API function PyEval_EvalCode let's you execute compiled Python code. I want to execute a block of Python code as if it were executing within the scope of a function, so that it has its own dictionary of local variables which don't affect the global state.
This seems easy enough to do, since PyEval_EvalCode lets you provide a Global and Local dictionary:
PyObject* PyEval_EvalCode(PyCodeObject *co, PyObject *globals, PyObject *locals)
The problem I run into has to do with how Python looks up variable names. Consider the following code, that I execute with PyEval_EvalCode:
myvar = 300
def func():
return myvar
func()
This simple code actually raises an error, because Python is unable to find the variable myvar from within func. Even though myvar is in the local dictionary in the outer scope, Python doesn't copy it into the local dictionary in the inner scope. The reason for this is as follows:
Whenever Python looks up a variable name, first it checks locals, then it checks globals, and finally it checks builtins. At module scope, locals and globals are the SAME dictionary object. So the statement x = 5 at module scope will place x in the the locals dictionary, which is also the globals dictionary. Now, a function defined at module scope which needs to lookup x won't find x within the function-scope locals, because Python doesn't copy module-scope locals into function-scope locals. But this normally isn't a problem, because it can find x in globals.
x = 5
def foo():
print(x) # This works because 'x' in globals() == True
It's only with nested functions, that Python seems to copy outer-scope locals into inner-scope locals. (It also seems to do so lazily, only if they are needed within the inner scope.)
def foo():
x = 5
def bar():
print(x) # Now 'x' in locals() == True
bar()
So the result of all this is that, when executing code at module scope, you HAVE to make sure that your global dictionary and local dictionary are the SAME object, otherwise module-scope functions won't be able to access module-scope variables.
But in my case, I don't WANT the global dictionary and local dictionary to be the same. So I need some way to tell the Python interpreter that I am executing code at function scope. Is there some way to do this? I looked at the PyCompileFlags as well as the additional arguments to PyEval_EvalCodeEx and can't find any way to do this.
Python doesn't actually copy outer-scope locals into inner-scope locals; the documentation for locals states:
Free variables are returned by locals() when it is called in function blocks, but not in class blocks.
Here "free" variables refers to variables closed over by a nested function. It's an important distinction.
The simplest fix for your situation is just to pass the same dict object as globals and locals:
code = """
myvar = 300
def func():
return myvar
func()
"""
d = {}
eval(compile(code, "<str>", "exec"), d, d)
Otherwise, you can wrap your code in a function and extract it from the compiled object:
s = 'def outer():\n ' + '\n '.join(code.strip().split('\n'))
exec(compile(s, '<str>', 'exec').co_consts[0], {}, {})
Related
Why does this code raise NameError?:
def func():
exec("my_var = 42")
print(my_var)
func()
There are many related questions, but I have not found a clear answer as to WHY. I don't want a workaround.
Also, I have noticed that the code works when I run:
exec("my_var = 42", globals(), globals())
But not really sure why.
The documentation for exec() states: "...if the optional parts are omitted [second and third arguments], the code is executed in the current scope.". The current scope is the func() function. Why can't I access my_var from this same scope?
The parser/compiler doesn't take the argument to exec() into account (since it could be a variable). So when it parses the function it doesn't see the assignment to my_var, so it treats it as a global variable. But the assignment in exec() will create a local variable. So the variable that was assigned is not the same one that it tries to print.
In addition, if you want changes to local variables to be visible, you have to pass the locals() dictionary explicitly. The [documentation] states:
Note: The default locals act as described for function locals() below: modifications to the default locals dictionary should not be attempted. Pass an explicit locals dictionary if you need to see effects of the code on locals after function exec() returns.
I tested this out with a program I wrote myself:
>>> def f():
f=['f',1,2]
def g():
g=1
print('this prints out f from f(): ',f)
print("id",id(f))
def x():
x=1
print('this also prints out f from f():',f)
print('id',id(f))
x()
g()
>>> f()#output
this prints out f from f(): ['f', 1, 2]
id 140601546763464
this also prints out f from f(): ['f', 1, 2]
id 140601546763464
From what I learned, the innermost x() function can only access its own local namespace, the enclosing namespace, the global, and finally the built-in namespace. I initially thought that trying to access the list f declared in function f() from function x() would raise an error, as the f() function's namespace cannot be classified as any of the aforementioned elements. After running the program, I realized you indeed can access the list f from the function x(). I don't quite understand how this works though. My guess is that checking the enclosing namespace not only checks the local namespace of the enclosing function but the enclosing function for it as well, in a process that works almost recursively. Can somebody please explain how this works?
Python resolves names using LEGB rule:(LEGB means Local, Enclosing, Global, and Built-in)
Local scope:
contains the names that are defined inside the function.
visible only inside the function
created at function call(If we call the function multiple times each call creates new local scope)
will be destroyed once function return
Enclosing or nonlocal:
exists for nested functions
contains names defined in the enclosing function
visible in inner and enclosing functions.
Global:
contains all the names defined at the top level of a program
visible from everywhere inside the code.
exist throut the life of code.
Built-in:
created whenever we run a script
contains keywords, functions, exceptions, etc that are built into Python
visible everywhere in the code
The LEGB rule is a rule which determines the order in which Python looks up names.
i.e Python will look the name sequentially in the local, enclosing, global, and built-in scope. And inner scope codes can outer scope names but outer scope codes cannot access inner scope names.
When we use nested functions the scope resolving is as follows:
check the local scope(inside the function)
If not found check enclosing scopes of outer functions from the innermost scope to the outermost scope
If not found look the global scope
If not found look built-ins
Still not found raise error
A little easy problem:
exec("a=3")
print(a)
# This will print 3
If I use this:
def func():
exec("a=3")
print(a)
func()
# NameError: name 'a' is not defined.
What happened?How could I use exec() to assign it a value in a function?
Edit:I found a question with the same trouble but still didn't solved.
why do you want to do that?
I know using exec() is bad and unsafe.But recently I try to solve a OP's problem.I met it.
Python knows several kinds of scope: module global, function local, nonlocal closures, class body. Notably, scope resolution is defined statically at byte code compile time – most importantly, whether names refer to local/nonlocal or global scope cannot be changed.
Of these scopes, only global scope is guaranteed to behave similar to a dict, and as such writeable. The local/nonlocal scope is generally not writeable, and new variables cannot be added to it.
exec will write to the global scope if locals is not passed in; globals must then explicitly be set to its default of globals().
def func():
exec("a='exec'", globals()) # access only global scope
print(a)
a = 'global'
func() # prints exec
However, once a name is local to a function, exec cannot modify it.
def func():
a = 'local' # assignment makes name local
exec("a='exec global'", globals())
exec("a='exec locals'", globals(), locals())
print(a)
a = 'global'
func() # prints local
While a dict-like representation of local/nonlocal scope exists, the interpreter is not required to honour changes to it.
locals()
Update and return a dictionary representing the current local symbol table. Free variables are returned by locals() when it is called in function blocks, but not in class blocks. Note that at the module level, locals() and globals() are the same dictionary.
Note: The contents of this dictionary should not be modified; changes may not affect the values of local and free variables used by the interpreter.
Even though exec does take locals as a dict, these are not treated like function locals/nonlocals. Attempts to modify the default locals (the result of locals()) are not defined.
exec()
... If globals and locals are given, they are used for the global and local variables, respectively. If provided, locals can be any mapping object. Remember that at module level, globals and locals are the same dictionary. If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition.
Note: The default locals act as described for function locals() below: modifications to the default locals dictionary should not be attempted. ...
execute help(exec) in your python3 REPL, you will get the following docuement:
Help on built-in function exec in module builtins:
exec(source, globals=None, locals=None, /)
Execute the given source in the context of globals and locals.
The source may be a string representing one or more Python statements
or a code object as returned by compile().
The globals must be a dictionary and locals can be any mapping,
defaulting to the current globals and locals.
If only globals is given, locals defaults to it.
so there are at least 2 options to provide the value of argument 'a':
assign a value to the variable 'a' in the module's global scope:
a = 1
def func():
exec("global a;a=3")
print(a)
pass a customized global or local context to exec:
def func():
my_context = {'a': 1}
exec("a=3", None, my_context)
print(my_context['a'])
NOTE: DONT USE eval or exec IN YOUR SERIOUS CODE UNLESS YOU KNOW WHAT YOU ARE DOING.
EDIT NOTE
the following solution(2th solution mentioned in the comments) wont work:
def func():
a = 1
exec("a=3")
print(a) # still get 1 here
A module holds a dictionary to keep track of itscontext, such as the names defined at some point of the execution. This dictionary can be accessed through vars(module) (or module.__dict__) if module was imported, or by a call to the locals built-in function in the module itself:
Update and return a dictionary representing the current local symbol table.
But I found myself a bit confused, when I tried accessing the locals dictionary from a function. The output of a script containing only the following is an empty dictionary:
def list_locals():
print(locals())
list_locals()
But on the other hand, if a script contains exclusively the following, the output is the expected dictionary, containing __name__, __doc__ and the other module-level variables:
print(locals())
So, when is the content of the locals dictionary set?
In addition, what does "update" mean in the definition of the locals function?
The namespace of a module is the global namespace, accessed through globals(). There is no separate locals namespace, so locals(), outside of functions, just returns the global namespace.
Only functions have a local namespace. Note that locals() is a one-way reflection of that namespace; the CPython implementation for functions is highly optimised and you can't add or alter local variables through the locals() dictionary. The dictionary returned by locals() is updated whenever the namespace has changed between calls to that function.
Note also that things like list / dict / set comprehensions, generator expressions and class bodies are in reality executed as functions too, albeit with different namespace rules. In such contexts locals() will also return the separate function local namespace.
If you call locals() more than once during the same function call, it will return the same dictionary.
>>> def f():
... d = locals()
... assert 'd' not in d
... locals()
... assert 'd' in d
... assert d is locals()
... print(d)
...
>>> f()
{'d': {...}}
"Update" in this case means that the contents of this dictionary are updated to reflect the current scope of local variables that exist, but if you kept a reference to this dictionary then that dictionary will still be used.
Notice also that the locals dictionary doesn't contain a key for 'f', even though you could have accessed it during the course of the function. In case it's not obvious, that's a global, not a local.
I came across some code which kind of puzzled me. Here's a minimal example which shows this:
# of course, the ... are not part of the actual code
some_var = {"key1":"value1" ... "keyN":"valueN"}
def some_func():
v = some_var["key1"]
The code works, but the fact that I can access some_var directly confuses me. The last time I had to write some Python code, I remember having to write some_func like this:
def some_func():
global some_var
v = some_var["key1"]
I am using Python 2.7.1 on a Windows 7 PC. Did something change in the 2.7 release that allows for this?
No, you just can't reassign some_var in a local scope. Consider the following:
some_var = {}
def some_func():
# some_var[5] = 6
some_var = {1:2}
some_var[3] = 4
some_func()
print (repr(some_var)) # {}
You'll see the assignment in some_func actually creates a local variable which shadows the global one. Therefore, uncommenting the line would result in an UnboundLocalError - you can't access variables before they're defined.
There's a difference between using (e.g. calling or using in an expression) a name from an outer scope and assigning it (and there's a difference between assigning a bare variable and assigning a member of an object pointed to by a variable - x.y = ... and x[...] = ... count as method calls!).
You only need to declare that a variable is from an outer scope if you're re-assigning it. In Python 2, you can only do that with global variables (via global var), in Python 3 you can do it for abritarily nested scopes (e.g. closures) using nonlocal var.
Using a nonlocal variable as in you example doesn't require global. It does, however, as soon as you assign the variable (with the aforementioned definition of assignment) anywhere within that scope - so add a line some_var = ... after the line where you're using it and you'll get an UnboundLocalError. Refer to the documentation for the nitty gritty details.
You only have to use global if you want to assign a new value to that variable.
Nested scope was introduced in Python 2.1 (and enabled by default in Python 2.2) (emphasis mine):
Put simply, when a given variable name is not assigned a value within a function (by an assignment, or the def, class, or import statements), references to the variable will be looked up in the local namespace of the enclosing scope. A more detailed explanation of the rules, and a dissection of the implementation, can be found in the PEP.
You only need to use global if you intend to assign to the variable, for reading the variable this is not necessary. This difference is not arbitrary, though it may seem like it at first glance.
When reading a value the interpreter can just look for a local variable named some_var, if it cannot find it then it looks for a global variable of that name. These are simple and straight forward semantics.
When assigning values to a variable the interpreter needs to know whether you intend to assign to a local variable some_var or a global variable. The interpreter assumes that some_var = 2 when called within a function is assigning to a local variable, this makes sense since this is the most common case. For the relatively rare times when you want to assign to a global variable from within a function then you use the global modifier global some_var = 2.
It depends on the usage of the variable in the function
Python variable scope error
Assigning a value to a name makes the name local, unless the name is explicitly declared global.
a = 12
def foo():
a = 42
print a # uses local
foo()
>>> 42
def foo():
global a
a = 42
foo()
print a
>>> 42
If a name is not assigned to, it is global.
a = 12
def foo():
print a # uses global
foo()
>>> 12
In short, you only have to explicitly declare a name global if you will be assigning to it. If you are just reading from it, you can use it at will. However, if you ever assign to the variable, it will be considered local in that function unless you declared it global.
b = 5
def foo():
print b
b = 7
foo()
>>> ???
Since b is assigned to in foo() and not declared global, Python decides at compile time that b is a local name. Therefore b is a local name throughout the whole function, including at the print statement before the assignment.
Therefore the print statement gives you an error, because the local name b has not been defined!