Develop Reference

How to call a python function from C++ with pybind11?

Please consider the following C++ pybind11 program:
#include <pybind11/embed.h>
namespace py = pybind11;
int main() {
py::scoped_interpreter guard{};
py::dict locals;
py::exec(R"(
import sys
def f():
print(sys.version)
)", py::globals(), locals);
locals["f"](); // <-- ERROR
}
The py::exec call and the enclosed import sys call both succeed, but the call locals["f"]() throws an exception:
NameError: name 'sys' is not defined
on the first line of function f.
Expected behaviour is that the program prints the python system version.
Any ideas?
Update:
I modified the program as suggested by #DavidW:
#include <pybind11/embed.h>
namespace py = pybind11;
int main() {
py::scoped_interpreter guard{};
py::dict globals = py::globals();
py::exec(R"(
import sys
def f():
print(sys.version)
)", globals, globals);
globals["f"](); // <-- WORKS NOW
}
and it now works.
I'm not 100% sure I understand what is going on, so I would appreciate an explanation.
(In particular does the modification of the common globals / locals dictionary impact any other scripts. Is there some global dictionary that is part of the python interpreter that the exec script is modifying? or does py::globals() take a copy of that state so the execed script is isolated from other scripts?)
Update 2:
So it looks like having globals and locals be the same dictionary is the default state:
$ python
>>> globals() == locals()
True
>>> from __main__ import __dict__ as x
>>> x == globals()
True
>>> x == locals()
True
...and that the default value for the two is __main__.__dict__, whatever that is (__main__.__dict__ is the dictionary returned by py::globals())
I'm still not clear what exactly __main__.__dict__ is.

So the initial problem (solved in the comments) was that having different globals and locals causes it to be evaluated as if it were in a class (see the Python documentation for exec - the PyBind11 function behaves basically the same):
Remember that at the module level, globals and locals are the same dictionary. If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition.
A function scope doesn't look up variables defined in its enclosing class - this wouldn't work
class C:
import sys
def f():
print(sys.version)
# but C.sys.version would work
and thus your code doesn't work.
pybind11::globals returns a dictionary that's shared in a number of places:
Return a dictionary representing the global variables in the current execution frame, or __main__.__dict__ if there is no frame (usually when the interpreter is embedded).
and thus any modifications to this dictionary will be persistent and stay (which probably isn't what you want!). In your case it's probably __main__.__dict__ but in general "the current execution frame" might change from call-to-call, depending on how much you're crossing the C++-Python boundary. For example, if a Python function calls a C++ function that modifies globals() then exactly what you modify depends on the caller.
My advice would be to create a new, empty dict instead and pass that to exec. This ensures that you run in a fresh, non-shared namespace.
__main__ is just a special module that represents the "top level code environment". Like any module is has a __dict__. When running in the REPL it's the global scope there. From the pybind11 point of view it's just a module with a dict, and you probably shouldn't be writing into it casually (unless you've really decided that you want to deliberately put something there to share it globally).
Regarding the __builtins__: the documentation for the Python exec function says
If the globals dictionary does not contain a value for the key __builtins__, a reference to the dictionary of the built-in module builtins is inserted under that key. That way you can control what builtins are available to the executed code by inserting your own __builtins__ dictionary into globals before passing it to exec().
and looking at the code for the PyRun_String that Pybind11 exec calls, the same applies there.
This dictionary seems to be sufficient for the builtin functions to be looked up correctly. (If that isn't the case then you can always do pybind11::dict(pybind11::module::import("builtins").attr("__dict__")) to make a copy of the builtin dict and use that instead. However, I don't believe it's necessary)

How pass a value to a variable in python function with using exec()?

A little easy problem:
exec("a=3")
print(a)
# This will print 3
If I use this:
def func():
exec("a=3")
print(a)
func()
# NameError: name 'a' is not defined.
What happened?How could I use exec() to assign it a value in a function?
Edit:I found a question with the same trouble but still didn't solved.
why do you want to do that?
I know using exec() is bad and unsafe.But recently I try to solve a OP's problem.I met it.

Python knows several kinds of scope: module global, function local, nonlocal closures, class body. Notably, scope resolution is defined statically at byte code compile time – most importantly, whether names refer to local/nonlocal or global scope cannot be changed.
Of these scopes, only global scope is guaranteed to behave similar to a dict, and as such writeable. The local/nonlocal scope is generally not writeable, and new variables cannot be added to it.
exec will write to the global scope if locals is not passed in; globals must then explicitly be set to its default of globals().
def func():
exec("a='exec'", globals()) # access only global scope
print(a)
a = 'global'
func() # prints exec
However, once a name is local to a function, exec cannot modify it.
def func():
a = 'local' # assignment makes name local
exec("a='exec global'", globals())
exec("a='exec locals'", globals(), locals())
print(a)
a = 'global'
func() # prints local
While a dict-like representation of local/nonlocal scope exists, the interpreter is not required to honour changes to it.
locals()
Update and return a dictionary representing the current local symbol table. Free variables are returned by locals() when it is called in function blocks, but not in class blocks. Note that at the module level, locals() and globals() are the same dictionary.
Note: The contents of this dictionary should not be modified; changes may not affect the values of local and free variables used by the interpreter.
Even though exec does take locals as a dict, these are not treated like function locals/nonlocals. Attempts to modify the default locals (the result of locals()) are not defined.
exec()
... If globals and locals are given, they are used for the global and local variables, respectively. If provided, locals can be any mapping object. Remember that at module level, globals and locals are the same dictionary. If exec gets two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition.
Note: The default locals act as described for function locals() below: modifications to the default locals dictionary should not be attempted. ...

execute help(exec) in your python3 REPL, you will get the following docuement:
Help on built-in function exec in module builtins:
exec(source, globals=None, locals=None, /)
Execute the given source in the context of globals and locals.
The source may be a string representing one or more Python statements
or a code object as returned by compile().
The globals must be a dictionary and locals can be any mapping,
defaulting to the current globals and locals.
If only globals is given, locals defaults to it.
so there are at least 2 options to provide the value of argument 'a':
assign a value to the variable 'a' in the module's global scope:
a = 1
def func():
exec("global a;a=3")
print(a)
pass a customized global or local context to exec:
def func():
my_context = {'a': 1}
exec("a=3", None, my_context)
print(my_context['a'])
NOTE: DONT USE eval or exec IN YOUR SERIOUS CODE UNLESS YOU KNOW WHAT YOU ARE DOING.
EDIT NOTE
the following solution(2th solution mentioned in the comments) wont work:
def func():
a = 1
exec("a=3")
print(a) # still get 1 here

How to get the same scope for imported classes in a Python script? [duplicate]

This question already has answers here:
Visibility of global variables in imported modules
(9 answers)
Closed 7 years ago.
It appears that a class defined in a script has a different scope to one that is imported into the script. For example:
In a file foo.py:
class foo(object):
def __init__(self):
print globals()
In my main file:
from foo import foo
class bar(object):
def __init__(self):
print globals()
classimport = foo()
classinternal = bar()
The list of globals returned from foo and bar are different - why is this?
It is making life difficult as any class that needs to access the main globals() has to reside in the main file. How do I ensure that the imported class has the same global scope? Some things that I have tried after reading other posts here and here include:
module = __import__("foo", fromlist="foo")
globals()["foo"] = getattr(module, "foo")
and
__builtin__.foo = foo
Any help appreciated!
[EDIT] ---
So per the link above, this is answered in a duplicate article. It turns out that scope is not shared across modules. It mentions several ways around this, but in my case I need to actually create / read / write global variables. So I created a routine in the main script and pass it as an object when foo and bar are initialized. For example:
def PrintGlobals():
print globals()
class bar(object):
def __init__(self, PrintGlobals):
self.PrintGlobals = PrintGlobals
self.PrintGlobals()
classinternal = bar(PrintGlobals)
(Not my choice of how all this should work, its a hack until I get some time with the application devs :-)

Here's what the Python 3 FAQ has to say:
In Python, variables that are only referenced inside a function are implicitly global. If a variable is assigned a value anywhere within the function’s body, it’s assumed to be a local unless explicitly declared as global.
Though a bit surprising at first, a moment’s consideration explains this. On one hand, requiring global for assigned variables provides a bar against unintended side-effects. On the other hand, if global was required for all global references, you’d be using global all the time. You’d have to declare as global every reference to a built-in function or to a component of an imported module. This clutter would defeat the usefulness of the global declaration for identifying side-effects.
To see globals in various scopes, try doing a print(globals()) at various points during your execution. For example: at the top-level module before any code is run, then in __init__.py if you have any code in there (because you import foo), at foo's module level, within each function, and before/after you modify any variables passed to the function.
This answer further explains:
I think the key thing you're missing here is that each module has its own "global" namespace. This can be a bit confusing at first, because in languages like C, there's a single global namespace shared by all external variables and functions. But once you get past that assumption, the Python way makes perfect sense.
Note however that all names assigned in a package __init__.py file are available in the package namespace when you import the package or a module in the package.

What are `globals` and `locals` parameters in Python __import__ function for?

There's a part of __import__ in Python documentation, which I don't understand:
__import__(name[, globals[, locals[, fromlist[, level]]]])
The function imports the module name, potentially using the given globals and locals to determine how to interpret the name in a package context. The standard implementation does not use its locals argument at all, and uses its globals only to determine the package context of the import statement.
What is there to "interpret" about the module name? What is package context?
An example call using those parameters looks like this:
spam = __import__('spam', globals(), locals(), [], -1)
Why does the example provide globals() and locals() to the function? What happens when I only provide globals()? Or neither?
I am probably missing some part of the namespace logic with relation to importing modules. Could you point me to an article that explains this/has examples with __import__ function?

The standard implementation does not use its locals argument at all, and uses its globals only to determine the package context of the import statement.
(from docs.python.org)
I still have no idea how globals are used; what global variable can ever affect the way import statement works?
EDIT: After looking at import.c in Python 2.5 source I found that __import__ expects to find either __name__ or __path__ in globals in order to augment import search path relative to path(s) found in one of these variables, in that order.

globals is used to determine the current context on where the import is being called. For example:
"""
/myproject/a/b.py
/myproject/a/foo.py
/myproject/c/d.py
/myproject/c/foo.py
"""
# Which foo gets imported?
import foo #1
foo = __import__('foo') #2
They are not the same, since there is no (easy) way on #2 to know from which module the import is being called from. The __import__ function needs to know which is the current module to actually import the correct foo.
Internally on __import__(), globals is used to get the reference on the current module invoking the import. From the __import__ source code:
Return the package that an import is being performed in. If globals comes
from the module foo.bar.bat (not itself a package), this returns the
sys.modules entry for foo.bar. If globals is from a package's init.py,
the package's entry in sys.modules is returned, as a borrowed reference.

What is there to "interpret" about the module name? What is package context?
When you enter
>>> a
Python must "interpret" that name. Is it a global? Is it a local?
>>> def f(x):
... return x * a
Now, x is clearly local. a has to be "interpreted". Global? Local?
Why does the example provide globals() and locals() to the function? What happens when I only provide globals()? Or neither?
Try it and see. Seriously. It's easier to play with it than it is to ask.
What's important is that things you do at the >>> prompt are global.
You'll need to define functions that will create a local context so you can see the differences between global and local.

Global variable with imports

first.py
myGlobal = "hello"
def changeGlobal():
myGlobal="bye"
second.py
from first import *
changeGlobal()
print myGlobal
The output I get is
hello
although I thought it should be
bye
Why doesn't the global variable myGlobal changes after the call to the changeGlobal() function?

Try:
def changeGlobal():
global myGlobal
myGlobal = "bye"
Actually, that doesn't work either. When you import *, you create a new local module global myGlobal that is immune to the change you intend (as long as you're not mutating the variable, see below). You can use this instead:
import nice
nice.changeGlobal()
print nice.myGlobal
Or:
myGlobal = "hello"
def changeGlobal():
global myGlobal
myGlobal="bye"
changeGlobal()
However, if your global is a mutable container, you're now holding a reference to a mutable and are able to see changes done to it:
myGlobal = ["hello"]
def changeGlobal():
myGlobal[0] = "bye"

I had once the same concern as yours and reading the following section from Norman Matloff's Quick and Painless Python Tutorial was really a good help. Here is what you need to understand (copied from Matloff's book):
Python does not truly allow global variables in the sense that C/C++ do. An imported Python module will not have direct access to the globals in the module which imports it, nor vice versa.
For instance, consider these two files, x.py,
# x.py
import y
def f():
global x
x = 6
def main():
global x
x = 3
f()
y.g()
if __name__ == ’__main__’:
main()
and y.py:
# y.py
def g():
global x
x += 1
The variable x in x.py is visible throughout the module x.py, but not in y.py. In fact, execution of the line
x += 1
in the latter will cause an error message to appear, “global name ’x’ is not defined.”
Indeed, a global variable in a module is merely an attribute (i.e. a member entity) of that module, similar to a class variable’s role within a class. When module B is imported by module A, B’s namespace is copied to A’s. If module B has a global variable X, then module A will create a variable of that name, whose initial value is whatever module B had for its variable of that name at the time of importing. But changes to X in one of the modules will NOT be reflected in the other.
Say X does change in B, but we want code in A to be able to get the latest value of X in B. We can do that by including a function, say named GetX() in B. Assuming that A imported everything from B, then A will get a function GetX() which is a copy of B’s function of that name, and whose sole purpose is to return the value of X. Unless B changes that function (which is possible, e.g. functions may be assigned), the functions in the two modules will always be the same, and thus A can use its function to get the value of X in B.

Python global variables are not global
As wassimans points out above they are essentially attributes within the scope of the module they are defined in (or the module that contains the function that defined them).
The first confusion(bug) people run into is not realizing that functions have a local name space and that setting a variable in a function makes it a local to the function even when they intended for it to change a (global) variable of the same name in the enclosing module. (declaring the name
in a 'global' statement in the function, or accessing the (global) variable before setting it.)
The second confusion(bug) people run into is that each module (ie imported file) contains its own so called 'global' name space. I guess python things the world(globe) is the module -- perhaps we are looking for 'universal' variables that span more than one globe.
The third confusion (that I'm starting to understand now) is where are the 'globals' in the __main__ module? Ie if you start python from the command line in interactive mode, or if you invoke python script (type the name of the foo.py from the command shell) -- there is no import of a module whose name you can use.
The contents of 'globals()' or globals().keys() -- which gives you a list of the globals -- seems to be accessible as: dir(sys.modules['__main__'])
It seems that the module for the loaded python script (or the interactive session with no loaded script), the one named in: __name__, has no global name, but is accessible as the module whose name is '__main__' in the system's list of all active modules, sys.modules