Develop Reference

globals from multiple modules not visible to exec code?

My app executes bits of python logic stored in a configuration file via exec, as in:
"foo() + 2"
This logic commonly references symbols that I store in a module named "standard". For example, in the above, you can see the logic accesses the method foo(), and foo is defined inside standard.py:
def foo():...
To provide the logic with access to the symbols in standard, I'm extracting out the methods from standard into a dictionary, like so:
import standard
my_globals = standard.__dict__
Then I'm adding in a few other relevant symbols to my_globals (which I don't show here) and providing them to the logic, when I execute it:
exec("foo() + 2", my_globals)
This is working. When I look at globals() from inside foo(), I can see other methods I defined in the module standard.py as well as the other relevant symbols I mentioned above and foo() can access all of those things.
The problem comes in when I want to make another module of functions available to the logic as well. Let's say I have a module named custom.py that has some other symbols I want the logic to access. I'm trying to make those symbols available as well by doing this:
import custom
my_globals.update(custom.__dict__)
Let's say my logic now is "bar() + 1", where "bar" is defined inside of custom.py. bar() also wants to access some of those relevant other symbols I added into my_globals.
The problem I'm running in to is that code inside of custom is only seeing the symbols defined inside custom.py, and not everything else stored in my_globals. IE, bar() can't see foo(), nor the other stuff I tucked away into my_globals.
Yet foo() can. It's code can see any other methods I defined in standard, as well as symbols defined in custom, as well as the "extra" symbols I plugged into my_globals.
Why is this happening? My expectation is that the logic being executed is run in the context of the contents of my_globals, so it seems like both foo() and bar() should be able to access any and all symbols in my_globals.
I suspect this has to do with how I'm creating my_globals. Any insight would be greatly appreciated!

Here is some insight:
"To provide the logic with access to the symbols in standard, I'm extracting out the methods from standard into a dictionary, like so:"
import standard
my_globals = standard.__dict__
Not exactly. You're just creating a local variable, my_globals that now points to standard.__dict__. Whenever you update my_globals, you're really just updating standard.__dict__.
When you add your other symbols to my_globals, again, you're just adding them to standard.__dict__.
Calling:
exec("foo() + 2", my_globals)
works great when foo is defined in standard.py, because you've added all the other methods to this module - you now have access to them all.
When you do:
import custom
my_globals.update(custom.__dict__)
You've added your "symbols" from custom.py to the standard module. All the functions in standard can access functions from custom.py after this point
Unfortunately, custom.py itself, doesn't have direct access to the methods in standard.py (unless you import them). From within custom.py, you can see that everything you've created is in standard now:
(from within custom.py):
import sys
def custom_foo():
print(dir(sys.modules['standard'])) # shows that you've put everything in here
sys.modules['standard'].foo() # calls foo from standard.py (assuming you've imported standard in your main pgm)
Above is really ugly though - you could just add a:
from standard import *
at the top of custom.py, and you would have access to everything you've added to its __dict__ instead.
I doubt you really want to do what you're attempting with the whole exec thing, but I'm not really sure what your use case is.
EDIT:
If you really want all the symbols you've attached to my_globals available to the methods of custom.py, you could call:
custom.__dict__.update(my_globals)
After this point, functions in custom.py would have access to everything you've added to the standard.dict (aka my_globals). (You've also overrode any functions defined in custom.py with functions of the same name in my_globals)
Please note, doing things this way is pretty atypical (read: somewhat ill advised).

How do you declare a global constant in Python?

I have some heavy calculations that I want to do when my program starts, and then I want to save the result (a big bumpy matrix) in memory so that I can use it again and again. My program contains multiple files and classes, and I would like to be able to access this variable from anywhere, and if possible define it as constant.
How do you define a global constant in Python?

You can just declare a variable on the module level and use it in the module as a global variable. An you can also import it to other modules.
#mymodule.py
GLOBAL_VAR = 'Magic String' #or matrix...
def myfunc():
print(GLOBAL_VAR)
Or in other modules:
from mymodule import GLOBAL_VAR

I do not think the marked as good answer solves the op question. The global keyword in Python is used to modify a global variable in a local context (as explained here). This means that if the op modifies SOME_CONSTANT within myfunc the change will affect also outside the function scope (globally).
Not using the global keyword at the begining of myfunc is closer to the sense of global constant than the one suggested. Despite there are no means to render a value constant or immutable in Python.

There is no way to declare a constant in Python. You can just use
SOME_CONSTANT = [...]
If the file name where it is declared is file1.py, then you can access to it from other files in the following way:
import file1
print file1.SOME_CONSTANT
Assuming that both files are in the same directory.

I am not sure what you mean by 'global constant'; because there are no constants in Python (there is no "data protection", all variables are accessible).
You can implement a singleton pattern, but you will have to regenerate this at runtime each time.
Your other option will be to store the results in an external store (like say, redis) which is accessible from all processes.
Depending on how big your data set is, storing it externally in a fast K/V like redis might offer a performance boost as well.
You would still have to transform and load it though, since redis would not know what a numpy array is (although it has many complex types that you can exploit).

How to add builtin functions?

I am new to python programming. How can I add new built-in functions and keywords to python interpreter using C or C++?

In short, it is technically possible to add things to Python's builtins†, but it is almost never necessary (and generally considered a very bad idea).
In longer, it's obviously possible to modify Python's source and add new builtins, keywords, etc… But the process for doing that is a bit out of the scope of the question as it stands.
If you'd like more detail on how to modify the Python source, how to write C functions which can be called from Python, or something else, please edit the question to make it more specific.
If you are new to Python programming and you feel like you should be modifying the core language in your day-to-day work, that's probably an indicator you should simply be learning more about it. Python is used, unmodified, for a huge number of different problem domains (for example, numpy is an extension which facilitates scientific computing and Blender uses it for 3D animation), so it's likely that the language can handle your problem domain too.
†: you can modify the __builtin__ module to “add new builtins”… But this is almost certainly a bad idea: any code which depends on it will be very difficult (and confusing) to use anywhere outside the context of its original application. Consider, for example, if you add a greater_than_zero “builtin”, then use it somewhere else:
$ cat foo.py
import __builtin__
__builtin__.greater_than_zero = lambda x: x > 0
def foo(x):
if greater_than_zero(x):
return "greater"
return "smaller"
Anyone who tries to read that code will be confused because they won't know where greater_than_zero is defined, and anyone who tries to use that code from an application which hasn't snuck greater_than_zero into __builtin__ won't be able to use it.
A better method is to use Python's existing import statement: http://docs.python.org/tutorial/modules.html

for python 3.6 onward use import builtins.
# example 1
import builtins
def f():
print('f is called')
builtins.g = f
g() # output = f is called
####################################
# example 2
import builtins
k = print
def f(s):
k('new print called : ' + s)
builtins.print = f
print('abc') # output = new print is called abc

While David Wolever's answer is perfect, it should be noted again that the asker is new to Python. Basically all he wants is a global function, which can be done in two different ways...
Define a function in your module and use it.
Define a function in a different module and import it using the "from module import *" statement.
I think the asker's solution is the 2nd option and anyone new to Python having this question should look in to the same.
For an advance user, I would agree with Wolever's suggestion that it is a bad idea to insert a new function in to the builtin module. However, may be the user is looking for a way to avoid importing an always-used module in every script in the project. And that is a valid use case. Of course the code will not make sense to people who aren't part of the project but that shouldn't be a concern. Anyways, such users should look in to the PYTHONSTARTUP environment variable. I would suggest looking it up in the Index of the Python documentation and look at all links that talks about this environment variable and see which page serves your purpose. However, this solution works for interactive mode only and does not work for sub-main script.
For an all around solution look in to this function that I have implemented: https://drive.google.com/file/d/19lpWd_h9ipiZgycjpZW01E34hbIWEbpa/view
Yet another way is extending or embedding Python and it is a relatively complex topic. It is best to read the Python documentation on the same. For basic users, all I would say is that...
Extending means adding new builtin modules to the Python interpreter.
Embedding means inserting Python interpreter into your application.
And advanced users already know what they are doing!

You can use builtins module.
Example 1:
import builtins
def write(x):
print(x)
builtins.write = write
write("hello")
# output:
# Hello
Example 2:
import builtins
def hello(*name):
print(f"Hello, {' '.join(name)}!")
builtins.hello = hello
hello("Clark", "Kent")
# output:
# Hello, Clark Kent!

Sharing variables acoss several python files

I have tasked to modify a wx python gui based program which has several .py files.
I would like to share some variables defined in a.py and use them in b.py
The 10 -15 variables are of this form:
Amode = [SINGLE]
Format = [A] etc...
I would like to use them in b.py.
How do I go about it? I read about Pickle but still not clear how to use it well.

import a
// do something with a.Amode
// do something with a.Format

Generally, the best idea, in this case, is to either place the variables on the module directly or use some shared dataStore. I like the Borg pattern for this.
Basically do this:
#in borg.py
class Borg:
__shared_state = {}
def __init__(self):
self.__dict__ = self.__shared_state
Everywhere else:
import borg
drone = borg.Borg()
drone.foo = 1;
Obviously, you can limit this by defining __set__.
As to placing variables on modules directly, well, I'm not really a fan of having stateful variables publicly accessible on modules, but that is probably mostly me.

Modules are singletons (no matter how many times it's imported, it's only actually imported once, and that once is shared), so what I often do for this use case is to create a modules named, say, "shared.py", and put the data I want shared across other modules in it. Then, in those other modules:
import shared
# Use a shared variable/object
print shared.Amode
# Changes to the shared data are seen in all modules where it's imported.
shared.Amode = aNewValue
This has the nice effect of keeping all my shared data in its own namespace, "shared".

How to make a cross-module variable?

The __debug__ variable is handy in part because it affects every module. If I want to create another variable that works the same way, how would I do it?
The variable (let's be original and call it 'foo') doesn't have to be truly global, in the sense that if I change foo in one module, it is updated in others. I'd be fine if I could set foo before importing other modules and then they would see the same value for it.

If you need a global cross-module variable maybe just simple global module-level variable will suffice.
a.py:
var = 1
b.py:
import a
print a.var
import c
print a.var
c.py:
import a
a.var = 2
Test:
$ python b.py
# -> 1 2
Real-world example: Django's global_settings.py (though in Django apps settings are used by importing the object django.conf.settings).

I don't endorse this solution in any way, shape or form. But if you add a variable to the __builtin__ module, it will be accessible as if a global from any other module that includes __builtin__ -- which is all of them, by default.
a.py contains
print foo
b.py contains
import __builtin__
__builtin__.foo = 1
import a
The result is that "1" is printed.
Edit: The __builtin__ module is available as the local symbol __builtins__ -- that's the reason for the discrepancy between two of these answers. Also note that __builtin__ has been renamed to builtins in python3.

I believe that there are plenty of circumstances in which it does make sense and it simplifies programming to have some globals that are known across several (tightly coupled) modules. In this spirit, I would like to elaborate a bit on the idea of having a module of globals which is imported by those modules which need to reference them.
When there is only one such module, I name it "g". In it, I assign default values for every variable I intend to treat as global. In each module that uses any of them, I do not use "from g import var", as this only results in a local variable which is initialized from g only at the time of the import. I make most references in the form g.var, and the "g." serves as a constant reminder that I am dealing with a variable that is potentially accessible to other modules.
If the value of such a global variable is to be used frequently in some function in a module, then that function can make a local copy: var = g.var. However, it is important to realize that assignments to var are local, and global g.var cannot be updated without referencing g.var explicitly in an assignment.
Note that you can also have multiple such globals modules shared by different subsets of your modules to keep things a little more tightly controlled. The reason I use short names for my globals modules is to avoid cluttering up the code too much with occurrences of them. With only a little experience, they become mnemonic enough with only 1 or 2 characters.
It is still possible to make an assignment to, say, g.x when x was not already defined in g, and a different module can then access g.x. However, even though the interpreter permits it, this approach is not so transparent, and I do avoid it. There is still the possibility of accidentally creating a new variable in g as a result of a typo in the variable name for an assignment. Sometimes an examination of dir(g) is useful to discover any surprise names that may have arisen by such accident.

Define a module ( call it "globalbaz" ) and have the variables defined inside it. All the modules using this "pseudoglobal" should import the "globalbaz" module, and refer to it using "globalbaz.var_name"
This works regardless of the place of the change, you can change the variable before or after the import. The imported module will use the latest value. (I tested this in a toy example)
For clarification, globalbaz.py looks just like this:
var_name = "my_useful_string"

You can pass the globals of one module to onother:
In Module A:
import module_b
my_var=2
module_b.do_something_with_my_globals(globals())
print my_var
In Module B:
def do_something_with_my_globals(glob): # glob is simply a dict.
glob["my_var"]=3

Global variables are usually a bad idea, but you can do this by assigning to __builtins__:
__builtins__.foo = 'something'
print foo
Also, modules themselves are variables that you can access from any module. So if you define a module called my_globals.py:
# my_globals.py
foo = 'something'
Then you can use that from anywhere as well:
import my_globals
print my_globals.foo
Using modules rather than modifying __builtins__ is generally a cleaner way to do globals of this sort.

You can already do this with module-level variables. Modules are the same no matter what module they're being imported from. So you can make the variable a module-level variable in whatever module it makes sense to put it in, and access it or assign to it from other modules. It would be better to call a function to set the variable's value, or to make it a property of some singleton object. That way if you end up needing to run some code when the variable's changed, you can do so without breaking your module's external interface.
It's not usually a great way to do things — using globals seldom is — but I think this is the cleanest way to do it.

I wanted to post an answer that there is a case where the variable won't be found.
Cyclical imports may break the module behavior.
For example:
first.py
import second
var = 1
second.py
import first
print(first.var) # will throw an error because the order of execution happens before var gets declared.
main.py
import first
On this is example it should be obvious, but in a large code-base, this can be really confusing.

I wondered if it would be possible to avoid some of the disadvantages of using global variables (see e.g. http://wiki.c2.com/?GlobalVariablesAreBad) by using a class namespace rather than a global/module namespace to pass values of variables. The following code indicates that the two methods are essentially identical. There is a slight advantage in using class namespaces as explained below.
The following code fragments also show that attributes or variables may be dynamically created and deleted in both global/module namespaces and class namespaces.
wall.py
# Note no definition of global variables
class router:
""" Empty class """
I call this module 'wall' since it is used to bounce variables off of. It will act as a space to temporarily define global variables and class-wide attributes of the empty class 'router'.
source.py
import wall
def sourcefn():
msg = 'Hello world!'
wall.msg = msg
wall.router.msg = msg
This module imports wall and defines a single function sourcefn which defines a message and emits it by two different mechanisms, one via globals and one via the router function. Note that the variables wall.msg and wall.router.message are defined here for the first time in their respective namespaces.
dest.py
import wall
def destfn():
if hasattr(wall, 'msg'):
print 'global: ' + wall.msg
del wall.msg
else:
print 'global: ' + 'no message'
if hasattr(wall.router, 'msg'):
print 'router: ' + wall.router.msg
del wall.router.msg
else:
print 'router: ' + 'no message'
This module defines a function destfn which uses the two different mechanisms to receive the messages emitted by source. It allows for the possibility that the variable 'msg' may not exist. destfn also deletes the variables once they have been displayed.
main.py
import source, dest
source.sourcefn()
dest.destfn() # variables deleted after this call
dest.destfn()
This module calls the previously defined functions in sequence. After the first call to dest.destfn the variables wall.msg and wall.router.msg no longer exist.
The output from the program is:
global: Hello world!
router: Hello world!
global: no message
router: no message
The above code fragments show that the module/global and the class/class variable mechanisms are essentially identical.
If a lot of variables are to be shared, namespace pollution can be managed either by using several wall-type modules, e.g. wall1, wall2 etc. or by defining several router-type classes in a single file. The latter is slightly tidier, so perhaps represents a marginal advantage for use of the class-variable mechanism.

This sounds like modifying the __builtin__ name space. To do it:
import __builtin__
__builtin__.foo = 'some-value'
Do not use the __builtins__ directly (notice the extra "s") - apparently this can be a dictionary or a module. Thanks to ΤΖΩΤΖΙΟΥ for pointing this out, more can be found here.
Now foo is available for use everywhere.
I don't recommend doing this generally, but the use of this is up to the programmer.
Assigning to it must be done as above, just setting foo = 'some-other-value' will only set it in the current namespace.

I use this for a couple built-in primitive functions that I felt were really missing. One example is a find function that has the same usage semantics as filter, map, reduce.
def builtin_find(f, x, d=None):
for i in x:
if f(i):
return i
return d
import __builtin__
__builtin__.find = builtin_find
Once this is run (for instance, by importing near your entry point) all your modules can use find() as though, obviously, it was built in.
find(lambda i: i < 0, [1, 3, 0, -5, -10]) # Yields -5, the first negative.
Note: You can do this, of course, with filter and another line to test for zero length, or with reduce in one sort of weird line, but I always felt it was weird.

I could achieve cross-module modifiable (or mutable) variables by using a dictionary:
# in myapp.__init__
Timeouts = {} # cross-modules global mutable variables for testing purpose
Timeouts['WAIT_APP_UP_IN_SECONDS'] = 60
# in myapp.mod1
from myapp import Timeouts
def wait_app_up(project_name, port):
# wait for app until Timeouts['WAIT_APP_UP_IN_SECONDS']
# ...
# in myapp.test.test_mod1
from myapp import Timeouts
def test_wait_app_up_fail(self):
timeout_bak = Timeouts['WAIT_APP_UP_IN_SECONDS']
Timeouts['WAIT_APP_UP_IN_SECONDS'] = 3
with self.assertRaises(hlp.TimeoutException) as cm:
wait_app_up(PROJECT_NAME, PROJECT_PORT)
self.assertEqual("Timeout while waiting for App to start", str(cm.exception))
Timeouts['WAIT_JENKINS_UP_TIMEOUT_IN_SECONDS'] = timeout_bak
When launching test_wait_app_up_fail, the actual timeout duration is 3 seconds.