Develop Reference

globals from multiple modules not visible to exec code?

My app executes bits of python logic stored in a configuration file via exec, as in:
"foo() + 2"
This logic commonly references symbols that I store in a module named "standard". For example, in the above, you can see the logic accesses the method foo(), and foo is defined inside standard.py:
def foo():...
To provide the logic with access to the symbols in standard, I'm extracting out the methods from standard into a dictionary, like so:
import standard
my_globals = standard.__dict__
Then I'm adding in a few other relevant symbols to my_globals (which I don't show here) and providing them to the logic, when I execute it:
exec("foo() + 2", my_globals)
This is working. When I look at globals() from inside foo(), I can see other methods I defined in the module standard.py as well as the other relevant symbols I mentioned above and foo() can access all of those things.
The problem comes in when I want to make another module of functions available to the logic as well. Let's say I have a module named custom.py that has some other symbols I want the logic to access. I'm trying to make those symbols available as well by doing this:
import custom
my_globals.update(custom.__dict__)
Let's say my logic now is "bar() + 1", where "bar" is defined inside of custom.py. bar() also wants to access some of those relevant other symbols I added into my_globals.
The problem I'm running in to is that code inside of custom is only seeing the symbols defined inside custom.py, and not everything else stored in my_globals. IE, bar() can't see foo(), nor the other stuff I tucked away into my_globals.
Yet foo() can. It's code can see any other methods I defined in standard, as well as symbols defined in custom, as well as the "extra" symbols I plugged into my_globals.
Why is this happening? My expectation is that the logic being executed is run in the context of the contents of my_globals, so it seems like both foo() and bar() should be able to access any and all symbols in my_globals.
I suspect this has to do with how I'm creating my_globals. Any insight would be greatly appreciated!

Here is some insight:
"To provide the logic with access to the symbols in standard, I'm extracting out the methods from standard into a dictionary, like so:"
import standard
my_globals = standard.__dict__
Not exactly. You're just creating a local variable, my_globals that now points to standard.__dict__. Whenever you update my_globals, you're really just updating standard.__dict__.
When you add your other symbols to my_globals, again, you're just adding them to standard.__dict__.
Calling:
exec("foo() + 2", my_globals)
works great when foo is defined in standard.py, because you've added all the other methods to this module - you now have access to them all.
When you do:
import custom
my_globals.update(custom.__dict__)
You've added your "symbols" from custom.py to the standard module. All the functions in standard can access functions from custom.py after this point
Unfortunately, custom.py itself, doesn't have direct access to the methods in standard.py (unless you import them). From within custom.py, you can see that everything you've created is in standard now:
(from within custom.py):
import sys
def custom_foo():
print(dir(sys.modules['standard'])) # shows that you've put everything in here
sys.modules['standard'].foo() # calls foo from standard.py (assuming you've imported standard in your main pgm)
Above is really ugly though - you could just add a:
from standard import *
at the top of custom.py, and you would have access to everything you've added to its __dict__ instead.
I doubt you really want to do what you're attempting with the whole exec thing, but I'm not really sure what your use case is.
EDIT:
If you really want all the symbols you've attached to my_globals available to the methods of custom.py, you could call:
custom.__dict__.update(my_globals)
After this point, functions in custom.py would have access to everything you've added to the standard.dict (aka my_globals). (You've also overrode any functions defined in custom.py with functions of the same name in my_globals)
Please note, doing things this way is pretty atypical (read: somewhat ill advised).

In Python, how to access a function-internal inline function (intention: to unit test)?

I have a body of python code that contains inline functions within functions. I'd like to unit test the make_exciting inner function, so I'm trying to figure out how to invoke it directly.
def say_something_exciting(name, phrase):
def make_exciting(phrase):
return phrase + "!"
return "%s says '%s'" % (name, make_exciting(phrase))
Function say_something_exciting is written at the top level of a .py file, and is not inside a class. The py file is in the org.something module. Tried:
Invoking the function directly via org.something.say_something_exciting.make_exciting("Hello") - error: 'function' object has no attribute 'make_exciting'
Inspecting dir(org.something.say_something_exciting) and org.something.say_something_exciting.__dict__ for any paths to traverse, didn't see make_exciting anywhere.
internal_function = org.something.say_something_exciting.__dict__.get('make_exciting'), but internal_function is None.
How can I access (unit test) this inner function? This may suggest what I'm asking isn't possible. I'm generally familiar with unit testing and how to use the unittest module; accessing the function is the problem. If it's not possible, how should I re-write this code to support testing (if other than promote the inner function to a top-level function)?. Thanks!
UPDATE: In Java I often give class methods default/package visibility so they're less visible but still accessible to unit tests, looking for a python equivalent.

I don't think you can.
You'll need to return the inner function, or bind it elsewhere. It gets ugly though. You can bind the declared function to somewhere else, from inside the function:
import testmodule
testmodule.testfunc = make_exciting
Or you can return it, for example when a keyword-argument is given, such as export=True.

How to change the string to class object in another file

I already use this function to change some string to class object.
But now I have defined a new module. How can I implement the same functionality?
def str2class(str):
return getattr(sys.modules[__name__], str)
I want to think some example, but it is hard to think. Anyway, the main problem is maybe the file path problem.
If you really need an example, the GitHub code is here.
The Chain.py file needs to perform an auto action mechanism. Now it fails.
New approach:
Now I put all files under one filefold, and it works, but if I use the modules concept, it fails. So if the problem is in a module file, how can I change the string object to relative class object?
Thanks for your help.

You can do this by accessing the namespace of the module directly:
import module
f = module.__dict__["func_name"]
# f is now a function and can be called:
f()
One of the greatest things about Python is that the internals are accessible to you, and that they fit the language paradigm. A name (of a variable, class, function, whatever) in a namespace is actually just a key in a dictionary that maps to that name's value.
If you're interested in what other language internals you can play with, try running dir() on things. You'd be surprised by the number of hidden methods available on most of the objects.

You probably should write this function like this:
def str2class(s):
return globals()[s]
It's really clearer and works even if __name__ is set to __main__.

Python: uniquely identify a function from a module

I am not really a programmer but a computational statistician, so I may understand complex algorithms but not simple programming constructs.
My original problem is to check within a function if a module function is callable. I looked around and decided to go for a try (call function) - except (import module) to make it simple. I'd love to search sys.mod for this but I am running in some identifiability problems.
My current problem is that there are many ways of importing a function from a module: import module will define the function as module.function but from module import function will define it as function. Not to mention from module import function as myfunction. Therefore the same function can be called in several different ways.
My question is: is there a unique "signature" for a function that can be traced if the module is loaded? It would be fantastic to have the actual call alias to it.
ps besides: mod is mathematical function and sys.mod returns a list of loaded modules, but python (2.7) does not complain when you shadow the built-in mod function by doing the following, from sys import mod. I find this a bit awkward - is there any way to avoid this sort of shadowing programatically?

My original problem is to check within a function if a module function is callable.
By definition, all functions are callable. This will test if an object is callable: http://docs.python.org/library/functions.html#callable
Therefore the same function can be called in several different ways.
Yes, but it will be the same object. You can just use f is g to test if f and g are the same object.
Update: Why would you need to use a unique ID? Seriously, don't do this. You have is for identity tests, and the __hash__ method to define the hash function applicable.
It would be fantastic to have the actual call alias to it.
Not sure at all what you mean, but I think you just want it to always be one object. Which it is already.
mod is mathematical function and sys.mod returns a list of loaded modules, but python (2.7) does not complain to from sys import mod. I find this a bit awkward?
Then don't do that. You know about the import ... as syntax. Also mod is not by default in the global namespace (the operator % is for that).
Finally, python does complain about your import line:
>>> from sys import mod
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name mod
(Thanks to kindall for pointing this out).

Assume I have a module with the following:
def foo(): pass
bar = foo
You can easily see that they're the same functions by using is or id():
>>> import demo
>>> from demo import *
>>> demo.foo is foo
True
>>> id(demo.foo) == id(foo)
True
>>> demo.bar is foo
True
They all refer to the same code object, it's just stored under different names in the scope dictionary.

# define modulus f
def mod(a,b):
return b % a
print mod(5,2)
alias:
modulus=mod
print modulus(5,2)
this is pretty pythonic construct, and it is pretty intuitive for mathematicians
different ways of import serve to help you place a function into different "name space" for later use in your program, sometimes you wish to use a function a lot so you choose variant that is shorter to write.
you can also do something like:
myat=math.atanh
to make alias in another "name space"
and use it as:
myat(x)
as it would use math.atanh(x) - becomes shorter to write
Typical programmers approach would be define all you want to use and then use it. What you are trying in my belief is to do it "lazy" => import module when you need a function. That is why you wish to know if function is "callable".
Python is not functional programming language (e.g. like haskel) so that you can load or refer "on demand".
hope this helps.

How can I figure out in my module if the main program uses a specific variable?

I know this does not sound Pythonic, but bear with me for a second.
I am writing a module that depends on some external closed-source module. That module needs to get instantiated to be used (using module.create()).
My module attempts to figure out if my user already loaded that module (easy to do), but then needs to figure out if the module was instantiated. I understand that checking out the type() of each variable can tell me this, but I am not sure how I can get the names of variables defined by the main program. The reason for this is that when one instantiates the model, they also set a bunch of parameters that I do not want to overwrite for any reason.
My attempts so far involved using sys._getframe().f_globals and iterating through the elements, but in my testing it doesn't work. If I instantiate the module as modInst and then call the function in my module, it fails to show the modInst variable. Is there another solution to this? Sample code provided below.
import sys
if moduleName not in sys.modules:
import moduleName
modInst = moduleName.create()
else:
globalVars = sys._getframe().f_globals
for key, value in globalVars:
if value == "Module Name Instance":
return key
return moduleName.create()
EDIT: Sample code included.

Looks like your code assumes that the .create() function was called, if at all, by the immediate/direct caller of your function (which you show only partially, making it pretty hard to be sure about what's going on) and the results placed in a global variable (of the module where the caller of your function resides). It all seems pretty fragile. Doesn't that third-party module have some global variables of its own that are affected by whether the module's create has been called or not? I imagine it would -- where else is it keeping the state-changes resulting from executing the create -- and I would explore that.
To address a specific issue you raise,
I am not sure how I can get the names
of variables defined by the main
program
that's easy -- the main program is found, as a module, in sys.modules['__main__'], so just use vars(sys.modules['__main__']) to get the global dictionary of the main program (the variable names are the keys in that dictionary, along of course with names of functions, classes, etc -- the module, like any other module, has exactly one top-level/global namespace, not one for variables, a separate one for functions, etc).

Suppose the external closed-sourced module is called extmod.
Create my_extmod.py:
import extmod
INSTANTIATED=False
def create(*args,**kw):
global INSTANTIATED
INSTANTIATED=True
return extmod.create(*args,**kw)
Then require your users to import my_extmod instead of extmod directly.
To test if the create function has been called, just check the value of extmod.INSTANTIATED.
Edit: If you open up an IPython session and type import extmod, then type
extmod.[TAB], then you'll see all the top-level variables in the extmod namespace. This might help you find some parameter that changes when extmod.create is called.
Barring that, and barring the possibility of training users to import my_extmod, then perhaps you could use something like the function below. find_extmod_instance searches through all modules in sys.modules.
def find_instance(cls):
for modname in sys.modules:
module=sys.modules[modname]
for value in vars(module).values():
if isinstance(value,cls):
return value
x=find_instance(extmod.ExtmodClass) or extmod.create()