Today I wrote an "alias import function" for myself, because I need to write a script to do variable value checks for different python files.
# filename: zen_basic.py
import importlib
def from_module_import_alias(module_name, var_name, alias):
""" equal to from module import a as b """
agent = importlib.import_module(module_name)
globals()[alias] = vars(agent)[var_name]
The weird thing is, if I start a Python interactive shell, I can't import things by using this function. But by using its content outside of the function, it works.
>>> from zen_basic import *
>>> module_name = 'autor'
>>> var_name = 'food'
>>> alias = 'fd'
>>> from_module_import_alias(moduele_name, var_name, alias)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'moduele_name' is not defined
>>> from_module_import_alias(module_name, var_name, alias)
>>> fd
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'fd' is not defined
>>> agent = importlib.import_module(module_name)
>>> globals()[alias] = vars(agent)[var_name]
>>> fd
'cake'
>>>
I did 3 more experiments after:
Do python -i test.py
import zen_basic in interactive shell
call from_module_import_alias function, failed.
Do python -i zen_basic.py
call from_module_import_alias function, success.
add code which import zen_basic and call the from_module_import_alias function to test.py
Do python -i test.py, success
What's the reason that directly using from_module_import_alias function in a Python interactive shell failed?
You are updating the wrong globals. You are updating the globals of your zen_basic module, not the __main__ module (the namespace for your script or the interactive interpreter). The globals() function always returns the globals of the module in which the code was defined, not the module that called your function.
You'd have to retrieve the globals of the calling frame. Note that it rarely is advisable to modify the globals of the calling frame, but if you must then you can retrieve the calling frame by using the sys._getframe() function:
import sys
def from_module_import_alias(module_name, var_name, alias):
""" equal to from module import a as b """
agent = importlib.import_module(module_name)
calling_globals = sys._getframe(1).f_globals
calling_globals[alias] = vars(agent)[var_name]
In your experiments, python -i zen_basic.py runs zen_basic as the main script entry, and thus globals() references the __main__ module.
Related
I have some code in the form of a string and would like to make a module out of it without writing to disk.
When I try using imp and a StringIO object to do this, I get:
>>> imp.load_source('my_module', '', StringIO('print "hello world"'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: load_source() argument 3 must be file, not instance
>>> imp.load_module('my_module', StringIO('print "hello world"'), '', ('', '', 0))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: load_module arg#2 should be a file or None
How can I create the module without having an actual file? Alternatively, how can I wrap a StringIO in a file without writing to disk?
UPDATE:
NOTE: This issue is also a problem in python3.
The code I'm trying to load is only partially trusted. I've gone through it with ast and determined that it doesn't import anything or do anything I don't like, but I don't trust it enough to run it when I have local variables running around that could get modified, and I don't trust my own code to stay out of the way of the code I'm trying to import.
I created an empty module that only contains the following:
def load(code):
# Delete all local variables
globals()['code'] = code
del locals()['code']
# Run the code
exec(globals()['code'])
# Delete any global variables we've added
del globals()['load']
del globals()['code']
# Copy k so we can use it
if 'k' in locals():
globals()['k'] = locals()['k']
del locals()['k']
# Copy the rest of the variables
for k in locals().keys():
globals()[k] = locals()[k]
Then you can import mymodule and call mymodule.load(code). This works for me because I've ensured that the code I'm loading does not use globals. Also, the global keyword is only a parser directive and can't refer to anything outside of the exec.
This really is way too much work to import the module without writing to disk, but if you ever want to do this, I believe it's the best way.
Here is how to import a string as a module (Python 2.x):
import sys,imp
my_code = 'a = 5'
mymodule = imp.new_module('mymodule')
exec my_code in mymodule.__dict__
In Python 3, exec is a function, so this should work:
import sys,imp
my_code = 'a = 5'
mymodule = imp.new_module('mymodule')
exec(my_code, mymodule.__dict__)
Now access the module attributes (and functions, classes etc) as:
print(mymodule.a)
>>> 5
To ignore any next attempt to import, add the module to sys:
sys.modules['mymodule'] = mymodule
imp.new_module is deprecated since python 3.4, but it still works as of python 3.9
imp.new_module was replaced with importlib.util.module_from_spec
importlib.util.module_from_spec
is preferred over using types.ModuleType to create a new module as
spec is used to set as many import-controlled attributes on the module
as possible.
importlib.util.spec_from_loader
uses available loader APIs, such as InspectLoader.is_package(), to
fill in any missing information on the spec.
these module attributes are __builtins__ __doc__ __loader__ __name__ __package__ __spec__
import sys, importlib.util
def import_module_from_string(name: str, source: str):
"""
Import module from source string.
Example use:
import_module_from_string("m", "f = lambda: print('hello')")
m.f()
"""
spec = importlib.util.spec_from_loader(name, loader=None)
module = importlib.util.module_from_spec(spec)
exec(source, module.__dict__)
sys.modules[name] = module
globals()[name] = module
# demo
# note: "if True:" allows to indent the source string
import_module_from_string('hello_module', '''if True:
def hello():
print('hello')
''')
hello_module.hello()
You could simply create a Module object and stuff it into sys.modules and put your code inside.
Something like:
import sys
from types import ModuleType
mod = ModuleType('mymodule')
sys.modules['mymodule'] = mod
exec(mycode, mod.__dict__)
If the code for the module is in a string, you can forgo using StringIO and use it directly with exec, as illustrated below with a file named dynmodule.py.
Works in Python 2 & 3.
from __future__ import print_function
class _DynamicModule(object):
def load(self, code):
execdict = {'__builtins__': None} # optional, to increase safety
exec(code, execdict)
keys = execdict.get(
'__all__', # use __all__ attribute if defined
# else all non-private attributes
(key for key in execdict if not key.startswith('_')))
for key in keys:
setattr(self, key, execdict[key])
# replace this module object in sys.modules with empty _DynamicModule instance
# see Stack Overflow question:
# https://stackoverflow.com/questions/5365562/why-is-the-value-of-name-changing-after-assignment-to-sys-modules-name
import sys as _sys
_ref, _sys.modules[__name__] = _sys.modules[__name__], _DynamicModule()
if __name__ == '__main__':
import dynmodule # name of this module
import textwrap # for more readable code formatting in sample string
# string to be loaded can come from anywhere or be generated on-the-fly
module_code = textwrap.dedent("""\
foo, bar, baz = 5, 8, 2
def func():
return foo*bar + baz
__all__ = 'foo', 'bar', 'func' # 'baz' not included
""")
dynmodule.load(module_code) # defines module's contents
print('dynmodule.foo:', dynmodule.foo)
try:
print('dynmodule.baz:', dynmodule.baz)
except AttributeError:
print('no dynmodule.baz attribute was defined')
else:
print('Error: there should be no dynmodule.baz module attribute')
print('dynmodule.func() returned:', dynmodule.func())
Output:
dynmodule.foo: 5
no dynmodule.baz attribute was defined
dynmodule.func() returned: 42
Setting the '__builtins__' entry to None in the execdict dictionary prevents the code from directly executing any built-in functions, like __import__, and so makes running it safer. You can ease that restriction by selectively adding things to it you feel are OK and/or required.
It's also possible to add your own predefined utilities and attributes which you'd like made available to the code thereby creating a custom execution context for it to run in. That sort of thing can be useful for implementing a "plug-in" or other user-extensible architecture.
you could use exec or eval to execute python code as a string. see here, here and here
The documentation for imp.load_source says (my emphasis):
The file argument is the source file, open for reading as text, from the beginning. It must currently be a real file object, not a user-defined class emulating a file.
... so you may be out of luck with this method, I'm afraid.
Perhaps eval would be enough for you in this case?
This sounds like a rather surprising requirement, though - it might help if you add some more to your question about the problem you're really trying to solve.
I have some code in the form of a string and would like to make a module out of it without writing to disk.
When I try using imp and a StringIO object to do this, I get:
>>> imp.load_source('my_module', '', StringIO('print "hello world"'))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: load_source() argument 3 must be file, not instance
>>> imp.load_module('my_module', StringIO('print "hello world"'), '', ('', '', 0))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: load_module arg#2 should be a file or None
How can I create the module without having an actual file? Alternatively, how can I wrap a StringIO in a file without writing to disk?
UPDATE:
NOTE: This issue is also a problem in python3.
The code I'm trying to load is only partially trusted. I've gone through it with ast and determined that it doesn't import anything or do anything I don't like, but I don't trust it enough to run it when I have local variables running around that could get modified, and I don't trust my own code to stay out of the way of the code I'm trying to import.
I created an empty module that only contains the following:
def load(code):
# Delete all local variables
globals()['code'] = code
del locals()['code']
# Run the code
exec(globals()['code'])
# Delete any global variables we've added
del globals()['load']
del globals()['code']
# Copy k so we can use it
if 'k' in locals():
globals()['k'] = locals()['k']
del locals()['k']
# Copy the rest of the variables
for k in locals().keys():
globals()[k] = locals()[k]
Then you can import mymodule and call mymodule.load(code). This works for me because I've ensured that the code I'm loading does not use globals. Also, the global keyword is only a parser directive and can't refer to anything outside of the exec.
This really is way too much work to import the module without writing to disk, but if you ever want to do this, I believe it's the best way.
Here is how to import a string as a module (Python 2.x):
import sys,imp
my_code = 'a = 5'
mymodule = imp.new_module('mymodule')
exec my_code in mymodule.__dict__
In Python 3, exec is a function, so this should work:
import sys,imp
my_code = 'a = 5'
mymodule = imp.new_module('mymodule')
exec(my_code, mymodule.__dict__)
Now access the module attributes (and functions, classes etc) as:
print(mymodule.a)
>>> 5
To ignore any next attempt to import, add the module to sys:
sys.modules['mymodule'] = mymodule
imp.new_module is deprecated since python 3.4, but it still works as of python 3.9
imp.new_module was replaced with importlib.util.module_from_spec
importlib.util.module_from_spec
is preferred over using types.ModuleType to create a new module as
spec is used to set as many import-controlled attributes on the module
as possible.
importlib.util.spec_from_loader
uses available loader APIs, such as InspectLoader.is_package(), to
fill in any missing information on the spec.
these module attributes are __builtins__ __doc__ __loader__ __name__ __package__ __spec__
import sys, importlib.util
def import_module_from_string(name: str, source: str):
"""
Import module from source string.
Example use:
import_module_from_string("m", "f = lambda: print('hello')")
m.f()
"""
spec = importlib.util.spec_from_loader(name, loader=None)
module = importlib.util.module_from_spec(spec)
exec(source, module.__dict__)
sys.modules[name] = module
globals()[name] = module
# demo
# note: "if True:" allows to indent the source string
import_module_from_string('hello_module', '''if True:
def hello():
print('hello')
''')
hello_module.hello()
You could simply create a Module object and stuff it into sys.modules and put your code inside.
Something like:
import sys
from types import ModuleType
mod = ModuleType('mymodule')
sys.modules['mymodule'] = mod
exec(mycode, mod.__dict__)
If the code for the module is in a string, you can forgo using StringIO and use it directly with exec, as illustrated below with a file named dynmodule.py.
Works in Python 2 & 3.
from __future__ import print_function
class _DynamicModule(object):
def load(self, code):
execdict = {'__builtins__': None} # optional, to increase safety
exec(code, execdict)
keys = execdict.get(
'__all__', # use __all__ attribute if defined
# else all non-private attributes
(key for key in execdict if not key.startswith('_')))
for key in keys:
setattr(self, key, execdict[key])
# replace this module object in sys.modules with empty _DynamicModule instance
# see Stack Overflow question:
# https://stackoverflow.com/questions/5365562/why-is-the-value-of-name-changing-after-assignment-to-sys-modules-name
import sys as _sys
_ref, _sys.modules[__name__] = _sys.modules[__name__], _DynamicModule()
if __name__ == '__main__':
import dynmodule # name of this module
import textwrap # for more readable code formatting in sample string
# string to be loaded can come from anywhere or be generated on-the-fly
module_code = textwrap.dedent("""\
foo, bar, baz = 5, 8, 2
def func():
return foo*bar + baz
__all__ = 'foo', 'bar', 'func' # 'baz' not included
""")
dynmodule.load(module_code) # defines module's contents
print('dynmodule.foo:', dynmodule.foo)
try:
print('dynmodule.baz:', dynmodule.baz)
except AttributeError:
print('no dynmodule.baz attribute was defined')
else:
print('Error: there should be no dynmodule.baz module attribute')
print('dynmodule.func() returned:', dynmodule.func())
Output:
dynmodule.foo: 5
no dynmodule.baz attribute was defined
dynmodule.func() returned: 42
Setting the '__builtins__' entry to None in the execdict dictionary prevents the code from directly executing any built-in functions, like __import__, and so makes running it safer. You can ease that restriction by selectively adding things to it you feel are OK and/or required.
It's also possible to add your own predefined utilities and attributes which you'd like made available to the code thereby creating a custom execution context for it to run in. That sort of thing can be useful for implementing a "plug-in" or other user-extensible architecture.
you could use exec or eval to execute python code as a string. see here, here and here
The documentation for imp.load_source says (my emphasis):
The file argument is the source file, open for reading as text, from the beginning. It must currently be a real file object, not a user-defined class emulating a file.
... so you may be out of luck with this method, I'm afraid.
Perhaps eval would be enough for you in this case?
This sounds like a rather surprising requirement, though - it might help if you add some more to your question about the problem you're really trying to solve.
I have a project that tries to create a new module dynamically and then in a subsequent exec statement tries to import that module.
import imp
s="""
class MyClass(object):
def __init__(self):
pass
def foo(self):
pass
"""
mod = imp.new_module("testmodule.testA")
exec s in mod.__dict__
exec "import testmodule.testA"
But this throws this exception:
Traceback (most recent call last):
File "test.py", line 14, in <module>
exec "import testmodule.testA"
File "<string>", line 1, in <module>
ImportError: No module named testmodule.testA
I've tried several things: adding it to sys.modules, creating a scope dict containing the name and the module as well. But no dice. I can see testmodule.testA when I do a print locals() in my exec statement, but I can't import it. What am I missing here?
Thank you.
You'll need to add your module to the sys.modules structure for import to find it:
import sys
sys.modules['testmodule.testA'] = mod
You already have the module object however and there is not much point in importing it again. mod is a reference to what Python would otherwise 'import' already.
The following would work without an import call:
mod.MyClass
If you need the module in an exec call, add it to the namespace there:
exec 'instance = MyClass()' in {'MyClass': mod.MyClass}
I have two modules misc.py and main.py and would like to define all classes present in misc.py in main.py.
Below is the code
#misc.py
class dummy:
def __init__(self):
pass
def dummyPrint(self):
print "Welcome to python"
#main.py
import misc
dummyObj = dummy()
dummyObj.dummyPrint()
Is this the right way to go ? I do not see any output i.e., Welcome to python
$python misc_main.py misc.py
EDIT: I added the statement from misc import dummy and i am getting the following error
$python misc_main.py main.py
Traceback (most recent call last):
File "misc_main.py", line 5, in <module>
dummyObj = dummmy()
NameError: name 'dummmy' is not defined
When you do the following command, you are calling misc_main.py from the interpreter with misc.py as an argument.
python misc_main.py misc.py
Since misc_main is not reading command line arguments, this is equivalent to
python misc_main.py
I am surprised that you do not get errors, in either case. You need to import the actual class if you want to get output.
from misc import dummy
dummyObj = dummy()
dummyObj.dummyPrint()
Note, I am assuming your main file is actually in called misc_main.py rather than main.py as you have stated in your question. Otherwise you are not invoking the correct file.
I'm using Python for a weeks now and i'm confronted to an issue with dynamic import.
I have a file Test.py that in which a class is defined. I would like to use this class after the dynamic import of Test.py from another file.
My final goal is more complex but I simplified it but i still get the same problem.
File : Test.py
class Test :
def __init__ ( self ) :
print ( "instance" )
File : Main.py
def allImports ( ) :
__import__ ( "Test" )
What i get :
>>> import Main
>>> Main.allImports()
>>> myInstance = Test ()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'Test' is not defined
I cannot specify in the fromlist which element from Test.py i have to import because i'm not supposed to know them.
What should i do ?
For a solution closer to your intent:
import importlib
def allImports(globals):
mod = importlib.import_module('Test', globals['__name__'])
try:
keys = mod.__all__
except AttributeError:
keys = dir(mod)
for key in keys:
if not key.startswith('_'):
globals[key] = getattr(mod, key)
# …
allImports(globals())
Test # should work, you can also look into dir(Test) to find the class.
If your module doesn't have an __all__ the code above /will/ clobber your namespace something fierce. Either make sure you define __all__, or modify allImports() to only import the things you want. (E.g. only classes, or only classes defined in the module. This part really depends on your use case.)
When using __import__() to load a module, you have to look it up in sys.modules:
>>> import sys
>>> import Main
>>> Main.allImports()
>>> myInstance = sys.modules['Test'].Test()
instance
>>>
More information in the documentation and here, here, and here.
this code makes __import__ ( "Test" ) a local variable, so you can't access it outside the function.
def allImports ( ) :
__import__ ( "Test" )
try:
def allImports ( ) :
test= __import__ ( "Test" )
return test #return the module
>>> import Main
>>> x=Main.allImports() #store the returned module in x
>>> myInstance = x.Test ()
instance
>>>myInstance
<Test.Test instance at 0x011D7F80>
__import__ doesn't modify magically neither global nor local namespaces.
Modules and classes are first class citizens in Python i.e., you can use them as any other object in Python (bind to a name, pass as a parameter to a function, return as a value from a function).
def allImports():
return __import__("Test")
Test_module = allImports()
Test = Test_module.Test # class
test_instance = Test()
If the above code is inside a function then to put Test into global namespace: globals()['Test'] = Test. Note most probably you don't need it and there are better ways to do whatever you want without modifying global namespace inside a function.
Usage of __import__() is discouraged use importlib.import_module() instead.
If the name of the module and the class are known you could just write at the module level:
from Test import Test