I am trying to create a dynamic method executor, where I have a list that will always contain two elements. The first element is the name of the file, the second element is the name of the method to execute.
How can I achieve this?
My below code unfortunately doesn't work, but it will give you an good indication of what I am trying to achieve.
from logic.intents import CenterCapacity
def method_executor(event):
call_reference = ['CenterCapacity', 'get_capacity']
# process method call
return call_reference[0].call_reference[1]
Thanks!
You can use __import__ to look up the module by name and then then use getattr to find the method. For example if the following code is in a file called exec.py then
def dummy(): print("dummy")
def lookup(mod, func):
module = __import__(mod)
return getattr(module, func)
if __name__ == "__main__":
lookup("exec","dummy")()
will output
dummy
Addendum
Alternatively importlib.import_module can be used, which although a bit more verbose, may be easier to use.
The most important difference between these two functions is that import_module() returns the specified package or module (e.g. pkg.mod), while __import__() returns the top-level package or module (e.g. pkg).
def lookup(mod, func):
import importlib
module = importlib.import_module(mod)
return getattr(module, func)
starting from:
from logic.intents import CenterCapacity
def method_executor(event):
call_reference = ['CenterCapacity', 'get_capacity']
# process method call
return call_reference[0].call_reference[1]
Option 1
We have several options, the first one is using a class reference and the getattr. For this we have to remove the ' around the class and instantiate the class before calling a reference (you do not have to instantiate the class when the method is a staticmethod.)
def method_executor(event):
call_reference = [CenterCapacity, 'get_capacity'] # We now store a class reference
# process method call
return getattr(call_reference[0](), call_reference[1])
option 2
A second option is based on this answer. It revolves around using the getattr method twice. We firstly get module using sys.modules[__name__] and then get the class from there using getattr.
import sys
def method_executor(event):
call_reference = ['CenterCapacity', 'get_capacity']
class_ref = getattr(sys.modules[__name__], call_reference[0])
return getattr(class_ref, call_reference[1])
Option 3
A third option could be based on a full import path and use __import__('module.class'), take a look at this SO post.
(Note: This answer assumes that the necessary imports have already happened, and you just need a mechanism to invoke the functions of the imported modules. If you also want the import do be done by some program code, I will have to add that part, using importlib library)
You can do this:
globals()[call_reference[0]].__dict__[call_reference[1]]()
Explanation:
globals() returns a mapping between global variable names and their referenced objects. The imported module's name counts as one of these global variables of the current module.
Indexing this mapping object with call_reference[0] returns the module object containing the function to be called.
The module object's __dict__ maps each attribute-name of the module to the object referenced by that attribute. Functions defined in the module also count as attributes of the module.
Thus, indexing __dict__ with the function name call_reference[1] returns the function object.
Related
I wish to create my own variation of amoffat'ssh module, where it can import pretty much any command from user's UNIX path, such as:
from sh import hg
However, I am having a hard time finding a way to intercept / override python's own import [...] and from [...] import [...]. At this point I simply need a way to at least get [the name of] the object of the from import, at which point I can simply setattr() and partial() my way from there, I hope. I'm at a complete loss of how to do this at the moment, however, and hence, have no code to show for it.
The gist of what I'm going for:
from test import t # Even though "t" doesn't exist in the module (yet)
Any help with the full code would be greatly appreciated!
Final Answer, consolidated:
def __getattr__(name):
if name == '__path__': raise AttributeError
print(name)
There is actually a straightforward way if you are on Python 3.7+, PEP-562, which allows you to define __getattr__ at the module level:
def __getattr__(name):
if name == "t":
return "magic"
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
There is also a function __dir__ that you can define to declare what the builtin dir() will say about names in your module.
What sh does is more sophisticated, as they want to support versions below 3.7: Modifying sys.modules and replacing the module with a special object that pretends to be a module.
As #L3viathan pointed out, this is easy starting with Python 3.7: just define a __getattr__ function in your special module. So, for example, you could create an "echo" module (just returns the name of the object you requested) like this:
echo.py (Python >=3.7)
def __getattr__(name):
return name
Then you could use it like this:
from echo import x
print(repr(x))
# 'x'
On earlier versions of Python, you have to subclass the module, as hinted in PEP-562. This also works in Python 3.7.
echo.py (Python >=2)
import sys, types
class EchoModule(types.ModuleType):
def __getattr__(self, name):
return name
sys.modules[__name__] = EchoModule(__name__)
You would use this the same way as the 3.7 version: from echo import something.
Update
For some reason Python tries to retrieve the attribute twice for each from echo import <x> call. It also calls __getattr__('__path__') when the module is loaded. You can avoid side effects in these cases with the following code:
echo.py (only define attributes once)
import sys, types
class EchoModule(types.ModuleType):
def __getattr__(self, name):
# don't define __path__ attribute
if name == '__path__':
raise AttributeError
print("importing {}".format(name))
# create the attribute in case it's required again
setattr(self, name, name)
# return the new attribute
return getattr(self, name)
sys.modules[__name__] = EchoModule(__name__)
This code creates an attribute in the echo module each time a previously unused attribute is imported (sort of like collections.defaultdict). Then, if Python tries to import that same attribute again later, it will pull it directly from the module instead of calling __getattr__ (this is normal behavior for object attributes).
There is also some code here to avoid setting a spurious __path__ attribute; this also avoids running your code when __path__ is requested. Note that this may actually be the most important part; when I tested, just raising AttributeError for __path__ was enough to prevent the double-access to the named attribute.
What is the equivalent of import * in Python using functions (presumably from importlib)?
I know that you can import a module with mod = __import__(...), which will delegate to whatever the currently configured implementation is. You can also do something like
mod_spec = importlib.utl.spec_from_file_location(...)
mod = importlib.util.module_from_spec(mod_spec)
mod_spec.loader.exec_module(mod)
which allows you to do crazy things like injecting things into the module by inserting them before the call to exec_module. (Courtesy of https://stackoverflow.com/a/67692/2988730 and https://stackoverflow.com/a/38650878/2988730)
However, my question remains. How does import * work in function form? What function determines which names to load from a module depending on the presence/contents of __all__?
There's no function for from whatever import *. In fact, there's no function for import whatever, either! When you do
mod = __import__(...)
the __import__ function is only responsible for part of the job. It provides you with a module object, but you have to assign that module object to a variable separately. There's no function that will import a module and assign it to a variable the way import whatever does.
In from whatever import *, there are two parts:
prepare the module object for whatever
assign variables
The "prepare the module object" part is almost identical to in import whatever, and it can be handled by the same function, __import__. There's a minor difference in that import * will load any not-yet-loaded submodules in a package's __all__ list; __import__ will handle this for you if you provide fromlist=['*']:
module = __import__('whatever', fromlist=['*'])
The part about assigning names is where the big differences occur, and again, you have to handle that yourself. It's fairly straightforward, as long as you're at global scope:
if hasattr(module, '__all__'):
all_names = module.__all__
else:
all_names = [name for name in dir(module) if not name.startswith('_')]
globals().update({name: getattr(module, name) for name in all_names})
Function scopes don't support assigning variables determined at runtime.
I'm trying to implement an "import hook" in Python 3. The hook is supposed to add an attribute to every class that is imported. (Not really every class, but for the sake of simplifying the question, let's assume so.)
I have a loader defined as follows:
import sys
class ConfigurableImports(object):
def find_module(self, fullname, path):
return self
def create_module(self, spec):
# ???
def exec_module(self, module):
# ???
sys.meta_path = [ConfigurableImports()]
The documentation states that as of 3.6, loaders will have to implement both create_module and exec_module. However, the documentation also has little indication what one should do to implement these, and no examples. My use case is very simple because I'm only loading Python modules and the behavior of the loader is supposed to be almost exactly the same as the default behavior.
If I could, I'd just use importlib.import_module and then modify the module contents accordingly; however, since importlib leverages the import hook, I get an infinite recursion.
EDIT: I've also tried using the imp module's load_module, but this is deprecated.
Is there any easy way to implement this functionality with import hooks, or am I going about this the wrong way?
Imho, if you only need to alter the module, that is, play with it after it has been found and loaded, there's no need to actually create a full hook that finds, loads and returns the module; just patch __import__.
This can easily be done in a few lines:
import builtins
from inspect import getmembers, isclass
old_imp = builtins.__import__
def add_attr(mod):
for name, val in getmembers(mod):
if isclass(val):
setattr(val, 'a', 10)
def custom_import(*args, **kwargs):
m = old_imp(*args, **kwargs)
add_attr(m)
return m
builtins.__import__ = custom_import
Here, __import__ is replaced by your custom import that calls the original __import__ to get the loaded module and then calls a function add_attr that does the actual modification of the classes in a module (with getmembers and isclass from inspect) before returning the module.
Of course this is created in a way that when you import the script, the changes are made.
You can and probably should create auxiliary functions that restore and change it again if needed i.e things like:
def revert(): builtins.__import__ = old_imp
def apply(): builtins.__import__ = custom_import
A context-manager would also make this implementation cleaner.
I have a python function that takes a imported module as a parameter:
def printModule(module):
print("That module is named '%s'" % magic(module))
import foo.bar.baz
printModule(foo.bar.baz)
What I want is to be able to extract the module name (in this case foo.bar.baz) from a passed reference to the module. In the above example, the magic() function is a stand-in for the function I want.
__name__ would normally work, but that requires executing in the context of the passed module, and I'm trying to extract this information from merely a reference to the module.
What is the proper procedure here?
All I can think of is either doing string(module), and then some text hacking, or trying to inject a function into the module that I can then call to have return __name__, and neither of those solutions is elegant. (I tried both these, neither actually work. Modules can apparently permute their name in the string() representation, and injecting a function into the module and then calling it just returns the caller's context.)
The __name__ attribute seems to work:
def magic(m):
return m.__name__
If you have a string with the module name, you can use pkgutil.
import pkgutil
pkg = pkgutil.get_loader(module_name)
print pkg.fullname
From the module itself,
import pkgutil
pkg = pkgutil.get_loader(module.__name__)
print pkg.fullname
I have a function called get_account(param1,param2)
in run time I need to replace this function with the function mock_get_account(param1,param2)
so when the system calls get_account(param1,param2) I need the mock_get_account(param1,param2) to be called instead.
I tried this code:
package.get_account=self.mock_get_account
package.get_account(x,y)
but still the get_account runs instead of the mock_get_account
I'm new to python and I don't know if this is even possible but I have seen the lamda function and I know that function programming is possible in python. Thanks
Edit:
if i do the following:
package.get_account=self.mock_get_account
package.get_account(x,y)
then every thing is ok, meaning the mock_get_account is called, but in mu code I the following code i do a post self.client.post(url, data=data, follow=True) that triggers the package.get_account and this is not working:
package.get_account=self.mock_get_account
package.get_account(x,y)
#the folowing call will trigger the package.get_account(x,y) function in a django url #callback
self.client.post(url, data=data, follow=True)
meaning it calls the old function, also get_account(param1,param2) is defined in side a file, and is not a child function of a class and mock_get_account(self,param1,param2) is defined in a class Test and is called inside the Test.test_account - function
This is very opinionated and does not (directly) answer your question, but hopefully solves your problem.
A better practice is to use a subclass with your mock_get_account's implementation override the parent get_account method, example below:
class A(object):
def get_account(self):
return 1
def post(self):
return self.get_account()
class B(A):
def get_account(self):
return 2 # your original mock_get_account implementation
a = A()
print(a.get_account())
b = B()
print(b.post()) # this .post will trigger the overridden implementation of get_account
My guess is that the code implementing self.client.post has access to get_account through an import statement that looks like from package import get_account.
from package import get_account will first load package if it hasn't been already imported. Then it will look for a name get_account in that module, and whatever object that was bound to will be bound in the importing package's namespace, also under the name get_account. Thereafter the two names refer to the same object, but they are not the same name.
So if your mocking code comes along after this point, it sets the name get_account in package to instead refer to mock_get_account. But that'll only affect code that reads get_account from package again; anything that's already imported that name specially won't be affected.
If the code behind self.client.post instead had access only to package through import package, and was calling package.get_account it would work, because it's then only the object representing the package module that has been bound in the importing module's namespace. package.get_account would be reading an attribute of that object, and so would get whatever the current value is. If the from package import get_account appeared at function local scope rather than module scope, then this would behave similarly.
If I'm correct and your code is structured this way, then it's unfortunately not really package.get_account you need to rebind to a mock, but the get_account name in the module where self.client.post comes from (as well as any other modules which may call it).