Let's assume, that there are following minimalistic python classes inside one module, e.g. Module:
module/
__init__.py
db.py
document.py
db.py
import yaml
class DB(object):
config = {}
#classmethod
def load_config(cls, config_path):
cls.config = yaml.load(open(config_path, 'r').read())
and document.py
from .db import DB
class Document(object):
db = None
def __init__(self):
self.db = DB()
End-user is going to use such Module as follows:
from Module import DB, Document
DB.load_config('/path/to/config.yml')
Document.do_some_stuff()
doc1 = Document()
doc2 = Document.find(...)
doc2.update_something(...)
doc2.save()
It is expected that Document class and every instance of it will have internally an access to class DB with a config specified by user. However, since Document performs an internal import of DB class (from .db import DB) it receives a 'fresh' DB class with default config.
I did a lot of searches, most of questions and answers are about module-wide configs, but not specified by the end user.
How can I achieve such functionality? I guess that there is some architectural problem here, but what is the most simple way to solve it?
Perhaps this isn't the most appropriate answer, but a few months back I wrote a module called aconf for this exact purpose. It's a memory-based global configuration module for Python written in 8 lines. The idea is you can do the following:
You create a Config object to force the user to input the configuration your program requires (in this case it's inside config.py):
""" 'Config' class to hold our desired configuration parameters.
Note:
This is technically not needed. We do this so that the user knows what he/she should pass
as a config for the specific project. Note how we also take in a function object - this is
to demonstrate that one can have absolutely any type in the global config and is not subjected
to any limitations.
"""
from aconf import make_config
class Config:
def __init__(self, arg, func):
make_config(arg=arg, func=func)
You consume your configuration throughout your module (in this case, inside functionality.py):
""" Use of the global configuration through the `conf` function. """
from aconf import conf
class Example:
def __init__(self):
func = conf().func
arg = conf().arg
self.arg = func(arg)
And then use it (in this case inside main.py):
from project.config import Config
from project.functionality import Example
# Random function to demonstrate we can pass _anything_ to 'make_config' inside 'Config'.
def uppercase(words):
return words.upper()
# We create our custom configuration without saving it.
Config(arg="hello world", func=uppercase)
# We initialize our Example object without passing the 'Config' object to it.
example = Example()
print(example.arg)
# >>> "HELLO WORLD"
The entire aconf module is the following:
__version__ = "1.0.1"
import namedtupled
def make_config(**kwargs):
globals()["aconf"] = kwargs
conf = lambda: namedtupled.map(globals()["aconf"])
config = lambda: globals()["aconf"]
... in essence, you just save your configuration to globals() during runtime.
It's so stupid it makes me wonder if you should even be allowed to do this. I wrote aconf for fun, but have never personally used it in a big project. The reality is, you might run into the problem of making your code weird for other developers.
Related
I am creating a commands system in Python. I have module vkcommands that has a class that processes commands from chat (this is a chat-bot), and inside it, I also have class VKCommand with attributes like name, usage, min_rank, etc. Then I have module vkcmds with submodules that implement these commands:
...
vkcommands.py
vkcmds
|- __init__.py # empty
|- add_group.py
|- another_cmd.py
|- ...
Implementations of commands (e.g. add_group) look like this:
import ranks
import vkcommands
from vkcommands import VKCommand
class AddGroup(VKCommand):
def __init__(self, kristy):
VKCommand.__init__(self, kristy,
label='create',
# ... (other attributes)
min_rank=ranks.Rank.USER)
def execute(self, chat, peer, sender, args=None, attachments=None):
# implementation (called from vkcommands.py)
When a user sends a message in the chat, the command manager analyzes it and looks through the registered commands list to see if this is an ordinary message or a bot command. Currently I register all commands in the commands list manually like this:
class VKCommandsManager:
def __init__(self, kristy):
from vkcmds import (
add_group,
next_class
)
self.kristy = kristy
self.commands = (
add_group.AddGroup(kristy),
next_class.NextClass(kristy)
)
Now I would like all commands to be registered automatically using reflections instead. In Java, I'd iterate over all classes in my commands package, reflectively getConstructor of each, call it to retrieve the VKCommand object, and add it to the commands list.
How can I do so in Python? Again, what I need is to:
iterate over all submodules in module (folder) vkcmds/;
for each submodule, check if there is some class X that extends VKCommand inside;
if (2) is true, then call the constructor of that class with one argument (it is guaranteed that the constructor for all commands only has one argument of a known type (my bot's main class));
add the object (? extends VKCommand) constructed in (3) to the commands list that I can iterate over later.
With this file structure:
- Project
├─ commands
| ├─ base.py
| ├─ baz.py
| └─ foo_bar.py
|
└─ main.py
And the following inside the commands directory files:
base.py
class VKCommand:
""" We will inherit from this class if we want to include the class in commands. """
baz.py
from commands.base import VKCommand
class Baz(VKCommand):
pass
def baz():
""" Random function we do not want to retrieve.
foo_bar.py
from .base import VKCommand
class Foo(VKCommand):
""" We only want to retrieve this command. """
pass
class Bar:
""" We want to ignore this class. """
pass
def fizz():
""" Random function we do not want to retrieve. """
We can retrieve the class instances and names directly using the following code:
main.py
"""
Dynamically load all commands located in submodules.
This file is assumed to be at most 1 level higher than the
specified folder.
"""
import pyclbr
import glob
import os
def filter_class(classes):
inherit_from = 'VKCommand'
classes = {name: info for name, info in classes.items() if inherit_from in info.super}
return classes
# Locate all submodules and classes that it contains without importing it.
folder = 'commands' # `vkcmds`.
submodules = dict()
absolute_search_path = os.path.join(os.path.dirname(__file__), folder, '*.py')
for path in glob.glob(absolute_search_path):
submodule_name = os.path.basename(path)[:-3]
all_classes = pyclbr.readmodule(f"commands.{submodule_name}")
command_classes = filter_class(all_classes)
if command_classes:
submodules[submodule_name] = command_classes
# import the class and store an instance of the class into the command list
class_instances = dict()
for submodule_name, class_names in submodules.items():
module = __import__(f"{folder}.{submodule_name}")
submodule = getattr(module, submodule_name)
for class_name in class_names:
class_instance = getattr(submodule, class_name)
class_instances[class_name] = class_instance
print(class_instances)
Explanation
The solution is twofold. It first locates all submodules that have a class which inherit from VKCommand and are located in the folder 'commands`. This leads to the following output containing the module and the class that have to be imported and instantiated respectively:
{'baz': {'Baz': <pyclbr.Class object at 0x000002BF886357F0>}, 'foo_bar': {'Foo': <pyclbr.Class object at 0x000002BF88660668>}}
The second part of the code imports the correct module and class name at run time. The variable class_instance contains the class name and a reference to the class which can be used to instantiate it. The final output will be:
{'Baz': <class 'commands.baz.Baz'>, 'Foo': <class 'commands.foo_bar.Foo'>}
Important notes:
The code only works when importing modules that are 1 dictionary deeper. If you want to use it recursively, you will have to locate the relative path difference and update the pyclbr.readmodule and __import__ with the correct (full) relative import path.
Only the modules that contain a class which inherit from VKCommand get loaded. All other modules are not imported, and have to be imported manually.
I believe that you can make an array of all the commands you have in your folder and then go over them and instantiate the objects.
in __init__.py
all_commands = [AddGroup, AnotherCmd, ...]
instantiate them like this:
objects = [Cmd(arg1, arg2, ...) for Cmd in all_commands]
Edit:
you could also retrieve the class names with the method that you said you had of getting all class names in the folder.
Im trying to import files on Flask app in base of url route. I started to coding python few days ago so i havent idea if i doing it well. I write this on :
#app.route('/<file>')
def call(file):
__import__('controller.'+file)
hello = Example('Hello world')
return hello.msg
And i have other file called example.py into a controller folder that contains this:
class Example:
def __init__(self, msg):
self.msg = msg
So i start from terminal the app and i try to enter to localhost:5000/example.
Im trying to show in screen Hello world but give me the next error:
NameError: global name 'Example' is not defined
Thanks for all!
__import__ returns the newly imported module; names from that module are not added to your globals, so you need to get the Example class as an attribute from the returned module:
module = __import__('controller.'+file)
hello = module.Example('Hello world')
__import__ is rather low-level, you probably want to use importlib.import_module() instead:
import importlib
module = importlib.import_module('controller.'+file)
hello = module.Example('Hello world')
If you need to dynamically get the classname too, use getattr():
class_name = 'Example'
hello_class = getattr(module, class_name)
hello = hello_class('Hello world')
The Werkzeug package (used by Flask) offers a helpful functions here: werkzeug.utils.import_string() imports an object dynamically:
from werkzeug.utils import import_string
object_name = 'controller.{}:Example'.format(file)
hello_class = import_string(object_name)
This encapsulates the above process.
You'll need to be extremely careful with accepting names from web requests and using those as module names. Please do sanitise the file argument and only allow alphanumerics to prevent relative imports from being used.
You could use the werkzeug.utils.find_modules() function to limit the possible values for file here:
from werkzeug.utils import find_modules, import_string
module_name = 'controller.{}'.format(file)
if module_name not in set(find_modules('controller')):
abort(404) # no such module in the controller package
hello_class = import_string(module_name + ':Example')
I think you might not add the directory to the file, add the following code into the previous python program
# Add another directory
import sys
sys.path.insert(0, '/your_directory')
from Example import Example
There are two ways for you to do imports in Python:
import example
e = example.Example('hello world')
or
from example import Example
e = Example('hello world')
I have a third-party module (cx_Oracle) that I'd like to import whose location is unknown from environment to environment. I am currently using pythons configparser so I thought it would be a neat trick to set the location of the module within the config parser, append that location to path, and then import the third-party module from there.
This worked all fine and dandy until I began to refactor my code and started to split out logic into their own class/methods:
class Database:
def __init__(self, config):
self.CONFIG=config
sys.path.append(self.CONFIG.cx_oracle_path)
from cx_Oracle import cx_Oracle
self.open()
def open(self):
self.CONNECTION = cx_Oracle.Connection(self.CONFIG.username,
self.CONFIG.password,
self.CONFIG.db_sid)
self.CURSOR = self.CONNECTION.cursor()
....
....
....
Of course, the open method does not know what to do because cx_Oracle was defined in init and so the open method cannot see it.
I can't picture the proper way to do this, so I'm assuming I am over thinking this. What should I do instead so that open (and all other methods within the Database class) can see the imported module?
Thank you.
If you only need to use cx_Oracle within that class, you can just set it as an attribute on this instance, for example:
class Database:
def __init__(self, config):
self.CONFIG=config
sys.path.append(self.CONFIG.cx_oracle_path)
from cx_Oracle import cx_Oracle
self.cx_Oracle = cx_Oracle
self.open()
def open(self):
self.CONNECTION = self.cx_Oracle.Connection(self.CONFIG.username,
self.CONFIG.password,
self.CONFIG.db_sid)
self.CURSOR = self.CONNECTION.cursor()
As a side note, if you are creating multiple Database instances this is an odd approach, since you would end up adding multiple identical entries to sys.path.
I am using jython with a third party application. The third party application has some builtin libraries foo. To do some (unit) testing we want to run some code outside of the application. Since foo is bound to the application we decided to write our own mock implementation.
However there is one issue, we implemented our mock class in python while their class is in java. Thus to use their code one would do import foo and foo is the mock class afterwards. However if we import the python module like this we get the module attached to the name, thus one has to write foo.foo to get to the class.
For convenience reason we would love to be able to write from ourlib.thirdparty import foo to bind foo to the foo-class. However we would like to avoid to import all the classes in ourlib.thirdparty directly, since the loading time for each file takes quite a while.
Is there any way to this in python? ( I did not get far with Import hooks I tried simply returning the class from load_module or overwriting what I write to sys.modules (I think both approaches are ugly, particularly the later))
edit:
ok: here is what the files in ourlib.thirdparty look like simplified(without magic):
foo.py:
try:
import foo
except ImportError:
class foo
....
Actually they look like this:
foo.py:
class foo
....
__init__.py in ourlib.thirdparty
import sys
import os.path
import imp
#TODO: 3.0 importlib.util abstract base classes could greatly simplify this code or make it prettier.
class Importer(object):
def __init__(self, path_entry):
if not path_entry.startswith(os.path.join(os.path.dirname(__file__), 'thirdparty')):
raise ImportError('Custom importer only for thirdparty objects')
self._importTuples = {}
def find_module(self, fullname):
module = fullname.rpartition('.')[2]
try:
if fullname not in self._importTuples:
fileObj, self._importTuples[fullname] = imp.find_module(module)
if isinstance(fileObj, file):
fileObj.close()
except:
print 'backup'
path = os.path.join(os.path.join(os.path.dirname(__file__), 'thirdparty'), module+'.py')
if not os.path.isfile(path):
return None
raise ImportError("Could not find dummy class for %s (%s)\n(searched:%s)" % (module, fullname, path))
self._importTuples[fullname] = path, ('.py', 'r', imp.PY_SOURCE)
return self
def load_module(self, fullname):
fp = None
python = False
print fullname
if self._importTuples[fullname][1][2] in (imp.PY_SOURCE, imp.PY_COMPILED, imp.PY_FROZEN):
fp = open( self._importTuples[fullname][0], self._importTuples[fullname][1][1])
python = True
try:
imp.load_module(fullname, fp, *self._importTuples[fullname])
finally:
if python:
module = fullname.rpartition('.')[2]
#setattr(sys.modules[fullname], module, getattr(sys.modules[fullname], module))
#sys.modules[fullname] = getattr(sys.modules[fullname], module)
if isinstance(fp, file):
fp.close()
return getattr(sys.modules[fullname], module)
sys.path_hooks.append(Importer)
As others have remarked, it is such a plain thing in Python that the import statement iself has a syntax for that:
from foo import foo as original_foo, for example -
or even import foo as module_foo
Interesting to note is that the import statemente binds a name to the imported module or object ont he local context - however, the dictionary sys.modules (on the moduels sys of course), is a live reference to all imported modules, using their names as a key. This mechanism plays a key role in avoding that Python re-reads and re-executes and already imported module , when running (that is, if various of yoru modules or sub-modules import the samefoo` module, it is just read once -- the subsequent imports use the reference stored in sys.modules).
And -- besides the "import...as" syntax, modules in Python are just another object: you can assign any other name to them in run time.
So, the following code would also work perfectly for you:
import foo
original_foo = foo
class foo(Mock):
...
There are given two versions of a storage. Based on the version, I need to select the proper interface module to get the result.
The file structure looks like this:
lib/
__init__.py
provider.py
connection.py
device.py
storage/
__init__.py
interface_v1.py # interface for storage of version 1
interface_v2.py # interface for storage of version 2
main.py
The main.py imports provider.py, that should import one of the interfaces listed in the storage subpackage depending on the version of the storage.
main.py:
from lib.provider import Provider
from lib.connection import Connection
from lib.device import Device
connection = Connection.establish(Device)
storage_version = Device.get_storage_version()
massage = Provider.get_data(connection)
provider.py should import an interface to the storage based on storage_version and implement provide some functions:
from storage import interface
class Provider(object):
def __init_(self):
self.storage = interface.Storage
def get_data(self, connection):
return self.storage.get_data()
def clear_storage(self, connection):
self.storage.clear_storage()
This example is not complete, but should be sufficient for the problem explanation.
Additional question:
Is it possible to use storage.__init__ to use import just the
subpackage?
How to proper implement Factory in Python?
Assuming the interface_v1 and interface_v2 bith impleement a class StorageClassI guess something like this:
import storage.interface_v1
import storage.interface_v2
class Provider(object):
def __init__(self , version):
if version == 1:
self.storage = storage.interface_v1.StorageClass
else:
self.storage = storage.interface_v2.StorageClass
Would be the best solution - but https://docs.python.org/2/library/functions.html#import should provide a way to import a module based on name.