I am creating a commands system in Python. I have module vkcommands that has a class that processes commands from chat (this is a chat-bot), and inside it, I also have class VKCommand with attributes like name, usage, min_rank, etc. Then I have module vkcmds with submodules that implement these commands:
...
vkcommands.py
vkcmds
|- __init__.py # empty
|- add_group.py
|- another_cmd.py
|- ...
Implementations of commands (e.g. add_group) look like this:
import ranks
import vkcommands
from vkcommands import VKCommand
class AddGroup(VKCommand):
def __init__(self, kristy):
VKCommand.__init__(self, kristy,
label='create',
# ... (other attributes)
min_rank=ranks.Rank.USER)
def execute(self, chat, peer, sender, args=None, attachments=None):
# implementation (called from vkcommands.py)
When a user sends a message in the chat, the command manager analyzes it and looks through the registered commands list to see if this is an ordinary message or a bot command. Currently I register all commands in the commands list manually like this:
class VKCommandsManager:
def __init__(self, kristy):
from vkcmds import (
add_group,
next_class
)
self.kristy = kristy
self.commands = (
add_group.AddGroup(kristy),
next_class.NextClass(kristy)
)
Now I would like all commands to be registered automatically using reflections instead. In Java, I'd iterate over all classes in my commands package, reflectively getConstructor of each, call it to retrieve the VKCommand object, and add it to the commands list.
How can I do so in Python? Again, what I need is to:
iterate over all submodules in module (folder) vkcmds/;
for each submodule, check if there is some class X that extends VKCommand inside;
if (2) is true, then call the constructor of that class with one argument (it is guaranteed that the constructor for all commands only has one argument of a known type (my bot's main class));
add the object (? extends VKCommand) constructed in (3) to the commands list that I can iterate over later.
With this file structure:
- Project
├─ commands
| ├─ base.py
| ├─ baz.py
| └─ foo_bar.py
|
└─ main.py
And the following inside the commands directory files:
base.py
class VKCommand:
""" We will inherit from this class if we want to include the class in commands. """
baz.py
from commands.base import VKCommand
class Baz(VKCommand):
pass
def baz():
""" Random function we do not want to retrieve.
foo_bar.py
from .base import VKCommand
class Foo(VKCommand):
""" We only want to retrieve this command. """
pass
class Bar:
""" We want to ignore this class. """
pass
def fizz():
""" Random function we do not want to retrieve. """
We can retrieve the class instances and names directly using the following code:
main.py
"""
Dynamically load all commands located in submodules.
This file is assumed to be at most 1 level higher than the
specified folder.
"""
import pyclbr
import glob
import os
def filter_class(classes):
inherit_from = 'VKCommand'
classes = {name: info for name, info in classes.items() if inherit_from in info.super}
return classes
# Locate all submodules and classes that it contains without importing it.
folder = 'commands' # `vkcmds`.
submodules = dict()
absolute_search_path = os.path.join(os.path.dirname(__file__), folder, '*.py')
for path in glob.glob(absolute_search_path):
submodule_name = os.path.basename(path)[:-3]
all_classes = pyclbr.readmodule(f"commands.{submodule_name}")
command_classes = filter_class(all_classes)
if command_classes:
submodules[submodule_name] = command_classes
# import the class and store an instance of the class into the command list
class_instances = dict()
for submodule_name, class_names in submodules.items():
module = __import__(f"{folder}.{submodule_name}")
submodule = getattr(module, submodule_name)
for class_name in class_names:
class_instance = getattr(submodule, class_name)
class_instances[class_name] = class_instance
print(class_instances)
Explanation
The solution is twofold. It first locates all submodules that have a class which inherit from VKCommand and are located in the folder 'commands`. This leads to the following output containing the module and the class that have to be imported and instantiated respectively:
{'baz': {'Baz': <pyclbr.Class object at 0x000002BF886357F0>}, 'foo_bar': {'Foo': <pyclbr.Class object at 0x000002BF88660668>}}
The second part of the code imports the correct module and class name at run time. The variable class_instance contains the class name and a reference to the class which can be used to instantiate it. The final output will be:
{'Baz': <class 'commands.baz.Baz'>, 'Foo': <class 'commands.foo_bar.Foo'>}
Important notes:
The code only works when importing modules that are 1 dictionary deeper. If you want to use it recursively, you will have to locate the relative path difference and update the pyclbr.readmodule and __import__ with the correct (full) relative import path.
Only the modules that contain a class which inherit from VKCommand get loaded. All other modules are not imported, and have to be imported manually.
I believe that you can make an array of all the commands you have in your folder and then go over them and instantiate the objects.
in __init__.py
all_commands = [AddGroup, AnotherCmd, ...]
instantiate them like this:
objects = [Cmd(arg1, arg2, ...) for Cmd in all_commands]
Edit:
you could also retrieve the class names with the method that you said you had of getting all class names in the folder.
Related
The objective:
I have a package with submodules that I would like to be accessible in the most straightforward way possible. The submodules contain classes to take advantage of the class structure, but don't need to be initialized (as they contain static and class methods). So, ideally, I would like to access them as follows:
from myPackage.subModule import someMethod
print (someMethod)
from myPackage import subModule
print (subModule.someMethod)
import myPackage
print(myPackage.subModule.someMethod)
Here is the package structure:
myPackage ─┐
__init__.py
subModule
subModule2
etc.
Example of a typical submodule:
# submodule.py
class SomeClass():
someAttr = list(range(10))
#classmethod
def someMethod(cls):
pass
#staticmethod
def someMethod2():
pass
Here is the code I have in my '__init __.py': In order to achieve the above; it attempts to set attributes for each class at the package level, and the same for it's methods at the sub-module level.
# __init__.py
def import_submodules(package, filetypes=('py', 'pyc', 'pyd'), ignoreStartingWith='_'):
'''Import submodules to the given package, expose any classes at the package level
and their respective class methods at submodule level.
:Parameters:
package (str)(obj) = A python package.
filetypes (str)(tuple) = Filetype extension(s) to include.
ignoreStartingWith (str)(tuple) = Ignore submodules starting with given chars.
'''
if isinstance(package, str):
package = sys.modules[package]
if not package:
return
pkg_dir = os.path.dirname(os.path.abspath(package.__file__))
sys.path.append(pkg_dir) #append this dir to the system path.
for mod_name in os.listdir(pkg_dir):
if mod_name.startswith(ignoreStartingWith):
continue
elif os.path.isfile(os.path.join(pkg_dir, mod_name)):
mod_name, *mod_ext = mod_name.rsplit('.', 1)
if filetypes:
if not mod_ext or mod_ext[0] not in filetypes:
continue
mod = importlib.import_module(mod_name)
vars(package)[mod_name] = mod
classes = inspect.getmembers(mod, inspect.isclass)
for cls_name, clss in classes:
vars(package)[cls_name] = clss
methods = inspect.getmembers(clss, inspect.isfunction)
for method_name, method in methods:
vars(mod)[method_name] = method
del mod_name
import_submodules(__name__)
At issue is this line:
vars(mod)[method_name] = method
Which ultimately results in: (indicating that the attribute was not set)
from myPackage.subModule import someMethod
ImportError: cannot import name 'someMethod' from 'myPackage.subModule'
I am able to set the methods as attributes to the module within that module, but setting them from outside (ie. in the package __init __), isn't working as written. I understand this isn't ideal to begin with, but my current logic is; that the ease of use, outweighs any perceived issues with namespace pollution. I am, of course, always open to counter-arguments.
I just checked it on my machine.
Created a package myPackage with a module subModule that has a function someMethod.
I run a python shell with working directory in the same directory that the myPackage is in, and to get these 3 import statements to work:
from myPackage.subModule import someMethod
from myPackage import subModule
import myPackage
All I had to do was to create an __init__.py with this line in it:
from . import subModule
Found a nice "hacky" solution -
subModule.py:
class myClass:
#staticmethod
def someMethod():
print("I have a bad feeling about this")
myInstance = myClass()
someMethod = myInstance.someMethod
init.py is empty
Still scratching my head of why I am unable to do this from the package __init __, but this solution works with the caveat it has to be called at the end of each submodule. Perhaps someone, in the future, someone can chime in as to why this wasn't working when completely contained in the __init __.
def addMembers(module, ignoreStartingWith='_'):
'''Expose class members at module level.
:Parameters:
module (str)(obj) = A python module.
ignoreStartingWith (str)(tuple) = Ignore class members starting with given chars.
ex. call: addMembers(__name__)
'''
if isinstance(module, str):
module = sys.modules[module]
if not module:
return
classes = inspect.getmembers(module, inspect.isclass)
for cls_name, clss in classes:
cls_members = [(o, getattr(clss, o)) for o in dir(clss) if not o.startswith(ignoreStartingWith)]
for name, mem in cls_members:
vars(module)[name] = mem
This is the solution I ended up going with. It needs to be put at the end of each submodule of your package. But, it is simple and in addition to all the standard ways of importing, allows you to import a method directly:
def __getattr__(attr):
'''Attempt to get a class attribute.
:Parameters:
attr (str): A name of a class attribute.
:Return:
(obj) The attribute.
'''
try:
return getattr(Someclass, attr)
except AttributeError as error:
raise AttributeError(f'{__file__} in __getattr__\n\t{error} ({type(attr).__name__})')
from somePackage.someModule import someMethod
I have a directory of code filled in like this:
root_dir
├── root.py
├── same_step_0.py
├── same_step_1.py
├── same_step_2.py
└── utils
├── __init__.py
├── file_one.py
└── file_two.py
The root.py file references the functions in the same_step files. For example:
# In root.py
same_step_0.generated_function()
The "step" files have references to functions in the 'utils' directory, with a simple import and call - e.g.
from utils.file_one import function_one
function_one(val="somelongstring")
Ok, now we get to the problem - I'm working against an API which effectively requires each "step" to be a single python file. So, what I need to do is, somehow, append all functions pulled in by each step file into a single file which I can then submit to the service.
IDEALLY, I'd love to do this with as little changes to the underlying code. So, if possible, in same_step_0 I'd like to leave it as:
from utils.file_one import function_one
function_one(val="somelongstring")
and not have to change it to
# Note removing the 'utils' prefix
from file_one import function_one
function_one(val="somelongstring")
So, the question is, is there a way I can append all the dependent files or functions together into a single file such that it mimic'd import behavior? Or at least function call behavior? Ideas I've thought about:
Pickling all subfiles into a serialization format and inserting at the end of the code
Raw appending of all python files - perhaps inside an inline module?
Scraping all files for functions and attaching them manually (however, this would not preserve namespacing)
I haven't dug into these yet - I'm curious if this is even possible
I can do a lot of generation and wrapping of code, but I'd prefer not to touch the inner code (the function_one call, for example).
Goal
To create a minimal set of scripts, and an interchange format that can be stored in a Python executable to recreate a module structure.
Source
Ok here's a simple example that abuses types.ModuleType, sys, and globals() to do this:
import os
import sys
import types
class Module:
def __init__(self, name, source=None, path=None, modules=None):
self.name = name
self.source = source
self.path = path
self.modules = modules
def from_file(name, path):
return Module(name, open(path).read(), path)
def from_dir(name, path):
return Module(name, None, path, [])
def __repr__(self):
return f'Module(name={repr(self.name)}, source={repr(self.source)}, path={repr(self.path)}, modules={self.modules})'
def to_dict(self):
data = { 'name': self.name }
if self.source is not None:
data['source'] = self.source
if self.path is not None:
data['path'] = self.path
if self.modules is not None:
data['modules'] = [i.to_dict() for i in self.modules]
return data
#staticmethod
def from_dict(data):
modules = None
if 'modules' in data:
modules = [Module.from_dict(i) for i in data.pop('modules')]
return Module(**data, modules=modules)
def get_modname(parent, module):
if parent is None:
return module
if module == '__init__':
return parent
return f'{parent}.{module}'
def compile_module(module, parent=None):
modname = get_modname(parent, module.name)
mod = types.ModuleType(modname)
exec(module.source, mod.__dict__)
path = os.path.realpath(module.path)
mod.__path__ = os.path.dirname(path)
mod.__file__ = path
sys.modules[modname] = mod
globals()[modname] = mod
return mod
def compile_module_recursive(package, parent=None):
# Need to do this recursively.
mod = compile_module(package, parent)
if not package.modules:
return mod
for submodule in package.modules:
submod = compile_module_recursive(submodule, parent=get_modname(parent, package.name))
if not hasattr(mod, submodule.name):
setattr(mod, submodule.name, submod)
return mod
def read_module_recursive(directory, parent=None):
# The module order is first the:
# 1. All submodules
# 2. Then definitions inside
# Then, need to define `__name__`, `__path__` and `__file__`.
cwd = os.getcwd()
realpath = os.path.realpath(directory)
parent_dir = os.path.dirname(realpath)
base_dir = os.path.basename(realpath)
if parent is None:
parent = Module.from_dir(base_dir, realpath)
os.chdir(realpath)
for entry in os.listdir(realpath):
path = os.path.join(realpath, entry)
if os.path.isfile(path):
name, ext = os.path.splitext(entry)
if not ext == '.py':
continue
if name == '__init__':
parent.path = path
parent.source = open(path).read()
else:
parent.modules.append(Module.from_file(name, path))
elif os.path.isdir(path):
if entry == '__pycache__':
continue
path = os.path.join(realpath, entry)
# Must have processed __init__.py
if not os.path.isfile(f'{path}/__init__.py'):
continue
module = Module.from_dir(entry, path)
parent.modules.append(module)
read_module_recursive(entry, module)
os.chdir(cwd)
return parent
# SAMPLE USE
# ----------
# module = read_module_recursive('mylib')
# data = module.to_dict() # can store as a Python dict.
# module = Module.from_dict(data)
# compile_module_recursive(module)
How it works
Basically, it reads all .py files from all modules, recursively. It currently does not support extension types, only pure Python files. It then creates an intermediate, tree-like type for the entire module tree. It provides easy serialization to and from dict, so it's much easier to copy-and-paste it into your executable file.
Example use
Say I have the following directory structure:
mylib/
__init__.py
a.py
b.py
c/
__init__.py
d.py
It properly respects the import order, and how it variable assignments override imports or submodules. Please note that it imports everything recursively all at once, which differs than the traditional import structure.
mylib/init.py
a = 'e'
mylib/a.py
def afunc(a):
return str(a)
class AClass:
def x(self):
return 1
mylib/b.py
def bfunc(b):
return repr(b)
class BClass:
def x(self):
return 1
mylib/c/init.py
mylib/c/d.py
def dfunc(d):
return str(d)
class DClass:
def x(self):
return 1
Example
>>> module = read_module_recursive('mylib')
>>> data = module.to_dict() # can store as a Python dict.
>>> module = Module.from_dict(data)
>>> compile_module_recursive(module)
>>> mylib
<module 'mylib' from 'C:\\Users\\user\\OneDrive\\Desktop\\lib\\mylib\\__init__.py'>
>>> mylib.a
'e'
>>> from mylib.a import afunc # still works
>>> afunc(54)
'54'
>>> import mylib # works because `'mylib'` is in `sys.modules`.
Congrats: this is a hack, but it works, and it works nicely, and it uses Python's own packaging system to do everything, so it's much more resilient than copying and pasting. After compiling, it also respects imports (due to adding them to sys.modules). It also respects __doc__, and other attributes.
License
Public Domain or Unlicensed. Use as you see fit. Attribution would be nice, but not required.
I have to admit that I am fairly a newbie to python. This question is more related to code organization.
For example, here is app.py file
import backtracker as bt
# Create a class
class TestTracker(bt.Strategy):
def log(self, txt, dt=None):
''' Logging function for this strategy '''
dt = dt or self.datas[0].datetime.date(0)
print('%s, %s' % (dt.isoformat(), txt))
def __init__(self):
self.dataclose = self.datas[0].close
self.datapoint = self.datas[0]
def next(self):
self.log('Close, %2f' %self.dataclose[0])
Now, I would like to separate class file to testStrategy.py and import it into app.py. The folder structure would look something like this
|
|-strategies
| |-testStrategy.py
|-app.py
How do I reference base class bt.strategy in my new class file ?
Thank you for your help.
There are two ways you can go about it.
Add the folder strategies to sys.path and then import your class.
# This gives you the folder in which current file resides
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
if BASE_DIR + '/strategies' not in sys.path:
sys.path.append(BASE_DIR + '/strategies')
from testStrategy import TestTracker
Add __init__.py in every folder and then use your code as a python package. This allows you to import in the following way.
import strategies.testStrategy.TestTracker
Be wary of the trap though.
I have this main.py, in it:
import uuid
class tools(object):
def generate_uuid(self):
return self.uuid.uuid4()
in my calling program callmain.py, I have
import main
result = main.tool.generate_uuid()
print ("result")
if I run my callmain.py: I get
"TypeError: generate_uuid() missing 1 required positional argument:
'self'
if I add self to the line
result = main.tool.generate_uuid(self): I get
NameError: name 'self' is not defined
How to fix this? thank for help.
because you should make a object from your class first. then call your sub function like this:
import main
result = tools()
result.generate_uuid()
print(result) # "result" is a string! you should just call result without any "".
If you want to use the module of a class you have to create an instance of that class first and call if from that instance, that way the self argument is passed a valid reference to an instance of that class. For example:
import main
tools_instance = main.tools()
result = tools_instance.generate_uuid()
The style of importing shown in your question looks like a package. In packages a folder of python files __init__.py can be arranged in a particular way, documented here Python Packages. An example from the docs
parent/
__init__.py
one/
__init__.py
two/
__init__.py
three/
__init__.py
So an package of the format
main/
__init__.py
tools/
__init__.py # Add function 'generate_uuid' in this file
Could be utilized as follow:
import main
result = main.tools.generate_uuid()
print(result)
Let's assume, that there are following minimalistic python classes inside one module, e.g. Module:
module/
__init__.py
db.py
document.py
db.py
import yaml
class DB(object):
config = {}
#classmethod
def load_config(cls, config_path):
cls.config = yaml.load(open(config_path, 'r').read())
and document.py
from .db import DB
class Document(object):
db = None
def __init__(self):
self.db = DB()
End-user is going to use such Module as follows:
from Module import DB, Document
DB.load_config('/path/to/config.yml')
Document.do_some_stuff()
doc1 = Document()
doc2 = Document.find(...)
doc2.update_something(...)
doc2.save()
It is expected that Document class and every instance of it will have internally an access to class DB with a config specified by user. However, since Document performs an internal import of DB class (from .db import DB) it receives a 'fresh' DB class with default config.
I did a lot of searches, most of questions and answers are about module-wide configs, but not specified by the end user.
How can I achieve such functionality? I guess that there is some architectural problem here, but what is the most simple way to solve it?
Perhaps this isn't the most appropriate answer, but a few months back I wrote a module called aconf for this exact purpose. It's a memory-based global configuration module for Python written in 8 lines. The idea is you can do the following:
You create a Config object to force the user to input the configuration your program requires (in this case it's inside config.py):
""" 'Config' class to hold our desired configuration parameters.
Note:
This is technically not needed. We do this so that the user knows what he/she should pass
as a config for the specific project. Note how we also take in a function object - this is
to demonstrate that one can have absolutely any type in the global config and is not subjected
to any limitations.
"""
from aconf import make_config
class Config:
def __init__(self, arg, func):
make_config(arg=arg, func=func)
You consume your configuration throughout your module (in this case, inside functionality.py):
""" Use of the global configuration through the `conf` function. """
from aconf import conf
class Example:
def __init__(self):
func = conf().func
arg = conf().arg
self.arg = func(arg)
And then use it (in this case inside main.py):
from project.config import Config
from project.functionality import Example
# Random function to demonstrate we can pass _anything_ to 'make_config' inside 'Config'.
def uppercase(words):
return words.upper()
# We create our custom configuration without saving it.
Config(arg="hello world", func=uppercase)
# We initialize our Example object without passing the 'Config' object to it.
example = Example()
print(example.arg)
# >>> "HELLO WORLD"
The entire aconf module is the following:
__version__ = "1.0.1"
import namedtupled
def make_config(**kwargs):
globals()["aconf"] = kwargs
conf = lambda: namedtupled.map(globals()["aconf"])
config = lambda: globals()["aconf"]
... in essence, you just save your configuration to globals() during runtime.
It's so stupid it makes me wonder if you should even be allowed to do this. I wrote aconf for fun, but have never personally used it in a big project. The reality is, you might run into the problem of making your code weird for other developers.