Develop Reference

Set a module's class method as an attribute of that module from outside that module

The objective:
I have a package with submodules that I would like to be accessible in the most straightforward way possible. The submodules contain classes to take advantage of the class structure, but don't need to be initialized (as they contain static and class methods). So, ideally, I would like to access them as follows:
from myPackage.subModule import someMethod
print (someMethod)
from myPackage import subModule
print (subModule.someMethod)
import myPackage
print(myPackage.subModule.someMethod)
Here is the package structure:
myPackage ─┐
__init__.py
subModule
subModule2
etc.
Example of a typical submodule:
# submodule.py
class SomeClass():
someAttr = list(range(10))
#classmethod
def someMethod(cls):
pass
#staticmethod
def someMethod2():
pass
Here is the code I have in my '__init __.py': In order to achieve the above; it attempts to set attributes for each class at the package level, and the same for it's methods at the sub-module level.
# __init__.py
def import_submodules(package, filetypes=('py', 'pyc', 'pyd'), ignoreStartingWith='_'):
'''Import submodules to the given package, expose any classes at the package level
and their respective class methods at submodule level.
:Parameters:
package (str)(obj) = A python package.
filetypes (str)(tuple) = Filetype extension(s) to include.
ignoreStartingWith (str)(tuple) = Ignore submodules starting with given chars.
'''
if isinstance(package, str):
package = sys.modules[package]
if not package:
return
pkg_dir = os.path.dirname(os.path.abspath(package.__file__))
sys.path.append(pkg_dir) #append this dir to the system path.
for mod_name in os.listdir(pkg_dir):
if mod_name.startswith(ignoreStartingWith):
continue
elif os.path.isfile(os.path.join(pkg_dir, mod_name)):
mod_name, *mod_ext = mod_name.rsplit('.', 1)
if filetypes:
if not mod_ext or mod_ext[0] not in filetypes:
continue
mod = importlib.import_module(mod_name)
vars(package)[mod_name] = mod
classes = inspect.getmembers(mod, inspect.isclass)
for cls_name, clss in classes:
vars(package)[cls_name] = clss
methods = inspect.getmembers(clss, inspect.isfunction)
for method_name, method in methods:
vars(mod)[method_name] = method
del mod_name
import_submodules(__name__)
At issue is this line:
vars(mod)[method_name] = method
Which ultimately results in: (indicating that the attribute was not set)
from myPackage.subModule import someMethod
ImportError: cannot import name 'someMethod' from 'myPackage.subModule'
I am able to set the methods as attributes to the module within that module, but setting them from outside (ie. in the package __init __), isn't working as written. I understand this isn't ideal to begin with, but my current logic is; that the ease of use, outweighs any perceived issues with namespace pollution. I am, of course, always open to counter-arguments.

I just checked it on my machine.
Created a package myPackage with a module subModule that has a function someMethod.
I run a python shell with working directory in the same directory that the myPackage is in, and to get these 3 import statements to work:
from myPackage.subModule import someMethod
from myPackage import subModule
import myPackage
All I had to do was to create an __init__.py with this line in it:
from . import subModule

Found a nice "hacky" solution -
subModule.py:
class myClass:
#staticmethod
def someMethod():
print("I have a bad feeling about this")
myInstance = myClass()
someMethod = myInstance.someMethod
init.py is empty

Still scratching my head of why I am unable to do this from the package __init __, but this solution works with the caveat it has to be called at the end of each submodule. Perhaps someone, in the future, someone can chime in as to why this wasn't working when completely contained in the __init __.
def addMembers(module, ignoreStartingWith='_'):
'''Expose class members at module level.
:Parameters:
module (str)(obj) = A python module.
ignoreStartingWith (str)(tuple) = Ignore class members starting with given chars.
ex. call: addMembers(__name__)
'''
if isinstance(module, str):
module = sys.modules[module]
if not module:
return
classes = inspect.getmembers(module, inspect.isclass)
for cls_name, clss in classes:
cls_members = [(o, getattr(clss, o)) for o in dir(clss) if not o.startswith(ignoreStartingWith)]
for name, mem in cls_members:
vars(module)[name] = mem

This is the solution I ended up going with. It needs to be put at the end of each submodule of your package. But, it is simple and in addition to all the standard ways of importing, allows you to import a method directly:
def __getattr__(attr):
'''Attempt to get a class attribute.
:Parameters:
attr (str): A name of a class attribute.
:Return:
(obj) The attribute.
'''
try:
return getattr(Someclass, attr)
except AttributeError as error:
raise AttributeError(f'{__file__} in __getattr__\n\t{error} ({type(attr).__name__})')
from somePackage.someModule import someMethod

Python — Init class reflectively

I am creating a commands system in Python. I have module vkcommands that has a class that processes commands from chat (this is a chat-bot), and inside it, I also have class VKCommand with attributes like name, usage, min_rank, etc. Then I have module vkcmds with submodules that implement these commands:
...
vkcommands.py
vkcmds
|- __init__.py # empty
|- add_group.py
|- another_cmd.py
|- ...
Implementations of commands (e.g. add_group) look like this:
import ranks
import vkcommands
from vkcommands import VKCommand
class AddGroup(VKCommand):
def __init__(self, kristy):
VKCommand.__init__(self, kristy,
label='create',
# ... (other attributes)
min_rank=ranks.Rank.USER)
def execute(self, chat, peer, sender, args=None, attachments=None):
# implementation (called from vkcommands.py)
When a user sends a message in the chat, the command manager analyzes it and looks through the registered commands list to see if this is an ordinary message or a bot command. Currently I register all commands in the commands list manually like this:
class VKCommandsManager:
def __init__(self, kristy):
from vkcmds import (
add_group,
next_class
)
self.kristy = kristy
self.commands = (
add_group.AddGroup(kristy),
next_class.NextClass(kristy)
)
Now I would like all commands to be registered automatically using reflections instead. In Java, I'd iterate over all classes in my commands package, reflectively getConstructor of each, call it to retrieve the VKCommand object, and add it to the commands list.
How can I do so in Python? Again, what I need is to:
iterate over all submodules in module (folder) vkcmds/;
for each submodule, check if there is some class X that extends VKCommand inside;
if (2) is true, then call the constructor of that class with one argument (it is guaranteed that the constructor for all commands only has one argument of a known type (my bot's main class));
add the object (? extends VKCommand) constructed in (3) to the commands list that I can iterate over later.

With this file structure:
- Project
├─ commands
| ├─ base.py
| ├─ baz.py
| └─ foo_bar.py
|
└─ main.py
And the following inside the commands directory files:
base.py
class VKCommand:
""" We will inherit from this class if we want to include the class in commands. """
baz.py
from commands.base import VKCommand
class Baz(VKCommand):
pass
def baz():
""" Random function we do not want to retrieve.
foo_bar.py
from .base import VKCommand
class Foo(VKCommand):
""" We only want to retrieve this command. """
pass
class Bar:
""" We want to ignore this class. """
pass
def fizz():
""" Random function we do not want to retrieve. """
We can retrieve the class instances and names directly using the following code:
main.py
"""
Dynamically load all commands located in submodules.
This file is assumed to be at most 1 level higher than the
specified folder.
"""
import pyclbr
import glob
import os
def filter_class(classes):
inherit_from = 'VKCommand'
classes = {name: info for name, info in classes.items() if inherit_from in info.super}
return classes
# Locate all submodules and classes that it contains without importing it.
folder = 'commands' # `vkcmds`.
submodules = dict()
absolute_search_path = os.path.join(os.path.dirname(__file__), folder, '*.py')
for path in glob.glob(absolute_search_path):
submodule_name = os.path.basename(path)[:-3]
all_classes = pyclbr.readmodule(f"commands.{submodule_name}")
command_classes = filter_class(all_classes)
if command_classes:
submodules[submodule_name] = command_classes
# import the class and store an instance of the class into the command list
class_instances = dict()
for submodule_name, class_names in submodules.items():
module = __import__(f"{folder}.{submodule_name}")
submodule = getattr(module, submodule_name)
for class_name in class_names:
class_instance = getattr(submodule, class_name)
class_instances[class_name] = class_instance
print(class_instances)
Explanation
The solution is twofold. It first locates all submodules that have a class which inherit from VKCommand and are located in the folder 'commands`. This leads to the following output containing the module and the class that have to be imported and instantiated respectively:
{'baz': {'Baz': <pyclbr.Class object at 0x000002BF886357F0>}, 'foo_bar': {'Foo': <pyclbr.Class object at 0x000002BF88660668>}}
The second part of the code imports the correct module and class name at run time. The variable class_instance contains the class name and a reference to the class which can be used to instantiate it. The final output will be:
{'Baz': <class 'commands.baz.Baz'>, 'Foo': <class 'commands.foo_bar.Foo'>}
Important notes:
The code only works when importing modules that are 1 dictionary deeper. If you want to use it recursively, you will have to locate the relative path difference and update the pyclbr.readmodule and __import__ with the correct (full) relative import path.
Only the modules that contain a class which inherit from VKCommand get loaded. All other modules are not imported, and have to be imported manually.

I believe that you can make an array of all the commands you have in your folder and then go over them and instantiate the objects.
in __init__.py
all_commands = [AddGroup, AnotherCmd, ...]
instantiate them like this:
objects = [Cmd(arg1, arg2, ...) for Cmd in all_commands]
Edit:
you could also retrieve the class names with the method that you said you had of getting all class names in the folder.

Maintain separate class files in python

I have to admit that I am fairly a newbie to python. This question is more related to code organization.
For example, here is app.py file
import backtracker as bt
# Create a class
class TestTracker(bt.Strategy):
def log(self, txt, dt=None):
''' Logging function for this strategy '''
dt = dt or self.datas[0].datetime.date(0)
print('%s, %s' % (dt.isoformat(), txt))
def __init__(self):
self.dataclose = self.datas[0].close
self.datapoint = self.datas[0]
def next(self):
self.log('Close, %2f' %self.dataclose[0])
Now, I would like to separate class file to testStrategy.py and import it into app.py. The folder structure would look something like this
|
|-strategies
| |-testStrategy.py
|-app.py
How do I reference base class bt.strategy in my new class file ?
Thank you for your help.

There are two ways you can go about it.
Add the folder strategies to sys.path and then import your class.
# This gives you the folder in which current file resides
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
if BASE_DIR + '/strategies' not in sys.path:
sys.path.append(BASE_DIR + '/strategies')
from testStrategy import TestTracker
Add __init__.py in every folder and then use your code as a python package. This allows you to import in the following way.
import strategies.testStrategy.TestTracker
Be wary of the trap though.

Unpickling python objects with a changed module path

I'm trying to integrate a project Project A built by a colleague into another python project. Now this colleague has not used relative imports in his code but instead done
from packageA.moduleA import ClassA
from packageA.moduleA import ClassB
and consequently pickled the classes with cPickle. For neatness I'd like to hide the package that his (Project A) built inside my project. This however changes the path of the classes defined in packageA. No problem, I'll just redefine the import using
from ..packageA.moduleA import ClassA
from ..packageA.moduleA import ClassB
but now the un pickling the classes fails with the following message
with open(fname) as infile: self.clzA = cPickle.load(infile)
ImportError: No module named packageA.moduleA
So why doesn't cPickle apparently see the module defs. Do I need to add the root of packageA to system path? Is this the correct way to solve the problem?
The cPickled file looks something like
ccopy_reg
_reconstructor
p1
(cpackageA.moduleA
ClassA
p2
c__builtin__
object
p3
NtRp4
The old project hierarchy is of the sort
packageA/
__init__.py
moduleA.py
moduleB.py
packageB/
__init__.py
moduleC.py
moduleD.py
I'd like to put all of that into a WrapperPackage
MyPackage/
.. __init__.py
.. myModuleX.py
.. myModuleY.py
WrapperPackage/
.. __init__.py
.. packageA/
.. __init__.py
.. moduleA.py
.. moduleB.py
.. packageB/
.. __init__.py
.. moduleC.py
.. moduleD.py

You'll need to create an alias for the pickle import to work; the following to the __init__.py file of the WrapperPackage package:
from .packageA import * # Ensures that all the modules have been loaded in their new locations *first*.
from . import packageA # imports WrapperPackage/packageA
import sys
sys.modules['packageA'] = packageA # creates a packageA entry in sys.modules
It may be that you'll need to create additional entries though:
sys.modules['packageA.moduleA'] = moduleA
# etc.
Now cPickle will find packageA.moduleA and packageA.moduleB again at their old locations.
You may want to re-write the pickle file afterwards, the new module location will be used at that time. The additional aliases created above should ensure that the modules in question have the new location name for cPickle to pick up when writing the classes again.

In addition to #MartinPieters answer the other way of doing this is to define the find_global method of the cPickle.Unpickler class, or extend the pickle.Unpickler class.
def map_path(mod_name, kls_name):
if mod_name.startswith('packageA'): # catch all old module names
mod = __import__('WrapperPackage.%s'%mod_name, fromlist=[mod_name])
return getattr(mod, kls_name)
else:
mod = __import__(mod_name)
return getattr(mod, kls_name)
import cPickle as pickle
with open('dump.pickle','r') as fh:
unpickler = pickle.Unpickler(fh)
unpickler.find_global = map_path
obj = unpickler.load() # object will now contain the new class path reference
with open('dump-new.pickle','w') as fh:
pickle.dump(obj, fh) # ClassA will now have a new path in 'dump-new'
A more detailed explanation of the process for both pickle and cPickle can be found here.

One possible solution is to directly edit the pickle file (if you have access). I ran into this same problem of a changed module path, and I had saved the files as pickle.HIGHEST_PROTOCOL so it should be binary in theory, but the module path was sitting at the top of the pickle file in plain text. So I just did a find replace on all of the instances of the old module path with the new one and voila, they loaded correctly.
I'm sure this solution is not for everyone, especially if you have a very complex pickled object, but it is a quick and dirty data fix that worked for me!

This is my basic pattern for flexible unpickling - via an unambiguous and fast transition map - as there are usually just a few known classes besides the primitive data-types relevant for pickling. This also protects unpickling against erroneous or maliciously constructed data, which after all can execute arbitrary python code (!) upon a simple pickle.load() (with or without error-prone sys.modules fiddling).
Python 2 & 3:
from __future__ import print_function
try:
import cPickle as pickle, copy_reg as copyreg
except:
import pickle, copyreg
class OldZ:
a = 1
class Z(object):
a = 2
class Dangerous:
pass
_unpickle_map_safe = {
# all possible and allowed (!) classes & upgrade paths
(__name__, 'Z') : Z,
(__name__, 'OldZ') : Z,
('old.package', 'OldZ') : Z,
('__main__', 'Z') : Z,
('__main__', 'OldZ') : Z,
# basically required
('copy_reg', '_reconstructor') : copyreg._reconstructor,
('__builtin__', 'object') : copyreg._reconstructor,
}
def unpickle_find_class(modname, clsname):
print("DEBUG unpickling: %(modname)s . %(clsname)s" % locals())
try:
return _unpickle_map_safe[(modname, clsname)]
except KeyError:
raise pickle.UnpicklingError(
"%(modname)s . %(clsname)s not allowed" % locals())
if pickle.__name__ == 'cPickle': # PY2
def SafeUnpickler(f):
u = pickle.Unpickler(f)
u.find_global = unpickle_find_class
return u
else: # PY3 & Python2-pickle.py
class SafeUnpickler(pickle.Unpickler):
find_class = staticmethod(unpickle_find_class)
def test(fn='./z.pkl'):
z = OldZ()
z.b = 'teststring' + sys.version
pickle.dump(z, open(fn, 'wb'), 2)
pickle.dump(Dangerous(), open(fn + 'D', 'wb'), 2)
# load again
o = SafeUnpickler(open(fn, 'rb')).load()
print(pickle, "loaded:", o, o.a, o.b)
assert o.__class__ is Z
try:
raise SafeUnpickler(open(fn + 'D', 'rb')).load() and AssertionError
except pickle.UnpicklingError:
print('OK: Dangerous not allowed')
if __name__ == '__main__':
test()

How can I discover classes in a specific package in python?

I have a package of plug-in style modules. It looks like this:
/Plugins
/Plugins/__init__.py
/Plugins/Plugin1.py
/Plugins/Plugin2.py
etc...
Each .py file contains a class that derives from PluginBaseClass. So I need to list every module in the Plugins package and then search for any classes that implement PluginBaseClass. Ideally I want to be able to do something like this:
for klass in iter_plugins(project.Plugins):
action = klass()
action.run()
I have seen some other answers out there, but my situation is different. I have an actual import to the base package (ie: import project.Plugins) and I need to find the classes after discovering the modules.

Edit: here's a revised solution. I realised I was making a mistake while testing my previous one, and it doesn't really work the way you would expect. So here is a more complete solution:
import os
from imp import find_module
from types import ModuleType, ClassType
def iter_plugins(package):
"""Receives package (as a string) and, for all of its contained modules,
generates all classes that are subclasses of PluginBaseClass."""
# Despite the function name, "find_module" will find the package
# (the "filename" part of the return value will be None, in this case)
filename, path, description = find_module(package)
# dir(some_package) will not list the modules within the package,
# so we explicitly look for files. If you need to recursively descend
# a directory tree, you can adapt this to use os.walk instead of os.listdir
modules = sorted(set(i.partition('.')[0]
for i in os.listdir(path)
if i.endswith(('.py', '.pyc', '.pyo'))
and not i.startswith('__init__.py')))
pkg = __import__(package, fromlist=modules)
for m in modules:
module = getattr(pkg, m)
if type(module) == ModuleType:
for c in dir(module):
klass = getattr(module, c)
if (type(klass) == ClassType and
klass is not PluginBaseClass and
issubclass(klass, PluginBaseClass)):
yield klass
My previous solution was:
You could try something like:
from types import ModuleType
import Plugins
classes = []
for item in dir(Plugins):
module = getattr(Plugins, item)
# Get all (and only) modules in Plugins
if type(module) == ModuleType:
for c in dir(module):
klass = getattr(module, c)
if isinstance(klass, PluginBaseClass):
classes.append(klass)
Actually, even better, if you want some modularity:
from types import ModuleType
def iter_plugins(package):
# This assumes "package" is a package name.
# If it's the package itself, you can remove this __import__
pkg = __import__(package)
for item in dir(pkg):
module = getattr(pkg, item)
if type(module) == ModuleType:
for c in dir(module):
klass = getattr(module, c)
if issubclass(klass, PluginBaseClass):
yield klass

You may (and probably should) define __all__ in __init__.py as a list of the submodules in your package; this is so that you support people doing from Plugins import *. If you have done so, you can iterate over the modules with
import Plugins
import sys
modules = { }
for module in Plugins.__all__:
__import__( module )
modules[ module ] = sys.modules[ module ]
# iterate over dir( module ) as above
The reason another answer posted here fails is that __import__ imports the lowest-level module, but returns the top-level one (see the docs). I don't know why.

Scanning modules isn't good idea. If you need class registry you should look at metaclasses or use existing solutions like zope.interface.
Simple solution through metaclasses may look like that:
from functools import reduce
class DerivationRegistry(type):
def __init__(cls,name,bases,cls_dict):
type.__init__(cls,name,bases,cls_dict)
cls._subclasses = set()
for base in bases:
if isinstance(base,DerivationRegistry):
base._subclasses.add(cls)
def getSubclasses(cls):
return reduce( set.union,
( succ.getSubclasses() for succ in cls._subclasses if isinstance(succ,DerivationRegistry)),
cls._subclasses)
class Base(object):
__metaclass__ = DerivationRegistry
class Cls1(object):
pass
class Cls2(Base):
pass
class Cls3(Cls2,Cls1):
pass
class Cls4(Cls3):
pass
print(Base.getSubclasses())

If you don't know what's going to be in Plugins ahead of time, you can get a list of python files in the package's directory, and import them like so:
# compute a list of modules in the Plugins package
import os
import Plugins
plugin_modules = [f[:-3] for f in os.listdir(os.path.dirname(Plugins.__file__))
if f.endswith('.py') and f != '__init__.py']
Sorry, that comprehension might be a mouthful for someone relatively new to python. Here's a more verbose version (might be easier to follow):
plugin_modules = []
package_path = Plugins.__file__
file_list = os.listdir(os.path.dirname(package_path))
for file_name in file_list:
if file_name.endswith('.py') and file_name != '__init__.py':
plugin_modules.append(file_name)
Then you can use __import__ to get the module:
# get the first one
plugin = __import__('Plugins.' + plugin_modules[0])