I'm writing a Python application which stores some data. For storing the data i've wrote a Connection class with abstract methods (using Python's abc module). This class is the super class all storage back-ends derive from. Each storage back-end has only one purpose, e.g. storing the data in plain text files or in a XML file.
All storage backends (inclusive the module where the super class is located) are in one package called 'data_handler'. And each back-end is in one module.
My application should be able the store data in multiple back-ends simultaneously and determinate at runtime which storage back-ends are available. To do this i had the idea to write a singleton class where each back-end have to register at their import. But this seems to be not so good in a dynamic language (please correct me if I misinterpret this). Another way could be the import of the package with import data_handler and then get the __file__ attribute of the package and search all Python files in the dir for subclasses of the super Connection class.
What method should I use, or are there other (maybe better) methods to do this.
Stefan
Is discovering the back-ends at runtime a strict requirement or would static
enumeration of them in the code do?
This feature will be nice to note have to edit the code when I add a new back-end
But should your application always write to all backends?
I will have a class where I can register available handler. And the data shall be written to each registered handler. But not all available handlers have to be registered.
Do not walk the filesystem (!) and scan the Python source code of the backends! That's an ugly hack at the best of times, and even worse here because you don't need anything like it at all! Registering all the classes on import is perfectly OK.
Store the backends in a class attribute instead of an instance attribute; that way, all Storage instances will look at the same set of backends:
>>> class Storage(object):
... backends = set()
...
... def register(self, backend):
... self.backends.add(backend)
...
Every backend can register itself by instantiating its own Storage, which has access to the class-level backends attribute:
>>> foo = Storage()
>>> foo.register("text")
>>> bar = Storage()
>>> bar.register("xml")
You can read this attribute by instantiating another Storage, which will read the same variable:
>>> baz = Storage()
>>> baz.backends
{'xml', 'text'}
You could even store the backend instances in a class attribute of Connection, and register each backend upon instantiation:
>>> class Connection(object,metaclass=abc.ABCMeta):
... #abc.abstractmethod
... def register(self, backend):
... pass
...
... backends = set()
...
>>> class TextBackend(Connection):
... def register(self):
... super().backends.add(self)
...
... def __init__(self):
... self.register()
...
>>> class XMLBackend(Connection):
... def register(self):
... super().backends.add(self)
...
... def __init__(self):
... self.register()
...
>>> foo = TextBackend()
>>> bar = XMLBackend()
>>> Connection.backends
{<__main__.XMLBackend object at 0x027ADAB0>, \
<__main__.TextBackend object at 0x027ADA50>}
If these backends are going to be distributed in various Python distributions, you might want to look at setuptools/distribute entry points. Here's an article on how you might use these for dynamic plugin finding services:
http://aroberge.blogspot.com/2008/12/plugins-part-6-setuptools-based.html
But should your application always write to all backends? If not, you could use (as usual) another layer of indirection, e.g.
storage = Storage()
storage.use(TextBackend, XMLBackend, YamlBackend)
storage.write(data)
Or something like this, with Storage being a dispatcher, which would just loop over the backends and call the appropriate serializer.
This is of course, very coarse.
you could use a function like this one:
def loadClass(fullclassname):
sepindex=fullclassname.rindex('.')
classname=fullclassname[sepindex+1:]
modname=fullclassname[:sepindex]
#dynmically import the class in the module
imod=__import__(modname,None,None,classname)
classtype=getattr(imod,classname)
return classtype
where fullclassname is the fully dotted qualifiant for the class you want load.
example (pseudo code,but the idea is there):
for package availability scanning, only perform some globbing , then for finding final class name, you may declare a Plugin class in each of your modules that has a getStorage()
#scan for modules , getPluginPackagesUnder (to be defined) returns the dotted name for all packages under a root path (using some globbing, listdir or whatever method)
pluginpackages=getPluginPackagesUnder("x/y/z")
storagelist=[]
for pgpck in plunginpackages:
pluginclass=loadClass("%s.Plugin"%pgpck)
storageinstance=Plugin().getStorage()
storagelist.append(storageinstance)
so, you can dynamically scan for your existing storage plugins
Related
I designed my framework so it will contain devices in it (classes with metadata) and a list of interfaces (objects) in it which are responsible to connect to the specific interface on the device (using the device metadata) and execute command on it and return the output.
This interfaces usually contatins methods that is handling the output that coming from the device.
For example: executing some show command creating dataframe with the output and returning this dataframe.
Now I've encountered that this output can be different regarding to the version installed on the device.
So I want to create modules that will contain this specific methods (that aren't generic) and add them dynamically regarding the device version (Monkey patching).
For example: if the device version is 1.8.20-6 I will search for 1.8.20.py module and add it.
Now I found how to decorate those functions #some_decorater so they will be added to the interface class on run time.
In case I'm adding those modules by full path with the following code:
import importlib.util
spec = importlib.util.spec_from_file_location("module.name", "/path/to/file.py")
foo = importlib.util.module_from_spec(spec)
spec.loader.exec_module(foo)
Does the python interpreter will automatically search for all decorated functions and will add them to interface class?
Or is there another way to deal with this problem? maybe some python package that deals with it.
Another question I have is why pycharm IDE isn't recognizing me modules with name x.y.z.py or x_y_z.py when I trying to import them (import framework.Versions.x_y_z)?
By the way , I know this is not a typical code problem question but more "how to" question.
It is probably a better idea to build a specific interface for each device, and use a factory to choose & return the proper object:
calling get_device_adapter with the device as argument, will return the proper DeviceAdapter object.
pseudocode:
class AbstractDeviceAdapter(ABC):
def __init__(self, ...):
initiatlize the common attributes
...
def common_method_1(self):
pass
def common_method_2(self):
do 2
#abstractmethod
def device_specific_method_0(self):
pass
#abstractmethod
def device_specific_method_1(self):
pass
class Device01Adapter(AbstractAdapterInterface):
def device_specific_method_0(self):
do specific stuff for device 01
def device_specific_method_1(self):
do specific stuff for device 01
class Device02Adapter(AbstractAdapterInterface):
def device_specific_method_0(self):
do specific stuff for device 02
def device_specific_method_1(self):
do specific stuff for device 02
def get_device_adapter(device, ...):
"""
returns the Adapter appropriate for the given device
"""
if '01' in device.__version__:
return Device01Adapter(...)
if '02' in device.__version__:
return Device02Adapter(...)
Introduction
For Java, Dependency Injection works as pure OOP, i.e. you provide an interface to be implemented and in your framework code accept an instance of a class that implements the defined interface.
Now for Python, you are able to do the same way, but I think that method was too much overhead right in case of Python. So then how would you implement it in the Pythonic way?
Use Case
Say this is the framework code:
class FrameworkClass():
def __init__(self, ...):
...
def do_the_job(self, ...):
# some stuff
# depending on some external function
The Basic Approach
The most naive (and maybe the best?) way is to require the external function to be supplied into the FrameworkClass constructor, and then be invoked from the do_the_job method.
Framework Code:
class FrameworkClass():
def __init__(self, func):
self.func = func
def do_the_job(self, ...):
# some stuff
self.func(...)
Client Code:
def my_func():
# my implementation
framework_instance = FrameworkClass(my_func)
framework_instance.do_the_job(...)
Question
The question is short. Is there any better commonly used Pythonic way to do this? Or maybe any libraries supporting such functionality?
UPDATE: Concrete Situation
Imagine I develop a micro web framework, which handles authentication using tokens. This framework needs a function to supply some ID obtained from the token and get the user corresponding to that ID.
Obviously, the framework does not know anything about users or any other application specific logic, so the client code must inject the user getter functionality into the framework to make the authentication work.
See Raymond Hettinger - Super considered super! - PyCon 2015 for an argument about how to use super and multiple inheritance instead of DI. If you don't have time to watch the whole video, jump to minute 15 (but I'd recommend watching all of it).
Here is an example of how to apply what's described in this video to your example:
Framework Code:
class TokenInterface():
def getUserFromToken(self, token):
raise NotImplementedError
class FrameworkClass(TokenInterface):
def do_the_job(self, ...):
# some stuff
self.user = super().getUserFromToken(...)
Client Code:
class SQLUserFromToken(TokenInterface):
def getUserFromToken(self, token):
# load the user from the database
return user
class ClientFrameworkClass(FrameworkClass, SQLUserFromToken):
pass
framework_instance = ClientFrameworkClass()
framework_instance.do_the_job(...)
This will work because the Python MRO will guarantee that the getUserFromToken client method is called (if super() is used). The code will have to change if you're on Python 2.x.
One added benefit here is that this will raise an exception if the client does not provide a implementation.
Of course, this is not really dependency injection, it's multiple inheritance and mixins, but it is a Pythonic way to solve your problem.
The way we do dependency injection in our project is by using the inject lib. Check out the documentation. I highly recommend using it for DI. It kinda makes no sense with just one function but starts making lots of sense when you have to manage multiple data sources etc, etc.
Following your example it could be something similar to:
# framework.py
class FrameworkClass():
def __init__(self, func):
self.func = func
def do_the_job(self):
# some stuff
self.func()
Your custom function:
# my_stuff.py
def my_func():
print('aww yiss')
Somewhere in the application you want to create a bootstrap file that keeps track of all the defined dependencies:
# bootstrap.py
import inject
from .my_stuff import my_func
def configure_injection(binder):
binder.bind(FrameworkClass, FrameworkClass(my_func))
inject.configure(configure_injection)
And then you could consume the code this way:
# some_module.py (has to be loaded with bootstrap.py already loaded somewhere in your app)
import inject
from .framework import FrameworkClass
framework_instance = inject.instance(FrameworkClass)
framework_instance.do_the_job()
I'm afraid this is as pythonic as it can get (the module has some python sweetness like decorators to inject by parameter etc - check the docs), as python does not have fancy stuff like interfaces or type hinting.
So to answer your question directly would be very hard. I think the true question is: does python have some native support for DI? And the answer is, sadly: no.
Some time ago I wrote dependency injection microframework with a ambition to make it Pythonic - Dependency Injector. That's how your code can look like in case of its usage:
"""Example of dependency injection in Python."""
import logging
import sqlite3
import boto.s3.connection
import example.main
import example.services
import dependency_injector.containers as containers
import dependency_injector.providers as providers
class Platform(containers.DeclarativeContainer):
"""IoC container of platform service providers."""
logger = providers.Singleton(logging.Logger, name='example')
database = providers.Singleton(sqlite3.connect, ':memory:')
s3 = providers.Singleton(boto.s3.connection.S3Connection,
aws_access_key_id='KEY',
aws_secret_access_key='SECRET')
class Services(containers.DeclarativeContainer):
"""IoC container of business service providers."""
users = providers.Factory(example.services.UsersService,
logger=Platform.logger,
db=Platform.database)
auth = providers.Factory(example.services.AuthService,
logger=Platform.logger,
db=Platform.database,
token_ttl=3600)
photos = providers.Factory(example.services.PhotosService,
logger=Platform.logger,
db=Platform.database,
s3=Platform.s3)
class Application(containers.DeclarativeContainer):
"""IoC container of application component providers."""
main = providers.Callable(example.main.main,
users_service=Services.users,
auth_service=Services.auth,
photos_service=Services.photos)
Here is a link to more extensive description of this example - http://python-dependency-injector.ets-labs.org/examples/services_miniapp.html
Hope it can help a bit. For more information please visit:
GitHub https://github.com/ets-labs/python-dependency-injector
Docs http://python-dependency-injector.ets-labs.org/
Dependency injection is a simple technique that Python supports directly. No additional libraries are required. Using type hints can improve clarity and readability.
Framework Code:
class UserStore():
"""
The base class for accessing a user's information.
The client must extend this class and implement its methods.
"""
def get_name(self, token):
raise NotImplementedError
class WebFramework():
def __init__(self, user_store: UserStore):
self.user_store = user_store
def greet_user(self, token):
user_name = self.user_store.get_name(token)
print(f'Good day to you, {user_name}!')
Client Code:
class AlwaysMaryUser(UserStore):
def get_name(self, token):
return 'Mary'
class SQLUserStore(UserStore):
def __init__(self, db_params):
self.db_params = db_params
def get_name(self, token):
# TODO: Implement the database lookup
raise NotImplementedError
client = WebFramework(AlwaysMaryUser())
client.greet_user('user_token')
The UserStore class and type hinting are not required for implementing dependency injection. Their primary purpose is to provide guidance to the client developer. If you remove the UserStore class and all references to it, the code still works.
After playing around with some of the DI frameworks in python, I've found they have felt a bit clunky to use when comparing how simple it is in other realms such as with .NET Core. This is mostly due to the joining via things like decorators that clutter the code and make it hard to simply add it into or remove it from a project, or joining based on variable names.
I've recently been working on a dependency injection framework that instead uses typing annotations to do the injection called Simple-Injection. Below is a simple example
from simple_injection import ServiceCollection
class Dependency:
def hello(self):
print("Hello from Dependency!")
class Service:
def __init__(self, dependency: Dependency):
self._dependency = dependency
def hello(self):
self._dependency.hello()
collection = ServiceCollection()
collection.add_transient(Dependency)
collection.add_transient(Service)
collection.resolve(Service).hello()
# Outputs: Hello from Dependency!
This library supports service lifetimes and binding services to implementations.
One of the goals of this library is that it is also easy to add it to an existing application and see how you like it before committing to it as all it requires is your application to have appropriate typings, and then you build the dependency graph at the entry point and run it.
Hope this helps. For more information, please see
github: https://github.com/BradLewis/simple-injection
docs: https://simple-injection.readthedocs.io/en/latest/
pypi: https://pypi.org/project/simple-injection/
A very easy and Pythonic way to do dependency injection is importlib.
You could define a small utility function
def inject_method_from_module(modulename, methodname):
"""
injects dynamically a method in a module
"""
mod = importlib.import_module(modulename)
return getattr(mod, methodname, None)
And then you can use it:
myfunction = inject_method_from_module("mypackage.mymodule", "myfunction")
myfunction("a")
In mypackage/mymodule.py you define myfunction
def myfunction(s):
print("myfunction in mypackage.mymodule called with parameter:", s)
You could of course also use a class MyClass iso. the function myfunction. If you define the values of methodname in a settings.py file you can load different versions of the methodname depending on the value of the settings file. Django is using such a scheme to define its database connection.
I think that DI and possibly AOP are not generally considered Pythonic because of typical Python developers preferences, rather that language features.
As a matter of fact you can implement a basic DI framework in <100 lines, using metaclasses and class decorators.
For a less invasive solution, these constructs can be used to plug-in custom implementations into a generic framework.
There is also Pinject, an open source python dependency injector by Google.
Here is an example
>>> class OuterClass(object):
... def __init__(self, inner_class):
... self.inner_class = inner_class
...
>>> class InnerClass(object):
... def __init__(self):
... self.forty_two = 42
...
>>> obj_graph = pinject.new_object_graph()
>>> outer_class = obj_graph.provide(OuterClass)
>>> print outer_class.inner_class.forty_two
42
And here is the source code
Due to Python OOP implementation, IoC and dependency injection are not standard practices in the Python world. But the approach seems promising even for Python.
To use dependencies as arguments is a non-pythonic approach. Python is an OOP language with beautiful and elegant OOP model, that provides more straightforward ways to maintain dependencies.
To define classes full of abstract methods just to imitate interface type is weird too.
Huge wrapper-on-wrapper workarounds create code overhead.
I also don't like to use libraries when all I need is a small pattern.
So my solution is:
# Framework internal
def MetaIoC(name, bases, namespace):
cls = type("IoC{}".format(name), tuple(), namespace)
return type(name, bases + (cls,), {})
# Entities level
class Entity:
def _lower_level_meth(self):
raise NotImplementedError
#property
def entity_prop(self):
return super(Entity, self)._lower_level_meth()
# Adapters level
class ImplementedEntity(Entity, metaclass=MetaIoC):
__private = 'private attribute value'
def __init__(self, pub_attr):
self.pub_attr = pub_attr
def _lower_level_meth(self):
print('{}\n{}'.format(self.pub_attr, self.__private))
# Infrastructure level
if __name__ == '__main__':
ENTITY = ImplementedEntity('public attribute value')
ENTITY.entity_prop
EDIT:
Be careful with the pattern. I used it in a real project and it showed itself a not that good way. My post on Medium about my experience with the pattern.
I have the following file structure, which implements a very simple plugin architecture using Yapsy.
Plugins/BasePlugin.py
from yapsy.IPlugin import IPlugin
class BasePlugin(IPlugin):
def process(self):
# Do some stuff common to all plugins
pass
Plugins/TestPlugin.py
from Plugins.BasePlugin import BasePlugin
class TestPlugin(BasePlugin):
def process(self, info):
super(TestPlugin, self).process()
# Do stuff
return "done"
test.py
from yapsy.PluginManager import PluginManager
from Plugins.BasePlugin import BasePlugin
import logging
logging.basicConfig(level=logging.DEBUG)
# See note 1
manager = PluginManager() # Does not work
manager = PluginManager(categories_filter={'BasePlugin': BasePlugin}) # Works
def init_plugins():
# Load the plugins from the plugin directory.
manager.setPluginPlaces(["Plugins"])
manager.collectPlugins()
# Loop round the plugins and print their names.
for plugin in manager.getAllPlugins():
manager.activatePluginByName(plugin.name, "BasePlugin")
print "Plugin path: {}".format(plugin.path)
print "Plugin obj: {}".format(plugin.plugin_object)
result = plugin.plugin_object.process(info)
There is also the correct .yapsy-plugin information file for TestPlugin.
Without the filter (at note 1) Yapsy tries to instantiate a BasePlugin instead of my TestPlugin, despite the plugin's name being "TestPlugin". For example:
Plugin path: /home/user/python/Plugins/TestPlugin
Plugin obj: <Plugins.BasePlugin.BasePlugin object at 0x7f159af22050>
If I include the filter then plugins are loaded correctly and my TestPlugin class is instantiated and used instead.
DEBUG:yapsy:Activating plugin: BasePlugin.test
Plugin path: /home/david/python/Plugins/TestPlugin
Plugin obj: <yapsy_loaded_plugin_test_0.TestPlugin object at 0x7f4dad7d4050>
Am I doing something wrong with inheritance, or is this just how Yapsy works? I can't see why it would try to use BasePlugin instead of TestPlugin despite finding the correct file.
Do I need to implement categories to extend IPlugin and provide my own base class?
You've come across a known problem of yapsy which is related to the way it detects plugins in files and categorizes them.
This is explained in the troubleshooting section of yapsy's doc at https://yapsy.readthedocs.org/en/latest/Advices.html#plugin-class-detection-caveat.
Since these are already my own words, I'll just copy-paste them here, but if something's unclear, feel free to ask for precisions.
There must be only one plugin defined per module. This means that you can’t have two plugin description files pointing at the same module for instance.
Because of the “categorization by inheritance” system, you mustn’t directly import the subclass of IPlugin in the main plugin file, instead import its containing module and make your plugin class inherit from ContainingModule.SpecificPluginClass as in the following example.
The following code won’t work (the class MyBasePluginClass will be detected as the plugin’s implementation instead of MyPlugin):
from myapp.plugintypes import MyBasePluginClass
class MyPlugin(MyBasePluginClass):
pass
Instead you should do the following:
import myapp.plugintypes as plugintypes
class MyPlugin(plugintypes.MyBasePluginClass):
pass
I'm currently writing an application which allows the user to extend it via a 'plugin' type architecture. They can write additional python classes based on a BaseClass object I provide, and these are loaded against various application signals. The exact number and names of the classes loaded as plugins is unknown before the application is started, but are only loaded once at startup.
During my research into the best way to tackle this I've come up with two common solutions.
Option 1 - Roll your own using imp, pkgutil, etc.
See for instance, this answer or this one.
Option 2 - Use a plugin manager library
Randomly picking a couple
straight.plugin
yapsy
this approach
My question is - on the proviso that the application must be restarted in order to load new plugins - is there any benefit of the above methods over something inspired from this SO answer and this one such as:
import inspect
import sys
import my_plugins
def predicate(c):
# filter to classes
return inspect.isclass(c)
def load_plugins():
for name, obj in inspect.getmembers(sys.modules['my_plugins'], predicate):
obj.register_signals()
Are there any disadvantages to this approach compared to the ones above? (other than all the plugins must be in the same file) Thanks!
EDIT
Comments request further information... the only additional thing I can think to add is that the plugins use the blinker library to provide signals that they subscribe to. Each plugin may subscribe to different signals of different types and hence must have its own specific "register" method.
Since Python 3.6 a new class method __init_subclass__ is added, that is called on a base class, whenever a new subclass is created.
This method can further simplify the solution offered by will-hart above, by removing the metaclass.
The __init_subclass__ method was introduced with PEP 487: Simpler customization of class creation. The PEP comes with a minimal example for a plugin architecture:
It is now possible to customize subclass creation without using a
metaclass. The new __init_subclass__ classmethod will be called on
the base class whenever a new subclass is created:
class PluginBase:
subclasses = []
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
cls.subclasses.append(cls)
class Plugin1(PluginBase):
pass
class Plugin2(PluginBase):
pass
The PEP example above stores references to the classes in the Plugin.plugins field.
If you want to store instances of the plugin classes, you can use a structure like this:
class Plugin:
"""Base class for all plugins. Singleton instances of subclasses are created automatically and stored in Plugin.plugins class field."""
plugins = []
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
cls.plugins.append(cls())
class MyPlugin1(Plugin):
def __init__(self):
print("MyPlugin1 instance created")
def do_work(self):
print("Do something")
class MyPlugin2(Plugin):
def __init__(self):
print("MyPlugin2 instance created")
def do_work(self):
print("Do something else")
for plugin in Plugin.plugins:
plugin.do_work()
which outputs:
MyPlugin1 instance created
MyPlugin2 instance created
Do something
Do something else
The metaclass approach is useful for this issue in Python < 3.6 (see #quasoft's answer for Python 3.6+). It is very simple and acts automatically on any imported module. In addition, complex logic can be applied to plugin registration with very little effort. It requires:
The metaclass approach works like the following:
1) A custom PluginMount metaclass is defined which maintains a list of all plugins
2) A Plugin class is defined which sets PluginMount as its metaclass
3) When an object deriving from Plugin - for instance MyPlugin is imported, it triggers the __init__ method on the metaclass. This registers the plugin and performs any application specific logic and event subscription.
Alternatively if you put the PluginMount.__init__ logic in PluginMount.__new__ it is called whenver a new instance of a Plugin derived class is created.
class PluginMount(type):
"""
A plugin mount point derived from:
http://martyalchin.com/2008/jan/10/simple-plugin-framework/
Acts as a metaclass which creates anything inheriting from Plugin
"""
def __init__(cls, name, bases, attrs):
"""Called when a Plugin derived class is imported"""
if not hasattr(cls, 'plugins'):
# Called when the metaclass is first instantiated
cls.plugins = []
else:
# Called when a plugin class is imported
cls.register_plugin(cls)
def register_plugin(cls, plugin):
"""Add the plugin to the plugin list and perform any registration logic"""
# create a plugin instance and store it
# optionally you could just store the plugin class and lazily instantiate
instance = plugin()
# save the plugin reference
cls.plugins.append(instance)
# apply plugin logic - in this case connect the plugin to blinker signals
# this must be defined in the derived class
instance.register_signals()
Then a base plugin class which looks like:
class Plugin(object):
"""A plugin which must provide a register_signals() method"""
__metaclass__ = PluginMount
Finally, an actual plugin class would look like the following:
class MyPlugin(Plugin):
def register_signals(self):
print "Class created and registering signals"
def other_plugin_stuff(self):
print "I can do other plugin stuff"
Plugins can be accessed from any python module that has imported Plugin:
for plugin in Plugin.plugins:
plugin.other_plugin_stuff()
See the full working example
The approach from will-hart was the most useful one to me!
For i needed more control I wrapped the Plugin Base class in a function like:
def get_plugin_base(name='Plugin',
cls=object,
metaclass=PluginMount):
def iter_func(self):
for mod in self._models:
yield mod
bases = not isinstance(cls, tuple) and (cls,) or cls
class_dict = dict(
_models=None,
session=None
)
class_dict['__iter__'] = iter_func
return metaclass(name, bases, class_dict)
and then:
from plugin import get_plugin_base
Plugin = get_plugin_base()
This allows to add additional baseclasses or switching to another metaclass.
What are the best practices for extending an existing Python module – in this case, I want to extend the python-twitter package by adding new methods to the base API class.
I've looked at tweepy, and I like that as well; I just find python-twitter easier to understand and extend with the functionality I want.
I have the methods written already – I'm trying to figure out the most Pythonic and least disruptive way to add them into the python-twitter package module, without changing this modules’ core.
A few ways.
The easy way:
Don't extend the module, extend the classes.
exttwitter.py
import twitter
class Api(twitter.Api):
pass
# override/add any functions here.
Downside : Every class in twitter must be in exttwitter.py, even if it's just a stub (as above)
A harder (possibly un-pythonic) way:
Import * from python-twitter into a module that you then extend.
For instance :
basemodule.py
class Ball():
def __init__(self,a):
self.a=a
def __repr__(self):
return "Ball(%s)" % self.a
def makeBall(a):
return Ball(a)
def override():
print "OVERRIDE ONE"
def dontoverride():
print "THIS WILL BE PRESERVED"
extmodule.py
from basemodule import *
import basemodule
def makeBalls(a,b):
foo = makeBall(a)
bar = makeBall(b)
print foo,bar
def override():
print "OVERRIDE TWO"
def dontoverride():
basemodule.dontoverride()
print "THIS WAS PRESERVED"
runscript.py
import extmodule
#code is in extended module
print extmodule.makeBalls(1,2)
#returns Ball(1) Ball(2)
#code is in base module
print extmodule.makeBall(1)
#returns Ball(1)
#function from extended module overwrites base module
extmodule.override()
#returns OVERRIDE TWO
#function from extended module calls base module first
extmodule.dontoverride()
#returns THIS WILL BE PRESERVED\nTHIS WAS PRESERVED
I'm not sure if the double import in extmodule.py is pythonic - you could remove it, but then you don't handle the usecase of wanting to extend a function that was in the namespace of basemodule.
As far as extended classes, just create a new API(basemodule.API) class to extend the Twitter API module.
Don't add them to the module. Subclass the classes you want to extend and use your subclasses in your own module, not changing the original stuff at all.
Here’s how you can directly manipulate the module list at runtime – spoiler alert: you get the module type from types module:
from __future__ import print_function
import sys
import types
import typing as tx
def modulize(namespace: tx.Dict[str, tx.Any],
modulename: str,
moduledocs: tx.Optional[str] = None) -> types.ModuleType:
""" Convert a dictionary mapping into a legit Python module """
# Create a new module with a trivially namespaced name:
namespacedname: str = f'__dynamic_modules__.{modulename}'
module = types.ModuleType(namespacedname, moduledocs)
module.__dict__.update(namespace)
# Inspect the new module:
name: str = module.__name__
doc: tx.Optional[str] = module.__doc__
contents: str = ", ".join(sorted(module.__dict__.keys()))
print(f"Module name: {name}")
print(f"Module contents: {contents}")
if doc:
print(f"Module docstring: {doc}")
# Add to sys.modules, as per import machinery:
sys.modules.update({ modulename : module })
# Return the new module instance:
return module
… you could then use such a function like so:
ns = {
'func' : lambda: print("Yo Dogg"), # these can also be normal non-lambda funcs
'otherfunc' : lambda string=None: print(string or 'no dogg.'),
'__all__' : ('func', 'otherfunc'),
'__dir__' : lambda: ['func', 'otherfunc'] # usually this’d reference __all__
}
modulize(ns, 'wat', "WHAT THE HELL PEOPLE")
import wat
# Call module functions:
wat.func()
wat.otherfunc("Oh, Dogg!")
# Inspect module:
contents = ", ".join(sorted(wat.__dict__.keys()))
print(f"Imported module name: {wat.__name__}")
print(f"Imported module contents: {contents}")
print(f"Imported module docstring: {wat.__doc__}")
… You could also create your own module subclass, by specifying types.ModuleType as the ancestor of your newly declared class, of course; I have never personally found this necessary to do.
(Also, you don’t have to get the module type from the types module – you can always just do something like ModuleType = type(os) after importing os – I specifically pointed out this one source of the type because it is non-obvious; unlike many of its other builtin types, Python doesn’t offer up access to the module type in the global namespace.)
The real action is in the sys.modules dict, where (if you are appropriately intrepid) you can replace existing modules as well as adding your new ones.
Say you have an older module called mod that you use like this:
import mod
obj = mod.Object()
obj.method()
mod.function()
# and so on...
And you want to extend it, without replacing it for your users. Easily done. You can give your new module a different name, newmod.py or place it by same name at a deeper path and keep the same name, e.g. /path/to/mod.py. Then your users can import it in either of these ways:
import newmod as mod # e.g. import unittest2 as unittest idiom from Python 2.6
or
from path.to import mod # useful in a large code-base
In your module, you'll want to make all the old names available:
from mod import *
or explicitly name every name you import:
from mod import Object, function, name2, name3, name4, name5, name6, name7, name8, name9, name10, name11, name12, name13, name14, name15, name16, name17, name18, name19, name20, name21, name22, name23, name24, name25, name26, name27, name28, name29, name30, name31, name32, name33, name34, name35, name36, name37, name38, name39
I think the import * will be more maintainable for this use-case - if the base module expands functionality, you'll seamlessly keep up (though you might shade new objects with the same name).
If the mod you are extending has a decent __all__, it will restrict the names imported.
You should also declare an __all__ and extend it with the extended module's __all__.
import mod
__all__ = ['NewObject', 'newfunction']
__all__ += mod.__all__
# if it doesn't have an __all__, maybe it's not good enough to extend
# but it could be relying on the convention of import * not importing
# names prefixed with underscores, (_like _this)
Then extend the objects and functionality as you normally would.
class NewObject(object):
def newmethod(self):
"""this method extends Object"""
def newfunction():
"""this function builds on mod's functionality"""
If the new objects provide functionality you intend to replace (or perhaps you are backporting the new functionality into an older code base) you can overwrite the names
May I suggest not to reinvent the Wheel here? I'm building a >6k line Twitter Client for 2 month now, at first I checked python-twitter too, but it's lagging a lot behind the recent API changes,, Development doesn't seem to be that active either, also there was(at least when I last checked) no support for OAuth/xAuth).
So after searching around a bit more I discovered tweepy:
http://github.com/joshthecoder/tweepy
Pros: Active development, OAauth/xAuth and up to date with the API.
Chances are high that what you need is already in there.
So I suggest going with that, it's working for me, the only thing I had to add was xAuth(that got merge back to tweepy :)
Oh an a shameless plug, if you need to parse Tweets and/or format them to HTML use my python version of the twitter-text-* libraries:
http://github.com/BonsaiDen/twitter-text-python
This thing is unittestetd an guaranteed to parse Tweets just like Twitter.com does it.
Define a new class, and instead of inherit it from the class you want to extend from the original module, add an instance of the original class as an attribute to your new class.
And here comes the trick: intercept all non-existing method calls on your new class and try to call it on the instance of the old class.
In your NewClass just define new or overridden methods as you like:
import originalmodule
class NewClass:
def __init__(self, *args, **kwargs):
self.old_class_instance = originalmodule.create_oldclass_instance(*args, **kwargs)
def __getattr__(self, methodname):
"""This is a wrapper for the original OldClass class.
If the called method is not part of this NewClass class,
the call will be intercepted and replaced by the method
in the original OldClass instance.
"""
def wrapper(*args, **kwargs):
return getattr(self.old_class_instance, methodname)(*args, **kwargs)
return wrapper
def new_method(self, arg1):
"""Does stuff with the OldClass instance"""
thing = self.old_class_instance.get_somelist(arg1)
# returns the first element only
return thing[0]
def overridden_method(self):
"""Overrides an existing method, if OldClass has a method with the same name"""
print("This message is coming from the NewClass and not from the OldClass")
In my case I used this solution when simple inheritance from the old class was not possible, because an instance had to be created not by its constructor, but with an init script from an other class/module. (It is the originalmodule.create_oldclass_instance in the example above.)