I am building a tool that takes directories as inputs and performs actions where necessary. These actions vary depeding on certain variables so I created a few class objects which help me with my needs in an organised fashion.
However, I hit a wall figuring out how to best design the following scenario.
For the sake of simplicity, let's assume there are only directories (no files). Also, the below is a heavily simplified example.
I have the following parent class:
# directory.py
from pathlib import Path
class Directory:
def __init__(self, absolute_path):
self.path = Path(absolute_path)
def content(self):
return [Directory(c) for c in self.path.iterdir()]
So, I have a method in the parent class that returns Directory instances for each directory inside the initial directory in absolute_path
What the above does, is hold all methods that can be performed on all directories. Now, I have a separate class that inherits from the above and adds further methods.
# special_directory.py
from directory import Directory
class SpecialDirectory(Directory):
def __init__(self, absolute_path):
super().__init__(absolute_path)
# More methods
I am using an Object Factory like approach to build one or the other based on a condition like so:
# directory_factory.py
from directory import Directory
from special_directory import SpecialDirectory
def pick(path):
return SpecialDirectory(path) if 'foo' in path else Directory(path)
So, if 'foo' exists in the path, it should be a SpecialDirectory instead allowing it to do everything Directory does plus more.
The problem I'm facing is with the content() method. Both should be able to do that but I don't want it to be limited to making a list of Directory instances. If any of its content has "foo*", it should be a SpecialDirectory.
Directory doesn't (and shouldn't) know about SpecialDirectory, so I tried importing and using the factory but it complains about some circular import (which makes sense).
I am not particularly stuck as I have come up with a temp fix, but it isn't pretty. So I was hoping I could get some tips as to what would be an effective and clean solution for this specific situation.
What you need is sometimes called a "virtual constructor" which is a way to allow subclasses to determine what type of class instance is created when calling the base class constructor. There's no such thing in Python (or C++ for that matter), but you can simulate them. Below is an example of a way of doing this.
Note this code is very similar to what's in my answer to the question titled Improper use of __new__ to generate classes? (which has more information about the technique). Also see the one to What exactly is a Class Factory?
from pathlib import Path
class Directory:
subclasses = []
#classmethod
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
cls.subclasses.append(cls)
def __init__(self, absolute_path):
self.path = Path(absolute_path)
def __new__(cls, path):
""" Create instance of appropriate subclass. """
for subclass in cls.subclasses:
if subclass.pick(path):
return object.__new__(subclass)
else:
return object.__new__(cls) # Default is this base class.
def content(self):
return [Directory(c) for c in self.path.iterdir()]
def __repr__(self):
classname = type(self).__name__
return f'{classname}(path={self.path!r})'
# More methods
...
class SpecialDirectory(Directory):
def __init__(self, absolute_path):
super().__init__(absolute_path)
#classmethod
def pick(cls, path):
return 'foo' in str(path)
# More methods
...
if __name__ == '__main__':
root = './_obj_factory_test'
d = Directory(root)
print(d.content())
I'd like to describe my problem with code to make it clear:
class MyBaseClass(object):
def __init__(self):
print 'foobar'
def __call__(self):
print 'spameggs'
def __is_used__(self): # This is only a pseudo method
print 'I\'m being used! - MyBaseClass'
class MySubClass(MyBaseClass):
def __init__(self):
print 'monty python'
Now I'd like to know if there is a similar magic method __is_used__ for a class object to know if it is being use as a parent/base class of another class (sub)?
Example usage:
class_a = MySubClass()
# Output
# monty python
# I'm being used! - MyBaseClass
Use Case
To avoid confusion (I apologize). A best example would be a mixin. Example an S3Mixin.
An S3Mixin has a capabilities to upload and download file to S3 buckets.
class S3Mixin(object):
def upload(self):
def download(self):
Now i want to use it to ImageFile and VideoFile classes.
class ImageFile(S3Mixin):
# omitted lengthy properties
class VideoFile(S3Mixin):
# omitted lengthy properties
Now each object has a function to use the s3 basic functionalities. Now the real problem arise when I try to use another module inside a S3Mixin which cause a circular dependency issue. Now to avoid it I have to import it inside each function of S3Mixin. I tried putting it on the __init__ method and __call__ method which is obviously not going to work.
I don't want to do that. Instead I wanted to know if there is available method so I can import all the conflicted module preferable on a magic method of an S3Mixin.
Note:
I'm not asking for a checking of a class that is a subclass of another class. That is far from the question. I would like to know if there is a MAGIC METHOD so i can further create a logic in it when a base class is used.
How to get the filename of the subclass?
Example:
base.py:
class BaseClass:
def __init__(self):
# How to get the path "./main1.py"?
main1.py:
from base import BaseClass
class MainClass1(BaseClass):
pass
Remember that self in BaseClass.__init__ is an instance of the actual class that's being initialised. Therefore, one solution, is to ask that class which module it came from, and then from the path for that module:
import importlib
class BaseClass:
def __init__(self):
m = importlib.import_module(self.__module__)
print m.__file__
I think there are probably a number of way you could end up with a module that you can't import though; this doesn't feel like the most robust solution.
If all you're trying to do is identify where the subclass came from, then probably combining the module name and class name is sufficient, since that should uniquely identify it:
class BaseClass:
def __init__(self):
print "{}.{}".format(
self.__module__,
self.__class__.__name__
)
You could do it by reaching back through the calling stack to get the global namespace of the caller of the BaseClass.__init__() method, and from that you can extract the name of the file it is in by using the value of the __file__ key in that namespace.
Here's what I mean:
base.py:
import sys
class BaseClass(object):
def __init__(self):
print('In BaseClass.__init__()')
callers_path = sys._getframe(1).f_globals['__file__']
print(' callers_path:', callers_path)
main1.py:
from base import BaseClass
class MainClass1(BaseClass):
def __init(self):
super().__init__()
mainclass1 = MainClass1()
Sample output of running main1.py:
In BaseClass.__init__()
callers_path: the\path\to\main1.py
I think you're looking to the wrong mechanism for your solution. Your comments suggest that what you want is an exception handler with minimal trace-back capability. This is not something readily handled within the general class mechanism.
Rather, you should look into Python's stack inspection capabilities. Very simply, you want your __init__ method to report the file name of the calling sub-class. You can hammer this by requiring the caller to pass its own __file__ value. In automated fashion, you can dig back one stack frame and access __file__ via that context record. Note that this approach assumes that the only time you need this information is when __init__ is called is directly from a sub-class method.
Is that enough to get you to the right documentation?
I have the following code, which is a command line test
from cmd2 import Cmd
class App(Cmd, object):
def __init__(self, *args, **kwargs):
pass
def do_test(self, line):
'test'
print "parent test"
class App2():
def __init__(self, *args, **kwargs):
pass
def do_test2(self, line):
print "Test2"
app = App()
app.cmdloop()
Is there a possibility to extend the App class with extra functions?
I know there is the following solution
class App2(App):
....
app = App2()
app.cmdloop()
but in my case I would like to run only the App and extend it if it is possible.
In general this is not a good idea, because when people (including you in six months) read the code they will expect App to be the class they know about, and for an extended version of the class to have a different name. However, there's nothing preventing you from naming the subclass the same as the original:
class App(App):
# etc.
Python's smart enough to know that the App in parentheses means the one it already knows about, and then, since the new class has the same name, it replaces the original one. Don't worry, the new class contains a reference to the old one, so the old class doesn't go away entirely (if it did, nothing inherited by the subclass would work).
If the class came from some module that you've imported, you can even monkey-patch the replacement class back into the original module, so that all code that imports that module uses your replacement class. (Though I would recommend against it!)
import appmodule
class App(appmodule.App):
# etc.
appmodule.App = App
Of course, this gets tricky, because some modules may already have imported a reference to the original class, if you don't do this first thing in your script. And if other modules are also trying to patch the same class, all hell can break loose. Still, if you want to confuse yourself and those who will maintain your code, Python will let you do it!
It is worth noting that you can always augment the class dictionary, therefore extending it at runtime.
class App(...):
def __init__(self, a, b):
pass
def do_something(self, a):
pass
app_instance = App()
def do_something_else(self, b):
pass
App.do_something_else = do_something_else
app_instance.do_something_else('b')
You have to think how python does lookups at runtime. First looks at the instance of yourclass, then looks at the __mro__ (starting with type(yourclass)), and on up until it gets to object.
Since classes ARE objects, you can extend them by adding attributes, which will then be found during attribute lookups. Make sure you do this ONCE (eg, during an import of another file).
Here is a real example:
>>> class foo():
... pass
...
>>> x = foo()
>>>
>>> # Define a function and attach it
>>>
>>> def bar(self, a):
... print(a)
...
>>> foo.bar = bar
>>>
>>> x.bar('a')
a
this isnt an exact solution but it allows you to always access it by App name
rename App class to _App
then where you want to use it
from blah import _App as App
and when you extend it
from blah import App2 as App
What are the best practices for extending an existing Python module – in this case, I want to extend the python-twitter package by adding new methods to the base API class.
I've looked at tweepy, and I like that as well; I just find python-twitter easier to understand and extend with the functionality I want.
I have the methods written already – I'm trying to figure out the most Pythonic and least disruptive way to add them into the python-twitter package module, without changing this modules’ core.
A few ways.
The easy way:
Don't extend the module, extend the classes.
exttwitter.py
import twitter
class Api(twitter.Api):
pass
# override/add any functions here.
Downside : Every class in twitter must be in exttwitter.py, even if it's just a stub (as above)
A harder (possibly un-pythonic) way:
Import * from python-twitter into a module that you then extend.
For instance :
basemodule.py
class Ball():
def __init__(self,a):
self.a=a
def __repr__(self):
return "Ball(%s)" % self.a
def makeBall(a):
return Ball(a)
def override():
print "OVERRIDE ONE"
def dontoverride():
print "THIS WILL BE PRESERVED"
extmodule.py
from basemodule import *
import basemodule
def makeBalls(a,b):
foo = makeBall(a)
bar = makeBall(b)
print foo,bar
def override():
print "OVERRIDE TWO"
def dontoverride():
basemodule.dontoverride()
print "THIS WAS PRESERVED"
runscript.py
import extmodule
#code is in extended module
print extmodule.makeBalls(1,2)
#returns Ball(1) Ball(2)
#code is in base module
print extmodule.makeBall(1)
#returns Ball(1)
#function from extended module overwrites base module
extmodule.override()
#returns OVERRIDE TWO
#function from extended module calls base module first
extmodule.dontoverride()
#returns THIS WILL BE PRESERVED\nTHIS WAS PRESERVED
I'm not sure if the double import in extmodule.py is pythonic - you could remove it, but then you don't handle the usecase of wanting to extend a function that was in the namespace of basemodule.
As far as extended classes, just create a new API(basemodule.API) class to extend the Twitter API module.
Don't add them to the module. Subclass the classes you want to extend and use your subclasses in your own module, not changing the original stuff at all.
Here’s how you can directly manipulate the module list at runtime – spoiler alert: you get the module type from types module:
from __future__ import print_function
import sys
import types
import typing as tx
def modulize(namespace: tx.Dict[str, tx.Any],
modulename: str,
moduledocs: tx.Optional[str] = None) -> types.ModuleType:
""" Convert a dictionary mapping into a legit Python module """
# Create a new module with a trivially namespaced name:
namespacedname: str = f'__dynamic_modules__.{modulename}'
module = types.ModuleType(namespacedname, moduledocs)
module.__dict__.update(namespace)
# Inspect the new module:
name: str = module.__name__
doc: tx.Optional[str] = module.__doc__
contents: str = ", ".join(sorted(module.__dict__.keys()))
print(f"Module name: {name}")
print(f"Module contents: {contents}")
if doc:
print(f"Module docstring: {doc}")
# Add to sys.modules, as per import machinery:
sys.modules.update({ modulename : module })
# Return the new module instance:
return module
… you could then use such a function like so:
ns = {
'func' : lambda: print("Yo Dogg"), # these can also be normal non-lambda funcs
'otherfunc' : lambda string=None: print(string or 'no dogg.'),
'__all__' : ('func', 'otherfunc'),
'__dir__' : lambda: ['func', 'otherfunc'] # usually this’d reference __all__
}
modulize(ns, 'wat', "WHAT THE HELL PEOPLE")
import wat
# Call module functions:
wat.func()
wat.otherfunc("Oh, Dogg!")
# Inspect module:
contents = ", ".join(sorted(wat.__dict__.keys()))
print(f"Imported module name: {wat.__name__}")
print(f"Imported module contents: {contents}")
print(f"Imported module docstring: {wat.__doc__}")
… You could also create your own module subclass, by specifying types.ModuleType as the ancestor of your newly declared class, of course; I have never personally found this necessary to do.
(Also, you don’t have to get the module type from the types module – you can always just do something like ModuleType = type(os) after importing os – I specifically pointed out this one source of the type because it is non-obvious; unlike many of its other builtin types, Python doesn’t offer up access to the module type in the global namespace.)
The real action is in the sys.modules dict, where (if you are appropriately intrepid) you can replace existing modules as well as adding your new ones.
Say you have an older module called mod that you use like this:
import mod
obj = mod.Object()
obj.method()
mod.function()
# and so on...
And you want to extend it, without replacing it for your users. Easily done. You can give your new module a different name, newmod.py or place it by same name at a deeper path and keep the same name, e.g. /path/to/mod.py. Then your users can import it in either of these ways:
import newmod as mod # e.g. import unittest2 as unittest idiom from Python 2.6
or
from path.to import mod # useful in a large code-base
In your module, you'll want to make all the old names available:
from mod import *
or explicitly name every name you import:
from mod import Object, function, name2, name3, name4, name5, name6, name7, name8, name9, name10, name11, name12, name13, name14, name15, name16, name17, name18, name19, name20, name21, name22, name23, name24, name25, name26, name27, name28, name29, name30, name31, name32, name33, name34, name35, name36, name37, name38, name39
I think the import * will be more maintainable for this use-case - if the base module expands functionality, you'll seamlessly keep up (though you might shade new objects with the same name).
If the mod you are extending has a decent __all__, it will restrict the names imported.
You should also declare an __all__ and extend it with the extended module's __all__.
import mod
__all__ = ['NewObject', 'newfunction']
__all__ += mod.__all__
# if it doesn't have an __all__, maybe it's not good enough to extend
# but it could be relying on the convention of import * not importing
# names prefixed with underscores, (_like _this)
Then extend the objects and functionality as you normally would.
class NewObject(object):
def newmethod(self):
"""this method extends Object"""
def newfunction():
"""this function builds on mod's functionality"""
If the new objects provide functionality you intend to replace (or perhaps you are backporting the new functionality into an older code base) you can overwrite the names
May I suggest not to reinvent the Wheel here? I'm building a >6k line Twitter Client for 2 month now, at first I checked python-twitter too, but it's lagging a lot behind the recent API changes,, Development doesn't seem to be that active either, also there was(at least when I last checked) no support for OAuth/xAuth).
So after searching around a bit more I discovered tweepy:
http://github.com/joshthecoder/tweepy
Pros: Active development, OAauth/xAuth and up to date with the API.
Chances are high that what you need is already in there.
So I suggest going with that, it's working for me, the only thing I had to add was xAuth(that got merge back to tweepy :)
Oh an a shameless plug, if you need to parse Tweets and/or format them to HTML use my python version of the twitter-text-* libraries:
http://github.com/BonsaiDen/twitter-text-python
This thing is unittestetd an guaranteed to parse Tweets just like Twitter.com does it.
Define a new class, and instead of inherit it from the class you want to extend from the original module, add an instance of the original class as an attribute to your new class.
And here comes the trick: intercept all non-existing method calls on your new class and try to call it on the instance of the old class.
In your NewClass just define new or overridden methods as you like:
import originalmodule
class NewClass:
def __init__(self, *args, **kwargs):
self.old_class_instance = originalmodule.create_oldclass_instance(*args, **kwargs)
def __getattr__(self, methodname):
"""This is a wrapper for the original OldClass class.
If the called method is not part of this NewClass class,
the call will be intercepted and replaced by the method
in the original OldClass instance.
"""
def wrapper(*args, **kwargs):
return getattr(self.old_class_instance, methodname)(*args, **kwargs)
return wrapper
def new_method(self, arg1):
"""Does stuff with the OldClass instance"""
thing = self.old_class_instance.get_somelist(arg1)
# returns the first element only
return thing[0]
def overridden_method(self):
"""Overrides an existing method, if OldClass has a method with the same name"""
print("This message is coming from the NewClass and not from the OldClass")
In my case I used this solution when simple inheritance from the old class was not possible, because an instance had to be created not by its constructor, but with an init script from an other class/module. (It is the originalmodule.create_oldclass_instance in the example above.)