Develop Reference

Typing an unimported module

I have a Python package that has an optional [extras] dependency, yet I want to adhere to typing on all methods.
The situation is that in my file, I have this
class MyClass:
def __init__(self, datastore: Datastore): # <- Datastore is azureml.core.Datastore
...
def my_func(self):
from azureml.core import Datastore
...
I import from within the function because there are other classes in the same file that should be imported when not using the extras (extras being azureml).
So this obviously fails, because I refer to Datastore before importing it. Removing the Datastore typing from the __init__ method obviously solves the problem.
So in general my question is whether it is possible, and if so how, to use typing when typing an optional (extras) package.
Notice, that importing in the class definition (below the class MyClass statement) is not a valid solution, as this code is called when the module is imported

You can use TYPE_CHECKING:
A special constant that is assumed to be True by 3rd party static type
checkers. It is False at runtime.
It is False at runtime: So it doesn't affect your module's behavior.
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from azureml.core import Datastore
class MyClass:
def __init__(self, datastore: Datastore):
...
def my_func(self):
from azureml.core import Datastore
...
Since I want to show this in action, I will use operator.itemgetter as an instance because it's recognizable for type checkers, but azureml.core is not:
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from operator import itemgetter
class MyClass:
def __init__(self, datastore: itemgetter):
...
def my_func(self):
from operator import itemgetter
...
obj1 = MyClass(itemgetter(1)) # line 16
obj2 = MyClass(10) # line 17
Here is the Mypy error:
main.py:17: error: Argument 1 to "MyClass" has incompatible type "int"; expected "itemgetter[Any]"
Found 1 error in 1 file (checked 1 source file)
Which shows it works as excepted.

Just to add my two cents:
While it is certainly a solution, I consider the use of the TYPE_CHECKING constant a red flag regarding the project structure. It typically (though not always) either shows the presence of circular dependencies or poor separation of concerns.
In your case it seems to be the latter, as you state this:
I import from within the function because there are other classes in the same file that should be imported when not using the extras
If MyClass provides optional functionality to your package, it should absolutely reside in its own module and not alongside other classes that provide core functionality.
When you put MyClass into its own module (say my_class), you can place its dependencies at the top with all the other imports. Then you put the import from my_class inside a function that handles the logic of loading internal optional dependencies.
Aside from visibility and arguably better style, one advantage of such a setup over the one you presented is that the my_class module will be consistent in itself and fail on import, if the extra azureml dependency is missing (or broken/renamed/deprecated), rather than at runtime only when MyClass.my_func is called.
You'd be surprised how easy it is to accidentally forget to install all extra dependencies (even in a production environment). Then you'll thank the stars, when the code fails immediately and transparently, rather than causing errors at some point later at runtime.

Sphinx python: autodoc does not import class methods defined in other source files

In my project, I have a class EnergyModel defined in energymodel.py. There is also generate_data class method imported from another source file:
#energy_model.py
from .energy_model_data_generation import generate_data
class EnergyModel(BaseEstimator):
def __init__(...):
EnergyModel.generate_data=generate_data
And there is another source file where the method is defined:
#energy_model_data_generation.py
def generate_data(self, ...):
...
I separated methods into different source files to keep the files sufficiently small - this is convenient for me. Now, I want to generate class documentation with Sphinx. My implementation.rst is the following:
Implementation
==============
.. _Energymodel_class:
Energymodel class
-----------------
.. autoclass:: neuralflow.EnergyModel
:members:
Where neuralflow is the name of the package (imported in conf.py). This does not generate documentation for generate_data methods. I did not find how to include it, I only found how you can cross-refence methods, which is not what I want.
Edit: There is also __init__.py file in my directory with the source file, so it is a package. It looks like this:
from .energy_model import EnergyModel
# Also import other modules/packages that are not part of EnergyModel class
__all__ = ['EnergyModel', ...(other stuff that are not part of EnergyModel)]
Also I've added root directory to the path in conf.py file

The problem is that the generate_data() method is added dynamically to the class in the constructor. Because Sphinx nevers call the constructor (as #mzjn points out), this method is simply not part of the class when Sphinx is building the documentation. The solution is to define the method at the class level:
import energy_model_generation as emg
class EnergyModel(BaseEstimator):
generate_data = emg.generate_data
As more of a code-review type comment, it doesn't really make sense to dynamically create methods in the constructor like this. Methods should only be defined once, but the constructor can be called many times—once for each instance of the class. It's easy to imagine this being the source of obscure bugs.

Python abstract base classes

As I dig further into Python internals, I start to see abc's more often in the documentation. Unfortunately the docs don't explain how they can be used. I haven't even been able to use the "concrete implementations" of these abstract base classes.
For example, reading about class importlib.abc.SourceLoader, one learns that "is_package" is a concrete implementation of InspectLoader.is_package(). But what if I'd like to use that in my code? Is it possible? I've tried many ways but the method can't be imported.
ExtensionFileLoader is documented as a concrete implementation of importlib.abc.ExecutionLoader, but if I try to use it (such as: from importlib import machinery.ExecutionLoader), once again it can't be found.
If these methods can't be imported, why are they documented? is there any sample code to show how they can be used? Example:
import importlib.abc.SourceLoader # doesn't work
class try_pkg_check():
def main(self, source_file_name):
possible_pkgs = ['math', 'numpy']
for posbl_pkg in possible_pkgs:
answer = SourceLoader.is_package(posbl_pkg)
print("For {}, the answer is: {}".format(posbl_pkg, answer))
return None
if __name__ == "__main__":
instantiated_obj = try_pkg_check()
instantiated_obj.main()
People might comment that I shouldn't try to import an abstract class. But "is_package" is documented as concrete, so I should be able to use it somehow, which is my question.

import importlib.abc.SourceLoader
The error message that this line produces should give you a hint where you've gone wrong:
ModuleNotFoundError: No module named 'importlib.abc.SourceLoader'; 'importlib.abc' is not a package
"import foo" requires that foo be a module, but SourceLoader is a class inside a module. You need to instead write:
from importlib.abc import SourceLoader
However, there are further problems with this line:
answer = SourceLoader.is_package(posbl_pkg)
First of all, SourceLoader.is_package is an instance method, not a class or static method; it has to be called on an instance of SourceLoader, not on the class itself. However, SourceLoader is an abstract class, so it can't be directly instantiated; you need to use a concrete subclass like SourceFileLoader instead. (When the docs call SourceLoader.is_package a "concrete implementation" of InspectLoader.is_package, I believe what they mean is that SourceLoader provides a default implementation for is_package so that its subclasses don't need to override it in order to be non-abstract.)
Hence, you need to write:
from importlib.machinery import SourceFileLoader
...
answer = SourceFileLoader(fullname, path).is_package(fullname)
where fullname is "a fully resolved name of the module the loader is to handle" and path is "the path to the file for the module."

Dynamically loaded subclass can still instanstiate even without all abstract methods overridden

Setup: Python 3.3
I have a base class, called SourceBase, which defines abstract methods and values:
import abc
class SourceBase(object):
__metaclass__=abc.ABCMeta
pluginid='undefined' #OVERRIDE THIS IN YOUR SUBCLASS. If you don't, the program will ignore your plugin.
#abc.abstractmethod
def get_images(self):
'''This method should return a list of URLs.'''
return
#abc.abstractmethod
def get_source_info(self):
'''This method should return a list containing a human friendly name at index 0, and a human readable url describing the source for this repository.
For example, the EarthPorn subreddit returns a list ['EarthPorn Subreddit', 'http://reddit.com/r/EarthPorn'].
This is used to populate the treeview object with your source information.'''
return
#abc.abstractmethod
def get_pluginid(self):
'''This method should return a string that represents this plugins ID.
The pluginid is used to make calls to this plugin when necessary. It should be unique as ids are in a shared pool,
so make sure the id is unique. The id should remain the same even when updated as some settings with the pluginid
are persisted by the main application, and they will be lost if the id changes.
'''
return
This is the superclass of some python plugins I wrote, which subclass this. They are dynamically loaded at runtime, and all of this works, except that even though I added a new abstract method to my SourceBase, the plugins still load. They shouldn't, since none of them have my new method. (I gave it the #abc.abstractmethod marking).
My google-fu doesn't really show anything, so I'm not sure why I can still instanstiate these plugins even though the superclass says they are abstract.
For example, in SourceBase, I added:
#abc.abstractmethod
def get_dependencies(self):
print('ERROR: THIS PLUGIN SHOULD NOT HAVE LOADED.')
'''This method should return a list of package names. The host program will check if these packages are available.
If they are not available, the plugin will be disabled, but the user will be able to elect to install these packages.'''
return
I did not define this method in my plugins, but I still get this output on the terminal:
....
Screen Height:1080
Screen Width:1920
files: ['bingIOTD.py', 'redditEP.py', 'redditLP.py', '__init__.py']
ERROR: THIS PLUGIN SHOULD NOT HAVE LOADED. <--abstract method
I'm not sure why it is ignoring it, am I missing something...? I've done it before with normal classes that aren't dynamically loaded. Any help is appreciated. I understand I can probably make a workaround (make a default return, check for that), but that doesn't seem to be the right way.
If you need more sourcecode my project is on SourceForge here.

In Python3 the metaclass is specified by
class SourceBase(metaclass=abc.ABCMeta):
not
class SourceBase(object):
__metaclass__=abc.ABCMeta
The code is ignoring the abstractmethod decorator because as far as Python3 is concerned, SourceBase is simply a standard class (instance of type) with an attribute name __metaclass__ rather than being an instance of abc.ABCMeta.

Python Plugin-System - HOWTO

I'm writing a Python application which stores some data. For storing the data i've wrote a Connection class with abstract methods (using Python's abc module). This class is the super class all storage back-ends derive from. Each storage back-end has only one purpose, e.g. storing the data in plain text files or in a XML file.
All storage backends (inclusive the module where the super class is located) are in one package called 'data_handler'. And each back-end is in one module.
My application should be able the store data in multiple back-ends simultaneously and determinate at runtime which storage back-ends are available. To do this i had the idea to write a singleton class where each back-end have to register at their import. But this seems to be not so good in a dynamic language (please correct me if I misinterpret this). Another way could be the import of the package with import data_handler and then get the __file__ attribute of the package and search all Python files in the dir for subclasses of the super Connection class.
What method should I use, or are there other (maybe better) methods to do this.
Stefan
Is discovering the back-ends at runtime a strict requirement or would static
enumeration of them in the code do?
This feature will be nice to note have to edit the code when I add a new back-end
But should your application always write to all backends?
I will have a class where I can register available handler. And the data shall be written to each registered handler. But not all available handlers have to be registered.

Do not walk the filesystem (!) and scan the Python source code of the backends! That's an ugly hack at the best of times, and even worse here because you don't need anything like it at all! Registering all the classes on import is perfectly OK.
Store the backends in a class attribute instead of an instance attribute; that way, all Storage instances will look at the same set of backends:
>>> class Storage(object):
... backends = set()
...
... def register(self, backend):
... self.backends.add(backend)
...
Every backend can register itself by instantiating its own Storage, which has access to the class-level backends attribute:
>>> foo = Storage()
>>> foo.register("text")
>>> bar = Storage()
>>> bar.register("xml")
You can read this attribute by instantiating another Storage, which will read the same variable:
>>> baz = Storage()
>>> baz.backends
{'xml', 'text'}
You could even store the backend instances in a class attribute of Connection, and register each backend upon instantiation:
>>> class Connection(object,metaclass=abc.ABCMeta):
... #abc.abstractmethod
... def register(self, backend):
... pass
...
... backends = set()
...
>>> class TextBackend(Connection):
... def register(self):
... super().backends.add(self)
...
... def __init__(self):
... self.register()
...
>>> class XMLBackend(Connection):
... def register(self):
... super().backends.add(self)
...
... def __init__(self):
... self.register()
...
>>> foo = TextBackend()
>>> bar = XMLBackend()
>>> Connection.backends
{<__main__.XMLBackend object at 0x027ADAB0>, \
<__main__.TextBackend object at 0x027ADA50>}

If these backends are going to be distributed in various Python distributions, you might want to look at setuptools/distribute entry points. Here's an article on how you might use these for dynamic plugin finding services:
http://aroberge.blogspot.com/2008/12/plugins-part-6-setuptools-based.html

But should your application always write to all backends? If not, you could use (as usual) another layer of indirection, e.g.
storage = Storage()
storage.use(TextBackend, XMLBackend, YamlBackend)
storage.write(data)
Or something like this, with Storage being a dispatcher, which would just loop over the backends and call the appropriate serializer.
This is of course, very coarse.

you could use a function like this one:
def loadClass(fullclassname):
sepindex=fullclassname.rindex('.')
classname=fullclassname[sepindex+1:]
modname=fullclassname[:sepindex]
#dynmically import the class in the module
imod=__import__(modname,None,None,classname)
classtype=getattr(imod,classname)
return classtype
where fullclassname is the fully dotted qualifiant for the class you want load.
example (pseudo code,but the idea is there):
for package availability scanning, only perform some globbing , then for finding final class name, you may declare a Plugin class in each of your modules that has a getStorage()
#scan for modules , getPluginPackagesUnder (to be defined) returns the dotted name for all packages under a root path (using some globbing, listdir or whatever method)
pluginpackages=getPluginPackagesUnder("x/y/z")
storagelist=[]
for pgpck in plunginpackages:
pluginclass=loadClass("%s.Plugin"%pgpck)
storageinstance=Plugin().getStorage()
storagelist.append(storageinstance)
so, you can dynamically scan for your existing storage plugins