I have an abstract class called IDataStream and it has one method. I create two implementations for this abstract class called IMUDataStream, GPSDataStream. In future there is a possibility that I may add another implementation of IDataStream abstract class. I have another class called DataVisualizer that visualizes all the data pulled by different DataStream classes.
In future if I add another DataStream implementation of the IDataStream abstract class, I should not be modifying the DataVisualizer class to visualize the data. Is there a way to create objects of all the derived classes of the IDataStream class, add it to a list and iterate through the list and use it to call the methods that will give me the data ?
Please note that I'm new to python and design patterns. Trying to learn. This may be a complete dumb question and total madness. I actually have a requirement for this. If this can be achieved through a design pattern I request the reader to point me to the material. Help much appreciated. Thanks !
#!/usr/bin/env python3
from abc import ABC, abstractmethod
class IDataStream(ABC):
def get_data(self):
pass
class IMUDataStream(IDataStream):
def __init__(self):
self.__text = "this is IMU data"
def get_data(self):
print(self.__text)
class GPSDataStream(IDataStream):
def __init__(self):
self.__text = "this is GPS data"
def get_data(self):
print(self.__text)
class DataVisualizer:
def __init__(self):
# somehow create objects of all derived classes of IDataStream here and call the get_data() function
# even if I add another derived class in the future. I should not be modifying the code here
What you're asking is to be able to find all instantiated objects that are in memory, and then filter them for only a particular class/subclass/parent class/whatever, take a look at this stack-overflow question regarding how to get all current objects and methods from memory.
That said... Any time you have to ask yourself how to find ALL instances of something GLOBALLY in memory, you should be stopping yourself and asking (which it seems like you did, so kudos) is there a better/easier way?
Most of the time, you'd want to make a data visualizer independent, such that it only consumes the data stream (which is specified during construction), see below:
ds = myDataStream()
vis = myDataVisualizer(ds)
vis.show() # or whatever
or
ds = myDataStream()
vis = myDataVisualizer()
vis.show(ds)
If you want your data visualizer to be data agnostic, at runtime (like have data coming from multiple sources), then you have a couple choices. Add methods for removing and adding data sources, or, you can link them together using something like the producer-consumer pattern using Queues and Processes (This is how I do it).
BUT, if you really must be managing your own memory, entirely (like through a map, or heap, or whatever). Then there are design patterns that can help you:
Factory
Abstract Factory
Decorator
Or maybe some other one, look at the catalog at refactoring.guru
First, you probably want method get_data to return data rather than print it (else it is doing its own visualization). This may do what you want. The following code will figure out all the subclassess of IDataStream, instantiate an instance of the class if it is not an abstract class, call method get_data on the instance and append the return values in a list:
#!/usr/bin/env python3
from abc import ABC, abstractmethod
class IDataStream(ABC):
#abstractmethod # you probably ment to add this
def get_data(self):
pass
class IMUDataStream(IDataStream):
def __init__(self):
self.__text = "this is IMU data"
def get_data(self):
return self.__text
class GPSDataStream(IDataStream):
def __init__(self):
self.__text = "this is GPS data"
def get_data(self):
return self.__text
def is_abstract(cls):
return bool(getattr(cls, "__abstractmethods__", False))
def get_all_non_abstract_subclasses(cls):
all_subclasses = []
for subclass in cls.__subclasses__():
if not is_abstract(subclass):
all_subclasses.append(subclass)
all_subclasses.extend(get_all_non_abstract_subclasses(subclass))
return all_subclasses
class DataVisualizer:
def __init__(self):
data = [cls().get_data() for cls in get_all_non_abstract_subclasses(IDataStream)]
print(data)
dv = DataVisualizer()
Prints:
['this is IMU data', 'this is GPS data']
Related
I am writing a python class to store data and then another class will create an instance of that class to print different variables. Some class variables require a lot of formatting which may take multiple lines of code to get it in its "final state".
Is it bad practice to just access the variables from outside the class with this structure?
class Data():
def __init__(self):
self.data = "data"
Or is it better practice to use an #property method to access variables?
class Data:
#property
def data(self):
return "data"
Be careful, if you do:
class Data:
#property
def data(self):
return "data"
d = Data()
d.data = "try to modify data"
will give you error:
AttributeError: can't set attribute
And as I see in your question, you want to be able to transform the data until its final state, so, go for the other option
class Data2():
def __init__(self):
self.data = "data"
d2 = Data2()
d2.data = "now I can be modified"
or modify the previus:
class Data:
def __init__(self):
self._data = "data"
#property
def data(self):
return self._data
#data.setter
def data(self, value):
self._data = value
d = Data()
d.data = "now I can be modified"
Common Practice
The normal practice in Python is to exposure the attributes directly. A property can be added later if additional actions are required when getting or setting.
Most of the modules in the standard library follow this practice. Public variables (not prefixed with an underscore) typically don't use property() unless there is a specific reason (such as making an attribute read-only).
Rationale
Normal attribute access (without property) is simple to implement, simple to understand, and runs very fast.
The possibility of use property() afterwards means that we don't have to practice defensive programming. We can avoid having to prematurely implement getters and setters which bloats the code and makes accesses slower.
Basically you could hide lot of complexity in the property and make it look like an attribute. This increases code readability.
Also, you need to understand the difference between property and attribute.
Please refer What's the difference between a Python "property" and "attribute"?
Is it possible to modify/extend an inherited method from the middle. I realize I can call super and get the original method, then either put code before or after that call which will extend the original. Is there a technique of doing something similar but from the middle of a method?
class Base():
def __init__(self):
self.size = 4
def get_data(self):
data = []
for num in range(self.size):
data.append("doing stuff")
data.append("doing stuff")
### add here from child##
data.append("doing stuff")
data.append("doing stuff")
return data
class MyClass(Base):
def __init__(self):
super().__init__()
def get_data(self):
# inherited parent code
# Do something else here
# inherited parent code
Despite Python's powerful introspection and code-modifying capabilities, there is no "clean" way of doing this. It could be done only by directly modifying the bytecode in the original function and shoehorsing a new method call in there - which would also implying in creating a new code and function objects - definitely not something to do in production code - even because bytecode is not guaranteed to be unchanged across Python versions or Python implementations.
Refactoring the original method:
But it can be done if the original method is coded in a way it is "aware" of points were subclasses might want to run additional code (maybe even being split up in several methods):
For your example, you'd have something like:
class Base():
def __init__(self):
self.size = 4
def get_data(self):
self.data = data = []
for num in range(self.size):
data.append("doing stuff")
data.append("doing stuff")
self.do_extra_things_with_data()
data.append("doing stuff")
data.append("doing stuff")
return data
def do_extra_things_with_data():
"""Override this on subclasses"""
class MyClass(Base):
def __init__(self):
super().__init__()
def do_extra_things_with_data():
print(len(self.data), "objects defined so far")
One technical name for this is "slot". (It is used for templating in certain web frameworks - the derived page uses the parent template for columns and general layout, and defines "slots" for the content areas)
One other thing to watch are descriptors such as "properties": you can't change the superclass'method code - but if the code retrieves instance attributes for its computations, you can define these attributes as properties on the subclasses to run custom code.
Using descriptors:
One other way of doing that is to use descriptors such as "properties": you can't change the superclass'method code - but if the code retrieves instance attributes for its computations, you can define these attributes as properties on the subclasses to run custom code.
Let's suppose your method makes use of the self.size attribute, but it is exactly for calculating it that you might want to run more code - keeping exactly the same Base class you can do:
class MyClass(Base):
#property
def size(self):
# put extr calculation to retrieve the dynamc value
of self.size here
return value
Is there a technique of doing something similar but from the middle of
a method?
Not really. The def compiles into a function object that has a self-contained code object that is usually treated as being opaque.
When a need like this arises, it is usually an indication that the parent method needs to be split into reusable components that can be called separately.
If you can't refactor the parent method, then the unfortunate alternative is that the subclass will have to override the method and duplicate some of the code from the parent.
In short, Pythonic object oriented design is treats methods and attributes as the atomic units of composability.
I have a class that looks something like the following:
# Class violates the Single Responsibility Principle
class Baz:
data = [42]
def do_foo_to_data(self):
# call a dozen functions that do complicated stuff to data
def do_bar_to_data(self):
# call other functions that do different stuff to data
I want to break it into two separate classes because it violates the SRP. The functions called by do_foo_to_data() are completely distinct from those called by do_bar_to_data(). Yet they must operate on the same data.
I've come up with a bunch of solutions, but they're all ugly. Is there a way to do this cleanly, preferably in Python 3 (though 2.7 is OK too)?
The best of my "solutions" is below:
# I find this hard to read and understand
class Baz:
data = [42]
def create_foo(self):
return Baz.Foo()
def create_bar(self):
return Baz.Bar()
class Foo:
def do_foo_to_data(self):
# call foo functions
class Bar:
def do_bar_to_data(self):
# call bar functions
Note: It's not essential to me that the data member be a class member.
I only expect to create one instance of Baz; but I didn't want to ask two questions in one post and start a discussion about singletons.
This is not an elegant solution. You better pass a reference to the object you want them to operate on. So something like:
class Foo:
def __init__(self,data):
self.data = data
def do_foo_to_data(self):
#...
self.data[0] = 14
pass
class Bar:
def __init__(self,data):
self.data = data
def do_bar_to_data(self):
#...
self.data.append(15)
pass
(I added sample manipulations like self.data[0] = 14 and self.data.append(15))
And now you construct the data. For instance:
data = [42]
Next you construct a Foo and a Bar and pass a reference to data like:
foo = Foo(data)
bar = Bar(data)
__init__ is what most programming languages call the constructor and as you have seen in the first fragment, it requires an additional parameter data (in this case it is a reference to our constructed data).
and then you can for instance call:
foo.do_foo_to_data()
which will set data to [14] and
bar.do_bar_to_data()
which will result in data being equal to [14,15].
Mind that you cannot state self.data = ['a','new','list'] or something equivalent in do_foo_to_data or do_bar_to_data because this would change the reference to a new object. Instead you could for instance .clear() the list, and append new elements to it like:
def do_foo_to_data(self): #alternative version
#...
self.data.clear()
self.data.append('a')
self.data.append('new')
self.data.append('list')
Finally to answer your remark:
preferably in Python 3 (though 2.7 is OK too)?
The technique demonstrated is almost universal (meaning it is available in nearly every programming language). So this will work in both python-3.x and python-2.7.
Why do you even need a class for that? All you want is two separated functions which do some job on some data.
data = [42]
def foo(data):
data.append('sample operation foo')
def bar(data):
data.append('sample operation bar')
Problem solved.
You can pull out the distinct groups of functionality to separate mix-in classes:
class Foo:
"""Mixin class.
Requires self.data (must be provided by classes extending this class).
"""
def do_foo_to_data(self):
# call a dozen functions that do complicated stuff to data
class Bar:
"""Mixin class.
Requires self.data (must be provided by classes extending this class).
"""
def do_bar_to_data(self):
# call other functions that do different stuff to data
class Baz(Foo, Baz):
data = [42]
This relies on Python's duck-typing behavior. You should only apply the Foo and Bar mix-ins to classes that actually provide self.data, like the Baz class here does.
This might be suitable where certain classes are by convention required to provide certain attributes anyway, such as customized view classes in Django. However, when such conventions aren't already in place, you might not want to introduce new ones. It's too easy to miss the documentation and then have NameErrors at runtime. So let's make the dependency explicit, rather than only documenting it. How? With a mix-in for the mix-ins!
class Data:
"""Mixin class"""
data = [42]
class Foo(Data):
"""Mixin class"""
def do_foo_to_data(self):
# call a dozen functions that do complicated stuff to data
class Bar(Data):
"""Mixin class"""
def do_bar_to_data(self):
# call other functions that do different stuff to data
class Baz(Foo, Baz):
pass
Whether this is appropriate for your use-case is difficult to say at this level of abstraction. As RayLuo's answer shows, you might not need classes at all. Instead, you could put the different groups of functions into different modules or packages, to organize them.
I'm a beginner-intermediate self taught Python developer,
In most of the projects I completed, I can see the following procedure repeats. I don't have any outside home code experiences, I think the below code is not so professional as it is not reusable, and seems like it is not fitting all together in a container, but loosely coupled functions on different modules.
def get_query():
# returns the query string
pass
def make_request(query):
# makes and returns the request with query
pass
def make_api_call(request):
# calls the api and returns response
pass
def process_response(response):
# process the response and returns the details
pass
def populate_database(details):
# populates the database with the details and returns the status of population
pass
def log_status(status):
# logs the status so that developer knows whats happening
pass
query = get_query()
request = make_request(query)
response = make_api_call(request)
details = process_response(response)
status = populate_database(details)
log_status(status)
How do I design this procedure as a class based design?
If I understand correctly, you want these group of functions to be reused. Good approach to this would be create Abstract base class with these methods as shown below:
from abc import ABCMeta
class Generic(object):
__metaclass__ = ABCMeta
def get_query(self):
# returns the query string
pass
def make_request(self, query):
# makes and returns the request with query
pass
def make_api_call(self, request):
# calls the api and returns response
pass
def process_response(self, response):
# process the response and returns the details
pass
def populate_database(self, details):
# populates the database with the details and returns the status of population
pass
def log_status(self, status):
# logs the status so that developer knows whats happening
pass
Now whenever you need to use any of these methods in your project, inherit your class from this abstract class.
class SampleUsage(Generic):
def process_data(self):
# In any of your methods you can call these generic functions
self.get_query()
And then you can create object to actually get results which you want.
obj = SampleUsage()
obj.process_data()
You may have several classes here. To name a few, Query, Request, Response, Database, Logger
Some of your functions may map as follows:
make_query -> Query.make() constructor or class method
make_request -> Request.make(query) constructor or class method
make_api_call -> Request.make_api_call()
process_response -> Response.process()
populate_database -> Database.populate()
log_status -> Logger.status Consider using logging module
You have to think about your application and design it as an interaction of cooperating objects. This is just a starting point in order for you to be partition the functionality of the application between the classes.
Some of these Classes may be Singletons, meaning they are instantiated only once at the beginning of the application and accessed everywhere else. Database and Logger fit that role.
Here is some skeleton definitions:
class Query(object):
#classmethod
def make(cls, *args, **kwargs):
pass
class Request(object):
#classmethod
def make(cls, query):
pass
def make_api_call(self, *args, **kwargs):
# possibly return Response
pass
class Response(object):
def process_response(self):
pass
class Database(object):
_the_db = None
#classmethod
def get_db(cls):
# Simple man's singleton
if not cls._the_db:
cls._the_db = Database()
return cls._the_db
def populate(self):
pass
class Logger(object):
def log(self):
# consider using logging module
pass
I think what lacks in your question is the sense of purpose. You don't switch a perfectly fine procedural code to object-oriented code without a reason. Depending on the reason, there are several ways to do it. As this problem is quite a common one, there are some common techniques that are known to work well for some common reasons.
So, let's assume you encapsulate the main procedure in an object. What are your needs?
Allow re-using the procedure, possibly overriding some parts? See below the template method pattern.
Allow dynamically altering the behavior of the procedure at runtime depending on external factors? Look into the Strategy pattern.
Allow dynamically altering the behavior of the procedure at runtime depending on internal factors? For example, if some request may switch the procedure into "maintenance mode"? Look into the State pattern.
I'll just describe the template method pattern, which looks the closest to Marty's concerns. I cut down the example to 3 steps so it's easier to explain, but I made you a fully working example gist.
The template method
You want to provide a way to re-use the procedure, while allowing to override some well-defined parts? Let's create an empty, fill-in-the-blanks-style template:
class BaseRequestProcesor(object):
def get_query(self):
raise NotImplementedError()
def process_query(self, query):
raise NotImplementedError()
def log_status(self, status):
raise NotImplementedError()
def process(self): # main procedure
query = self.get_query()
status = self.process_query(query)
self.log_status(status)
__call__ = process # allow "calling" the requestprocessor
We have our basic template. Let's create some template fillers:
class DemoQueryReader(object):
def get_query(self):
return 'this is a query'
class HelloQueryProcessor(object):
def process_query(self, query):
return 'Hello World, {}!'.format(query)
class StdoutLogProcessor(object):
def log_status(self, status):
print(status)
Now build a full request processor from the bits we want. This is where the pieces comes together:
class DemonstrationProcessor(DemonQueryReader, HelloQueryProcessor, StdoutLogProcessor, BaseRequestProcessor):
pass
Demonstrating in the console:
>>> from marty_example import DemonstrationProcessor
>>> processor = DemonstrationProcessor()
>>> processor()
Hello World, this is a query!
This is the most pedantic example you can build. You could supply default implementations when that makes sense (doing nothing, mostly). And you can group together overrides, should that make sense.
The point is, you made your process a template, allowing easy override of chosen details, while still being in control of the overall workflow. This is a form of inversion of control.
You can also save a Python file with the class name, or you can create external modules with some functions, organizing them into the modules depending on what they do. Some modules will only contain one function; others will contain a lot.
With a class in Python, how do I define a function to print every single instance of the class in a format defined in the function?
I see two options in this case:
Garbage collector
import gc
for obj in gc.get_objects():
if isinstance(obj, some_class):
dome_something(obj)
This has the disadvantage of being very slow when you have a lot of objects, but works with types over which you have no control.
Use a mixin and weakrefs
from collections import defaultdict
import weakref
class KeepRefs(object):
__refs__ = defaultdict(list)
def __init__(self):
self.__refs__[self.__class__].append(weakref.ref(self))
#classmethod
def get_instances(cls):
for inst_ref in cls.__refs__[cls]:
inst = inst_ref()
if inst is not None:
yield inst
class X(KeepRefs):
def __init__(self, name):
super(X, self).__init__()
self.name = name
x = X("x")
y = X("y")
for r in X.get_instances():
print r.name
del y
for r in X.get_instances():
print r.name
In this case, all the references get stored as a weak reference in a list. If you create and delete a lot of instances frequently, you should clean up the list of weakrefs after iteration, otherwise there's going to be a lot of cruft.
Another problem in this case is that you have to make sure to call the base class constructor. You could also override __new__, but only the __new__ method of the first base class is used on instantiation. This also works only on types that are under your control.
Edit: The method for printing all instances according to a specific format is left as an exercise, but it's basically just a variation on the for-loops.
You'll want to create a static list on your class, and add a weakref to each instance so the garbage collector can clean up your instances when they're no longer needed.
import weakref
class A:
instances = []
def __init__(self, name=None):
self.__class__.instances.append(weakref.proxy(self))
self.name = name
a1 = A('a1')
a2 = A('a2')
a3 = A('a3')
a4 = A('a4')
for instance in A.instances:
print(instance.name)
You don't need to import ANYTHING! Just use "self". Here's how you do this
class A:
instances = []
def __init__(self):
self.__class__.instances.append(self)
print('\n'.join(A.instances)) #this line was suggested by #anvelascos
It's this simple. No modules or libraries imported
Very nice and useful code, but it has a big problem: list is always bigger and it is never cleaned-up, to test it just add print(len(cls.__refs__[cls])) at the end of the get_instances method.
Here a fix for the get_instances method:
__refs__ = defaultdict(list)
#classmethod
def get_instances(cls):
refs = []
for ref in cls.__refs__[cls]:
instance = ref()
if instance is not None:
refs.append(ref)
yield instance
# print(len(refs))
cls.__refs__[cls] = refs
or alternatively it could be done using WeakSet:
from weakref import WeakSet
__refs__ = defaultdict(WeakSet)
#classmethod
def get_instances(cls):
return cls.__refs__[cls]
Same as almost all other OO languages, keep all instances of the class in a collection of some kind.
You can try this kind of thing.
class MyClassFactory( object ):
theWholeList= []
def __call__( self, *args, **kw ):
x= MyClass( *args, **kw )
self.theWholeList.append( x )
return x
Now you can do this.
object= MyClassFactory( args, ... )
print MyClassFactory.theWholeList
Python doesn't have an equivalent to Smallktalk's #allInstances as the architecture doesn't have this type of central object table (although modern smalltalks don't really work like that either).
As the other poster says, you have to explicitly manage a collection. His suggestion of a factory method that maintains a registry is a perfectly reasonable way to do it. You may wish to do something with weak references so you don't have to explicitly keep track of object disposal.
It's not clear if you need to print all class instances at once or when they're initialized, nor if you're talking about a class you have control over vs a class in a 3rd party library.
In any case, I would solve this by writing a class factory using Python metaclass support. If you don't have control over the class, manually update the __metaclass__ for the class or module you're tracking.
See http://www.onlamp.com/pub/a/python/2003/04/17/metaclasses.html for more information.
In my project, I faced a similar problem and found a simple solution that may also work for you in listing and printing your class instances. The solution worked smoothly in Python version 3.7; gave partial errors in Python version 3.5.
I will copy-paste the relevant code blocks from my recent project.
```
instances = []
class WorkCalendar:
def __init__(self, day, patient, worker):
self.day = day
self.patient = patient
self.worker= worker
def __str__(self):
return f'{self.day} : {self.patient} : {self.worker}'
In Python the __str__ method in the end, determines how the object will be interpreted in its string form. I added the : in between the curly brackets, they are completely my preference for a "Pandas DataFrame" kind of reading. If you apply this small __str__ function, you will not be seeing some machine-readable object type descriptions- which makes no sense for human eyes. After adding this __str__ function you can append your objects to your list and print them as you wish.
appointment= WorkCalendar("01.10.2020", "Jane", "John")
instances.append(appointment)
For printing, your format in __str__ will work as default. But it is also possible to call all attributes separately:
for instance in instances:
print(instance)
print(instance.worker)
print(instance.patient)
For detailed reading, you may look at the source: https://dbader.org/blog/python-repr-vs-str