I am often writing scripts with boto3 and usually when writing functions I end up passing the boto3 client for the service(s) I need around my functions. So, for example
def main():
ec2 = create_client
long_function_with_lots_of_steps(ec2, ....)
def long_function_with_lots_of_steps(client):
....
This is not too bad, but it often feels repetitive and sometimes I will need to create a new client for a different service in the other function, for which I would like to use the original aws_session object.
Is there a way to do this more elegantly? I thought to make a class holding a boto3.session.Session() object but then you end up just passing that around.
How do you usually structure boto3 scripts?
I think you might have had some C or C++ programming experience. You are definitely getting language constructs confused. In Python function call arguments are passed by reference. So passing a reference is quick. You aren't passing the whole object.
This is in fact one of the better ways to pass in session info. Why is it better you may ask? Because of testing. You will need to test the thing and you don't always want to test the connections to 3rd party services. So you can do that with Mocks.
Try making a test where you are mocking out any one of those function arguments. Go ahead... I'll wait.
Easier... right?
Since you are basically asking for an opinion:
I usually go with your second approach. I build a base class with the session object, and build off of that. When working with a large program where I must maintain some "global" state, I make a class to house those items, and that becomes a member of my base class.
class ProgramState:
def __init__(self):
self.sesson = boto3.session.Session()
class Base:
def __init__(self, state: ProgramState):
self.state = state
class Firehose(Base):
def __init__(self, state: ProgramState):
Base.__init__(self, state)
self.client = self.state.session.client("firehose")
def do_something():
pass
class S3(Base):
def __init__(self, state: ProgramState):
Base.__init__(self, state)
self.client = self.state.session.client("s3")
def do_something_else():
pass
def main():
state = ProgramState()
firehose = Firehose(state)
s3 = S3(state)
firehose.do_something()
s3.do_something_else()
Full disclosure: I dislike Python.
Related
Suppose I have a python class with a large overhead
class some_class:
def __init__(self):
self.overhead = large_overhead
# Get new data
def read_new_data(self, data):
self.new_data = data
def do_something(self):
# DO SOMETHING.
Suppose I want to have it listen to output of another program, or multiple programs, and I have a way to maintain this steady stream of inputs. How do I not initiate a new instance every time given the overhead? Do I create a new script and package the class to maintain its 'live'? And if so, how do I capture the output of the programs if they cannot be in direct communication with the script I'm running without going through a middle storage like SQL or file?
You can use a class variable:
class some_class:
overhead = large_overhead
# Get new data
def read_new_data(self, data):
self.new_data = data
def do_something(self):
# DO SOMETHING.
now overhead is only evaluated once when the class is defined, and you can use self.overhead within any class instances.
Lacking specifics... Use asyncio to setup listeners/watchers and register your object's methods as callbacks for when the data comes in - run the whole thing in an event loop.
While that was easy to say and pretty abstract, I'm sure I would have a pretty steep learning curve to implement that, especially considering I'd want to implement some testing infrastructure. But it seems pretty straightforward.
I am a beginner in Python, so please be... kind?
Anyway, I need use a static method to call another method, which requires the use of "self" (and thus, a normal method I believe). I am working with Telethon, a Python implementation of Telegram. I have tried other questions on SO, but I just can't seem to find a solution to my problem.
An overview of the program (please correct me if I'm wrong):
1) interactive_telegram_client is a child class of telegram_client, and it creates an instance.
#interactive_telegram_client.py
class InteractiveTelegramClient(TelegramClient):
super().__init__(session_user_id, api_id, api_hash, proxy)
2) When the InteractiveTelegramClient runs, it adds an update_handler self.add_update_handler(self.update_handler) to constantly check for messages received/sent, and prints it to screen
#telegram_client.py
def add_update_handler(self, handler):
"""Adds an update handler (a function which takes a TLObject,
an update, as its parameter) and listens for updates"""
if not self.sender:
raise RuntimeError(
"You should connect at least once to add update handlers.")
self.sender.add_update_handler(handler)
#interactive_telegram_client.py
#staticmethod
def update_handler(update_object):
try:
if type(update_object) is UpdateShortMessage:
if update_object.out:
print('You sent {} to user #{}'.format(update_object.message,
update_object.user_id))
else:
print('[User #{} sent {}]'.format(update_object.user_id,
update_object.message))
Now, my aim here is to send back an auto-reply message upon receiving a message. Thus, I think that adding a call to method InteractiveTelegramClient.send_ack(update_object) in the update_handler method would serve my needs.
#interactive_telegram_client.py
def send_ack(self, update_object):
entity = update_object.user_id
message = update_object.message
msg, entities = parse_message_entities(message)
msg_id = utils.generate_random_long()
self.invoke(SendMessageRequest(peer=get_input_peer(entity),
message=msg,random_id=msg_id,entities=entities,no_webpage=False))
However, as you can see, I require the self to invoke this function (based on the readme, where I assume client to refer to the same thing as self). Since the method update_handler is a static one, self is not passed through, and as such I cannot invoke the call as such.
My possible strategies which have failed include:
1) Instantiating a new client for the auto-reply
- Creating a new client/conversation for each reply...
2) Making all the methods non-static
- Involves a tremendous amount of work since other methods modified as well
3) Observer pattern (sounds like a good idea, I tried, but due to a lack of skills, not succeeded)
I was wondering if there's any other way to tackle this problem? Or perhaps it's actually easy, just that I have some misconception somewhere?
Forgot to mention that due to some restrictions on my project, I can only use Telethon, as opposed to looking at other alternatives. Adopting another library (like an existing auto-reply one) is allowed, though I did not really look into that since merging that and Telethon may be too difficult for me...
based on the readme, where I assume client to refer to the same thing as self
Correct, since the InteractiveTelegramClient subclasses the TelegramClient and hence, self is an instance of the extended client.
Instantiating a new client for the auto-reply - Creating a new client/conversation for each reply
This would require you to create another authorization and send another code request to login, because you can't work with the same *.session at the same time.
Making all the methods non-static - Involves a tremendous amount of work since other methods modified as well
It doesn't require such amount of work. Consider the following example:
class Example:
def __init__(self, a):
self.a = a
def do_something(self):
Example.other_method()
#staticmethod
def other_method():
print('hello, world!')
Is equivalent to:
class Example:
def __init__(self, a):
self.a = a
def do_something(self):
self.other_method()
#staticmethod
def other_method():
print('hello, world!')
It doesn't matter whether you use self. or the class name to refer to a static method from within the class. Since the InteractiveClientExample already uses self., all you would have to do would be changing:
#staticmethod
def update_handler(update_object):
for
def update_handler(self, update_object):
For more on the #staticmethod decorator, you can refer to the docs.
I'm a beginner-intermediate self taught Python developer,
In most of the projects I completed, I can see the following procedure repeats. I don't have any outside home code experiences, I think the below code is not so professional as it is not reusable, and seems like it is not fitting all together in a container, but loosely coupled functions on different modules.
def get_query():
# returns the query string
pass
def make_request(query):
# makes and returns the request with query
pass
def make_api_call(request):
# calls the api and returns response
pass
def process_response(response):
# process the response and returns the details
pass
def populate_database(details):
# populates the database with the details and returns the status of population
pass
def log_status(status):
# logs the status so that developer knows whats happening
pass
query = get_query()
request = make_request(query)
response = make_api_call(request)
details = process_response(response)
status = populate_database(details)
log_status(status)
How do I design this procedure as a class based design?
If I understand correctly, you want these group of functions to be reused. Good approach to this would be create Abstract base class with these methods as shown below:
from abc import ABCMeta
class Generic(object):
__metaclass__ = ABCMeta
def get_query(self):
# returns the query string
pass
def make_request(self, query):
# makes and returns the request with query
pass
def make_api_call(self, request):
# calls the api and returns response
pass
def process_response(self, response):
# process the response and returns the details
pass
def populate_database(self, details):
# populates the database with the details and returns the status of population
pass
def log_status(self, status):
# logs the status so that developer knows whats happening
pass
Now whenever you need to use any of these methods in your project, inherit your class from this abstract class.
class SampleUsage(Generic):
def process_data(self):
# In any of your methods you can call these generic functions
self.get_query()
And then you can create object to actually get results which you want.
obj = SampleUsage()
obj.process_data()
You may have several classes here. To name a few, Query, Request, Response, Database, Logger
Some of your functions may map as follows:
make_query -> Query.make() constructor or class method
make_request -> Request.make(query) constructor or class method
make_api_call -> Request.make_api_call()
process_response -> Response.process()
populate_database -> Database.populate()
log_status -> Logger.status Consider using logging module
You have to think about your application and design it as an interaction of cooperating objects. This is just a starting point in order for you to be partition the functionality of the application between the classes.
Some of these Classes may be Singletons, meaning they are instantiated only once at the beginning of the application and accessed everywhere else. Database and Logger fit that role.
Here is some skeleton definitions:
class Query(object):
#classmethod
def make(cls, *args, **kwargs):
pass
class Request(object):
#classmethod
def make(cls, query):
pass
def make_api_call(self, *args, **kwargs):
# possibly return Response
pass
class Response(object):
def process_response(self):
pass
class Database(object):
_the_db = None
#classmethod
def get_db(cls):
# Simple man's singleton
if not cls._the_db:
cls._the_db = Database()
return cls._the_db
def populate(self):
pass
class Logger(object):
def log(self):
# consider using logging module
pass
I think what lacks in your question is the sense of purpose. You don't switch a perfectly fine procedural code to object-oriented code without a reason. Depending on the reason, there are several ways to do it. As this problem is quite a common one, there are some common techniques that are known to work well for some common reasons.
So, let's assume you encapsulate the main procedure in an object. What are your needs?
Allow re-using the procedure, possibly overriding some parts? See below the template method pattern.
Allow dynamically altering the behavior of the procedure at runtime depending on external factors? Look into the Strategy pattern.
Allow dynamically altering the behavior of the procedure at runtime depending on internal factors? For example, if some request may switch the procedure into "maintenance mode"? Look into the State pattern.
I'll just describe the template method pattern, which looks the closest to Marty's concerns. I cut down the example to 3 steps so it's easier to explain, but I made you a fully working example gist.
The template method
You want to provide a way to re-use the procedure, while allowing to override some well-defined parts? Let's create an empty, fill-in-the-blanks-style template:
class BaseRequestProcesor(object):
def get_query(self):
raise NotImplementedError()
def process_query(self, query):
raise NotImplementedError()
def log_status(self, status):
raise NotImplementedError()
def process(self): # main procedure
query = self.get_query()
status = self.process_query(query)
self.log_status(status)
__call__ = process # allow "calling" the requestprocessor
We have our basic template. Let's create some template fillers:
class DemoQueryReader(object):
def get_query(self):
return 'this is a query'
class HelloQueryProcessor(object):
def process_query(self, query):
return 'Hello World, {}!'.format(query)
class StdoutLogProcessor(object):
def log_status(self, status):
print(status)
Now build a full request processor from the bits we want. This is where the pieces comes together:
class DemonstrationProcessor(DemonQueryReader, HelloQueryProcessor, StdoutLogProcessor, BaseRequestProcessor):
pass
Demonstrating in the console:
>>> from marty_example import DemonstrationProcessor
>>> processor = DemonstrationProcessor()
>>> processor()
Hello World, this is a query!
This is the most pedantic example you can build. You could supply default implementations when that makes sense (doing nothing, mostly). And you can group together overrides, should that make sense.
The point is, you made your process a template, allowing easy override of chosen details, while still being in control of the overall workflow. This is a form of inversion of control.
You can also save a Python file with the class name, or you can create external modules with some functions, organizing them into the modules depending on what they do. Some modules will only contain one function; others will contain a lot.
I'm working on a module that I'm hoping to be somewhat dynamic, in that anyone can add features relatively easily.
The basic idea is to have a class, CriticBase, which handles all criticisms for this deployment. The critics would be any class that has inherited from CriticBase.
Pseudo Example:
class CriticBase(Object) {
def self.Execute():
for critic in self.__subclasses__: critic.run()
}
class DatabaseCritic(CriticBase) { def run( //things ) }
class DiskSpaceCritic(CriticBase) { def run( //things ) }
etc...
def DoWork():
Controller = CriticBase()
a = DatabaseCritic()
b = DiskSpaceCritic()
...
Controller.Execute()
I hope that kind of makes sense. Basically the idea is to have a framework that's fairly straightforward for other devs to add to. All you need to do is define some subclass of CriticBase, and everything else is handled for you by the critic framework.
However, it's pretty ugly to me to just assign these classes to something that's never going to be used. Is there such a thing as lingering objects in Python? Could I do away with the assignment, and still have the reference to the instantiated class from the base class? Or do I have to have it assigned to something, otherwise it will be garbage collected?
My understanding is that you don't want other devs to instantiate the sub-classes. Actually, they don't need to do that, as long as the method run() is a class method:
# The framework provides this base class
class CriticBase(object):
#classmethod
def execute(cls):
for critic_class in cls.__subclasses__():
critic_class.run()
# Devs only need to provide a definition of subclass
class DatabaseCritic(CriticBase):
#classmethod
def run(cls):
# do something specific about database
class DiskSpaceCritic(CriticBase):
#classmethod
def run(cls):
# do something specific about disk space
# now the base class do all the work
def DoWork():
CriticBase.execute()
With this approach, you use python's inheritance machinery to collect the subclasses into a list, and your code is free from useless instantiations.
Well, you could achieve this using a Publish and Subscribe pattern. Roughly, it could be:
class CriticServer:
def __init__(self):
self.clients = []
def insert_client(self, client):
self.clients.append(client)
def execute(self):
for client in self.clients:
client.run()
class CriticClient:
# move what you would inherit from server to here
def __init__(self, server):
server.insert_client(self)
class DatabaseCriticClient(CriticClient):
def run(self):
pass
class DiskSpaceCriticClient(CriticClient):
def run(self):
pass
def main():
server = CriticServer()
DiskSpaceCriticClient(server)
DatabaseCriticClient(server)
server.execute()
Since I don't know too much details about your project, I'm tempt to say that would be a better idea to create a base class for clients instead of sub-classing the server.
Maybe it has not too much magic, but it works nice and it is easy to understand and to extend (which is sometimes better than pure magic).
My Situation
I'm currently writing on a project in python which I want to use to learn a bit more about software architecture. I've read a few texts and watched a couple of talks about dependency injection and learned to love how clear constructor injection shows the dependencies of an object.
However, I'm kind of struggling how to get a dependency passed to an object. I decided NOT to use a DI framework since:
I don't have enough knowledge of DI to specify my requirements and thus cannot choose a framework.
I want to keep the code free of more "magical" stuff since I have the feeling that introducing a seldom used framework drastically decreases readability. (More code to read of which only a small part is used).
Thus, I'm using custom factory functions to create objects and explicitly pass their dependencies:
# Business and Data Objects
class Foo:
def __init__(self,bar):
self.bar = bar
def do_stuff(self):
print(self.bar)
class Bar:
def __init__(self,prefix):
self.prefix = prefix
def __str__(self):
return str(self.prefix)+"Hello"
# Wiring up dependencies
def create_bar():
return Bar("Bar says: ")
def create_foo():
return Foo(create_bar())
# Starting the application
f = create_foo()
f.do_stuff()
Alternatively, if Foo has to create a number of Bars itself, it gets the creator function passed through its constructor:
# Business and Data Objects
class Foo:
def __init__(self,create_bar):
self.create_bar = create_bar
def do_stuff(self,times):
for _ in range(times):
bar = self.create_bar()
print(bar)
class Bar:
def __init__(self,greeting):
self.greeting = greeting
def __str__(self):
return self.greeting
# Wiring up dependencies
def create_bar():
return Bar("Hello World")
def create_foo():
return Foo(create_bar)
# Starting the application
f = create_foo()
f.do_stuff(3)
While I'd love to hear improvement suggestions on the code, this is not really the point of this post. However, I feel that this introduction is required to understand
My Question
While the above looks rather clear, readable and understandable to me, I run into a problem when the prefix dependency of Bar is required to be identical in the context of each Foo object and thus is coupled to the Foo object lifetime. As an example consider a prefix which implements a counter (See code examples below for implementation details).
I have two Ideas how to realize this, however, none of them seems perfect to me:
1) Pass Prefix through Foo
The first idea is to add a constructor parameter to Foo and make it store the prefix in each Foo instance.
The obvious drawback is, that it mixes up the responsibilities of Foo. It controls the business logic AND provides one of the dependencies to Bar. Once Bar does not require the dependency any more, Foo has to be modified. Seems like a no-go for me. Since I don't really think this should be a solution, I did not post the code here, but provided it on pastebin for the very interested reader ;)
2) Use Functions with State
Instead of placing the Prefix object inside Foo this approach is trying to encapsulate it inside the create_foo function. By creating one Prefix for each Foo object and referencing it in a nameless function using lambda, I keep the details (a.k.a there-is-a-prefix-object) away from Foo and inside my wiring-logic. Of course a named function would work, too (but lambda is shorter).
# Business and Data Objects
class Foo:
def __init__(self,create_bar):
self.create_bar = create_bar
def do_stuff(self,times):
for _ in range(times):
bar = self.create_bar()
print(bar)
class Bar:
def __init__(self,prefix):
self.prefix = prefix
def __str__(self):
return str(self.prefix)+"Hello"
class Prefix:
def __init__(self,name):
self.name = name
self.count = 0
def __str__(self):
self.count +=1
return self.name+" "+str(self.count)+": "
# Wiring up dependencies
def create_bar(prefix):
return Bar(prefix)
def create_prefix(name):
return Prefix(name)
def create_foo(name):
prefix = create_prefix(name)
return Foo(lambda : create_bar(prefix))
# Starting the application
f1 = create_foo("foo1")
f2 = create_foo("foo2")
f1.do_stuff(3)
f2.do_stuff(2)
f1.do_stuff(2)
This approach seems much more useful to me. However, I'm not sure about common practices and thus fear that having state inside functions is not really recommended. Coming from a java/C++ background, I'd expect a function to be dependent on its parameters, its class members (if it's a method) or some global state. Thus, a parameterless function that does not use global state would have to return exactly the same value every time it is called. This is not the case here. Once the returned object is modified (which means that counter in prefix has been increased), the function returns an object which has a different state than it had when beeing returned the first time.
Is this assumption just caused by my restricted experience in python and do I have to change my mindset, i.e. don't think of functions but of something callable? Or is supplying functions with state an unintended misuse of lambda?
3) Using a Callable Class
To overcome my doubts on stateful functions I could use callable classes where the create_foo function of approach 2 would be replaced by this:
class BarCreator:
def __init__(self, prefix):
self.prefix = prefix
def __call__(self):
return create_bar(self.prefix)
def create_foo(name):
return Foo(BarCreator(create_prefix(name)))
While this seems a usable solution for me, it is sooo much more verbose.
Summary
I'm not absolutely sure how to handle the situation. Although I prefer number 2 I still have my doubts. Furthermore, I'm still hope that anyone comes up with a more elegant way.
Please comment, if there is anything you think is too vague or can be possibly misunderstood. I will improve the question as far as my abilities allow me to do :)
All examples should run under python2.7 and python3 - if you experience any problems, please report them in the comments and I'll try to fix my code.
If you want to inject a callable object but don't want it to have a complex setup -- if, as in your example, it's really just binding to a single input value -- you could try using functools.partial to provide a function <> value pair:
def factory_function(arg):
#processing here
return configurted_object_base_on_arg
class Consumer(object):
def __init__(self, injection):
self._injected = injection
def use_injected_value():
print self._injected()
injectable = functools.partial(factory_function, 'this is the configuration argument')
example = Consumer(injectable)
example.use_injected_value() # should return the result of your factory function and argument
As an aside, if you're creating a dependency injection setup like your option 3, you probably want to put the knwledge about how to do the configuration into a factory class rather than doing it inline as you're doing here. That way you can swap out factories if you want to choose between strategies. It's not functionally very different (unless the creation is more complex than this example and involves persistent state) but it's more flexible down the road if the code looks like
factory = FooBarFactory()
bar1 = factory.create_bar()
alt_factory = FooBlahFactory(extra_info)
bar2 = alt_factory.create_bar()