I'm trying to understand how to make RPC calls using Python. I have a stupid server that defines a class and exposes a method that create instances of that class:
# server.py
class Greeter(object):
def __init__(self, name):
self.name = name
def greet(self):
return "Hi {}!".format(self.name)
def greeter_factory(name):
return Greeter(name)
some_RPC_framework.register(greeter_factory)
and a client that wants to get an instance of the Greeter:
# client.py
greeter_factory = some_RPC_framework.proxy(uri_given_by_server)
h = greeter_factory("Heisemberg")
print("Server returned:", h.greet())
The problem is that I've found no framework that allows to return instances of user-defined objects, or that only returns a dict with the attributes of the object (for example, Pyro4).
In the past I've used Java RMI, where you can specify a codebase on the server where the client can download the compiled classes, if it needs to. Is there something like this for Python? Maybe some framework that can serialize objects along with the class bytecode to let the client have a full-working instance of the class?
Pyro can do this to a certain extent. You can register custom class (de)serializers when using the default serializer. Or you can decide to use the pickle serializer, but that has severe security implications. See http://pythonhosted.org/Pyro4/clientcode.html#serialization
What Pyro won't do for you, even when using the pickle serializer, is transfer the actual bytecode that makes up the module definition. The client, in your case, has to be able to import the module defining your classes in the regular way. There's no code transportation.
You can consider using
payload = CPickle.dump(Greeter(name))
on server side and on client side once the payload is received do -
h = CPickle.load(payload)
to get the instance of Greeter object that server has created.
Related
I am often writing scripts with boto3 and usually when writing functions I end up passing the boto3 client for the service(s) I need around my functions. So, for example
def main():
ec2 = create_client
long_function_with_lots_of_steps(ec2, ....)
def long_function_with_lots_of_steps(client):
....
This is not too bad, but it often feels repetitive and sometimes I will need to create a new client for a different service in the other function, for which I would like to use the original aws_session object.
Is there a way to do this more elegantly? I thought to make a class holding a boto3.session.Session() object but then you end up just passing that around.
How do you usually structure boto3 scripts?
I think you might have had some C or C++ programming experience. You are definitely getting language constructs confused. In Python function call arguments are passed by reference. So passing a reference is quick. You aren't passing the whole object.
This is in fact one of the better ways to pass in session info. Why is it better you may ask? Because of testing. You will need to test the thing and you don't always want to test the connections to 3rd party services. So you can do that with Mocks.
Try making a test where you are mocking out any one of those function arguments. Go ahead... I'll wait.
Easier... right?
Since you are basically asking for an opinion:
I usually go with your second approach. I build a base class with the session object, and build off of that. When working with a large program where I must maintain some "global" state, I make a class to house those items, and that becomes a member of my base class.
class ProgramState:
def __init__(self):
self.sesson = boto3.session.Session()
class Base:
def __init__(self, state: ProgramState):
self.state = state
class Firehose(Base):
def __init__(self, state: ProgramState):
Base.__init__(self, state)
self.client = self.state.session.client("firehose")
def do_something():
pass
class S3(Base):
def __init__(self, state: ProgramState):
Base.__init__(self, state)
self.client = self.state.session.client("s3")
def do_something_else():
pass
def main():
state = ProgramState()
firehose = Firehose(state)
s3 = S3(state)
firehose.do_something()
s3.do_something_else()
Full disclosure: I dislike Python.
I am a beginner in Python, so please be... kind?
Anyway, I need use a static method to call another method, which requires the use of "self" (and thus, a normal method I believe). I am working with Telethon, a Python implementation of Telegram. I have tried other questions on SO, but I just can't seem to find a solution to my problem.
An overview of the program (please correct me if I'm wrong):
1) interactive_telegram_client is a child class of telegram_client, and it creates an instance.
#interactive_telegram_client.py
class InteractiveTelegramClient(TelegramClient):
super().__init__(session_user_id, api_id, api_hash, proxy)
2) When the InteractiveTelegramClient runs, it adds an update_handler self.add_update_handler(self.update_handler) to constantly check for messages received/sent, and prints it to screen
#telegram_client.py
def add_update_handler(self, handler):
"""Adds an update handler (a function which takes a TLObject,
an update, as its parameter) and listens for updates"""
if not self.sender:
raise RuntimeError(
"You should connect at least once to add update handlers.")
self.sender.add_update_handler(handler)
#interactive_telegram_client.py
#staticmethod
def update_handler(update_object):
try:
if type(update_object) is UpdateShortMessage:
if update_object.out:
print('You sent {} to user #{}'.format(update_object.message,
update_object.user_id))
else:
print('[User #{} sent {}]'.format(update_object.user_id,
update_object.message))
Now, my aim here is to send back an auto-reply message upon receiving a message. Thus, I think that adding a call to method InteractiveTelegramClient.send_ack(update_object) in the update_handler method would serve my needs.
#interactive_telegram_client.py
def send_ack(self, update_object):
entity = update_object.user_id
message = update_object.message
msg, entities = parse_message_entities(message)
msg_id = utils.generate_random_long()
self.invoke(SendMessageRequest(peer=get_input_peer(entity),
message=msg,random_id=msg_id,entities=entities,no_webpage=False))
However, as you can see, I require the self to invoke this function (based on the readme, where I assume client to refer to the same thing as self). Since the method update_handler is a static one, self is not passed through, and as such I cannot invoke the call as such.
My possible strategies which have failed include:
1) Instantiating a new client for the auto-reply
- Creating a new client/conversation for each reply...
2) Making all the methods non-static
- Involves a tremendous amount of work since other methods modified as well
3) Observer pattern (sounds like a good idea, I tried, but due to a lack of skills, not succeeded)
I was wondering if there's any other way to tackle this problem? Or perhaps it's actually easy, just that I have some misconception somewhere?
Forgot to mention that due to some restrictions on my project, I can only use Telethon, as opposed to looking at other alternatives. Adopting another library (like an existing auto-reply one) is allowed, though I did not really look into that since merging that and Telethon may be too difficult for me...
based on the readme, where I assume client to refer to the same thing as self
Correct, since the InteractiveTelegramClient subclasses the TelegramClient and hence, self is an instance of the extended client.
Instantiating a new client for the auto-reply - Creating a new client/conversation for each reply
This would require you to create another authorization and send another code request to login, because you can't work with the same *.session at the same time.
Making all the methods non-static - Involves a tremendous amount of work since other methods modified as well
It doesn't require such amount of work. Consider the following example:
class Example:
def __init__(self, a):
self.a = a
def do_something(self):
Example.other_method()
#staticmethod
def other_method():
print('hello, world!')
Is equivalent to:
class Example:
def __init__(self, a):
self.a = a
def do_something(self):
self.other_method()
#staticmethod
def other_method():
print('hello, world!')
It doesn't matter whether you use self. or the class name to refer to a static method from within the class. Since the InteractiveClientExample already uses self., all you would have to do would be changing:
#staticmethod
def update_handler(update_object):
for
def update_handler(self, update_object):
For more on the #staticmethod decorator, you can refer to the docs.
Introduction
For Java, Dependency Injection works as pure OOP, i.e. you provide an interface to be implemented and in your framework code accept an instance of a class that implements the defined interface.
Now for Python, you are able to do the same way, but I think that method was too much overhead right in case of Python. So then how would you implement it in the Pythonic way?
Use Case
Say this is the framework code:
class FrameworkClass():
def __init__(self, ...):
...
def do_the_job(self, ...):
# some stuff
# depending on some external function
The Basic Approach
The most naive (and maybe the best?) way is to require the external function to be supplied into the FrameworkClass constructor, and then be invoked from the do_the_job method.
Framework Code:
class FrameworkClass():
def __init__(self, func):
self.func = func
def do_the_job(self, ...):
# some stuff
self.func(...)
Client Code:
def my_func():
# my implementation
framework_instance = FrameworkClass(my_func)
framework_instance.do_the_job(...)
Question
The question is short. Is there any better commonly used Pythonic way to do this? Or maybe any libraries supporting such functionality?
UPDATE: Concrete Situation
Imagine I develop a micro web framework, which handles authentication using tokens. This framework needs a function to supply some ID obtained from the token and get the user corresponding to that ID.
Obviously, the framework does not know anything about users or any other application specific logic, so the client code must inject the user getter functionality into the framework to make the authentication work.
See Raymond Hettinger - Super considered super! - PyCon 2015 for an argument about how to use super and multiple inheritance instead of DI. If you don't have time to watch the whole video, jump to minute 15 (but I'd recommend watching all of it).
Here is an example of how to apply what's described in this video to your example:
Framework Code:
class TokenInterface():
def getUserFromToken(self, token):
raise NotImplementedError
class FrameworkClass(TokenInterface):
def do_the_job(self, ...):
# some stuff
self.user = super().getUserFromToken(...)
Client Code:
class SQLUserFromToken(TokenInterface):
def getUserFromToken(self, token):
# load the user from the database
return user
class ClientFrameworkClass(FrameworkClass, SQLUserFromToken):
pass
framework_instance = ClientFrameworkClass()
framework_instance.do_the_job(...)
This will work because the Python MRO will guarantee that the getUserFromToken client method is called (if super() is used). The code will have to change if you're on Python 2.x.
One added benefit here is that this will raise an exception if the client does not provide a implementation.
Of course, this is not really dependency injection, it's multiple inheritance and mixins, but it is a Pythonic way to solve your problem.
The way we do dependency injection in our project is by using the inject lib. Check out the documentation. I highly recommend using it for DI. It kinda makes no sense with just one function but starts making lots of sense when you have to manage multiple data sources etc, etc.
Following your example it could be something similar to:
# framework.py
class FrameworkClass():
def __init__(self, func):
self.func = func
def do_the_job(self):
# some stuff
self.func()
Your custom function:
# my_stuff.py
def my_func():
print('aww yiss')
Somewhere in the application you want to create a bootstrap file that keeps track of all the defined dependencies:
# bootstrap.py
import inject
from .my_stuff import my_func
def configure_injection(binder):
binder.bind(FrameworkClass, FrameworkClass(my_func))
inject.configure(configure_injection)
And then you could consume the code this way:
# some_module.py (has to be loaded with bootstrap.py already loaded somewhere in your app)
import inject
from .framework import FrameworkClass
framework_instance = inject.instance(FrameworkClass)
framework_instance.do_the_job()
I'm afraid this is as pythonic as it can get (the module has some python sweetness like decorators to inject by parameter etc - check the docs), as python does not have fancy stuff like interfaces or type hinting.
So to answer your question directly would be very hard. I think the true question is: does python have some native support for DI? And the answer is, sadly: no.
Some time ago I wrote dependency injection microframework with a ambition to make it Pythonic - Dependency Injector. That's how your code can look like in case of its usage:
"""Example of dependency injection in Python."""
import logging
import sqlite3
import boto.s3.connection
import example.main
import example.services
import dependency_injector.containers as containers
import dependency_injector.providers as providers
class Platform(containers.DeclarativeContainer):
"""IoC container of platform service providers."""
logger = providers.Singleton(logging.Logger, name='example')
database = providers.Singleton(sqlite3.connect, ':memory:')
s3 = providers.Singleton(boto.s3.connection.S3Connection,
aws_access_key_id='KEY',
aws_secret_access_key='SECRET')
class Services(containers.DeclarativeContainer):
"""IoC container of business service providers."""
users = providers.Factory(example.services.UsersService,
logger=Platform.logger,
db=Platform.database)
auth = providers.Factory(example.services.AuthService,
logger=Platform.logger,
db=Platform.database,
token_ttl=3600)
photos = providers.Factory(example.services.PhotosService,
logger=Platform.logger,
db=Platform.database,
s3=Platform.s3)
class Application(containers.DeclarativeContainer):
"""IoC container of application component providers."""
main = providers.Callable(example.main.main,
users_service=Services.users,
auth_service=Services.auth,
photos_service=Services.photos)
Here is a link to more extensive description of this example - http://python-dependency-injector.ets-labs.org/examples/services_miniapp.html
Hope it can help a bit. For more information please visit:
GitHub https://github.com/ets-labs/python-dependency-injector
Docs http://python-dependency-injector.ets-labs.org/
Dependency injection is a simple technique that Python supports directly. No additional libraries are required. Using type hints can improve clarity and readability.
Framework Code:
class UserStore():
"""
The base class for accessing a user's information.
The client must extend this class and implement its methods.
"""
def get_name(self, token):
raise NotImplementedError
class WebFramework():
def __init__(self, user_store: UserStore):
self.user_store = user_store
def greet_user(self, token):
user_name = self.user_store.get_name(token)
print(f'Good day to you, {user_name}!')
Client Code:
class AlwaysMaryUser(UserStore):
def get_name(self, token):
return 'Mary'
class SQLUserStore(UserStore):
def __init__(self, db_params):
self.db_params = db_params
def get_name(self, token):
# TODO: Implement the database lookup
raise NotImplementedError
client = WebFramework(AlwaysMaryUser())
client.greet_user('user_token')
The UserStore class and type hinting are not required for implementing dependency injection. Their primary purpose is to provide guidance to the client developer. If you remove the UserStore class and all references to it, the code still works.
After playing around with some of the DI frameworks in python, I've found they have felt a bit clunky to use when comparing how simple it is in other realms such as with .NET Core. This is mostly due to the joining via things like decorators that clutter the code and make it hard to simply add it into or remove it from a project, or joining based on variable names.
I've recently been working on a dependency injection framework that instead uses typing annotations to do the injection called Simple-Injection. Below is a simple example
from simple_injection import ServiceCollection
class Dependency:
def hello(self):
print("Hello from Dependency!")
class Service:
def __init__(self, dependency: Dependency):
self._dependency = dependency
def hello(self):
self._dependency.hello()
collection = ServiceCollection()
collection.add_transient(Dependency)
collection.add_transient(Service)
collection.resolve(Service).hello()
# Outputs: Hello from Dependency!
This library supports service lifetimes and binding services to implementations.
One of the goals of this library is that it is also easy to add it to an existing application and see how you like it before committing to it as all it requires is your application to have appropriate typings, and then you build the dependency graph at the entry point and run it.
Hope this helps. For more information, please see
github: https://github.com/BradLewis/simple-injection
docs: https://simple-injection.readthedocs.io/en/latest/
pypi: https://pypi.org/project/simple-injection/
A very easy and Pythonic way to do dependency injection is importlib.
You could define a small utility function
def inject_method_from_module(modulename, methodname):
"""
injects dynamically a method in a module
"""
mod = importlib.import_module(modulename)
return getattr(mod, methodname, None)
And then you can use it:
myfunction = inject_method_from_module("mypackage.mymodule", "myfunction")
myfunction("a")
In mypackage/mymodule.py you define myfunction
def myfunction(s):
print("myfunction in mypackage.mymodule called with parameter:", s)
You could of course also use a class MyClass iso. the function myfunction. If you define the values of methodname in a settings.py file you can load different versions of the methodname depending on the value of the settings file. Django is using such a scheme to define its database connection.
I think that DI and possibly AOP are not generally considered Pythonic because of typical Python developers preferences, rather that language features.
As a matter of fact you can implement a basic DI framework in <100 lines, using metaclasses and class decorators.
For a less invasive solution, these constructs can be used to plug-in custom implementations into a generic framework.
There is also Pinject, an open source python dependency injector by Google.
Here is an example
>>> class OuterClass(object):
... def __init__(self, inner_class):
... self.inner_class = inner_class
...
>>> class InnerClass(object):
... def __init__(self):
... self.forty_two = 42
...
>>> obj_graph = pinject.new_object_graph()
>>> outer_class = obj_graph.provide(OuterClass)
>>> print outer_class.inner_class.forty_two
42
And here is the source code
Due to Python OOP implementation, IoC and dependency injection are not standard practices in the Python world. But the approach seems promising even for Python.
To use dependencies as arguments is a non-pythonic approach. Python is an OOP language with beautiful and elegant OOP model, that provides more straightforward ways to maintain dependencies.
To define classes full of abstract methods just to imitate interface type is weird too.
Huge wrapper-on-wrapper workarounds create code overhead.
I also don't like to use libraries when all I need is a small pattern.
So my solution is:
# Framework internal
def MetaIoC(name, bases, namespace):
cls = type("IoC{}".format(name), tuple(), namespace)
return type(name, bases + (cls,), {})
# Entities level
class Entity:
def _lower_level_meth(self):
raise NotImplementedError
#property
def entity_prop(self):
return super(Entity, self)._lower_level_meth()
# Adapters level
class ImplementedEntity(Entity, metaclass=MetaIoC):
__private = 'private attribute value'
def __init__(self, pub_attr):
self.pub_attr = pub_attr
def _lower_level_meth(self):
print('{}\n{}'.format(self.pub_attr, self.__private))
# Infrastructure level
if __name__ == '__main__':
ENTITY = ImplementedEntity('public attribute value')
ENTITY.entity_prop
EDIT:
Be careful with the pattern. I used it in a real project and it showed itself a not that good way. My post on Medium about my experience with the pattern.
I have following class in Python:
class SDK(object):
URL = 'http://example.com'
def checkUrl(self, url):
#some code
class api:
def innerMethod(self, url):
data = self.checkUrl(url)
#rest of code
but when I try to access checkUrl from api, I get error. I try to call nested method by:
sdk = SDK()
sdk.api.innerMethod('http://stackoverflow.com')
Is there any simple way to call inner class methods, or (if not) structurize methods into inner objects? Any suggestion will be appreciated.
Edit:
class code:
class SDK(object):
def run(self, method, *param):
pass
class api:
def checkDomain(self, domain):
json = self.run('check', domain)
return json
run code:
sdk = SDK()
result = sdk.api().checkDomain('stackoverflow.com')
The SDK class is not a parent of the api class in your example, i.e. api does not inherit from SDK, they are merely nested.
Therefore the self object in your api.innerMethod method is only an instance of the api class and doesn't provide access to methods of the SDK class.
I strongly recommend getting more knowledgeable about object-oriented programming concepts and grasp what the issue is here. It will help you tremendously.
As for using modules to achieve something along these lines, you can, for example, pull everything from the SDK class to sdk.py file, which would be the sdk module.
sdk.py:
URL = 'http://example.com'
def checkUrl(url):
#some code
class api:
def innerMethod(self, url):
data = checkUrl(url)
#rest of code
main.py:
import sdk
api = sdk.api()
api.innerMethod('http://stackoverflow.com')
Or you may go even further and transform sdk to a package with api being a module inside it.
See https://docs.python.org/2/tutorial/modules.html for details on how to use modules and packages.
If you want a method to act as a classmethod, you have to tell python about it:
class SDK:
class api:
#classmethod
def foo(cls):
return 1
Then you have access like
SDK.api.foo()
Depending on what you're trying to do, this smells kind of un-pythonic. If it's just the namespace you care about, you'd typically use a module.
I have an object that takes a parameter in the constructor. I was wondering how I can serve this from Pyro4. An Example:
import Pyro4
class MyPyroThing(object):
def __init__(self, theNumber):
self.Number = theNumber
Pyro4.Daemon.serveSimple(
{
MyPyroThing(): None
},
ns=True, verbose=True)
This fails of course because the constructor must have a parameter.
And when this is solved, how do you invoke such object?
theThing = Pyro4.Proxy("PYRONAME:MyPyroThing")
EDIT:
I think this question was not written correctly, see my answer below.
The answers above where not what I was really asking, meaning I explained my question badly. Mea Culpa.
I wanted to invoke an instance on the client. But that is not how Pyro4 works at all. A class in instantiated on the server and this instance is transmitted over the wire.
After mailing Irmin (the original developer) it came clear to me how Pyro4 works.
So, what I do now is use a factory pattern where I ask the factory to give me an instance of an object. For instance:
psf = Pyro4.Proxy("PYRONAME:MyApp.Factories.ProductFactory")
product = psf.GetProductOnButton(buttonNoPressed, parentProductId)
product is an instance of the Product() class. Because the instance is registered in the Pyro daemon, i can call methods on this instance of Product() too. Look at the shoppingcart example to know where I got my eureka moment.
Instead of using Pyro4.Daemon.serveSimple you can:
Get the name server using Pyro4.locateNS
Create a Pyro4.Daemon object
Create the objects you need to expose
Use the daemon register method to make them available
Use the name server register method to provide a name to uri mapping
Start the daemon loop
The code would be more or less as follows:
import Pyro4
name_server = Pyro4.locateNS()
daemon = Pyro4.Daemon()
my_object = MyPyroThing(parameter)
my_object_uri = daemon.register(my_object)
name_server.register('MyPyroThing', my_object_uri)
daemon.requestLoop()
After this, my_object URI will be available in the name server as MyPyroThing.