I am trying make a pyro4 proxy indexable. To test this, I took the greeting example from http://pythonhosted.org/Pyro4/intro.html#simple-example and I modified it:
Server:
import Pyro4
class Test(object):
def __getitem__(self, index):
return index
test = Test()
print test[1]
print test[100]
daemon = Pyro4.Daemon()
uri = daemon.register(test)
print("Ready. Object uri =", uri)
daemon.requestLoop()
Client:
import Pyro4
uri = input("What is the Pyro uri of the object? ").strip()
test = Pyro4.Proxy(uri)
print test.__getitem__(1)
print test.__getitem__(100)
print test[1]
print test[100]
The [] notation works on the server, but not also on the client proxy. I get:
TypeError: 'Proxy' object does not support indexing
But calls directly to __getitem__ do work.
I've just run into this myself.
From what I can see looking at the source code, Pyro4 doesn't proxy the Python implicit __getitem__ call that index notation uses. It does proxy __getattr__, which is why calling the __getitem__ method directly works.
What you can do, though, is create (on the client side) a proxy to the Pyro proxy(!) that implements __getitem__ and lets all other method calls fall through:
class TestProxy(object):
def __init__(self, pyroTest):
self.pyroTest = pyroTest
def __getattr__(self, name):
return getattr(self.pyroTest, name)
def __getitem__(self, item):
return self.pyroTest.__getitem__(item)
Then you can use index notation on the TestProxy object, as well as call methods in the normal Pyro way.
(Disclaimer: there are probably all sorts of Pythonic edge cases that this simple solution fails to cover!)
This might be worth an enhancement request for Pyro.
While this could perhaps be added to the Pyro proxy, it actually promotes potentially horrible performing code.
Indexing an object usually is done because the object is a collection of some sort and you are probably iterating over it. Doing this on a Pyro proxy will result in terrible performance because every index lookup will be a remote call.
It is usually a lot faster and way more efficient to simply get the collection you want to iterate over all at once using one remote call, and then iterate over the resulting local object as usual.
YMMV, it depends on the situation ofcourse.
Related
I am often writing scripts with boto3 and usually when writing functions I end up passing the boto3 client for the service(s) I need around my functions. So, for example
def main():
ec2 = create_client
long_function_with_lots_of_steps(ec2, ....)
def long_function_with_lots_of_steps(client):
....
This is not too bad, but it often feels repetitive and sometimes I will need to create a new client for a different service in the other function, for which I would like to use the original aws_session object.
Is there a way to do this more elegantly? I thought to make a class holding a boto3.session.Session() object but then you end up just passing that around.
How do you usually structure boto3 scripts?
I think you might have had some C or C++ programming experience. You are definitely getting language constructs confused. In Python function call arguments are passed by reference. So passing a reference is quick. You aren't passing the whole object.
This is in fact one of the better ways to pass in session info. Why is it better you may ask? Because of testing. You will need to test the thing and you don't always want to test the connections to 3rd party services. So you can do that with Mocks.
Try making a test where you are mocking out any one of those function arguments. Go ahead... I'll wait.
Easier... right?
Since you are basically asking for an opinion:
I usually go with your second approach. I build a base class with the session object, and build off of that. When working with a large program where I must maintain some "global" state, I make a class to house those items, and that becomes a member of my base class.
class ProgramState:
def __init__(self):
self.sesson = boto3.session.Session()
class Base:
def __init__(self, state: ProgramState):
self.state = state
class Firehose(Base):
def __init__(self, state: ProgramState):
Base.__init__(self, state)
self.client = self.state.session.client("firehose")
def do_something():
pass
class S3(Base):
def __init__(self, state: ProgramState):
Base.__init__(self, state)
self.client = self.state.session.client("s3")
def do_something_else():
pass
def main():
state = ProgramState()
firehose = Firehose(state)
s3 = S3(state)
firehose.do_something()
s3.do_something_else()
Full disclosure: I dislike Python.
I have a method that builds a query, pass it to a _make_query method in charge of resolving that query (using dns resolver) and return the answer. Then, the parent method do some stuff from the answer. I'd like to unit test the parent method ; for that I guess the best way would be to mock the _make_query method to return different outcomes and test how the parent method respond to it.
However I'm having a hard time mocking the method to return the same object returned by the dns resolver.
Here is the _make_query method:
def _make_query(self, query):
query_resolver = resolver.Resolver()
return query_resolver.query(query, 'SRV')
code of the calling method :
def _get_all_databases(self, database_parameters):
query = self._format_dns_query(database_parameters)
answers = self._make_query(query)
databases = []
for answer in answers:
databases.append(
Database(
answer.target, answer.port, answer.weight,
database_parameters.db_name
))
return databases
(also private as the main method get_database has then to pick a database from the list returned)
I have a mock to return what I want from this method in my unit tests, however I don't know how to reproduce the object being returned by the resolver.query() method. It should return a dns.resolver.Answer, which in turn contains a list of dns.rdtypes.IN.SRV.SRV it seems. Is there a simple way to do it?
You can either mock the __make_query() method (a bit harder, since you need to manually mangle the name now to match the class-private namespace protection, see What is the meaning of single and double underscore before an object name?), or mock the Resolver() object.
You don't have to exactly match the instances produced here, you only need to produce enough of their attributes to pass muster. For the SRV class from the dnspython project, all you need is an object with port, priority, target and weight attributes, with target behaving like a dns.name.Name instance. The latter is a bit more complex, but you only need to stub out the things your code needs.
You can trivially do this with the unittest.mock library, with or without speccing out the objects precisely. For your code, all you use is 3 attributes, so your mock only ever needs to return a list with nothing more than that.
You can use the create_autospec() function to generate a mock object that's limited to the attributes the original class supports. This can help detect bugs where your code uses an attribute or method that the original classes would never allow. If you don't use a spec, then the default is to produce mock objects that allow all attributes, pretending that those attributes exist (and each such access would produce more mock objects).
So, if you need SRV instances, then I'd use:
import unittest
from unittest import mock
from dns.rdtypes.IN.SRV import SRV
def make_mock_srv(target, port, weight):
mock_srv = mock.create_autospec(SRV)
mock_name = mock.create_autospec(Name)
instance = mock_srv.return_value
instance.target = target
instance.port = port
instance.weight = weight
return instance
class TestMakeQuery(unittest.TestCase):
#mock.patch('dns.resolver.Resolver')
def test_make_query(self, mock_resolver):
mock_resolver_instance = mock_resolver.return_value # the object returned by Resolver()
mock_resolver_instance.query.return_value = [
make_mock_srv('foo.', 1234, 2),
make_mock_srv('bar.', 42, 4),
]
# run your test, which calls _make_query, which calls Resolver().query()
I am a beginner in Python, so please be... kind?
Anyway, I need use a static method to call another method, which requires the use of "self" (and thus, a normal method I believe). I am working with Telethon, a Python implementation of Telegram. I have tried other questions on SO, but I just can't seem to find a solution to my problem.
An overview of the program (please correct me if I'm wrong):
1) interactive_telegram_client is a child class of telegram_client, and it creates an instance.
#interactive_telegram_client.py
class InteractiveTelegramClient(TelegramClient):
super().__init__(session_user_id, api_id, api_hash, proxy)
2) When the InteractiveTelegramClient runs, it adds an update_handler self.add_update_handler(self.update_handler) to constantly check for messages received/sent, and prints it to screen
#telegram_client.py
def add_update_handler(self, handler):
"""Adds an update handler (a function which takes a TLObject,
an update, as its parameter) and listens for updates"""
if not self.sender:
raise RuntimeError(
"You should connect at least once to add update handlers.")
self.sender.add_update_handler(handler)
#interactive_telegram_client.py
#staticmethod
def update_handler(update_object):
try:
if type(update_object) is UpdateShortMessage:
if update_object.out:
print('You sent {} to user #{}'.format(update_object.message,
update_object.user_id))
else:
print('[User #{} sent {}]'.format(update_object.user_id,
update_object.message))
Now, my aim here is to send back an auto-reply message upon receiving a message. Thus, I think that adding a call to method InteractiveTelegramClient.send_ack(update_object) in the update_handler method would serve my needs.
#interactive_telegram_client.py
def send_ack(self, update_object):
entity = update_object.user_id
message = update_object.message
msg, entities = parse_message_entities(message)
msg_id = utils.generate_random_long()
self.invoke(SendMessageRequest(peer=get_input_peer(entity),
message=msg,random_id=msg_id,entities=entities,no_webpage=False))
However, as you can see, I require the self to invoke this function (based on the readme, where I assume client to refer to the same thing as self). Since the method update_handler is a static one, self is not passed through, and as such I cannot invoke the call as such.
My possible strategies which have failed include:
1) Instantiating a new client for the auto-reply
- Creating a new client/conversation for each reply...
2) Making all the methods non-static
- Involves a tremendous amount of work since other methods modified as well
3) Observer pattern (sounds like a good idea, I tried, but due to a lack of skills, not succeeded)
I was wondering if there's any other way to tackle this problem? Or perhaps it's actually easy, just that I have some misconception somewhere?
Forgot to mention that due to some restrictions on my project, I can only use Telethon, as opposed to looking at other alternatives. Adopting another library (like an existing auto-reply one) is allowed, though I did not really look into that since merging that and Telethon may be too difficult for me...
based on the readme, where I assume client to refer to the same thing as self
Correct, since the InteractiveTelegramClient subclasses the TelegramClient and hence, self is an instance of the extended client.
Instantiating a new client for the auto-reply - Creating a new client/conversation for each reply
This would require you to create another authorization and send another code request to login, because you can't work with the same *.session at the same time.
Making all the methods non-static - Involves a tremendous amount of work since other methods modified as well
It doesn't require such amount of work. Consider the following example:
class Example:
def __init__(self, a):
self.a = a
def do_something(self):
Example.other_method()
#staticmethod
def other_method():
print('hello, world!')
Is equivalent to:
class Example:
def __init__(self, a):
self.a = a
def do_something(self):
self.other_method()
#staticmethod
def other_method():
print('hello, world!')
It doesn't matter whether you use self. or the class name to refer to a static method from within the class. Since the InteractiveClientExample already uses self., all you would have to do would be changing:
#staticmethod
def update_handler(update_object):
for
def update_handler(self, update_object):
For more on the #staticmethod decorator, you can refer to the docs.
I'm trying to understand how to make RPC calls using Python. I have a stupid server that defines a class and exposes a method that create instances of that class:
# server.py
class Greeter(object):
def __init__(self, name):
self.name = name
def greet(self):
return "Hi {}!".format(self.name)
def greeter_factory(name):
return Greeter(name)
some_RPC_framework.register(greeter_factory)
and a client that wants to get an instance of the Greeter:
# client.py
greeter_factory = some_RPC_framework.proxy(uri_given_by_server)
h = greeter_factory("Heisemberg")
print("Server returned:", h.greet())
The problem is that I've found no framework that allows to return instances of user-defined objects, or that only returns a dict with the attributes of the object (for example, Pyro4).
In the past I've used Java RMI, where you can specify a codebase on the server where the client can download the compiled classes, if it needs to. Is there something like this for Python? Maybe some framework that can serialize objects along with the class bytecode to let the client have a full-working instance of the class?
Pyro can do this to a certain extent. You can register custom class (de)serializers when using the default serializer. Or you can decide to use the pickle serializer, but that has severe security implications. See http://pythonhosted.org/Pyro4/clientcode.html#serialization
What Pyro won't do for you, even when using the pickle serializer, is transfer the actual bytecode that makes up the module definition. The client, in your case, has to be able to import the module defining your classes in the regular way. There's no code transportation.
You can consider using
payload = CPickle.dump(Greeter(name))
on server side and on client side once the payload is received do -
h = CPickle.load(payload)
to get the instance of Greeter object that server has created.
I have an object that takes a parameter in the constructor. I was wondering how I can serve this from Pyro4. An Example:
import Pyro4
class MyPyroThing(object):
def __init__(self, theNumber):
self.Number = theNumber
Pyro4.Daemon.serveSimple(
{
MyPyroThing(): None
},
ns=True, verbose=True)
This fails of course because the constructor must have a parameter.
And when this is solved, how do you invoke such object?
theThing = Pyro4.Proxy("PYRONAME:MyPyroThing")
EDIT:
I think this question was not written correctly, see my answer below.
The answers above where not what I was really asking, meaning I explained my question badly. Mea Culpa.
I wanted to invoke an instance on the client. But that is not how Pyro4 works at all. A class in instantiated on the server and this instance is transmitted over the wire.
After mailing Irmin (the original developer) it came clear to me how Pyro4 works.
So, what I do now is use a factory pattern where I ask the factory to give me an instance of an object. For instance:
psf = Pyro4.Proxy("PYRONAME:MyApp.Factories.ProductFactory")
product = psf.GetProductOnButton(buttonNoPressed, parentProductId)
product is an instance of the Product() class. Because the instance is registered in the Pyro daemon, i can call methods on this instance of Product() too. Look at the shoppingcart example to know where I got my eureka moment.
Instead of using Pyro4.Daemon.serveSimple you can:
Get the name server using Pyro4.locateNS
Create a Pyro4.Daemon object
Create the objects you need to expose
Use the daemon register method to make them available
Use the name server register method to provide a name to uri mapping
Start the daemon loop
The code would be more or less as follows:
import Pyro4
name_server = Pyro4.locateNS()
daemon = Pyro4.Daemon()
my_object = MyPyroThing(parameter)
my_object_uri = daemon.register(my_object)
name_server.register('MyPyroThing', my_object_uri)
daemon.requestLoop()
After this, my_object URI will be available in the name server as MyPyroThing.