So, I'm trying to use SQS to pass a Python object between two EC2 instances. Here's my failed attempt:
import boto.sqs
from boto.sqs.message import Message
class UserInput(Message):
def set_email(self, email):
self.email = email
def set_data(self, data):
self.data = data
def get_email(self):
return self.email
def get_data(self):
return self.data
conn = boto.sqs.connect_to_region('us-west-2')
q = conn.create_queue('user_input_queue')
q.set_message_class(UserInput)
m = UserInput()
m.set_email('something#something.com')
m.set_data({'foo': 'bar'})
q.write(m)
It returns an error message saying that The request must contain the parameter MessageBody. Indeed, the tutorial tells us to do m.set_body('something') before writing the message to the queue. But here I'm not passing a string, I want to pass an instance of my UserInput class. So, what should MessageBody be? I've read the docs and they say that
The constructor for the Message class must accept a keyword parameter “body” which represents the content or body of the message. The format of this parameter will depend on the behavior of the particular Message subclass. For example, if the Message subclass provides dictionary-like behavior to the user the body passed to the constructor should be a dict-like object that can be used to populate the initial state of the message.
I guess the answer to my question might be in that paragraph, but I can't make sense of it. Could anyone provide a concrete code example to illustrate what they are talking about?
For an arbitrary Python object the answer is to serialize the object into a string, use SQS to send that string to the other EC2 instance and deserialize the string back to an instance of the same class.
For example, you could use JSON with base64 encoding to serialize an object into a string and that would be the body of your message.
Related
I am trying to store the content of a callback function, in order to access and manipulate the data within the script.
As far as I am concerned, the given function .subscribe() in my code below does not return anything (None). My function is only passed as a reference to the function as an argument. Is there a way to return the data from the function that calls my function?
My code is a simple example with roslibpy (a library for Python that interacts with the open-source robotics framework ROS through Websockets). It is mentioned, that the data is published as a stream via a Websocket each time a message is published into the topic /turtle1/pose. My goal here is to return the data that is being published into the topic. The print command provides a nice visualization of the data, which just works fine.
import roslibpy
client = roslibpy.Ros(host='localhost', port=9090)
client.run()
def myfunc(msg):
print(msg)
listener = roslibpy.Topic(client, '/turtle1/pose', 'turtlesim/Pose')
#my function is passed as an argument
listener.subscribe(myfunc)
try:
while True:
pass
except KeyboardInterrupt:
client.terminate()
The subscribe() method in the roslibpy library is defined as follows:
def subscribe(self, callback):
"""Register a subscription to the topic.
Every time a message is published for the given topic,
the callback will be called with the message object.
Args:
callback: Function to be called when messages of this topic are published.
"""
# Avoid duplicate subscription
if self._subscribe_id:
return
self._subscribe_id = 'subscribe:%s:%d' % (
self.name, self.ros.id_counter)
self.ros.on(self.name, callback)
self._connect_topic(Message({
'op': 'subscribe',
'id': self._subscribe_id,
'type': self.message_type,
'topic': self.name,
'compression': self.compression,
'throttle_rate': self.throttle_rate,
'queue_length': self.queue_length
}))
Is there common way to deal with such problems? Does it make more sense to store the output as an external source (e.g. .txt) and then access the source trough the script?
You can define a Python class that acts like a function, that can modify its own state when called, by defining the magic __call__ method. When obj(whatever) is done on a non-function obj, Python will run obj.__call__(whatever). subscribe only needs its input to be callable; whether it is an actual function or an object with a __call__ method does not matter to subscribe.
Here's an example of what you could do:
class MessageRecorder():
def __init__(self):
self.messages = []
# Magic python 'dunder' method
# Whenever a MessageRecorder is called as a function
# This function defined here will be called on it
# In this case, adds the message to a list of received messages
def __call__(self, msg):
self.messages.append(msg)
recorder = MessageRecorder()
# recorder can be called like a function
recorder("Hello")
listener.subscribe(recorder)
try:
while True:
pass
except KeyboardInterrupt:
client.terminate()
"""Now you can do whatever you'd like with recorder.messages,
which contains all messages received before termination,
in order of reception. If you wanted to print all of them, do:"""
for m in recorder.messages:
print(m)
This is a pretty basic question, but regardless its had me stumped for a bit. I'm trying to access a specific attribute of a tweet (documentation found here), such as "text". I tried accessing it via data["text"], however this gives me the following error TypeError: string indices must be integers.
So I tried parsing the data using json.loads(data) thinking this would allow me to access each attribute of the tweet. However this instead returns solely the text portion of the tweet, meaning when I do print(newData), it prints out the text. Although this is useful, I need to be able to access other attributes of the tweet such as "created_at".
So my question is, how do I parse the tweet or access it which allows me to pluck out individual attributes I need. To reiterate, I'm sure this is pretty simple, however I'm new to handling JSON objects, and other solutions I found simply told me to use loads(), which isn't what I want.
class TwitterStreamer():
"""
Class for streaming and processing live tweets for a given list of hashtags
"""
def stream_tweets(selfself, hashtag_list):
listener = StdOutListener()
auth = OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)
stream = Stream(auth, listener)
stream.filter(track=hashtag_list)
class StdOutListener(StreamListener):
def on_data(self, data):
print(data)
newData = json.loads(data)
print(newData["text"])
return True
def on_error(self, status):
print(status)
def main():
hashtag_list = ['Chelsea']
fetched_tweets_filename = "tweets.json"
twitter_streamer = TwitterStreamer()
twitter_streamer.stream_tweets(hashtag_list)
main()
Try using "." operator to access attributes of the tweet. I used it in my code as follow:
tweet = follow_user.status.created_at
In this I got the user in the form of JSON data "status" is an attribute of that JSON object "follow_user"
Try using json.load() to load the JSON as a Python object. The method json.loads() load the JSON as a string, that's why it gives you a TypeError Exception since string objects indices can only be integers.
I have made a function that connects to a twitter api.
This function returns an twitter object. I want to create a testing function that checks if the returned object is really a twitter object.
So this is my function:
def authenticate_twitter_api():
"""Make connection with twitters REST api"""
try:
logger.info('Starting Twitter Authentication')
twitter_api = twitter.Twitter(auth=twitter.OAuth(config.TWITTER_ACCESS_KEY, config.TWITTER_ACCESS_SECRET,
config.TWITTER_CONSUMER_KEY, config.TWITTER_CONSUMER_SECRET))
print twitter_api
logger.info("Service has started")
return twitter_api
except:
logger.error("Authentication Error. Could not connect to twitter api service")
When i run this function it returns:
<twitter.api.Twitter object at 0x7fc751783910>
Now, i want to create a testing function, maybe through numpy.testing in order to check if the type is a object.
numpy.testing.assert_equal(actual, desired, err_msg='')
actual = type(authenticate_twitter_api())
desired =<class 'twitter.api.Twitter'>
And here is the problem. I can't save an object to 'desired'.
What can i do ?
The desired object you are looking for is twitter.api.Twitter, just import it and pass the class the assert_equal.
However, it's more idiomatic to use isinstance:
from twitter.api import Twitter
if isinstance(authenticate_twitter_api(), Twitter):
print("It was a Twitter object.")
classes are objects themselves in Python. So you can assign your desired variable like this:
import twitter
# (...)
desired = twitter.api.Twitter
I have a problem making the simple (non-json) arguments work with POST. Just taking the simple example from their tutorials, I can't make a unit test where the task is passing as an argument. However task is never passed in. Its none.
class TodoList(Resource):
def __init__(self):
self.reqparse = reqparse.RequestParser()
self.reqparse.add_argument('task', type = str)
super(TodoList, self).__init__()
def post(self):
args = self.reqparse.parse_args()
#args['task'] is None, but why?
return TODOS[args['task']], 201
Unit test:
def test_task(self):
rv = self.app.post('todos', data='task=test')
self.check_content_type(rv.headers)
resp = json.loads(rv.data)
eq_(rv.status_code, 201)
What am I missing please?
When you use 'task=test' test_client do not set application/x-www-form-urlencoded content type, because you put string to input stream. So flask can't detect form and read data from form and reqparse will return None for any values for this case.
To fix it you must set up content type or use dict {'task': 'test'} or tuple.
Also for request testing better to use client = self.app.test_client() instead app = self.app.test_client(), if you use FlaskTesting.TestCase class, then just call self.client.post.
I want to use Pyro with an existing set of classes that involve a factory pattern, i.e. an object of Class A (typically there will only be one of these) is used to instantiate objects of Class B (there can be an arbitrary number of these) through a factory method . So, I'm exposing an object of Class A as the Pyro proxy object.
I've extended the Pyro introductory sample code to reflect roughly what I'm trying to do. The server-side code is as follows:
# saved as greeting.py
import Pyro4
import socket
class NewObj:
func_count = None
def __init__(self):
print "{0} ctor".format(self)
func_count = 0
def __del__(self):
print "{0} dtor".format(self)
def func(self):
print "{0} func call {1}".format(self, self.func_count)
self.func_count += 1
class GreetingMaker(object):
def __init__(self):
print "{0} ctor".format(self)
def __del__(self):
print "{0} dtor".format(self)
def get_fortune(self, name):
print "getting fortune"
return "Hello, {0}. Here is your fortune message:\n" \
"Behold the warranty -- the bold print giveth and the fine print taketh away.".format(name)
def make_obj(self):
return NewObj()
greeting_maker=GreetingMaker()
daemon=Pyro4.Daemon(host=socket.gethostbyname(socket.gethostname()), port=8080) # make a Pyro daemon
uri=daemon.register(greeting_maker, "foo") # register the greeting object as a Pyro object
print "Ready. Object uri =", uri # print the uri so we can use it in the client later
daemon.requestLoop() # start the event loop of the server to wait for calls
The client side code was also altered slightly:
# saved as client.py
import Pyro4
uri="PYRO:foo#10.2.129.6:8080"
name="foo"
greeting_maker=Pyro4.Proxy(uri) # get a Pyro proxy to the greeting object
print greeting_maker.get_fortune(name) # call method normally
print greeting_maker.make_obj()
My intention is to be able to create instances of NewObj and to manipulate them just as I can manipulate instances of GreetingMaker on the client side, but it looks as though what happens is when the make_obj method gets called, a NewObj is created on the server side, immediately falls out of scope, and is consequently garbage collected.
This is what the output looks like, server-side:
<__main__.GreetingMaker object at 0x2aed47e01110> ctor
/usr/lib/python2.6/site-packages/Pyro4-4.12-py2.6.egg/Pyro4/core.py:152: UserWarning: HMAC_KEY not set, protocol data may not be secure
warnings.warn("HMAC_KEY not set, protocol data may not be secure")
Ready. Object uri = PYRO:foo#10.2.129.6:8080
getting fortune
<__main__.NewObj instance at 0x175c8098> ctor
<__main__.NewObj instance at 0x175c8098> dtor
... and client-side:
/usr/local/lib/python2.6/dist-packages/Pyro4-4.12-py2.6.egg/Pyro4/core.py:152: UserWarning: HMAC_KEY not set, protocol data may not be secure
warnings.warn("HMAC_KEY not set, protocol data may not be secure")
Hello, foo. Here is your fortune message:
Behold the warranty -- the bold print giveth and the fine print taketh away.
Traceback (most recent call last):
File "client.py", line 9, in <module>
print greeting_maker.make_obj()
File "/usr/local/lib/python2.6/dist-packages/Pyro4-4.12-py2.6.egg/Pyro4/core.py", line 146, in __call__
return self.__send(self.__name, args, kwargs)
File "/usr/local/lib/python2.6/dist-packages/Pyro4-4.12-py2.6.egg/Pyro4/core.py", line 269, in _pyroInvoke
data=self._pyroSerializer.deserialize(data, compressed=flags & MessageFactory.FLAGS_COMPRESSED)
File "/usr/local/lib/python2.6/dist-packages/Pyro4-4.12-py2.6.egg/Pyro4/util.py", line 146, in deserialize
return self.pickle.loads(data)
AttributeError: 'module' object has no attribute 'NewObj'
I suspect I could hack around this problem by having the factory class (i.e. GreetingMaker) keep a reference to every NewObj that it creates, and add a cleanup method of some sort... but is that really necessary? Am I missing something in Pyro that can help me implement this?
(edited for clarity)
I recently came across this feature and am using it. It is crucial for my code which uses a similar factory pattern.
Pyro Server
class Foo(object):
def __init__(self, x=5):
self.x = x
class Server(object):
def build_foo(self, x=5):
foo = Foo(x)
# This line will register your foo instance as its own proxy
self._pyroDaemon.register(foo)
# Returning foo here returns the proxy, not the actual foo
return foo
#...
uri = daemon.register(Server()) # In the later versions, just use Server, not Server()
#...
The problem here is that pyro pickles the NewObj object in the server side, but it fails to unpickle it in the client side because the NewObj implementation is unknown to the client.
One way to fix the problem, would be to create a third module called, for example new_obj.py and, after that, import it in both the server and the client as follows:
from new_obj import NewObj
This will let the client unpickle the NewObj instance and work with it. Anyway, please note that it will be a real NewObj object living in the client, not a proxy to an object living in the server.