Develop Reference

msgpack-python failing to deserialize -> getting ExtraData error

I'm trying to make MQTTtoROS Bridge work, and i keep getting this error:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/animu/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 2627, in _thread_main
self.loop_forever(retry_first_connection=True)
File "/home/animu/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 1407, in loop_forever
rc = self.loop(timeout, max_packets)
File "/home/animu/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 923, in loop
rc = self.loop_read(max_packets)
File "/home/animu/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 1206, in loop_read
rc = self._packet_read()
File "/home/animu/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 1799, in _packet_read
rc = self._packet_handle()
File "/home/animu/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 2275, in _packet_handle
return self._handle_publish()
File "/home/animu/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 2461, in _handle_publish
self._handle_on_message(message)
File "/home/animu/.local/lib/python2.7/site-packages/paho/mqtt/client.py", line 2615, in _handle_on_message
t[1](self, self._userdata, message)
File "/home/animu/catkin_ws/src/mqtt_bridge-master/src/mqtt_bridge/bridge.py", line 114, in _callback_mqtt
ros_msg = self._create_ros_message(mqtt_msg)
File "/home/animu/catkin_ws/src/mqtt_bridge-master/src/mqtt_bridge/bridge.py", line 124, in _create_ros_message
msg_dict = self._deserialize(mqtt_msg.payload)
File "msgpack/_unpacker.pyx", line 143, in msgpack._unpacker.unpackb (msgpack/_unpacker.cpp:2143)
ExtraData: unpack(b) received extra data.
I can't find anything on it in the internet, as this bridge is i guess not commonly used. The only similar problems were in Salt and Kafka, but the solution is nowhere to be found. All python libraries are up to date, i double checked. The bridge sends messages from RoS to MQTT without any problems, both STR and BOOL types. Any message sent from MQTT ends up as this error with no reception from ROS.

Its a bit late but I'll give some advice for future readers.
First, make sure you have installed all requirements for the bridge to function. Check them by reading requirement.txt
Second, Edit mqtt_bridge configuration file to match topics from ROS and from your MQTT server. Also IP address/port of MQTT server.
Thats it.

Since Animu is a student, I assume that this was an assignment from his training or internship company. An answer is probably no use anymore, but since I also had this problem, I hereby offer a solution for future readers:
In the bridge repository there is a file called "demo_params.yaml". Or if you have already named it differently, then the .yaml file that contains your "settings".
This file includes the following:
mqtt:
client:
protocol: 4 # MQTTv311
connection:
host: localhost
postage: 1883
keepalive: 60
private_path: device / 001
serializer: msgpack: dumps
deserializer: msgpack: loads
bridge:
# ping pong
- factory: mqtt_bridge.bridge: RosToMqttBridge
msg_type: std_msgs.msg: Bool
topic_from: / ping
topic_to: ping
- factory: mqtt_bridge.bridge: MqttToRosBridge
msg_type: std_msgs.msg: Bool
topic_from: ping
topic_to: / pong
As you can see, it says msgpack is used to serialize and deserialize your messages that are sent back and forth. This mainly works for ROS to MQTT. The other way around, this does not work, as no correct actions are performed in the Python code. You have two solutions for this.
Continue to work with msgpack and make sure that the MQTT messages you publish are already encoded as msgpack would serialize them itself. (binary). This is a tricky solution, since you just want to keep MQTT messages human readable if you publish them manually. If you have written another program that publishes the MQTT messages, feel free to serialize the message first with msgpack and then publish. Then the bridge also works.
The other option is to have JSON serialization, instead of msgpack serialization. This is the default option of the bridge, but you can also specify this in your .yaml file. You do this by editing this:
serializer: msgpack: dumps
deserializer: msgpack: loads
to this:
serializer: json: dumps
deserializer: json: loads
Now you can publish mqtt messages both manually and with the help of software.
You do this as follows:
mosquitto_pub -t 'echo' -m '{"data": "test"}

Python-SocketIO on Windows: An operation was attempted on something that is not a socket

I'm in the process of writing a minimal websocket server with Python 3. I am using flask, socketio, and eventlet per the instructions on the latest docs. The problem is that when the webpage with the socket connection is reloaded, the server throws the following exception:
Traceback (most recent call last):
File "C:\Users\Noah\AppData\Local\Programs\Python\Python35-32\lib\site-packages\eventlet\greenpool.py", line 88, in _spawn_n_impl
func(*args, **kwargs)
File "C:\Users\Noah\AppData\Local\Programs\Python\Python35-32\lib\site-packages\eventlet\wsgi.py", line 734, in process_request
proto.__init__(sock, address, self)
File "C:\Users\Noah\AppData\Local\Programs\Python\Python35-32\lib\socketserver.py", line 686, in __init__
self.finish()
File "C:\Users\Noah\AppData\Local\Programs\Python\Python35-32\lib\site-packages\eventlet\wsgi.py", line 651, in finish
greenio.shutdown_safe(self.connection)
File "C:\Users\Noah\AppData\Local\Programs\Python\Python35-32\lib\site-packages\eventlet\greenio\base.py", line 479, in shutdown_safe
return sock.shutdown(socket.SHUT_RDWR)
OSError: [WinError 10038] An operation was attempted on something that is not a socket
I took a look at the source, and it seems like shutdown_safe is supposed to just catch any exceptions while shutting down a connection. In short, it seems like the author of this part of the library didn't foresee Windows throwing an OSError on shutdown.
Although this is a benign issue, I was wondering if there are any existing fixes/tweaks, and if not, whether I should submit this to the python-socketio GitHub issues list.

ImageKit async error - can't decode message body

I'm using Django 1.6 and Django-ImageKit 3.2.1.
I'm trying to generate images asynchronously with ImageKit. Async image generation works locally but not on the production server.
I'm using Celery and I've tried both:
IMAGEKIT_DEFAULT_CACHEFILE_BACKEND = 'imagekit.cachefiles.backends.Async'
IMAGEKIT_DEFAULT_CACHEFILE_BACKEND = 'imagekit.cachefiles.backends.Celery'
Using the Simple backend (synchronous) instead of Async or Celery works fine on the production server. So I don't understand why the asynchronous backend gives me the following ImportError (pulled from the Celery log):
[2014-04-05 21:51:26,325: CRITICAL/MainProcess] Can't decode message body: DecodeError(ImportError('No module named s3utils',),) [type:u'application/x-python-serialize' encoding:u'binary' headers:{}]
body: '\x80\x02}q\x01(U\x07expiresq\x02NU\x03utcq\x03\x88U\x04argsq\x04cimagekit.cachefiles.backends\nCelery\nq\x05)\x81q\x06}bcimagekit.cachefiles\nImageCacheFile\nq\x07)\x81q\x08}q\t(U\x11cachefile_backendq\nh\x06U\x12ca$
Traceback (most recent call last):
File "/opt/python/run/venv/lib/python2.6/site-packages/kombu/messaging.py", line 585, in _receive_callback
decoded = None if on_m else message.decode()
File "/opt/python/run/venv/lib/python2.6/site-packages/kombu/message.py", line 142, in decode
self.content_encoding, accept=self.accept)
File "/opt/python/run/venv/lib/python2.6/site-packages/kombu/serialization.py", line 184, in loads
return decode(data)
File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__
self.gen.throw(type, value, traceback)
File "/opt/python/run/venv/lib/python2.6/site-packages/kombu/serialization.py", line 59, in _reraise_errors
reraise(wrapper, wrapper(exc), sys.exc_info()[2])
File "/opt/python/run/venv/lib/python2.6/site-packages/kombu/serialization.py", line 55, in _reraise_errors
yield
File "/opt/python/run/venv/lib/python2.6/site-packages/kombu/serialization.py", line 184, in loads
return decode(data)
File "/opt/python/run/venv/lib/python2.6/site-packages/kombu/serialization.py", line 64, in pickle_loads
return load(BytesIO(s))
DecodeError: No module named s3utils
s3utils is what defines my AWS S3 bucket paths. I'll post it if need be, but the strange thing I think is that the synchronous backend has no problem importing s3utils while the asynchronous does... and asynchronous does ONLY on the production server, not locally.
I'd be SO greatful for any help debugging this. I've been wrestling this for days. I'm still learning Django and python so I'm hoping this is a stupid mistake on my part. My Google-fu has failed me.

As I hinted at in my comment above, this kind of thing is usually caused by forgetting to restart the worker.
It's a common gotcha with Celery. The workers are a separate process from your web server so they have their own versions of your code loaded. And just like with your web server, if you make a change to your code, you need to reload so it sees the change. The web server talks to your worker not by directly running code, but by passing serialized messages via the broker, which will say something like "call the function do_something()". Then the worker will read that message and—and here's the tricky part—call its version of do_something(). So even if you restart your webserver (so that it has a new version of your code), if you forget to reload the worker (which is what actually calls the function), the old version of the function will be called. In other words, you need to restart the worker any time you make a change to your tasks.
You might want to check out the autoreload option for development. It could save you some headaches.

Catching bottle server errors

I am trying to get my bottle server so that when one person in a game logs out, everyone can immediately see it. As I am using long polling, there is a request open with all the users.
The bit I am having trouble with is catching the exception that is thrown when the user leaves the page from the long polling that can no longer connect to the page. The error message is here.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 438, in handle_one_response
self.run_application()
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 425, in run_application
self.process_result()
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 416, in process_result
self.write(data)
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 373, in write
self.socket.sendall(msg)
File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 509, in sendall
data_sent += self.send(_get_memory(data, data_sent), flags)
File "/usr/lib/python2.7/dist-packages/gevent/socket.py", line 483, in send
return sock.send(data, flags)
error: [Errno 32] Broken pipe
<WSGIServer fileno=3 address=0.0.0.0:8080>: Failed to handle request:
request = GET /refreshlobby/1 HTTP/1.1 from ('127.0.0.1', 53331)
application = <bottle.Bottle object at 0x7f9c05672750>
127.0.0.1 - - [2013-07-07 10:59:30] "GET /refreshlobby/1 HTTP/1.1" 200 160 6.038377
The function to handle that page is this.
#route('/refreshlobby/<id>')
def refreshlobby(id):
while True:
yield lobby.refresh()
gevent.sleep(1)
I tried catching the exception within the function, and in a decorator which I put to wrap #route, neither of which worked. I tried making an #error(500) decorator, but that didn't trigger, either. It seems that this is to do with the internals of bottle.
Edit: I know now that I need to be catching socket.error, but I don't know whereabouts in my code

The WSGI runner
Look closely at the traceback: this in not happening in your function, but in the WSGI runner.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/gevent/pywsgi.py", line 438, in handle_one_response
self.run_application()
The way the WSGI runner works, in your case, is:
Receives a request
Gets a partial response from your code
Sends it to the client (this is where the exception is raised)
Repeats steps 2-3
You can't catch this exception
This error is not raised in your code.
It happens when you try to send a response to a client that closed the connection.
You'll therefore not be able to catch this error from within your code.
Alternate solutions
Unfortunately, it's not possible to tell from within the generator (your code) when it stops being consumed.
It's also not a good idea to rely on your generator being garbage collected.
You have a couple other solutions.
"Last seen"
Another way to know when an user disconnects would probably be to record a "last seen", after your yield statement.
You'll be able to identify clients that disconnected if their last seen is far in the past.
Other runner
Another, non-WSGI runner, will be more appropriate for a realtime application. You could give tornado a try.

Python Paste using Bottle framework Broken Pipe Error

I am using Bottle framework implementing the WSGI Request and response and because of the single thread issue, I changed the server into PythonWSGIServer and tested with Apache bench but the result consist of error broken pipe which is similar with this question How to prevent errno 32 broken pipe?.
I have tried the answer but to no avail.
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/paste/httpserver.py", line 1068, in process_request_in_thread
self.finish_request(request, client_address)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 323, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 641, in __init__
self.finish()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 694, in finish
self.wfile.flush()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
The server code is shown below, and I have no idea how to improve the connection, using thread pool?
from paste import httpserver
#route('/')
def index():
connection = pymongo.MongoClient(connectionString)
db = connection.test
collection = db.test
return str(collection.find_one())
application = default_app()
httpserver.serve(application, host='127.0.0.1', port=8082)

The problem is due to WSGIServer is a synchronous server, and it is not applicable for high concurrent users sending requests at the same time. In order to bypass these fallbacks, there are a lot of third-party frameworks can be used. Popular among them are Gevent greenlet libraries, Tornado, and CherryPy. All of them are based on Event-driven and asynchronous methodologies, enabling them to handle multiple concurrent users.