I am attempting to pass binary data over websockets, more specifically compressed strings over websockets. In my current setup I use tornado as the server with a websocket client transmitting the binary data. The binary data is formed by compressing the data with zlib. Both client and server are as simple as they get and are shown below.
Server:
import tornado.websocket
import tornado.httpserver
import tornado.ioloop
import tornado.web
class WebSocketServer(tornado.websocket.WebSocketHandler):
def open(self):
print 'OPEN'
def on_message(self, message):
print 'len = {}'.format(len(message))
print 'GOT MESSAGE: {}'.format(message.decode('zlib'))
def on_close(self):
print 'CLOSE'
app = tornado.web.Application([
(r'/', WebSocketServer)
])
http_server = tornado.httpserver.HTTPServer(app)
http_server.listen(9500)
tornado.ioloop.IOLoop.instance().start()
Client:
import websocket
host = 'localhost'
port_ws = 9500
ws = websocket.create_connection('ws://{}:{}/'.format(host, port_ws))
message = 'this is my message'.encode('zlib')
print 'Length of message is {}'.format(len(message))
ws.send(message)
The client does not throw any errors, it prints out that the message: Length of message is 24. The message is encoded as a str as per the zlib standard. The server on the other end does not show that it received any messages, it just understands that a client had connected, and then disconnected. Does anyone know where the problem is? I am not sure if the problem lays within tornado or the websockets library. Any suggestions?
EDIT: In response to the comment below (#plg), I modified the scripts above to show that:
Non-encoded messages can be send from client to the tornado server
Tornado can reply with an encoded message
Server:
import tornado.websocket
import tornado.httpserver
import tornado.ioloop
import tornado.web
class WebSocketServer(tornado.websocket.WebSocketHandler):
def open(self):
print 'OPEN'
def on_message(self, message):
print 'len = {}'.format(len(message))
print 'GOT MESSAGE: {}'.format(message)
self.write_message(message.encode('zlib'))
def on_close(self):
print 'CLOSE'
app = tornado.web.Application([
(r'/', WebSocketServer)
])
http_server = tornado.httpserver.HTTPServer(app)
http_server.listen(9500)
tornado.ioloop.IOLoop.instance().start()
Client:
import websocket
host = 'localhost'
port_ws = 9500
ws = websocket.create_connection('ws://{}:{}/'.format(host, port_ws))
#message = 'this is my message'.encode('zlib')
message = 'this is my message'
print 'Length of message is {}'.format(len(message))
ws.send(message)
assert ws.recv().decode('zlib') == message
The system works just fine. The assert does not throw an error. The decoded message matches the send message. So I guess there is a problem with either:
Sending an encoded message from the client
Tornado receiving encoded messages
To be quite honest, I do believe that the first option is more probable than tornado. In my opinion, I believe tornado would alert me if an incoming message is not properly decoded as per the websocket standard. Any more suggestions?
EDIT: More development on who is at fault. Instead of using my own server to relay back and fourth my connection, I relayed the connection to ws://echo.websocket.org/. My testing application is as shows:
import websocket
host = 'localhost'
port_ws = 9500
ws = websocket.create_connection('ws://echo.websocket.org/')
message = 'this is my message'
ws.send(message.encode('zlib'))
got = ws.recv().decode('zlib')
print 'GOT: {}'.format(got)
assert got == message
This actually passed the test, the data was received just fine. So I guess there is something wrong with tornado receiving the data?
After looking though the source code of the websocket library, I found that by default it is formatting the packets as text. By changing the line:
ws.send('message')
# to:
ws.send('message', opcode=websocket.ABNF.OPCODE_BINARY)
# or better yet:
ws.send_binary('message')
The packet will be sent over just fine. Tornado I guess was just ignoring the fake binary packets since they were marked as text and contained binary.
Thanks to this commit tornado supports websocket compression extensions.
Related
I have Python client which opens a websocket connection to a server and subscribes to particular topic using STOMP protocol, subscription goes just fine as i see on the server all is fine. However, When the server publishes a few messages the client does not receive any.
Here are the codes used:
Client
# coding: utf-8
import websocket
import stomp
import stomper
token = "eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhZG1pbiIsInByaW5jaXBhbF9uYW1lIjoiYWRtaW4iLCJpc3MiOiJBdGhlbmEiLCJ1c2VydHlwZSI6IkxPQ0FMIiwiYW9zX3ZlcnNpb24iOiJldXBocmF0ZXMtNS4xMS1zdGFibGUiLCJyZWdpb24iOiJlbi1VUyIsImV4cCI6MTczNDI4MDI3NywidXVpZCI6ImI4MzhjOGRkLWI4NmQtNGNkZS05ZTE4LTUxM2E1OTk4ODhhYyIsImlhdCI6MTU3NjYwMDI3NywiYXV0aG9yaXRpZXMiOiJST0xFX0NMVVNURVJfQURNSU4sUk9MRV9NVUxUSUNMVVNURVJfQURNSU4sUk9MRV9VU0VSX0FETUlOLFJPTEVfQ0xVU1RFUl9WSUVXRVIiLCJqdGkiOiI1NTU1ZjEwZC04NGQ5LTRkZGYtOThhNC1mZmI1OTM1ZTQwZWEifQ.LOMX6ppkcSBBS_UwW9Qo2ieWZAGrKqADQL6ZQuTi2oieYa_LzykNiGMWMYXY-uw40bixDcE-aVWyrIEZQbVsvA"
headers = {"Authorization": "Bearer " + token}
uri = "ws://127.0.0.1:8765/notifications/websocket"
def on_msg(ws, msg):
print(msg)
def on_error(ws, err):
print(err)
def on_closed(ws):
print("#Closed#")
def on_open(ws):
sub = stomper.subscribe("/user/queue/alert", "MyuniqueId", ack="auto")
ws.send(sub)
headers = {"Authorization": "Bearer " + token}
websocket.enableTrace(True)
ws = websocket.WebSocketApp(uri, header=headers, on_message=on_msg, on_error=on_error, on_close=on_closed)
ws.on_open = on_open
ws.run_forever()
Code server uses to publish the message:
for (WatchesSubscription s : subscriptions) {
template.convertAndSendToUser(s.getSession().getUser(), destination, dto);
}
When i checked out the value of the above variables i saw that destination was as expected queue/alerts. I have java client to test out as well and it works just fine. I have even tried this by subscribing to /topic/alerts and sending to it via template.convertAndSend(/topic/alerts), here too i received nothing. I am a drawing a complete blank on this and would appreciate any sort of help!
After many days of hair pulling I finally figured out the reason and the fix!
The java client I used was
WebSocketStompClient stompClient = new WebSocketStompClient(transport);.The stompClient.connect(URL, webSocketHttpHeaders, sessionHandler); method implicitly sends a stomp CONNECT\n\n\x00\n
The Springboot server which has been configured for STOMP understands this as a connection request and responds with a CONNECT_ACK.
When this ACK is sent it also updates it's local UserRegistry with the new user. So the internal message broker knows that there is a user who has subscribed to so-and-so topic.
In my Python code, i had merely opened a Websocket connection and after that directly sent a SUBSCRIBE message. So the broker never got a CONNECT so the user was never stored! This resulted in the messages later on being published to be merely discarded by the broker.
The fix was to send a CONNECT\n\n\x00\n after opening up the connection and before the subscription. Here is the code:
def on_open(ws):
#The magic happens here!
ws.send("CONNECT\naccept-version:1.0,1.1,2.0\n\n\x00\n")
sub = stomper.subscribe("/user/queue/alert", "MyuniqueId", ack="auto")
ws.send(sub)
I am trying to send messages on TCP/IP all on host machine. This is working, although for some reason the socket needs to be re-instantiated for every new message on the client side only. For example here is a basic client that sends three separate messages:
import socket
host = '127.0.0.1'
class Client:
def __init__(self):
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def connect(self):
self.sock.connect((host,12347))
def send(self,message):
self.sock.sendall(message)
def close(self):
self.sock.close()
if __name__ == "__main__":
message1 = "I am message 1"
message2 = "I am message 2"
message3 = "I am message 3"
#exp = Client()
#exp.connect()
for i in range(0,3):
try:
exp = Client()
exp.connect()
if i == 0:
txt = message1
elif i == 1:
txt = message2
elif i == 2:
txt = message3
exp.send(txt)
exp.close()
print i
exp.send(txt)
except:
pass
and the server that receives:
#!/usr/bin/env python
import socket
class communication:
def __init__(self):
try:
host = '127.0.0.1'
self.Server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.Server.bind((host,12347))
self.Server.listen(1)
finally:
print "setup finished"
def recieve(self):
(connection, client_address) = self.Server.accept()
data = connection.recv(128)
return data
def close(self):
self.server.close()
if __name__ == "__main__":
exp = communication()
while True:
try:
(connection,client_address) = exp.Server.accept()
message = connection.recv(128)
finally:
print message
if message == "I am message 3":
exp.close()
You see how I re-call the Client class in each iteration of the for loop. This seems to be necessary for sending messages 2 and 3. If the socket is instantiated only once at the start of the main code along with the connect() function, then the server hangs on the recv() after the first message has been sent.
I can't understand why this is happening and the socket only needs to be setup once on the server side. I am doing something wrong, or is this normal?
Thanks!
It's even worse than you think. Take a look at your server code. exp.Server.accept() accepts a connection from the client, but connection.receive() ignores that connection completely and does a second self.Server.accept(). You ignore half of your connections!
Next, your server only does a single receive.... Even if you tried to send more messages on the connection, the server would ignore them.
But you can't just add a recv loop. Your client and server need some way to mark message boundaries so the server knows how to pull them out. Some text based systems use a new line. Others send a message size or fixed size header that the server can read. HTTP for example uses a combination of new lines and data count.
If you want to learn sockets from the ground up just know that they are complicated and you'll need to study. There are lots of ways to build a server and you'll need to understand the trade-offs. Otherwise, there are many frameworks from XMLRPC to zeromq that do some of the heavy lifting for you.
I have a simple TCP server set up in Python which posts data to another server when it gets a request and returns the data it receives from the client back to the client. After a while the server stops receiving any requests and quietly dies.
import SocketServer
import requests
import time
class TCPHandler(SocketServer.BaseRequestHandler):
def handle(self):
print self.client_address
self.data = self.request.recv(1024).strip()
print (time.strftime("%d/%m/%Y %H:%M:%S "))
params = {
'hello': 'world'
}
requests.post('http://website.co.uk/test',data=params)
self.request.sendall(self.data.upper())
def finish(self):
print 'end request'
if __name__ == "__main__":
HOST,PORT = "192.168.2.211",343
server = SocketServer.TCPServer((HOST,PORT),TCPHandler)
server.timeout = None
server.serve_forever()
I'm just wondering if anyone knows what might be causing the server to stop receiving requests.
Thanks!
I have simple websocket server, which echoes all the message back to the client.
import gevent
from geventwebsocket.resource import WebSocketApplication
from geventwebsocket.server import WebSocketServer
from geventwebsocket.resource import Resource
import ams_pb2
class AMSWebSocketServer(WebSocketApplication):
def __init__(self, ws):
super(AMSWebSocketServer, self).__init__(ws)
pass
def on_open(self):
pass
def on_message(self, message):
print 'received message'
print message
if message is None:
print 'message none'
return
print 'echo message back'
self.ws.send(message)
def on_close(self, reason):
print "connection closed"
gevent.sleep(0)
resource = Resource({'/': AMSWebSocketServer})
The server is spawned using gunicorn command
gunicorn -k "geventwebsocket.gunicorn.workers.GeventWebSocketWorker" -b 127.0.0.1:9000 gunicorn_test:resource
I have a test client, which sends the websocket message to be echoed back
from ws4py.client.threadedclient import WebSocketClient
import ams_pb2
class DummySwitch(WebSocketClient):
def closed(self, code, reason=None):
pass
def received_message(self, msg):
if msg is None:
print 'none'
return
print 'received message'
ams_message = ams_pb2.AMSConfig()
ams_message.ParseFromString(msg)
print ams_message
print msg
if __name__ == '__main__':
end_point = 'ws://127.0.0.1:9000'
client = DummySwitch(
end_point,
headers=[
]
)
client.connect()
print 'sending message'
AMSConfig = ams_pb2.AMSConfig()
AMSConfig.CliConfig = True
print AMSConfig
msg = AMSConfig.SerializeToString()
#msg = 'Hello'
print msg
client.send(msg)
client.run_forever()
My protobuff file is :
package ams;
message AMSConfig {
optional bool CliConfig = 1;
}
Whenever my client send a protobuff message to the server, i am able to see it getting parsed in the server, but when the server echoes back the same message to the client, the client fails due to:
File "client_test.py", line 15, in received_message
ams_message.ParseFromString(msg)
File "/usr/lib/python2.6/site-packages/google/protobuf/message.py", line 186, in ParseFromString
self.MergeFromString(serialized)
File "/usr/lib/python2.6/site-packages/google/protobuf/internal/python_message.py", line 847, in MergeFromString
raise message_mod.DecodeError('Truncated message.')
DecodeError: Truncated message.
So, i modified the code to send a simple string and i see that the 'Hello' string being sent to the server is being echoed back and client is able to print the message. But, the client fails to parse protobuff message echoed back.
I am unable to understand why the echo back for a string works but for a protocol buffer it doesn't work in my example.
Thanks for help.
I encounter same problem on client side, but this might be solved by enlarge receiving buffer size in client side code. For example:
maxpkglen = 1024 * 1024 // original maxpkglen = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.sendto(req, (server, port))
rsp, server = s.recvfrom(maxpkglen)
s.close()
i also encounter problem like this in my client program, because i set the receiving buffer to a hard number 1024. and when i enlarge the receiving buffer size to 10*1024, it is solved. also thanks for #lulyon 's tip.
you may need a loop for reading all the data from the peer code!
I had this problem but for me the answer was to not use the msg variable directly from the received_message(self, msg), but rather use msg.data to actually get the data associated with the message and parse from that.
I am running a very simple python (3.x) client-server program (both locally on my PC) for a school project (not intended for the real world) which just sends messages back-and-forth (like view customers, add customer, delete customer, etc... real basic).
Sometimes the data can be multiple records which I had stored as namedTuples (just made sense) and then went down the path of using Pickle to transfer then.
So for example on the client I do something like this:
s.send(message.encode('utf-8'))
pickledResponse = s.recv(4096);
response = pickle.loads(pickledResponse)
Now ever so often I get the following error:
response = pickle.loads(pickledResponse)
EOFError: Ran out of input
My fear is that this has something to do with my socket (TCP) transfer and maybe somehow I am not getting all the data in time for my pickle.loads - make sense? If not I am really lost as to why this would be happening so inconsistently.
However, even if I am right I am not sure how to fix it (quickly), I was considering dropping pickle and just using strings (but couldn't this suffer from the same fate)? Does anyone have any suggestions?
Really my message are pretty basic - usually just a command and some small data like "1=John" which means command (1) which is FIND command and then "John" and it returns the record (name, age, etc...) of John (as a namedTuple - but honestly this isn't mandatory).
Any suggestions or help would be much appreciated, looking for a quick fix...
The problem with your code is that recv(4096), when used on a TCP socket, might return different amount of data from what you might have expected, as they are sliced at packet boundaries.
The easy solution is to prefix each message with length; for sending like
import struct
packet = pickle.dumps(foo)
length = struct.pack('!I', len(packet)
packet = length + packet
then for receiving
import struct
buf = b''
while len(buf) < 4:
buf += socket.recv(4 - len(buf))
length = struct.unpack('!I', buf)[0]
# now recv until at least length bytes are received,
# then slice length first bytes and decode.
However, Python standard library already has a support for message oriented pickling socket, namely multiprocessing.Connection, that supports sending and receiving pickles with ease using the Connection.send and Connection.recv respectively.
Thus you can code your server as
from multiprocessing.connection import Listener
PORT = 1234
server_sock = Listener(('localhost', PORT))
conn = server_sock.accept()
unpickled_data = conn.recv()
and client as
from multiprocessing.connection import Client
client = Client(('localhost', 1234))
client.send(['hello', 'world'])
For receiving everything the server sends until it closes its side of the connection try this:
import json
import socket
from functools import partial
def main():
message = 'Test'
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.connect(('127.0.0.1', 9999))
sock.sendall(message.encode('utf-8'))
sock.shutdown(socket.SHUT_WR)
json_response = b''.join(iter(partial(sock.recv, 4096), b''))
response = json.loads(json_response.decode('utf-8'))
print(response)
if __name__ == '__main__':
main()
I've used sendall() because send() has the same ”problem” as recv(): It's not guaranteed everything is sent. send() returns the number of bytes actually sent, and the programmer has to make sure that matches the length of the argument and if not to send the rest until everything is out. After sending the writing side of the connection is closed (shutdown()) so the server knows there is no more data coming from the client. After that, all data from the server is received until the server closes its side of the connection, resulting in the empty bytes object returned from the recv() call.
Here is a suitable socketserver.TCPServer for the client:
import json
from socketserver import StreamRequestHandler, TCPServer
class Handler(StreamRequestHandler):
def handle(self):
print('Handle request...')
message = self.rfile.read().decode('utf-8')
print('Received message:', message)
self.wfile.write(
json.dumps(
{'name': 'John', 'age': 42, 'message': message}
).encode('utf-8')
)
print('Finished request.')
def main():
address = ('127.0.0.1', 9999)
try:
print('Start server at', address, '...')
server = TCPServer(address, Handler)
server.serve_forever()
except KeyboardInterrupt:
print('Stopping server...')
if __name__ == '__main__':
main()
It reads the complete data from the client and puts it into a JSON encoded response with some other, fixed items. Instead of the low level socket operations it makes use of the more convenient file like objects the TCPServer offers for reading and writing from/to the connection. The connection is closed by the TCPServer after the handle() method finished.