Develop Reference

Camel not receiving messages from websocket

I am new to websockets and I created a spring boot application to receive websocket messages using Camel. The websocket server was created using Python. I have tested the server using PieSocket on Chrome and was able to receive the messages. However, I am having trouble receiving messages on the camel application. Below are my codes for both the server and client.
Python websocket server
import asyncio
import websockets
# create handler for each connection
async def handler(websocket, path):
while True:
greeting = f"Hello user!"
await websocket.send(greeting)
print(f"> {greeting}")
await asyncio.sleep(10)
start_server = websockets.serve(handler, "localhost", 8000)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
Java client
package example.org.websocket.route;
import org.apache.camel.builder.RouteBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.stereotype.Component;
#Component
public class DataRoute extends RouteBuilder {
#Override
public void configure() {
from("websocket://localhost:8000")
.log("Received message: ${body}");
}
}
Any help would be useful. Thank you!

Is there a way for communicating between two asyncio loops that are under different threads?

I can't find an option for communicating between two different asyncio event loops that are under a different thread.
when I try to communicate and use resources that are part of a different asyncio event loop I get an error of:
got Future <Future pending> attached to a different loop
I've built a program which does background tasks on the server and also runs a Web sockets server.
Clients are connecting to the Web sockets server and then emitting events for the server to perform.
When the server finish to perform the emitted event he notify all clients and send them the result back.
Diagram that demonstrate my program logic
The server is running on a synchronized function.
The task thread is also on synchronized function.
The web socket server uses an asynchronized function (Websockets library requires it to run on asyncio event loop)
Therefore for now to achieve this goal I'm creating another asyncio event loop under the task thread.
I've created a shared lock between the Websocket server and the Task thread so I wont cause bugs by using the "Send" method at the same time.
The problem is that I'm getting the error of:
got Future <Future pending> attached to a different loop
When I try to use the method of "client.send" even though I'm using a thread lock when calling it.
code for debugging:
Python code
import asyncio
import websockets
import json
import threading
import time
from random import randint
class webSocketServer:
INSTANCE = None
ADDR = "127.0.0.1"
PORT = 7001
def __init__(self):
self.clients = {}
self.loop = asyncio.new_event_loop()
self.lock = asyncio.Lock()
async def addClient(self, client):
async with self.lock:
client.lock = threading.Lock()
self.clients[client.id] = client
return self.clients[client.id]
async def removeClient(self, client):
async with self.lock:
if client.id in self.clients:
del self.clients[client.id]
async def handle_client(self, client):
client = await self.addClient(client)
while True:
try:
packet = await client.recv()
print("Packet received", packet)
except Exception as e:
break
await self.removeClient(client)
def run(self):
server = websockets.serve(self.handle_client, webSocketServer.ADDR, webSocketServer.PORT, loop=self.loop)
self.loop.run_until_complete(server)
self.loop.run_forever()
def notifyClients(webSocketServer):
while True:
for id in webSocketServer.clients:
client = webSocketServer.clients[id]
with client.lock:
loop = asyncio.new_event_loop()
loop.run_until_complete(send_message(client))
print(str(threading.current_thread().ident), "finished!")
time.sleep(randint(1, 3))
async def send_message(client):
print("Lock acquired... trying to send message")
try:
await client.send(str(threading.current_thread().ident)+" say hi")
print("Message sent")
except Exception as e:
print("Error occured", e)
if __name__ == "__main__":
webSocketServer = webSocketServer()
webSocket_thread = threading.Thread(target = webSocketServer.run)
webSocket_thread.start()
print("Running...")
for i in range(3):
t = threading.Thread(target=notifyClients, args=(webSocketServer,))
t.start()
html code
<body>
<script>
function initSocket() {
var socket = new WebSocket('ws://localhost:7001');
socket.addEventListener('open', function(event) {
socket.send("Connection open");
});
socket.addEventListener('message', (event) => {
console.log(event.data);
});
socket.addEventListener("close", function(event) {
setTimeout(function() {
initSocket();
}, 1000); // try to recreate socket in 1 second
});
}
initSocket();
</script>
</body>
</html>
I've thought of creating a queue and make a task on the WebsocketServer that perform tasks from the Queue.
I saw that asyncio have also a Queue option but it again requires an async function to perform it.. so I'll end up in the same situation.
Is there any option of making this communication without relaying on the event loop?
What is the right approach for this kind of situation ?
Is there a way for make a communication between two different event loops on different threads?
--- edit ---
I've added code for debugging above ^
It demonstrate the same environment and basically what I'm trying to achieve.
You can see that the javascript code hangs when receiving a message from the Websocket server...
but if we uses the asyncio event loop from the WebsocketServer handle_client function it will work smooth.
python code:
import asyncio
import websockets
import json
import threading
import time
from random import randint
class webSocketServer:
INSTANCE = None
ADDR = "127.0.0.1"
PORT = 7001
def __init__(self):
self.clients = {}
self.loop = asyncio.new_event_loop()
self.lock = asyncio.Lock()
async def addClient(self, client):
async with self.lock:
client.lock = threading.Lock()
self.clients[client.id] = client
return self.clients[client.id]
async def removeClient(self, client):
async with self.lock:
if client.id in self.clients:
del self.clients[client.id]
async def handle_client(self, client):
client = await self.addClient(client)
while True:
try:
packet = await client.recv()
print("Packet received", packet)
await client.send(packet)
print("Sending !")
time.sleep(2)
except Exception as e:
break
await self.removeClient(client)
def run(self):
server = websockets.serve(self.handle_client, webSocketServer.ADDR, webSocketServer.PORT, loop=self.loop)
self.loop.run_until_complete(server)
self.loop.run_forever()
def notifyClients(webSocketServer):
while True:
for id in webSocketServer.clients:
client = webSocketServer.clients[id]
with client.lock:
loop = asyncio.new_event_loop()
loop.run_until_complete(send_message(client))
print(str(threading.current_thread().ident), "finished!")
time.sleep(randint(1, 3))
async def send_message(client):
print("Lock acquired... trying to send message")
try:
await client.send(str(threading.current_thread().ident)+" say hi")
print("Message sent")
except Exception as e:
print("Error occured", e)
if __name__ == "__main__":
webSocketServer = webSocketServer()
webSocket_thread = threading.Thread(target = webSocketServer.run)
webSocket_thread.start()
print("Running...")
# for i in range(3):
# t = threading.Thread(target=notifyClients, args=(webSocketServer,))
# t.start()
html code
<html>
<body>
<script>
function initSocket() {
var socket = new WebSocket('ws://localhost:7001');
socket.addEventListener('open', function(event) {
socket.send("Connection open");
});
socket.addEventListener('message', (event) => {
console.log(event.data);
socket.send(event.data);
});
socket.addEventListener("close", function(event) {
setTimeout(function() {
initSocket();
}, 1000); // try to recreate socket in 1 second
});
}
initSocket();
</script>
</body>
</html>

python websockets, multiple requests, why only requests are received?

I am trying to test out if I send out multiple requests at the same moment using coroutine can cause the server side receives corrupted data.
My test is based on the sample code from: https://websockets.readthedocs.io/en/stable/intro.html
Somehow, for the following code, the server side only receive one requests? Anyone has some insights? thx
server (this is basically the same code from the websockets Getting Started webpage):
#!/usr/bin/env python3
# WS server example
import asyncio
import websockets
async def hello(websocket, path):
name = await websocket.recv()
print(f"< {name}")
greeting = f"Hello {name}!"
await websocket.send(greeting)
print(f"> {greeting}")
start_server = websockets.serve(hello, "localhost", 8765)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
Client, I created 1000 tasks, and schedule to run them as soon as possible:
#!/usr/bin/env python3
# WS client example
import asyncio
import websockets
uri = "ws://localhost:8765"
connection = None
async def hello():
global connection
name = "What's your name? "
await connection.send(name)
print(f"> {name}")
async def main():
global connection
connection = await websockets.connect(uri)
#asyncio.run(main())
if __name__ == "__main__":
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.run_until_complete(asyncio.wait(
[hello() for i in range(1000)], return_when=asyncio.ALL_COMPLETED
))
UPDATE
The solution is to use a loop.

I found the reason: the server side, handler should use a loop so that the corroutine will not finish immediately after received the first request.

The documentation you linked also includes this paragraph just below the server code:
On the server side, websockets executes the handler coroutine hello once for each WebSocket connection. It closes the connection when the handler coroutine returns.
The client code you linked creates one connection and sends messages on that connection. After the client sends the first message, the server closes the connection, so the next 999 messages you attempt to send are being sent on a closed connection.
If you update the hello handler to include a loop, you will see all messages.
import asyncio
import websockets
async def hello(websocket, path):
while True:
name = await websocket.recv()
print(f"< {name}")
greeting = f"Hello {name}!"
await websocket.send(greeting)
print(f"> {greeting}")
start_server = websockets.serve(hello, "localhost", 8765)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()

Simple Python multi threaded webserver with Asyncio and events called in main function

I have a simple Python program that I want to do three things:
Serve an HTTP document
Serve Websockets
Interact with the Websocket data
I am trying to use / grok asyncio. The issue is that I can't figure out how to access data acquired from a function in the main event loop.
For example in my code below I have two threads.
One thread is the HTTP server thread, one thread is the Websocket server thread and there is the main thread.
What I want to do is to print data captured in the websocket receiving thread in the main thread.
The only way I know how to do this is to use Queues to pass data between threads at which point I do not even know what the advantage of using asyncio is.
Similarly, it feels weird to pass the event loop to the serve_websocket function.
Can anyone please explain how to architect this to get data from the Websocket function into the main function?
It seems like / I want a way to do this without using the threading library at all, which seems possible. In an async project I would want to react to websocket events in different function than where they are called.
NOTE: I know there are other libraries for websockets and http serving with asyncio but this is an example to help me understarnd how to structure projects using this paradigm.
Thanks
#!/usr/bin/env python
import json
import socketserver
import threading
import http.server
import asyncio
import time
import websockets
SERVER_ADDRESS = '127.0.0.1'
HTTP_PORT = 8087
WEBSOCKET_PORT = 5678
def serve_http():
http_handler = http.server.SimpleHTTPRequestHandler
with socketserver.TCPServer(("", HTTP_PORT), http_handler) as httpd:
print(f'HTTP server listening on port {HTTP_PORT}')
httpd.serve_forever()
def serve_websocket(server, event_loop):
print(f'Websocket server listening on port {WEBSOCKET_PORT}')
event_loop.run_until_complete(server)
event_loop.run_forever()
async def ws_callback(websocket, path):
while True:
data = await websocket.recv()
# How do I access parsed_data in the main function below
parsed_data = json.loads(data)
await websocket.send(data)
def main():
event_loop = asyncio.get_event_loop()
ws_server = websockets.serve(ws_callback, SERVER_ADDRESS, WEBSOCKET_PORT)
threading.Thread(target=serve_http, daemon=True).start()
threading.Thread(target=serve_websocket, args=(ws_server, event_loop), daemon=True).start()
try:
while True:
# Keep alive - this is where I want to access the data from ws_callback
# i.e.
# print(data.values)
time.sleep(.01)
except KeyboardInterrupt:
print('Exit called')
if __name__ == '__main__':
main()

I believe that you should not mix asyncio and multithreading without special need. And in your case, use only asyncio tools.
In this case, you have no problem sharing data between coroutines, because they all run on the same thread using cooperative multitasking.
Your code can be rewtitten as:
#!/usr/bin/env python
import json
import socketserver
import threading
import http.server
import asyncio
import time
import websockets
SERVER_ADDRESS = '127.0.0.1'
HTTP_PORT = 8087
WEBSOCKET_PORT = 5678
parsed_data = {}
async def handle_http(reader, writer):
data = await reader.read(100)
message = data.decode()
writer.write(data)
await writer.drain()
writer.close()
async def ws_callback(websocket, path):
global parsed_data
while True:
data = await websocket.recv()
# How do I access parsed_data in the main function below
parsed_data = json.loads(data)
await websocket.send(data)
async def main():
ws_server = await websockets.serve(ws_callback, SERVER_ADDRESS, WEBSOCKET_PORT)
print(f'Websocket server listening on port {WEBSOCKET_PORT}')
http_server = await asyncio.start_server(
handle_http, SERVER_ADDRESS, HTTP_PORT)
print(f'HTTP server listening on port {HTTP_PORT}')
try:
while True:
if parsed_data:
print(parsed_data.values())
await asyncio.sleep(0.1)
except KeyboardInterrupt:
print('Exit called')
if __name__ == '__main__':
asyncio.run(main())

Sharing a variable between different classes in Tornado

I am trying to write a Tornado TCP + HTTP Server application.
My use case is a Tornado TCP + HTTP Server application which accepts data from a TCP client and pass the data to display it on a webpage hosted on the HTTP server.
Here is my tornado server code:
#!/usr/bin/env python
import os.path
import tornado.httpserver
import tornado.web
import logging
from tornado.ioloop import IOLoop
from tornado import gen
from tornado.iostream import StreamClosedError
from tornado.tcpserver import TCPServer
from tornado.options import options, define
define("port", default=6642, help="TCP port to listen on")
logger = logging.getLogger(__name__)
test = {}
class IndexHandler(tornado.web.RequestHandler):
def get(self):
global test
self.render('index.html', test=test)
class EchoServer(TCPServer):
#gen.coroutine
def handle_stream(self, stream, address):
global test
while True:
try:
test = yield stream.read_until("\n")
logger.info("Received bytes: %s", test)
except StreamClosedError:
logger.warning("Lost client at host %s", address[0])
break
except Exception as e:
print(e)
if __name__ == "__main__":
options.parse_command_line()
app = tornado.web.Application( handlers=[
(r'/', IndexHandler)],
static_path=os.path.join(os.path.dirname(__file__), "static"),
template_path=os.path.join(os.path.dirname(__file__), "templates"))
http_server = tornado.httpserver.HTTPServer(app)
http_server.listen(options.port)
server = EchoServer()
server.listen(6641)
logger.info("Listening on TCP port %d",6641)
IOLoop.current().start()
Here is the python client code :
# echo_client.py
import socket
import time
counter = 0
host = '192.168.43.59'
port = 6641 # The same port as used by the server
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host, port))
while True:
s.sendall("s\n")
counter = counter + 1
time.sleep(5)
I want to pass the data received from TCP client application into the variable "test" to the render template to display in index.html webpage but I am getting no data displayed.
I am using the global variable concept but no success since couldn't pass the updated "test" variable to index.html page.
If anyone could throw light on using common vaiables across different classes or handlers would help me.
Javascript file iam using is this :
/* test.js */
var test = ""
function set_test(val)
{
test=val
}
function show_test()
{
alert(test);
}
The HTML template used is this :
<!DOCTYPE html>
<html>
<meta http-equiv="refresh" content="30" />
<head>
<title>Test</title>
<script src="{{ static_url('scripts/test.js') }}"
type="application/javascript"></script>
</head>
<body>
<input type="button" onclick="show_test()" value="alert" />
<script type="application/javascript">
set_test("{{test}}");
</script>
</body>
</html>
Hi xyres,
Thank you for your spontaneous reply.I went through the link provided by you and after going through it i could understand that q.get() and q.put() can be used to store and retrieve data as you said.But i could not after modifying the tornado server code in the following manner I couldn't receive the data from TCP client , before this i could at least get data from TCP client .Can you let me know what is the mistake i have done in queue implementation
Here is my tornado server code:
#!/usr/bin/env python
import os.path
import tornado.httpserver
import tornado.web
import logging
from tornado.ioloop import IOLoop
from tornado import gen
from tornado.iostream import StreamClosedError
from tornado.tcpserver import TCPServer
from tornado.options import options, define
define("port", default=6642, help="TCP port to listen on")
logger = logging.getLogger(__name__)
#test = {}
q = Queue(maxsize=2)
class IndexHandler(tornado.web.RequestHandler):
def get(self):
#global test
test = yield q.get
self.render('index.html', test=test)
class EchoServer(TCPServer):
#gen.coroutine
def handle_stream(self, stream, address):
#global test
yield q.put(test)
yield q.join()
while True:
try:
test = yield stream.read_until("\n")
logger.info("Received bytes: %s", test)
except StreamClosedError:
logger.warning("Lost client at host %s", address[0])
break
except Exception as e:
print(e)
if __name__ == "__main__":
options.parse_command_line()
app = tornado.web.Application( handlers=[
(r'/', IndexHandler)],
static_path=os.path.join(os.path.dirname(__file__), "static"),
template_path=os.path.join(os.path.dirname(__file__), "templates"))
http_server = tornado.httpserver.HTTPServer(app)
http_server.listen(options.port)
server = EchoServer()
server.listen(6641)
logger.info("Listening on TCP port %d",6641)
IOLoop.current().start()
As per the tornado documentation it seems that queue can be applied to coroutines and here iam trying to replicate the same to two different classes.Is that a mistake ..Iam new to tornado so please bear my silly questions ..

You've a multiple options:
If you want to have a long-running connection, for example, if a client sends a request to IndexHandler and you want the client to wait until a message is in the queue, you can convert your handler to a coroutine.
If you want to return the response immediately, regardless of the availability of the data in the queue, you can use a queue's get_nowait() method.
Example for case #1:
from tornado.queues import Queue
q = Queue()
class IndexHandler(tornado.web.RequestHandler):
#gen.coroutine
def get(self):
self.data_future = q.get()
data = yield self.data_future
self.render('index.html', data=data)
def on_connection_close(self):
# set an empty result on the future
# if connection is closed so that
# the messages don't get removed from
# the queue unnecessariliy for
# closed connections
self.msg_future.set_result(None)
Example for case #2:
from tornado.queues import Queue, QueueEmpty
q = Queue()
def get(self):
try:
data = q.get_nowait()
except QueueEmpty:
data = None
self.render(...)