Develop Reference

How to detect gRPC server is down from gRPC AIO python client

I've been facing this issue where I have a gRPC AIO python client sending bunch of configuration changes to the gRPC server, though it's a bi-directional RPC, client is not expecting any message from gRPC server. So whenever there is a configuration change the client sends gRPC message containing the configuration. It keeps the channel open and it's not closed (not calling done_writing()) . But when it doesn't have anything to send, it polls a queue for any new message in a tight loop. During that time if the server goes down, the client is not able to detect that. But, as soon as some data is available in the queue, while pushing it to server, client is able to detect that server went down(exception thrown).
How to detect server went down while there is no data to send and client is waiting for data availability. Is there any gRPC API I can call on the channel while the client is waiting for data to send, in order to detect channel failure(basically if there is any api call which throws exception while the server went down is also good for me, I was not able to find any useful API)? I tried gRPC keepalive, but it didn't work for my scenario.
async with grpc.aio.insecure_channel('127.0.0.1:51234') as channel:
stream = hello_pb2.HelloStub(channel).Configure()
await stream.wait_for_connection()
while True:
if queue.empty():
continue
if not queue.empty():
item = queue.get()
await asyncio.sleep(0.001)
queue.task_done()
await stream.write(item)
await asyncio.sleep(0.01)
await stream.done_writing()
I tried to enable gRPS keepalive while forming insecure_channel. It didn't have desired effect.
Subsequently I tried calling channel_ready() inside the tight loop during queue empty, was expecting to throw some exception and come out of that loop, but it didn't work.
async with grpc.aio.insecure_channel('127.0.0.1:51234',
options = [
('grpc.keepalive_time_ms', 60000),
('grpc.keepalive_timeout_ms', 600000),
('grpc.keepalive_permit_without_calls', 1),
('grpc.http2.max_pings_without_data', 0),
('grpc.http2.min_time_between_pings_ms', 10000),
('grpc.http2.min_ping_interval_without_data_ms', 60000),
('grpc.max_connection_age_ms', 2147483647),
]) as channel:

I was able to solve it using channel.get_state(). Below is the code snippet
if queue.empty():
if channel.get_state() != grpc.ChannelConnectivity.READY:
break
time.sleep(5)

Python - Async callback/receiver for WebSocket

I am trying to implement WebSocket connection to a server (Python app <=> Django app)
Whole system runs in big Asyncio loop with many tasks. Code snippet is just very small testing part.
I am able to send any data to a server at any moment and many of them will be type request something and wait for response. But I would like to have some "always running" handler for all incoming messages. (When something in Django database will change I want to send changes to python app).
How can Include always running receiver/ or add callback to websocket? I am not able to find any solution for this.
My code snippet:
import asyncio, json, websockets, logging
class UpdateConnection:
async def connect(self,botName):
self.sock = await websockets.connect('ws://localhost:8000/updates/bot/'+botName)
async def send(self,data):
try:
await self.sock.send(json.dumps(data))
except:
logging.info("Websocket connection lost!")
# Find a way how to reconenct... or make socket reconnect automatically
if __name__ == '__main__':
async def DebugLoop(socketCon):
await socketCon.connect("dev")
print("Running..")
while True:
data = {"type": "debug"}
await socketCon.send(data)
await asyncio.sleep(1)
uSocket = UpdateConnection()
loop = asyncio.get_event_loop()
loop.create_task(DebugLoop(uSocket))
loop.run_forever()
My debug server after connection will start sending random messages to the client in random intervals and I would like to somehow handle them in async way.
Thanks for any help :)

You don't have to do it so complicated. First of all I suggest you use the context patterns offered by websockets module.
From the documentation:
connect() can be used as an infinite asynchronous iterator to reconnect automatically on errors:
async for websocket in websockets.connect(...):
try:
...
except websockets.ConnectionClosed:
continue
Additionally, you simply keep the connection alive by awaiting incoming messages:
my_websocket = None
async for websocket in websockets.connect('ws://localhost:8000/updates/bot/' + botName):
try:
my_websocket = websocket
async for message in websocket:
pass # here you could also process incoming messages
except websockets.ConnectionClosed:
my_websocket = None
continue
As you can see we have a nested loop here:
The outer loop constantly reconnects to the server
The inner loop processes one incoming message at a time
If you are connected, and no messages are coming in from the server, this will just sleep.
The other thing that happens here is that my_websocket is set to the active connection, and unset again when the connection is lost.
In other parts of your script you can use my_websocket to send data. Note that you will need to check if it is currently set wherever you use it:
async def send(data):
if my_websocket:
await my_websocket.send(json.dumps(data))
This is just an illustration, you can also keep the websocket object as an object member, or pass it to another component through a setter function, etc.

python/ websocket implementation of sending messages to the client that keeps the socket open

Help me figure out how to implement it correctly.
The bottom line: The client is charging stations that connect, open a socket and send messages to the server and receive responses.
Server - listens to the port, sees the connected station, receives messages and sends responses to them.
Question: When the client connects and sends headers, I can send a message to the client. But I need to periodically send messages to the client that keeps the socket open, I don't understand how to implement this. Can someone tell me?
sample sending code:
charge_point_id = path.strip('/')
cp = client_main(charge_point_id, websocket)
logging.info(charge_point_id)
print(charge_point_id)
print(path)
await websocket.send(json.dumps([2,"222", "GetLocalListVersion", {}]))
await cp.start()
example of receiving a message from a client:
class client_main(cp):
errors = False
if not errors:
#on('BootNotification')
def on_boot_notitication(self, charge_point_vendor, charge_point_model,charge_point_serial_number,firmware_version,
meter_type, **kwargs):
return call_result.BootNotificationPayload(
status="Accepted",
current_time=date_str.replace('+00:00','Z'),
interval=60
)
in this case, the charging station according to the ocpp protocol opens the connection and keeps it open, it should be possible to somehow write to him
how do i send a message to the client? My example:
#on('Heartbeat')
def on_getlocallistversion(self):
await self.route_message(json.dumps([2,"222","GetLocalListVersion",{}]))
def on_hearbeat(self):
return call_result.HeartbeatPayload(
current_time=datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%S')+"Z"
)
I get an error:
await self.route_message(json.dumps([2,"222",
"GetLocalListVersion",{}]))

This is impossible to do. You can send a message only when the client is connected.

1006 Connection closed abnormally error with python 3.7 websockets

I'm having the same problem as this github issue with python websockets:
https://github.com/aaugustin/websockets/issues/367
The proposed solution isn't working for me though. The error I'm getting is:
websockets.exceptions.ConnectionClosed: WebSocket connection is closed: code = 1006 (connection closed abnormally [internal]), no reason
This is my code:
async def get_order_book(symbol):
with open('test.csv', 'a+') as csvfile:
csvw = csv.writer(csvfile, delimiter=',', quotechar='|', quoting=csv.QUOTE_MINIMAL)
DT = Data(data=data, last_OB_id=ob_id, last_TR_id=tr_id, sz=10, csvw=csvw)
while True:
if not websocket.open:
print('Reconnecting')
websocket = await websockets.connect(ws_url)
else:
resp = await websocket.recv()
update = ujson.loads(resp)
DT.update_data(update)
async def get_order_books():
r = requests.get(url='https://api.binance.com/api/v1/ticker/24hr')
await asyncio.gather(*[get_order_book(data['symbol']) for data in r.json()])
if __name__ == '__main__':
asyncio.run(get_order_books())
The way I've been testing it is by closing my internet connection, but after a ten second delay it still returns the 1006 error.
I'm running Python 3.7 and Websockets 7.0.
Let me know what your thoughts are, thanks!

I encountered the same problem.
After digging a while I found multiple versions of the answer that tells to just reconnect, but I didn't think it was a reasonable route, so I dug some more.
Enabling DEBUG level logging I found out that python websockets default to sending ping packets, and failing to receive a response, timeouts the connection. I am not sure if this lines up with the standard, but at least javascript websockets are completely fine with the server my python script times out with.
The fix is simple: add another kw argument to connect:
websockets.connect(uri, ping_interval=None)
The same argument should also work for server side function serve.
More info at https://websockets.readthedocs.io/en/stable/api.html

So I found the solution:
When the connection closes, it breaks out of the while loop for some reason. So in order to keep the websocket running you have to surround
resp = await websocket.recv()
with try ... except and have
print('Reconnecting')
websocket = await websockets.connect(ws_url)
in the exception handling part.

I ran in to this same issue. The solution by shinola worked for awhile, but I would still get errors sometimes.
To handle this I put the connection into a while True: loop and added two separate try except blocks. The consumer variable is a function that processes the messages received from the websocket connection.
async def websocketConnect(uri, payload, consumer):
websocket = await websockets.connect(uri, ssl=True)
await websocket.send(json.dumps(payload))
while True:
if not websocket.open:
try:
print('Websocket is NOT connected. Reconnecting...')
websocket = await websockets.connect(uri, ssl=True)
await websocket.send(json.dumps(payload))
except:
print('Unable to reconnect, trying again.')
try:
async for message in websocket:
if message is not None:
consumer(json.loads(message))
except:
print('Error receiving message from websocket.')
I start the connection using:
def startWebsocket(uri, payload, consumer):
asyncio.run(websocketConnect(uri, payload, consumer))

I might be a year late but i was just having this issue. No connection issues on my html5 websocket client but .py test client would crash after around a minute (raising 1006 exceptions for both the client and server too). As a test i started to just await connection.recv()ing after every frame the client sends. No more issues. I didnt need to receive data for my .py test client but apparently it causes issues if you let it build up. It's also probably why my web version was working fine, since i was handling the .onmessage callbacks.
Im pretty sure this is why this error occurs. So this solution of just receiving the data is an actual solution and not disabling pinging and screwing up a vital function of the protocol.

I think they explained here: https://websockets.readthedocs.io/en/stable/faq.html
it means that the TCP connection was lost. As a consequence, the
WebSocket connection was closed without receiving a close frame, which
is abnormal.
You can catch and handle ConnectionClosed to prevent it from being
logged.
There are several reasons why long-lived connections may be lost:
End-user devices tend to lose network connectivity often and
unpredictably because they can move out of wireless network coverage,
get unplugged from a wired network, enter airplane mode, be put to
sleep, etc.
HTTP load balancers or proxies that aren’t configured for long-lived
connections may terminate connections after a short amount of time,
usually 30 seconds.

I solved this problem by uvicorn the table with a hypercorn.
hypercorn app:app

Sending and receiving frames over the same websocket connection without blocking

Sorry for the long post but I've been poking at this for over a week so I've tried a lot of different stuff. I know Python well enough but I don't have any experience with asyncio or non-blocking functions in Python.
I'm writing an API library/module/package/whatever for a web service that requires a websocket connection. There are many incoming messages to act on, and some control-related (web app level, not websocket control messages) that I need to send on occasion. I can easily receive messages over the connection and act on them. I can send messages, but only in response to received messages because the receive loop is always blocking waiting for messages. I don't want to wait for an incoming messages to process an outgoing one so the script doesn't have to hang on input until a new messages is received. In my struggles to get two-way communication working as desired I discovered I need to use something like Twisted, Tornado, or asyncio but so far every implementation I've tried has failed. Note that the sending has to happen over the same connection. Opening a short-lived connection outside of the receive loop will not work. Here's what I've done so far:
The first iteration of the websocket code was using the websocket-client package. It was very close to the example from the docs:
import websocket
try:
import thread
except ImportError:
import _thread as thread
import time
def on_message(ws, message):
# Send message frames to respective functions
# for sorting, objectification, and processing
def on_error(ws, error):
print(error)
def on_close(ws):
print("### closed ###")
def on_open(ws):
def run(*args):
# Send initial frames required for server to send the desired frames
thread.start_new_thread(run, ())
if __name__ == "__main__":
websocket.enableTrace(True)
ws = websocket.WebSocketApp(buildWebsocketURL()),
on_message = on_message,
on_error = on_error,
on_close = on_close)
ws.on_open = on_open
ws.run_forever()
This blocks any further execution outside of the loop. I tried learning up on the _thread module but I couldn't find any indication that I could "communicate" with the websocket thread from outside. I tried setting up a pub/sub listener function that would forward data to ws.send() from another sender function but it didn't work. No errors or anything, just no indication of any sent messages.
Next I tried the Websockets module. This one seems to be built from the ground up to utilize asyncio. Again, I got a client build that would send initial messages and act on received messages but the progress stopped there:
async def wsconnection():
async with websockets.connect(getWebsocketURL()) as websocket:
while True:
message = await websocket.recv()
if message == '{"type":"broadcaster.ready"}':
subscriptions = getSubscriptions() # Get subscriptions from ident data
logging.info('Sending bookmarks to server as subscription keys')
subscriptionupdate = '{{"type": "subscribe","subscription_keys": ["{0}"],"subscription_scope": "update"}}'.format(
'","'.join(subscriptions))
subscriptioncontent = '{{"subscription_keys": ["{0}"],"subscription_scope": "content","type": "subscribe"}}'.format(
'","'.join(subscriptions))
logging.debug(subscriptioncontent)
await websocket.send(subscriptionupdate)
await websocket.send(subscriptioncontent)
await websocket.send(
'{"type":"message_lobby.read","lobby_id":"1","message_id:"16256829"}')
sortframe(message)
asyncio.get_event_loop().run_until_complete(wsconnection())
I tried the aforementioned pub/sub listener applied here to no avail. Upon reading the docs for this module more thoroughly I tried getting the websocket protocol object (that contains the send() and recv() methods) outside of the loop then creating two coroutines(?), one listening for incoming messages and one listening for and sending outgoing messages. So far I've been completely unable to get the websocket protocol object without running the async with websockets.connect(getWebsocketURL()) as websocket: line within the scope of the wsconnection() function. I tried using websocket = websockets.client.connect() which according to the docs I thought would set the protocol object I need but it doesn't. All of the examples I can find don't seem to reveal any apparent way to structure the websockets sender and receiver in the way I require without extensive knowledge of asyncio.
I also poked around with autobahn with similar code structures as above using both asyncio and Twisted but I came up with all the same problems as above.
So far the closest I've gotten was with the Websockets package above. The docs have an example snippet for a send/recv connection but I can't really read what's going on there as it's all very specific to asyncio. I'm really having trouble wrapping my head around asyncio in general and I think a big problem is it seems to have very rapidly evolved recently so there is a ton of very version-specific information floating around that conflicts. Not good for learning, unfortunately. ~~~~This is what I tried using that example and it connects, receives initial messages, then the connection is lost/closed:
async def producer(message):
print('Sending message')
async def consumer_handler(websocket, path):
while True:
message = await websocket.recv()
await print(message)
await pub.sendMessage('sender', message)
async def producer_handler(websocket, path):
while True:
message = await producer()
await websocket.send(message)
async def wsconnect():
async with websockets.connect(getWebsocketURL()) as websocket:
path = "443"
async def handler(websocket, path):
consumer_task = asyncio.ensure_future(
consumer_handler(websocket, path))
producer_task = asyncio.ensure_future(
producer_handler(websocket, path))
done, pending = await asyncio.wait(
[consumer_task, producer_task],
return_when=asyncio.FIRST_COMPLETED,
)
for task in pending:
task.cancel()
pub.subscribe(producer, 'sender')
asyncio.get_event_loop().run_until_complete(wsconnect())
So how do I structure this code to get sending and receiving over the same websocket connection? I also have various API calls to make in the same script while the websocket connection is open which further complicates things.
I'm using Python 3.6.6 and this script is intended to be imported as a module into other scripts so the websocket functionality will need to be wrapped up in a function or class for external calls.

I am in the exact same situation as u. I know that this is a very inelegant solution
because it still isn't full-duplex but i can't seem to find any example on the internet or stackoverflow involving asyncio and the websockets module which i used.
I don't think i completely understand your websockets example (is that server-side or client-side code?) but i'm going to explain my situation and "solution" and maybe that would be usable for you too.
So i have a server main function that has a websocket listening for messages in a loop with recv(). When i send "start" it will start a function that will send data every second to the javascript client in the browser. But while the function is sending data i sometimes want to pause or stop the stream of data from my client be sending a stop message. The problem is that when i use recv() while the data sending has already begun the server stops sending data and only waits for a message. I tried threads,multiprocessing and some other stuff but eventually i came to the hopefully temporarily solution of sending a "pong" message to the server immediately after the client receives a piece of data so that the server continues sending data at the next loop iteration or stop sending data if the "pong" message is "stop" instead for example but yeah this is not real duplex just fast half-duplex...
code on my python "server"
async def start_server(self,websocket,webserver_path):
self.websocket = websocket
self.webserver_path = webserver_path
while True:
command = await self.websocket.recv()
print("received command")
if command == "start":
await self.analyze()
asyncio.sleep(1)
in my analyze function:
for i,row in enumerate(data)
await self.websocket.send(json.dumps(row))
msg = await self.websocket.recv()
if msg == "stop":
self.stopFlag = True
return
await asyncio.sleep(1)
main
start_server = websockets.serve(t.start_server, "127.0.0.1", 5678)
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()
code on the javascript client
var ws = new WebSocket("ws://127.0.0.1:5678/");
ws.onmessage = function (event) {
var datapoint = JSON.parse(event.data);
console.log(counter);
counter++;
data.push(datapoint);
if (data.length > 40){
var element = data.shift();
render(data);
}
ws.send("pong");//sending dummy message to let server continue
};
I know it is not THE solution and i hope somebody else provides a better one but since i have the same or very similar problem and there are no other answers i decided to post and i hope it helps.