I'm new in python and I have to create a program that keeps in linstening from web socket and pipe so I need two asynchronous functions.
Each of these functions call other method in different thrad that elaborate the content of received json.
i.e. I receive a message on socket thread, I get the message and throw a new thread to elaborate the message.
This is the actual code:
import asyncio
import sys
import json
import websockets
# Keep listening from web socket and pipe
async def socket_receiver():
"""Listening from web socket"""
file_socket = open(r"SocketReceived.txt", "w")
header = {"Authorization": r"Basic XXXXXXXXXXXXXX="}
async with websockets.connect(
'wss://XXXXXXXXX', extra_headers=header) as web_socket:
print("SOCKET receiving:")
greeting = await web_socket.recv()
json_message = json.loads(greeting)
file_socket.write(json_message)
print(json_message)
file_socket.close()
async def pipe_receiver():
"""Listening from pipe"""
file_pipe = open(r"ipeReceived.txt", "w")
while True:
print("PIPE receiving:")
line = sys.stdin.readline()
if not line:
break
jsonObj = json.loads(line);
file_pipe.write(jsonObj['prova'] + '\n')
# jsonValue = json.dump(str(line), file);
sys.stdout.flush()
file_pipe.close()
asyncio.get_event_loop().run_until_complete(socket_receiver())
asyncio.get_event_loop().run_until_complete(pipe_receiver())
run_until_complete method keep forever in my case (it waits the end of function), so only the socket starts.
How can I start both? Thanks
asyncio.gather does the trick, the only point is that both functions should share the same event loop, and both should be fully asynchronous.
asyncio.get_event_loop().run_until_complete(
asyncio.gather( socket_receiver(),pipe_receiver()))
From a quick reading of pipe_receiver, you will hang your event loop in sys.stdin.readline call, please consider using aioconsole to asynchronously handle the input.
Related
I am trying to implement WebSocket connection to a server (Python app <=> Django app)
Whole system runs in big Asyncio loop with many tasks. Code snippet is just very small testing part.
I am able to send any data to a server at any moment and many of them will be type request something and wait for response. But I would like to have some "always running" handler for all incoming messages. (When something in Django database will change I want to send changes to python app).
How can Include always running receiver/ or add callback to websocket? I am not able to find any solution for this.
My code snippet:
import asyncio, json, websockets, logging
class UpdateConnection:
async def connect(self,botName):
self.sock = await websockets.connect('ws://localhost:8000/updates/bot/'+botName)
async def send(self,data):
try:
await self.sock.send(json.dumps(data))
except:
logging.info("Websocket connection lost!")
# Find a way how to reconenct... or make socket reconnect automatically
if __name__ == '__main__':
async def DebugLoop(socketCon):
await socketCon.connect("dev")
print("Running..")
while True:
data = {"type": "debug"}
await socketCon.send(data)
await asyncio.sleep(1)
uSocket = UpdateConnection()
loop = asyncio.get_event_loop()
loop.create_task(DebugLoop(uSocket))
loop.run_forever()
My debug server after connection will start sending random messages to the client in random intervals and I would like to somehow handle them in async way.
Thanks for any help :)
You don't have to do it so complicated. First of all I suggest you use the context patterns offered by websockets module.
From the documentation:
connect() can be used as an infinite asynchronous iterator to reconnect automatically on errors:
async for websocket in websockets.connect(...):
try:
...
except websockets.ConnectionClosed:
continue
Additionally, you simply keep the connection alive by awaiting incoming messages:
my_websocket = None
async for websocket in websockets.connect('ws://localhost:8000/updates/bot/' + botName):
try:
my_websocket = websocket
async for message in websocket:
pass # here you could also process incoming messages
except websockets.ConnectionClosed:
my_websocket = None
continue
As you can see we have a nested loop here:
The outer loop constantly reconnects to the server
The inner loop processes one incoming message at a time
If you are connected, and no messages are coming in from the server, this will just sleep.
The other thing that happens here is that my_websocket is set to the active connection, and unset again when the connection is lost.
In other parts of your script you can use my_websocket to send data. Note that you will need to check if it is currently set wherever you use it:
async def send(data):
if my_websocket:
await my_websocket.send(json.dumps(data))
This is just an illustration, you can also keep the websocket object as an object member, or pass it to another component through a setter function, etc.
Can anybody explain, or better yet provide a working example, of how to write a Python client and server where the server can initiate a send via WebSocket at any arbitrary time based on processing logic, and the client can receive at any time? Both have to be freed up to perform other processing activities as their majority purpose.
I've Googled extensively for this, including looking into Tornado, which seems to be the classic Python option and websockets, and asyncio which seem to be the newer way. In either case, every working example I can find is based on the client initiating the communication. Some cover sending from server to client as an echo where the client's logic initiates the series of communication, but I need to get an example working where the server initiates the communication.
Many Tornado examples are timeout based which won't work in my case. The websockets and asyncio examples all are either client to server and/or have a line like somethingOrOther.run() which blocks execution on the client or server or both. Also, many examples have one of either the client or the server written in Python but the other written in something else like JavaScript (I need both in Python).
I don't need the connection to be full-duplex, that is to say, the client never has to send to the server on the same socket, except perhaps for sending back an acknowledgment.
This question seems to be similar, however, the poster's code is not complete, there is no accepted answer, and the one answer provided (not accepted) does not contain complete code, and the client is in JavaScript (I need both in Python).
Here is an example of what I'm trying to do using regular sockets:
server:
# server.py
import socket
import time
import random
# module-level variables
HOST='127.0.0.1'
PORT=65439
ACK_TEXT = 'ack_text'
def main():
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind((HOST, PORT))
sock.listen()
conn, addr = sock.accept()
while True:
# In an actual program some significant processing/logic would happen here that would take time
# and also produce a certain result, depending on which we may or may not need to send a message.
# To simulate this, we'll pause for a short and random amount of time, then get a random integer.
time.sleep(random.uniform(0.5, 1.0))
myRandInt = random.randint(0, 10)
# if the ranom integer is even, send a message
if myRandInt % 2 == 0:
message = str(myRandInt)
print('sending message: ' + message)
sendTextViaSocket(conn, message)
# end if
# end while
# end function
def sendTextViaSocket(conn, message):
# encode the message
encodedMessage = bytes(message, 'utf-8')
# send the encoded message
conn.sendall(encodedMessage)
# receive the encoded acknowledgement
encodedAck = conn.recv(1024)
# decode the acknowledgement
ack = encodedAck.decode('utf-8')
# check the received acknowledgement is correct, if not log an error
if ack == ACK_TEXT:
pass
else:
print('error: acknowledgement was received as ' + str(ack))
# end if
# end function
if __name__ == '__main__':
main()
client:
# client
import socket
import select
import time
import random
# module-level variables
HOST='127.0.0.1'
PORT=65439
ACK_TEXT = 'ack_text'
def main():
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# if the client is started 1st connect will crash, so continually try to connect
# in a try-catch until the server is available
connectionSuccessful = False
while not connectionSuccessful:
try:
sock.connect((HOST, PORT))
connectionSuccessful = True
except:
pass
# end try
# end while
socks = [ sock ]
# endlessly receive messages
while True:
readySocks, _, _ = select.select(socks, [], [], 0)
for readySock in readySocks:
message = receiveTextViaSocket(readySock)
print('received message = ' + str(message))
# end for
# In an actual program client would have other significant tasks to do
# To simulate this we'll pause for a short but random amount of time
time.sleep(random.uniform(0.5, 1.0))
# end while
# end function
def receiveTextViaSocket(sock):
# receive the encoded message
encodedMessage = sock.recv(1024)
# decode the message
message = encodedMessage.decode('utf-8')
# now send the acknowledgement
# encode the acknowledgement
encodedAck = bytes(ACK_TEXT, 'utf-8')
# send the encoded acknowledgement
sock.sendall(encodedAck)
return message
# end function
if __name__ == '__main__':
main()
How can I do the same thing, but with WebSockets ??
--- Edit ---
After spending most of today on this, this is the best I have done so far:
server:
# server.py
import threading
import asyncio
import websockets
import collections
import random
import time
sendMessageQueue = collections.deque()
def main():
webSocketServer = websockets.serve(sendMessages, 'localhost', 8765)
myLoop = asyncio.get_event_loop()
webSockThread = threading.Thread(target=webSockStart, args=(webSocketServer, myLoop,))
webSockThread.start()
while True:
myRandInt = random.randint(1, 10)
print('appending ' + str(myRandInt))
sendMessageQueue.append(str(myRandInt))
print('sendMessageQueue = ' + str(sendMessageQueue))
time.sleep(random.uniform(1.0, 2.0))
# end while
# end function
def webSockStart(webSocketServer, myLoop):
myLoop.run_until_complete(webSocketServer)
myLoop.run_forever()
# end function
async def sendMessages(websocket, path):
while len(sendMessageQueue) > 0:
await websocket.send(sendMessageQueue.popleft())
# end while
# end function
if __name__ == '__main__':
main()
client:
# client.py
import threading
import asyncio
import websockets
import collections
import random
import time
receiveMessageQueue = collections.deque()
def main():
receiveWebSockThread = threading.Thread(target=receiveWebSockStart)
receiveWebSockThread.start()
while True:
print('doing other stuff')
time.sleep(1.0)
while len(receiveMessageQueue) > 0:
message = receiveMessageQueue.popleft()
print('message = ' + str(message))
# end while
# end while
# end function
def receiveWebSockStart():
loop = asyncio.new_event_loop()
loop.run_until_complete(receiveMessages())
loop.run_forever()
# end function
async def receiveMessages():
while True:
uri = 'ws://localhost:8765'
async with websockets.connect(uri) as webSockConn:
message = await webSockConn.recv()
receiveMessageQueue.append(str(message))
# while True:
# end while
# end with
# end function
if __name__ == '__main__':
main()
server output:
$ python3 server.py
appending 3
sendMessageQueue = deque(['3'])
appending 8
sendMessageQueue = deque(['8'])
appending 8
sendMessageQueue = deque(['8', '8'])
appending 6
sendMessageQueue = deque(['8', '8', '6'])
appending 1
sendMessageQueue = deque(['8', '8', '6', '1'])
client output:
$ python3 client.py
doing other stuff
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "client.py", line 30, in receiveWebSockStart
loop.run_until_complete(receiveMessages())
File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
return future.result()
File "client.py", line 38, in receiveMessages
message = await webSockConn.recv()
File "/usr/local/lib/python3.6/dist-packages/websockets/protocol.py", line 509, in recv
await self.ensure_open()
File "/usr/local/lib/python3.6/dist-packages/websockets/protocol.py", line 812, in ensure_open
raise self.connection_closed_exc()
websockets.exceptions.ConnectionClosedOK: code = 1000 (OK), no reason
message = 3
doing other stuff
doing other stuff
doing other stuff
doing other stuff
The server seems to be working as intended, however the client is disconnecting after receiving one message and I'm not sure why or how to resolve it. I'm not sure if threads are even the correct approach, or if I should be using run_in_executor from concurrent.futures, but I couldn't get that to work at all. Anybody with specific knowledge on this topic please help.
--- Edit2 ---
Well, at least I found an answer that works (tested with Python 3.6.9 on Ubuntu 18.04), but I'm really not happy with it:
server:
# server.py
import threading
import asyncio
import websockets
import collections
import random
import time
sendMessageQueue = collections.deque()
def main():
# start the WebSocket sending on a separate thread so it doesn't block main
webSockSendThread = threading.Thread(target=sendWebSockStart)
webSockSendThread.start()
while True:
# make up a random integer and append it to the send queue
myRandInt = random.randint(1, 10)
print('appending ' + str(myRandInt))
sendMessageQueue.append(str(myRandInt))
# an actual program would have many other activities to do here, use a random sleep to simulate this
print('doing other stuff')
time.sleep(random.uniform(1.0, 2.0))
# end while
# end function
def sendWebSockStart():
# since we're in a separate thread now, call new_event_loop() (rather than the usual get_event_loop())
# and set the returned loop as the current loop
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
# instantiate the WebSocket server, note this also connects it to the sendMessages function
webSocketServer = websockets.serve(sendMessages, 'localhost', 8765)
# run the webSocketServer forever, which runs the sendMessages function forever
loop.run_until_complete(webSocketServer)
loop.run_forever() # note execution of this separate thread stays on this line forever
# end function
async def sendMessages(websocket, path):
while True:
while len(sendMessageQueue) > 0:
await websocket.send(sendMessageQueue.popleft())
# end while
# end while
# end function
if __name__ == '__main__':
main()
client:
# client.py
import threading
import asyncio
import websockets
import collections
import random
import time
receiveMessageQueue = collections.deque()
def main():
# start the WebSocket receiving on a separate thread so it doesn't block main
receiveWebSockThread = threading.Thread(target=receiveWebSockStart)
receiveWebSockThread.start()
while True:
# dequeue and print out all the messages in the receive queue currently
while len(receiveMessageQueue) > 0:
message = receiveMessageQueue.popleft()
print('message = ' + str(message))
# end while
# an actual program would have many other activities to do here, use a sleep to simulate this
print('doing other stuff')
time.sleep(1.0)
# end while
# end function
def receiveWebSockStart():
# since we're in a separate thread now, call new_event_loop() rather than the usual get_event_loop()
loop = asyncio.new_event_loop()
# run receiveMessages() forever
loop.run_until_complete(receiveMessages())
loop.run_forever() # note execution of this separate thread stays on this line forever
# end function
async def receiveMessages():
# set up the connection
uri = 'ws://localhost:8765'
async with websockets.connect(uri) as webSockConn:
# endlessly receive messages and append to the queue when received
while True:
message = await webSockConn.recv()
receiveMessageQueue.append(str(message))
# end while
# end with
# end function
if __name__ == '__main__':
main()
server output:
$ python3 server.py
appending 10
doing other stuff
appending 6
doing other stuff
appending 2
doing other stuff
appending 8
doing other stuff
appending 2
doing other stuff
appending 3
doing other stuff
appending 9
doing other stuff
client output:
$ python3 client.py
doing other stuff
message = 10
message = 6
doing other stuff
doing other stuff
message = 2
doing other stuff
doing other stuff
message = 8
doing other stuff
doing other stuff
message = 2
doing other stuff
doing other stuff
message = 3
doing other stuff
message = 9
doing other stuff
I'm using threading as an intermediary between main and the asyncio websockets function. Can anybody explain or better yet provide a working example of how to re-work this to use run_in_executor to eliminate the intermediary thread ??
I have a simple python tkinter gui with just a couple buttons. When the button is pressed all I want to do is start a websocket connection and start receiving. I can run the code normally but as soon as I try to put it into a thread I get errors
RuntimeError: There is no current event loop in thread
So first try:
import websockets
websocket = websockets.connect(uri, ssl = True)
websocket.recv()
I get the error
"Connect object has no attribute 'recv'"
Which is weird when I run it differently I don't get that error
When I follow the documentation exactly
def run_websockets2(self):
async def hello():
uri = Websocket_Feed
# with websockets.connect(uri, ssl=True) as websocket:
socket = await websockets.connect(uri, ssl=True)
self.web_socket = socket
while self.running:
greeting = await socket.recv()
print(f"< {greeting}")
asyncio.get_event_loop().run_until_complete(hello())
It works as long as I just call "websockets2()". But if I try to do
self.websocket_thread = threading.Thread(target=self.run_websockets2, args=())
self.websocket_thread.start()
I get the error
RuntimeError: There is no current event loop in thread 'web_sockets'
And when I make the whole function non async I get an error
def run_websockets(self):
uri = Websocket_Feed
# with websockets.connect(uri, ssl=True) as websocket:
socket = websockets.connect(uri, ssl=True)
self.web_socket = socket
while self.running:
greeting = socket.recv()
print(f"< {greeting}")
I get the error
RuntimeError: There is no current event loop in thread 'web_sockets'. on
socket = websockets.connect(uri, ssl=True)
I don't understand why I can't just simply run these non asynchronous in a thread. Any help is greatly appreciated
You have a couple of different errors here, which is confusing the picture somewhat. First, regarding:
"Connect object has no attribute 'recv'"
... this just says that websocket object has no method called recv
The main problem you have is trying to invoke run_websockets2() from a spawned thread. I.e. calling this method from main thread works, but calling this from a new thread fails.
This is expected behaviour. It is because in a spawned thread (i.e., thread other than main thread), there is no asyncio event loop defined. But there is one defined in the main thread, for convenience. So asyncio is aware of whether you are invoking from spawned thread, or main thread, and behaves differently. See this answer for detailed explantion. Why asyncio.get_event_loop method checks if the current thread is the main thread?
To solve your problem you could create a new event loop per spawned thread, so that the code would become:
event_loop = asyncio.new_event_loop()
event_loop.run_until_complete(hello())
instead of
asyncio.get_event_loop().run_until_complete(hello())
Or, you could store event_loop in a common place, and allow all spawned threads to reuse that event loop.
I wanted to post how I actually solved my code thanks to #Darren Smith. I simply added one line of code
"asyncio.set_event_loop(asyncio.new_event_loop())" to the top.
def run_websockets2(self):
asyncio.set_event_loop(asyncio.new_event_loop())
async def hello():
uri = Websocket_Feed
# with websockets.connect(uri, ssl=True) as websocket:
socket = await websockets.connect(uri, ssl=True)
self.web_socket = socket
while self.running:
greeting = await socket.recv()
print(f"< {greeting}")
asyncio.get_event_loop().run_until_complete(hello())
I have to admit, I love python, but to have code that fundamentally runs differently depending on its context and the fix is a one line that is not related to the context seems to be in really bad form.
I am trying to implement a simple echo server using asyncio. Here's my attempt:
import asyncio
async def process(reader, writer):
while True:
data = await reader.readline()
writer.write(data)
loop = asyncio.get_event_loop()
listener = asyncio.start_server(process, host=None, port=2500)
loop.run_until_complete(listener)
loop.run_forever()
This starts the server and I can connect from multiple clients, also the clients have their messages echoed back to them. The problem is when I close one of the clients, the echo messages stop appearing on the other connected clients too. Why does that happen and how can I prevent that?
The server hangs because the loop is not exited when a connection is closed. The convention is when a network read function returns no data, it means EOF. The server must handle it, e.g.:
while True:
data = await reader.readline()
if not data:
break
writer.write(data)
Without the break the loop is executed again and again, because the EOF state will not change.
Please note that each connection is handled by its own process which is called with a reader and a writer belonging to that connection.
I used asyncio for my non-stop server in python and implemented
connection_made , connection_lost , data_received
funtions in my ServerClientProtocol
I used this class first beacause of using multiple times repeatedly sending data to socket class socket
got closed and program exited
and second becuase I thought its async and have parallel answering multiple coming sockets in same time,
but it's not.
how should I use that in one async thread and parallel answering socket?
this is my code:
class ServerClientProtocol(asyncio.Protocol):
def connection_made(self,transport):
self.transport = transport
def connection_lost(self,exc):
pass
def data_received(self, data):
server.server(self,data)
def main(*args):
loop = get_event_loop()
coro = loop.create_server(ServerClientProtocol, '127.0.0.1', 50008)
srv = loop.run_until_complete(coro)
loop.run_forever()
if __name__ == '__main__':
main()
server.server() might be blocking the other connections. If this is a long-running call, try using asyncio.start_server (example here) instead, and call server.server() using await loop.run_in_executor(None, server.server, data)