I'm setting a python websocket client that should make send and receive request's as described:
Connect to the websocket.
Send the request to get current timestamp.
Receive back the current timestamp.
Compare times , if times are synced continue, if not reply ("not_synced!").
Send the machine name (in this case it is defined in the config file)
The server response back with a timestamp in the future when it is
Expecting a ping, the time is saved in config file
Close connection and wait for current time to match the time in the future!
By now, I have perfectly created functions for reading/saving in strings in the config file, comparing the received time with current time.
The only issue I can`t figure out how to solve it's the communication to the server, actually I want to define one function that should do all the communication through.
Tried defining function without asyncio, I couldn't return received message.
While using asyncio, I couldn't pass the argument in function (actually the message string!)
import asyncio
import websockets
async def connect(msg):
async with websockets.connect("ws://connect.websocket.in /xnode?room_id=19210") as socket: # the opencfg function reads a file, in this case, line 4 of config file where url is stored
await socket.send(msg)
result =await socket.recv()
return result
asyncio.get_event_loop().run_until_complete(connect())
def connect2(msg):
soc= websockets.connect("ws://connect.websocket.in /xnode?room_id=19210")
soc.send(msg)
result=soc.recv()
return result
print(connect2("gettime"))
If you would try to send "gettime" , you will receive back the current timestamp, and after sending the "|online" you should receive back a value that is equal to current timestamp + 10.
You have the websocketurl so try it for yourself.
I changed your code to use asynio.gather to get the return value and passed "gettime" to the function:
import asyncio
import websockets
address = "ws://connect.websocket.in/xnode?room_id=19210"
async def connect(msg):
async with websockets.connect(address) as socket:
await socket.send(msg)
result = await socket.recv()
return result
result = asyncio.get_event_loop().run_until_complete(asyncio.gather(connect("gettime")))
print(result)
Output
['1564626191']
You can reuse the code by putting it into a function definition:
def get_command(command):
loop = asyncio.get_event_loop()
result = loop.run_until_complete(asyncio.gather(connect(command)))
return result
result = get_command("gettime")
print(result)
Related
I have a scraper that requires the use of a websocket server (can't go into too much detail on why because of company policy) that I'm trying to turn into a template/module for easier use on other websites.
I have one main function that runs the loop of the server (e.g. ping-pongs to keep the connection alive and send work and stop commands when necessary) that I'm trying to turn into a generator that yields the HTML of scraped pages (asynchronously, of course). However, I can't figure out a way to turn the server into a generator.
This is essentially the code I would want (simplified to just show the main idea, of course):
import asyncio, websockets
needsToStart = False # Setting this to true gets handled somewhere else in the script
async def run(ws):
global needsToStart
while True:
data = await ws.recv()
if data == "ping":
await ws.send("pong")
elif "<html" in data:
yield data # Yielding the page data
if needsToStart:
await ws.send("work") # Starts the next scraping session
needsToStart = False
generator = websockets.serve(run, 'localhost', 9999)
while True:
html = await anext(generator)
# Do whatever with html
This, of course, doesn't work, giving the error "TypeError: 'Serve' object is not callable". But is there any way to set up something along these lines? An alternative I could try is creating an 'intermittent' object that holds the data which the end loop awaits, but that seems messier to me than figuring out a way to get this idea to work.
Thanks in advance.
I found a solution that essentially works backwards, for those in need of the same functionality: instead of yielding the data, I pass along the function that processes said data. Here's the updated example case:
import asyncio, websockets
from functools import partial
needsToStart = False # Setting this to true gets handled somewhere else in the script
def process(html):
pass
async def run(ws, htmlFunc):
global needsToStart
while True:
data = await ws.recv()
if data == "ping":
await ws.send("pong")
elif "<html" in data:
htmlFunc(data) # Processing the page data
if needsToStart:
await ws.send("work") # Starts the next scraping session
needsToStart = False
func = partial(run, htmlFunc=process)
websockets.serve(func, 'localhost', 9999)
I'm trying to understand if grpc server using streams is able to wait for all client messages to be read in prior to sending responses.
I have a trivial application where I send in several numbers I'd like to add and return.
I've set up a basic proto file to test this:
syntax = "proto3";
message CalculateRequest{
int64 x = 1;
int64 y = 2;
};
message CalculateReply{
int64 result = 1;
}
service Svc {
rpc CalculateStream (stream CalculateRequest) returns (stream CalculateReply);
}
On my server-side I have implemented the following code which returns the answer message as the message is received:
class CalculatorServicer(contracts_pb2_grpc.SvcServicer):
def CalculateStream(self, request_iterator, context):
for request in request_iterator:
resultToOutput = request.x + request.y
yield contracts_pb2.CalculateReply(result=resultToOutput)
def serve():
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
contracts_pb2_grpc.add_SvcServicer_to_server(
CalculatorServicer(), server)
server.add_insecure_port('localhost:9000')
server.start()
server.wait_for_termination()
if __name__ == '__main__':
print( "We're up")
logging.basicConfig()
serve()
I'd like to tweak this to first read in all the numbers and then send these out at a later stage - something like the following:
class CalculatorServicer(contracts_pb2_grpc.SvcServicer):
listToReturn = []
def CalculateStream(self, request_iterator, context):
for request in request_iterator:
listToReturn.append (request.x + request.y)
# ...
# do some other stuff first before returning
for item in listToReturn:
yield contracts_pb2.CalculateReply(result=resultToOutput)
Currently, my implementation to write out later doesn't work as the code at the bottom is never reached. Is this by design that the connection seems to "close" before reaching there?
The grpc.io website suggests that this should be possible with BiDirectional streaming:
for example, the server could wait to receive all the client messages before writing its responses, or it could alternately read a message then write a message, or some other combination of reads and writes.
Thanks in advance for any help :)
The issue here is the definition of "all client messages." At the transport level, the server has no way of knowing whether the client has finished independent of the client closing its connection.
You need to add some indication of the client's having finished sending requests to the protocol. Either add a bool field to the existing CalculateRequest or add a top-level oneof with one of the options being something like a StopSendingRequests
I'm curious about how you are supposed to express that you want a message delivered to a Kafka topic in faust. The example in their readme doesn't seem to write to a topic:
import faust
class Greeting(faust.Record):
from_name: str
to_name: str
app = faust.App('hello-app', broker='kafka://localhost')
topic = app.topic('hello-topic', value_type=Greeting)
#app.agent(topic)
async def hello(greetings):
async for greeting in greetings:
print(f'Hello from {greeting.from_name} to {greeting.to_name}')
#app.timer(interval=1.0)
async def example_sender(app):
await hello.send(
value=Greeting(from_name='Faust', to_name='you'),
)
if __name__ == '__main__':
app.main()
I would expect hello.send in the above code to publish a message to the topic, but it doesn't appear to.
There are many examples of reading from topics, and many examples of using the cli to push an ad-hoc message. After combing through the docs, I don't see any clear examples of publishing to topics in code. Am I just being crazy and the above code should work?
You can use sink to tell Faust where to deliver the results of an agent function. You can also use multiple topics as sinks at once if you want.
#app.agent(topic_to_read_from, sink=[destination_topic])
async def fetch(records):
async for record in records:
result = do_something(record)
yield result
The send() function is the correct one to call to write to topics. You can even specify a particular partition, just like the equivalent Java API call.
Here is the reference for the send() method:
https://faust.readthedocs.io/en/latest/reference/faust.topics.html#faust.topics.Topic.send
If you want a Faust producer only (not combined with a consumer/sink), the original question actually has the right bit of code, here's a fully functional script that publishes messages to a 'faust_test' Kafka topic that is consumable by any Kafka/Faust consumer.
Run the code below like this: python faust_producer.py worker
"""Simple Faust Producer"""
import faust
if __name__ == '__main__':
"""Simple Faust Producer"""
# Create the Faust App
app = faust.App('faust_test_app', broker='localhost:9092')
topic = app.topic('faust_test')
# Send messages
#app.timer(interval=1.0)
async def send_message(message):
await topic.send(value='my message')
# Start the Faust App
app.main()
So we just ran into the need to send a message to a topic other than the sink topics.
The easiest way we found was: foo = await my_topic.send_soon(value="wtfm8").
You can also use send directly like below using the asyncio event loop.
loop = asyncio.get_event_loop()
foo = await ttopic.send(value="wtfm8??")
loop.run_until_complete(foo)
Dont know how relevant this is anymore but I came across this issue when trying to learn Faust. From what I read, here is what is happening:
topic = app.topic('hello-topic', value_type=Greeting)
The misconception here is that the topic you have created is the topic you are trying to consume/read from. The topic you created currently does not do anything.
await hello.send(
value=Greeting(from_name='Faust', to_name='you'),
)
this essentially creates an intermediate kstream which sends the values to your hello(greetings) function. def hello(...) will be called when there is a new message to the stream and will process the message that is being sent.
#app.agent(topic)
async def hello(greetings):
async for greeting in greetings:
print(f'Hello from {greeting.from_name} to {greeting.to_name}')
This is receiving the kafka stream from hello.send(...) and simply printing it to the console (no output to the 'topic' created). This is where you can send a message to a new topic. so instead of printing you can do:
topic.send(value = "my message!")
Alternatively:
Here is what you are doing:
example_sender() sends a message to hello(...) (through intermediate kstream)
hello(...) picks up the message and prints it
NOTICE: no sending of messages to the correct topic
Here is what you can do:
example_sender() sends a message to hello(...) (through intermediate kstream)
hello(...) picks up the message and prints
hello(...) ALSO sends a new message to the topic created(assuming you are trying to transform the original data)
app = faust.App('hello-app', broker='kafka://localhost')
topic = app.topic('hello-topic', value_type=Greeting)
output_topic = app.topic('test_output_faust', value_type=str)
#app.agent(topic)
async def hello(greetings):
async for greeting in greetings:
new_message = f'Hello from {greeting.from_name} to {greeting.to_name}'
print(new_message)
await output_topic.send(value=new_message)
I found a solution to how to send data to kafka topics using Faust, but I don't really understand how it works.
There are several methods for this in Faust: send(), cast(), ask_nowait(), ask(). In the documentation they are called RPC operations.
After creating the sending task, you need to run the Faust application in the mode Client-Only Mode. (start_client(), maybe_start_client())
The following code (the produce() function) demonstrates their application (pay attention to the comments):
import asyncio
import faust
class Greeting(faust.Record):
from_name: str
to_name: str
app = faust.App('hello-app', broker='kafka://localhost')
topic = app.topic('hello-topic', value_type=Greeting)
result_topic = app.topic('result-topic', value_type=str)
#app.agent(topic)
async def hello(greetings):
async for greeting in greetings:
s = f'Hello from {greeting.from_name} to {greeting.to_name}'
print(s)
yield s
async def produce(to_name):
# send - universal method for sending data to a topic
await hello.send(value=Greeting(from_name='SEND', to_name=to_name), force=True)
await app.maybe_start_client()
print('SEND')
# cast - allows you to send data without waiting for a response from the agent
await hello.cast(value=Greeting(from_name='CAST', to_name=to_name))
await app.maybe_start_client()
print('CAST')
# ask_nowait - it seems to be similar to cast
p = await hello.ask_nowait(
value=Greeting(from_name='ASK_NOWAIT', to_name=to_name),
force=True,
reply_to=result_topic
)
# without this line, ask_nowait will not work; taken from the ask implementation
await app._reply_consumer.add(p.correlation_id, p)
await app.maybe_start_client()
print(f'ASK_NOWAIT: {p.correlation_id}')
# blocks the execution flow
# p = await hello.ask(value=Greeting(from_name='ASK', to_name=to_name), reply_to=result_topic)
# print(f'ASK: {p.correlation_id}')
if __name__ == '__main__':
loop = asyncio.get_event_loop()
loop.run_until_complete(produce('Faust'))
Starting Fast worker with the command faust -A <example> worker
Then we can launch the client part of the application and check that everything is working: python <example.py>
<example.py> output:
SEND
CAST
ASK_NOWAIT: bbbe6795-5a99-40e5-a7ad-a9af544efd55
It is worth noting that you will also see a traceback of some error that occurred after delivery, which does not interfere with the program (it seems so)
Faust worker output:
[2022-07-19 12:06:27,959] [1140] [WARNING] Hello from SEND to Faust
[2022-07-19 12:06:27,960] [1140] [WARNING] Hello from CAST to Faust
[2022-07-19 12:06:27,962] [1140] [WARNING] Hello from ASK_NOWAIT to Faust
I don't understand why it works this way, why it's so difficult and why very little is written about in the documentation 😓.
Lately, I've been trying to stream data from KDB to python.
I'm now using websockets and have gone through the doc https://code.kx.com/q/wp/kdb_and_websockets.pdf
On the python side, I've been trying ws4py, autobahn and websocket-client.
All do work fine, essentially my problem resides in the format of the message sent to the server to subscribe to the feed.
A little (open source) example:
class DummyClient(WebSocketClient):
def opened(self):
self.send(*what should I put here?*, binary=True)
def closed(self, code, reason=None):
print("Closed down", code, reason)
def received_message(self, m):
# not yet implemented
if __name__ == '__main__':
try:
ws = DummyClient('ws://host:port/', protocols=['http-only', 'chat'])
ws.connect()
ws.run_forever()
except KeyboardInterrupt:
ws.close()
When opening, I'm supposed to subscribe to the feed by calling the server function loadPage.
I've tried different things when to encode the list containing the function name and the argument, without success.
What I've tried:
np.array("['loadPage',[]]").tobytes()
"['loadPage',[]]".encode('utf8')
json formating
hexadecimal formating
Any help would be much appreciated!
Best,
Yael
I think what you'll need to do in this case is define .z.ws in your kdb server. This function will be called with whatever you pass over the websocket as an argument, so for example if you defined .z.ws like so:
.z.ws:{show x}
and then send a message over the WebSocket like so:
var ws = new WebSocket("ws://localhost:1234")
ws.send("Hello World")
This will be output in your kdb session e.g.
λ q
KDB+ 3.6 2018.06.14 Copyright (C) 1993-2018 Kx Systems
w64/ 4(16)core 8082MB jonat laptop-o8a8co1o 10.255.252.249 EXPIRE 2019.05.21 jonathon.mcmurray#aquaq.co.uk KOD #4160315
q).z.ws:{show x}
q)\p 1234
q)"Hello World"
So in your case if you want to call loadPage, it might be as simple as defining .z.ws like so:
.z.ws:loadPage
Then whatever you pass over the socket will be passed into loadPage (note this is will not update if you change loadPage elsewhere, you can instead use .z.ws:{loadPage x} if you need to be able to update it dynamically). For a niladic function, this will just ignore whatever is passed in
I'm trying to make a simple Slack bot using asyncio, largely using the example here for the asyncio part and here for the Slack bot part.
Both the examples work on their own, but when I put them together it seems my loop doesn't loop: it goes through once and then dies. If info is a list of length equal to 1, which happens when a message is typed in a chat room with the bot in it, the coroutine is supposed to be triggered, but it never is. (All the coroutine is trying to do right now is print the message, and if the message contains "/time", it gets the bot to print the time in the chat room it was asked in). Keyboard interrupt also doesn't work, I have to close the command prompt every time.
Here is my code:
import asyncio
from slackclient import SlackClient
import time, datetime as dt
token = "MY TOKEN"
sc = SlackClient(token)
#asyncio.coroutine
def read_text(info):
if 'text' in info[0]:
print(info[0]['text'])
if r'/time' in info[0]['text']:
print(info)
resp = 'The time is ' + dt.datetime.strftime(dt.datetime.now(),'%H:%M:%S')
print(resp)
chan = info[0]['channel']
sc.rtm_send_message(chan, resp)
loop = asyncio.get_event_loop()
try:
sc.rtm_connect()
info = sc.rtm_read()
if len(info) == 1:
asyncio.async(read_text(info))
loop.run_forever()
except KeyboardInterrupt:
pass
finally:
print('step: loop.close()')
loop.close()
I think it's the loop part that's broken, since it never seems to get to the coroutine. So maybe a shorter way of asking this question is what is it about my try: statement that prevents it from looping like in the asyncio example I followed? Is there something about sc.rtm_connect() that it doesn't like?
I'm new to asyncio, so I'm probably doing something stupid. Is this even the best way to try and go about this? Ultimately I want the bot to do some things that take quite a while to compute, and I'd like it to remain responsive in that time, so I think I need to use asyncio or threads in some variety, but I'm open to better suggestions.
Thanks a lot,
Alex
I changed it to the following and it worked:
import asyncio
from slackclient import SlackClient
import time, datetime as dt
token = "MY TOKEN"
sc = SlackClient(token)
#asyncio.coroutine
def listen():
yield from asyncio.sleep(1)
x = sc.rtm_connect()
info = sc.rtm_read()
if len(info) == 1:
if 'text' in info[0]:
print(info[0]['text'])
if r'/time' in info[0]['text']:
print(info)
resp = 'The time is ' + dt.datetime.strftime(dt.datetime.now(),'%H:%M:%S')
print(resp)
chan = info[0]['channel']
sc.rtm_send_message(chan, resp)
asyncio.async(listen())
loop = asyncio.get_event_loop()
try:
asyncio.async(listen())
loop.run_forever()
except KeyboardInterrupt:
pass
finally:
print('step: loop.close()')
loop.close()
Not entirely sure why that fixes it, but the key things I changed were putting the sc.rtm_connect() call in the coroutine and making it x = sc.rtm_connect(). I also call the listen() function from itself at the end, which appears to be what makes it loop forever, since the bot doesn't respond if I take it out. I don't know if this is the way this sort of thing is supposed to be set up, but it does appear to continue to accept commands while it's processing earlier commands, my slack chat looks like this:
me [12:21 AM]
/time
[12:21]
/time
[12:21]
/time
[12:21]
/time
testbotBOT [12:21 AM]
The time is 00:21:11
[12:21]
The time is 00:21:14
[12:21]
The time is 00:21:16
[12:21]
The time is 00:21:19
Note that it doesn't miss any of my /time requests, which it would if it weren't doing this stuff asynchronously. Also, if anyone is trying to replicate this you'll notice that slack brings up the built in command menu if you type "/". I got around this by typing a space in front.
Thanks for the help, please let me know if you know of a better way of doing this. It doesn't seem to be a very elegant solution, and the bot can't be restarted after I use the a cntrl-c keyboard interrupt to end it - it says
Task exception was never retrieved
future: <Task finished coro=<listen() done, defined at asynctest3.py:8> exception=AttributeError("'NoneType' object has no attribute 'recv'",)>
Traceback (most recent call last):
File "C:\Users\Dell-F5\AppData\Local\Programs\Python\Python35-32\Lib\asyncio\tasks.py", line 239, in _step
result = coro.send(None)
File "asynctest3.py", line 13, in listen
info = sc.rtm_read()
File "C:\Users\Dell-F5\Envs\sbot\lib\site-packages\slackclient\_client.py", line 39, in rtm_read
json_data = self.server.websocket_safe_read()
File "C:\Users\Dell-F5\Envs\sbot\lib\site-packages\slackclient\_server.py", line 110, in websocket_safe_read
data += "{0}\n".format(self.websocket.recv())
AttributeError: 'NoneType' object has no attribute 'recv'
Which I guess means it's not closing the websockets nicely. Anyway, that's just an annoyance, at least the main problem is fixed.
Alex
Making blocking IO calls inside a coroutine defeat the very purpose of using asyncio (e.g. info = sc.rtm_read()). If you don't have a choice, use loop.run_in_executor to run the blocking call in a different thread. Careful though, some extra locking might be needed.
However, it seems there's a few asyncio-based slack client libraries you could use instead:
slacker-asyncio - fork of slacker, based on aiohttp
butterfield - based on slacker and websockets
EDIT: Butterfield uses the Slack real-time messaging API. It even provides an echo bot example that looks very much like what you're trying to achieve:
import asyncio
from butterfield import Bot
#asyncio.coroutine
def echo(bot, message):
yield from bot.post(
message['channel'],
message['text']
)
bot = Bot('slack-bot-key')
bot.listen(echo)
butterfield.run(bot)