for a WebRTC project, I have to build up a signaling mechanism. I use Django Channels for that job. As you might know in WebRTC, there are Session Description objects that will be passed back and forth as "offer"/"answer" and also there are ICE candidate objects which will also be passed between two (or more) clients.
Playing with Django Channels a little bit, I have wrote the following consumer:
'''
A basic consumer that accepts WebSocket connections on the path
/ws/chat/ROOM_NAME/ that takes any message it receives on the
WebSocket and echos it back to the same WebSocket.
we want to have multiple instances of SignallingConsumer in the same room communicate
with each other. To do that we will have each SignallingConsumer add its channel
to a group whose name is based on the room name. That will allow SignallingConsumers
to transmit messages to all other SignallingConsumers in the same room.
'''
class SignallingConsumer(WebsocketConsumer):
def connect(self):
print("connect() is called.")
'''
Obtains the 'room_name' parameter from the URL route in chat/routing.py
that opened the WebSocket connection to the consumer.
Every consumer has a scope that contains information about its connection,
including in particular any positional or keyword arguments from the URL route
and the currently authenticated user if any.
'''
self.room_name = self.scope['url_route']['kwargs']['room_name']
'''
Constructs a Channels group name directly from the user-specified room name,
without any quoting or escaping.
Group names may only contain letters, digits, hyphens, and periods.
Therefore this example code will fail on room names that have other characters.
'''
self.room_group_name = 'chat_%s' % self.room_name
'''
Join room group by adding the channel name to the group.
The async_to_sync(…) wrapper is required because ChatConsumer is a synchronous
WebsocketConsumer but it is calling an asynchronous channel layer method.
(All channel layer methods are asynchronous.)
Group names are restricted to ASCII alphanumerics, hyphens, and periods only.
Since this code constructs a group name directly from the room name, it will
fail if the room name contains any characters that aren’t valid in a group
name.
'''
async_to_sync(self.channel_layer.group_add)(
self.room_group_name,
self.channel_name
)
'''
Accepts the WebSocket connection.
If you do not call accept() within the connect() method then the
connection will be rejected and closed. You might want to reject a
connection for example because the requesting user is not authorized
to perform the requested action.
It is recommended that accept() be called as the last action
in connect() if you choose to accept the connection.
'''
self.accept()
def disconnect(self, close_code):
print("disconnect() is called.")
# Leave room group
async_to_sync(self.channel_layer.group_discard)(
self.room_group_name,
self.channel_name
)
# Receive message from WebSocket
# text_data is of type String
def receive(self, text_data):
print("Received data: " + text_data)
# json.loads(): takes in a String and returns a JSON object
# text_data_json is of type JSONObject
text_data_json = json.loads(text_data)
async_to_sync(self.channel_layer.group_send)(
self.room_group_name,
{
'type': 'offer_answer_message',
'data': text_data_json
}
)
def offer_answer_message(self, event):
# we access the JSONObject via event['data'] and
# store it into data variable
data = event['data']
# json.dumps(): takes in a JSONObject and returns a String
self.send(json.dumps(data))
The comments are basically for myself. I have tweaked the code from the official site of Django Channels a little bit for my purpose. For my question, the receive() method is very important. The questions regarding this are the following ones:
So, what the consumer should do is to pass the offer (or answer)
from client A ( or client B ) to the other remote peer. Does this
work here ?
Also, the ICE candidates which contain the network information about a peer
should also be passed to the remote peer. Does this work here ?
The receive() method takes a String as parameter. For that reason, I have to convert my JSON objects to Strings on the client side before sending them. I don't like that solution. What I would rather do is passing the JSON objects directly to the receive() method on the server side. How I would do it ?
Yes, you can receive a offer or answer using this receive method say one is a host other is peer, you have already made a group... so just forward offer/answer accordingly to required consumer, keep track what type of message it is i.e. its a offer / answer.
Similarly when ever you set the remote description or the local description on client side a event onicecandidate must be handled by sending the candidate information to other client just like offer and answer..
PC.onicecandidate = (event)=>{
if(iceCand){
await Socket.send(JSON.stringify({
"who" : "user1",
"for" : "user2",
"candidate" : event.candidate
}));
}
};
You need to get it via JSON ... otherwise you get a no data for the candidate key in dict given in code above.. but for AsyncConsumer or SyncConsumer the receive method takes in a dict as far as I know, dict like given below:
event = {"type" : "websocket.receive", "text" : data}
Related
I'm trying to understand if grpc server using streams is able to wait for all client messages to be read in prior to sending responses.
I have a trivial application where I send in several numbers I'd like to add and return.
I've set up a basic proto file to test this:
syntax = "proto3";
message CalculateRequest{
int64 x = 1;
int64 y = 2;
};
message CalculateReply{
int64 result = 1;
}
service Svc {
rpc CalculateStream (stream CalculateRequest) returns (stream CalculateReply);
}
On my server-side I have implemented the following code which returns the answer message as the message is received:
class CalculatorServicer(contracts_pb2_grpc.SvcServicer):
def CalculateStream(self, request_iterator, context):
for request in request_iterator:
resultToOutput = request.x + request.y
yield contracts_pb2.CalculateReply(result=resultToOutput)
def serve():
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
contracts_pb2_grpc.add_SvcServicer_to_server(
CalculatorServicer(), server)
server.add_insecure_port('localhost:9000')
server.start()
server.wait_for_termination()
if __name__ == '__main__':
print( "We're up")
logging.basicConfig()
serve()
I'd like to tweak this to first read in all the numbers and then send these out at a later stage - something like the following:
class CalculatorServicer(contracts_pb2_grpc.SvcServicer):
listToReturn = []
def CalculateStream(self, request_iterator, context):
for request in request_iterator:
listToReturn.append (request.x + request.y)
# ...
# do some other stuff first before returning
for item in listToReturn:
yield contracts_pb2.CalculateReply(result=resultToOutput)
Currently, my implementation to write out later doesn't work as the code at the bottom is never reached. Is this by design that the connection seems to "close" before reaching there?
The grpc.io website suggests that this should be possible with BiDirectional streaming:
for example, the server could wait to receive all the client messages before writing its responses, or it could alternately read a message then write a message, or some other combination of reads and writes.
Thanks in advance for any help :)
The issue here is the definition of "all client messages." At the transport level, the server has no way of knowing whether the client has finished independent of the client closing its connection.
You need to add some indication of the client's having finished sending requests to the protocol. Either add a bool field to the existing CalculateRequest or add a top-level oneof with one of the options being something like a StopSendingRequests
I'm trying to get my Python Slack bot to automatically reply in a thread if I post it commands in one. However, regardless of where I post my commands - in a thread or otherwise, it still replies as a general message.
I wish to get it to reply in a thread. Here's my code so far (I've truncated most of the initializing and startup code for the sake of brevity):
import os, time, re, inspect, functools
from slackclient import SlackClient
class Bot(object):
def __init__(self, token):
...
def startup(self):
...
def parse_bot_commands(self, slack_events):
"""
Parses a list of events coming from the Slack RTM API to find bot commands.
If a bot command is found, this function returns a tuple of command and channel.
If its not found, then this function returns None, None.
"""
for event in slack_events:
if event["type"] == "message" and not "subtype" in event:
user_id, message = self.parse_direct_mention(event["text"])
if user_id == self.starterbot_id:
return message, event["channel"]
return None, None
def parse_direct_mention(self, message_text):
"""
Finds a direct mention (a mention that is at the beginning) in message text
and returns the user ID which was mentioned. If there is no direct mention, returns None
"""
matches = re.search(self.MENTION_REGEX, message_text)
# the first group contains the username, the second group contains the remaining message
return (matches.group(1), matches.group(2).strip()) if matches else (None, None)
def handle_command(self, command, channel):
"""
Executes bot command if the command is known
"""
# Default response is help text for the user
default_response = "Not sure what you mean. Try *{}*.".format(self.EXAMPLE_COMMAND)
# Finds and executes the given command, filling in response
response = None
# NOTE: This is where you start to implement more commands!
if command.lower().startswith("roll"):
response = 'Rock and Roll!"
# Sends the response back to the channel
self.slack_client.api_call("chat.postMessage", channel=channel, text=response or default_response)
'''START THE BOT!'''
#Initialize the token (when installing the app)
bot = Bot('xxx-xxx')
bot.startup()
Slash commands do not work properly in threads. Its a known issue which has so far not been fixed.
See also this answer: Can a Slack bot get the thread id that a slash command was sent from?
I am attempting to create a small dataset by pulling messages/responses from a slack channel I am a part of. I would like to use python to pull the data from the channel however I am having trouble figuring out my api key. I have created an app on slack but I am not sure how to find my api key. I see my client secret, signing secret, and verification token but can't find my api key
Here is a basic example of what I believe I am trying to accomplish:
import slack
sc = slack.SlackClient("api key")
sc.api_call(
"channels.history",
channel="C0XXXXXX"
)
I am willing to just download the data manually if that is possible as well. Any help is greatly appreciated.
messages
See below for is an example code on how to pull messages from a channel in Python.
It uses the official Python Slack library and calls
conversations_history with paging. It will therefore work with
any type of channel and can fetch large amounts of messages if
needed.
The result will be written to a file as JSON array.
You can specify channel and max message to be retrieved
threads
Note that the conversations.history endpoint will not return thread messages. Those have to be retrieved additionaly with one call to conversations.replies for every thread you want to retrieve messages for.
Threads can be identified in the messages for each channel by checking for the threads_ts property in the message. If it exists there is a thread attached to it. See this page for more details on how threads work.
IDs
This script will not replace IDs with names though. If you need that here are some pointers how to implement it:
You need to replace IDs for users, channels, bots, usergroups (if on a paid plan)
You can fetch the lists for users, channels and usergroups from the API with users_list, conversations_list and usergroups_list respectively, bots need to be fetched one by one with bots_info (if needed)
IDs occur in many places in messages:
user top level property
bot_id top level property
as link in any property that allows text, e.g. <#U12345678> for users or <#C1234567> for channels. Those can occur in the top level text property, but also in attachments and blocks.
Example code
import os
import slack
import json
from time import sleep
CHANNEL = "C12345678"
MESSAGES_PER_PAGE = 200
MAX_MESSAGES = 1000
# init web client
client = slack.WebClient(token=os.environ['SLACK_TOKEN'])
# get first page
page = 1
print("Retrieving page {}".format(page))
response = client.conversations_history(
channel=CHANNEL,
limit=MESSAGES_PER_PAGE,
)
assert response["ok"]
messages_all = response['messages']
# get additional pages if below max message and if they are any
while len(messages_all) + MESSAGES_PER_PAGE <= MAX_MESSAGES and response['has_more']:
page += 1
print("Retrieving page {}".format(page))
sleep(1) # need to wait 1 sec before next call due to rate limits
response = client.conversations_history(
channel=CHANNEL,
limit=MESSAGES_PER_PAGE,
cursor=response['response_metadata']['next_cursor']
)
assert response["ok"]
messages = response['messages']
messages_all = messages_all + messages
print(
"Fetched a total of {} messages from channel {}".format(
len(messages_all),
CHANNEL
))
# write the result to a file
with open('messages.json', 'w', encoding='utf-8') as f:
json.dump(
messages_all,
f,
sort_keys=True,
indent=4,
ensure_ascii=False
)
This is using the slack webapi. You would need to install requests package. This should grab all the messages in channel. You need a token which can be grabbed from apps management page. And you can use the getChannels() function. Once you grab all the messages you will need to see who wrote what message you need to do id matching(map ids to usernames) you can use getUsers() functions. Follow this https://api.slack.com/custom-integrations/legacy-tokens to generate a legacy-token if you do not want to use a token from your app.
def getMessages(token, channelId):
print("Getting Messages")
# this function get all the messages from the slack team-search channel
# it will only get all the messages from the team-search channel
slack_url = "https://slack.com/api/conversations.history?token=" + token + "&channel=" + channelId
messages = requests.get(slack_url).json()
return messages
def getChannels(token):
'''
function returns an object containing a object containing all the
channels in a given workspace
'''
channelsURL = "https://slack.com/api/conversations.list?token=%s" % token
channelList = requests.get(channelsURL).json()["channels"] # an array of channels
channels = {}
# putting the channels and their ids into a dictonary
for channel in channelList:
channels[channel["name"]] = channel["id"]
return {"channels": channels}
def getUsers(token):
# this function get a list of users in workplace including bots
users = []
channelsURL = "https://slack.com/api/users.list?token=%s&pretty=1" % token
members = requests.get(channelsURL).json()["members"]
return members
Lately, I've been trying to stream data from KDB to python.
I'm now using websockets and have gone through the doc https://code.kx.com/q/wp/kdb_and_websockets.pdf
On the python side, I've been trying ws4py, autobahn and websocket-client.
All do work fine, essentially my problem resides in the format of the message sent to the server to subscribe to the feed.
A little (open source) example:
class DummyClient(WebSocketClient):
def opened(self):
self.send(*what should I put here?*, binary=True)
def closed(self, code, reason=None):
print("Closed down", code, reason)
def received_message(self, m):
# not yet implemented
if __name__ == '__main__':
try:
ws = DummyClient('ws://host:port/', protocols=['http-only', 'chat'])
ws.connect()
ws.run_forever()
except KeyboardInterrupt:
ws.close()
When opening, I'm supposed to subscribe to the feed by calling the server function loadPage.
I've tried different things when to encode the list containing the function name and the argument, without success.
What I've tried:
np.array("['loadPage',[]]").tobytes()
"['loadPage',[]]".encode('utf8')
json formating
hexadecimal formating
Any help would be much appreciated!
Best,
Yael
I think what you'll need to do in this case is define .z.ws in your kdb server. This function will be called with whatever you pass over the websocket as an argument, so for example if you defined .z.ws like so:
.z.ws:{show x}
and then send a message over the WebSocket like so:
var ws = new WebSocket("ws://localhost:1234")
ws.send("Hello World")
This will be output in your kdb session e.g.
λ q
KDB+ 3.6 2018.06.14 Copyright (C) 1993-2018 Kx Systems
w64/ 4(16)core 8082MB jonat laptop-o8a8co1o 10.255.252.249 EXPIRE 2019.05.21 jonathon.mcmurray#aquaq.co.uk KOD #4160315
q).z.ws:{show x}
q)\p 1234
q)"Hello World"
So in your case if you want to call loadPage, it might be as simple as defining .z.ws like so:
.z.ws:loadPage
Then whatever you pass over the socket will be passed into loadPage (note this is will not update if you change loadPage elsewhere, you can instead use .z.ws:{loadPage x} if you need to be able to update it dynamically). For a niladic function, this will just ignore whatever is passed in
I've little question about Django Channels, WebSockets, and chat applications. Serving with google gets me to chatrooms, where people can connect and start a chat. But I don't know how one user can send another user instant message.
For example:
1) I add John to friends, and want to start chat.
2) On server side I can generate object Room, with me and John as members.
3) When I send message via WebSocket to this room, I know for who this message is, but I don't know how to get John's channel
#channel_session_user_from_http
def ws_connect(message):
rooms_with_user = Room.objects.filter(members=message.user)
for r in rooms_with_user:
Group('%s' % r.name).add(message.reply_channel)
#channel_session_user
def ws_receive(message):
prefix, label = message['path'].strip('/').split('/')
try:
room = Room.objects.get(name=label)
except Exception, e:
room = Room.objects.create(name=get_random_string(30))
for u in message.chmembers:
room.members.add(u)
# here can be somethis like this
# try
reply_channel = Channels.objects.get(online=True, user=u)
Group('%s' % r.name).add(reply_channel)
Group('%s' % room.name).send({
"text": "%s : %s" % (message.user.username, message['text']),
})
#channel_session_user
def ws_disconnect(message):
prefix, label = message['path'].strip('/').split('/')
Group(label).discard(message.reply_channel)
Simply make "automatic unique rooms" for user pairs. The rest stays the same. For example like this
def get_group_name(user1, user2):
return 'chat-{}-{}'.format(*sorted([user1.id, user2.id]))
Give it two user objects, and it returns a unique room for that pair of users, ordered the User.id, something like "chat-1-2" for the users with User.id "1" and "2".
That way, a user can connect with more than one logged-in device and still get the messages sent between the two users.
You can get the authenticated user's object from message.user.
For the receiving User object, I'd just sent the username along with the message. Then you can unpack it from the message['text'] the same way you unpack the actual message.
payload = json.loads(message.content['text'])
msg = payload['msg']
sender = message.user
receiver = get_object_or_404(User, username=payload['receiver'])
# ... here you could check if they have required permission ...
group_name = get_group_name(sender, receiver)
response = {'msg': msg}
Group(group_name).send({'text': json.dumps(response)})
# ... here you could persist the message in a database ...
So with that, you can drop all the "room" things from your example, including the room table etc. Because group names are always created on-the-fly when a message is send between two users.
Another important thing: One user will connect later than the other user, and may miss initial messages. So when you connect, you probably want to check some "chat_messages" database table, fetch the last 10 or 20 messages between the user pair, and send those back. So users can catch up on their past conversation.