Develop Reference

How to access data in documents from realtime listener python

Apologies if some of this doesn't make sense, I'm struggling to understand realtime listeners completely.
I'm trying to add a realtime listener to the chat part of my app, so I can add new messages to the screen as they come into the database. In the below code I load all current messages to the screen when the user opens the page and then I (try to) add the realtime listener in so any new messages can be added to the screen.
However, the doc_snapshot is just a list of the document ids, rather than the message_dict I have been using above, how do I access the data for each document in doc_snapshot, rather than just the id?
Or am I doing it completely wrong, should I not load do a one-time load of the messages when the screen is opened and just use a realtime listener to load the messages and listen for new messages?
self.local_id is the id of the user who has logged in, doc_id is the id of the person they're messaging.
def move_to_chat(self, doc_id):
group_id = self.local_id + ":"+ doc_id
doc_ref = self.my_firestore.db.collection(u'messages').document(group_id)
doc = doc_ref.get()
if doc.exists: # Check if the document exists. If it does, load the messages to the screen
get_messages = self.my_firestore.db.collection(u'messages').document(group_id).collection(group_id).order_by(u'Timestamp').limit(20)
messages = get_messages.stream()
for message in messages:
message_dict = message.to_dict()
try:
if message_dict['IdFrom'] == self.local_id:
#Add label to left of screen
else:
#Add label to right of screen
except:
pass
else: # If it doesn't, create it
self.my_firestore.db.collection(u'messages').document(group_id).set({
u'GroupId': group_id
})
add_to_doc = self.my_firestore.db.collection(u'messages').document(group_id).collection(group_id).document()
add_to_doc.set({
u'Timestamp': datetime.datetime.now()
})
# Watch for new messages
self.query_watch = self.my_firestore.db.collection(u'messages').document(group_id).collection(group_id)
# Watch the document
self.query_watch.on_snapshot(self.on_snapshot)
def on_snapshot(self, doc_snapshot, changes, read_time):
for doc in doc_snapshot:
#Here's where I'd like to access data from the documents, to find the message that has been added.

The Google Firestore documentation for Snapshot method explained the Classes for representing documents for the Google Cloud Firestore API.You can refer to this document to conform the returning values.

How to group Cloud Function log entries by execution?

In Google App Engine, first generation, logs are grouped automatically by request in Logs Viewer, and in the second generation it's easy enough to set up.
In background Cloud Functions I can't find any way of doing it (save manually filtering by executionId in Logs Viewer). From various articles around the web I read that the key is to set the trace argument to the Trace ID when calling the Stackdriver Logging API, and that in HTTP contexts this ID can be found in the X-Cloud-Trace-Context header.
There are no headers in background contexts (for example, called from Pub/Sub or Storage triggers). I've tried setting this to an arbitrary value, such as the event_id from the function context, but no grouping happens.
Here's a minified representation of how I've tried it:
from google.cloud.logging.resource import Resource
import google.cloud.logging
log_name = 'cloudfunctions.googleapis.com%2Fcloud-functions'
cloud_client = google.cloud.logging.Client()
cloud_logger = cloud_client.logger(log_name)
request_id = None
def log(message):
labels = {
'project_id': 'settle-leif',
'function_name': 'group-logs',
'region': 'europe-west1',
}
resource = Resource(type='cloud_function', labels=labels)
trace_id = f'projects/settle-leif/traces/{request_id}'
cloud_logger.log_text(message, trace=trace_id, resource=resource)
def main(_data, context):
global request_id
request_id = context.event_id
log('First message')
log('Second message')

This is currently possible.
It's on our roadmap to provide this support: https://github.com/GoogleCloudPlatform/functions-framework-python/issues/79

Pulling historical channel messages python

I am attempting to create a small dataset by pulling messages/responses from a slack channel I am a part of. I would like to use python to pull the data from the channel however I am having trouble figuring out my api key. I have created an app on slack but I am not sure how to find my api key. I see my client secret, signing secret, and verification token but can't find my api key
Here is a basic example of what I believe I am trying to accomplish:
import slack
sc = slack.SlackClient("api key")
sc.api_call(
"channels.history",
channel="C0XXXXXX"
)
I am willing to just download the data manually if that is possible as well. Any help is greatly appreciated.

messages
See below for is an example code on how to pull messages from a channel in Python.
It uses the official Python Slack library and calls
conversations_history with paging. It will therefore work with
any type of channel and can fetch large amounts of messages if
needed.
The result will be written to a file as JSON array.
You can specify channel and max message to be retrieved
threads
Note that the conversations.history endpoint will not return thread messages. Those have to be retrieved additionaly with one call to conversations.replies for every thread you want to retrieve messages for.
Threads can be identified in the messages for each channel by checking for the threads_ts property in the message. If it exists there is a thread attached to it. See this page for more details on how threads work.
IDs
This script will not replace IDs with names though. If you need that here are some pointers how to implement it:
You need to replace IDs for users, channels, bots, usergroups (if on a paid plan)
You can fetch the lists for users, channels and usergroups from the API with users_list, conversations_list and usergroups_list respectively, bots need to be fetched one by one with bots_info (if needed)
IDs occur in many places in messages:
user top level property
bot_id top level property
as link in any property that allows text, e.g. <#U12345678> for users or <#C1234567> for channels. Those can occur in the top level text property, but also in attachments and blocks.
Example code
import os
import slack
import json
from time import sleep
CHANNEL = "C12345678"
MESSAGES_PER_PAGE = 200
MAX_MESSAGES = 1000
# init web client
client = slack.WebClient(token=os.environ['SLACK_TOKEN'])
# get first page
page = 1
print("Retrieving page {}".format(page))
response = client.conversations_history(
channel=CHANNEL,
limit=MESSAGES_PER_PAGE,
)
assert response["ok"]
messages_all = response['messages']
# get additional pages if below max message and if they are any
while len(messages_all) + MESSAGES_PER_PAGE <= MAX_MESSAGES and response['has_more']:
page += 1
print("Retrieving page {}".format(page))
sleep(1) # need to wait 1 sec before next call due to rate limits
response = client.conversations_history(
channel=CHANNEL,
limit=MESSAGES_PER_PAGE,
cursor=response['response_metadata']['next_cursor']
)
assert response["ok"]
messages = response['messages']
messages_all = messages_all + messages
print(
"Fetched a total of {} messages from channel {}".format(
len(messages_all),
CHANNEL
))
# write the result to a file
with open('messages.json', 'w', encoding='utf-8') as f:
json.dump(
messages_all,
f,
sort_keys=True,
indent=4,
ensure_ascii=False
)

This is using the slack webapi. You would need to install requests package. This should grab all the messages in channel. You need a token which can be grabbed from apps management page. And you can use the getChannels() function. Once you grab all the messages you will need to see who wrote what message you need to do id matching(map ids to usernames) you can use getUsers() functions. Follow this https://api.slack.com/custom-integrations/legacy-tokens to generate a legacy-token if you do not want to use a token from your app.
def getMessages(token, channelId):
print("Getting Messages")
# this function get all the messages from the slack team-search channel
# it will only get all the messages from the team-search channel
slack_url = "https://slack.com/api/conversations.history?token=" + token + "&channel=" + channelId
messages = requests.get(slack_url).json()
return messages
def getChannels(token):
'''
function returns an object containing a object containing all the
channels in a given workspace
'''
channelsURL = "https://slack.com/api/conversations.list?token=%s" % token
channelList = requests.get(channelsURL).json()["channels"] # an array of channels
channels = {}
# putting the channels and their ids into a dictonary
for channel in channelList:
channels[channel["name"]] = channel["id"]
return {"channels": channels}
def getUsers(token):
# this function get a list of users in workplace including bots
users = []
channelsURL = "https://slack.com/api/users.list?token=%s&pretty=1" % token
members = requests.get(channelsURL).json()["members"]
return members

Test Environment with Mocked REST API

Lets say I have a very simple web app which is presented as blue if the current president is a democrat and red if they are a republican. A REST API is used to get the current president, via the endpoint:
/presidents/current
which currently returns the json object:
{name: "Donald Trump", party: "Republican"}
So when my page loads I call the endpoint and I show red or blue depending on who is returned.
I wish to test this HTML/javascript page and I wish to mock the back-end so that I can control from within the test environment the API responses. For example:
def test_republican():
# configure the response for this test that the web app will receive when it connects to this endpoint
configure_endpoint(
"/presidents/current",
jsonify(
name="Donald Trump",
party="Republican"
)
)
# start the web app in the browser using selenium
load_web_app(driver, "http://localhost:8080")
e = driver.find_element_by_name("background")
assert(e.getCssValue("background-color") == "red")
def test_democrat():
# configure the response for this test that the web app will receive when it connects to this endpoint
configure_endpoint(
"/presidents/current",
jsonify(
name="Barack Obama",
party="Democrat"
)
)
# start the web app in the browser using selenium
load_web_app(driver, "http://localhost:8080")
e = driver.find_element_by_name("background")
assert(e.getCssValue("background-color") == "blue")
So the question is how should I implement the function configure_endpoint() and what libraries can you recommend me?

As #Kie mentioned, configure_endpoint implementation won't be enough, if you're going to stub the whole server-side within Selenium Python code. You would need a web server or whatever that will response via HTTP to requests from within testing environment.
It looks like the question is partially about testing of client-side code. What I see is that you're trying to make unit-test for client-side logic, but use integration testing suite in order to check this logic (it's strange).
The main idea is as follows.
You're trying to test client-side code. So, let's make mocks client-side too! Because this part of code is completely client-side related stuff.
If you actually want to have mocks, not stubs (watch the difference here: https://stackoverflow.com/a/3459491/882187) it is a better way to mock out HTTP requests inside your Javascript code. Just because you're testing a client-side piece of code, not some parts of server-side logic.
Having it isolated from whatever server-side is - is a great idea that you would love when your project become grow, while more and more endpoints will be appearing.
For example, you can use the following approach:
var restResponder = function() { // the original responder your client-side app will use
this.getCurrentPresident = function(successCallback) {
$.get('/presidents/current', callback);
}
};
var createMockResponder = function(president, party){ // factory that creates mocks
var myPresident = president;
var myParty = party;
return function() {
this.getCurrentPresident = function (successCallback) {
successCallback({"name": myPresident, "party": myParty});
}
};
}
// somewhere swap the original restResponder with new mockResponder created by 'createMockResponder'
// then use it in your app:
function drawColor(restResponder, backgroundEl) {
restResponder.getCurrentPresident(function(data){
if (data.party == "Democrat") $(backgroundEl).style('background-color', 'blue')
else if (data.party == "Republican") $(backgroundEl).style('background-color', 'red')
else console.info('Some strange response from server... Nevermind...');
});
}
Practically, this implementation depends on what do you have at the client-side as a framework. If jQuery, then my example is enough, but it looks very wordy. In case you have something more advanced, like AngularJS, you can do the same in 2-3 lines of code:
// Set up the mock http service responses
$httpBackend = $injector.get('$httpBackend');
// backend definition common for all tests
authRequestHandler = $httpBackend.when('GET', '/auth.py')
.respond({userId: 'userX'}, {'A-Token': 'xxx'});
Check out the docs: https://docs.angularjs.org/api/ngMock/service/$httpBackend
If you're still stick to the idea, that you need mocks inside Selenium tests, please
try this project: https://turq.readthedocs.io/en/latest/
It serves with Python DSL for describing REST responders.
Using turq your mocks will look as follows:
path('/presidents/current').json({'name':'Barack Obama', 'party': 'Democrat'}, jsonp=False)
Also, I would recommend to try stubs instead of mocks and use this Python module: mock-server https://pypi.python.org/pypi/mock-server/0.3.7
You are required to create the directory layout containing corresponding pre-populated JSON responses and to add some boilerplate code in order to make the mock-server respond on 'localhost:8080'. The directory layout for your example will look like this:
stub_obama/
presidents/
current/
GET_200.json # will contain {"name": "Barack Obama", "party": "Democrat"}
stub_trump/
presidents/
current/
GET_200.json # will contain {"name": "Donald Trump", "party": "Republican"}
But the mock_server is based on Tornado, it is very heavy solution for using in tests I think.
I hope, my answer is helpful and informative. Welcome to discuss it! I made tons of projects with Selenium, big and small tests, tested client-side and server-side.

I would use tornado web framework.
import json
import functools
import operator
from tornado import ioloop, web, gen
from tornado.options import define, options
define("data_file", default='default/mock.json', type=str)
class Handler(web.RequestHandler):
def data_received(self, chunk):
pass
def initialize(self, data):
self.data = data
#gen.coroutine
def get(self, *args, **kwargs):
path = self.request.path.split("/")[1:]
path = functools.reduce(
operator.add,
[[k, v[0].decode("utf-8")] for k, v in self.request.query_arguments.items()],
path
)
try:
self.write(functools.reduce(operator.getitem, path, self.data))
except KeyError:
self.set_status(404)
class Application(web.Application):
def __init__(self):
data = {}
with open(options.data_file) as data_file:
data = json.load(data_file)
handlers = [
('(.*)', Handler, {"data": data})
]
settings = dict(
gzip=True,
static_hash_cache=True,
)
web.Application.__init__(self, handlers, **settings)
def main():
io_loop = ioloop.IOLoop.instance()
backend_application = Application()
backend_application.listen(8001)
io_loop.start()
if __name__ == "__main__":
main()
This is a code I used for mocking a REST-API which is a standalone script, but it can be embedded into your test environment as well.
I defined a JSON file which defines the different path components and what should be returned. Like this:
{
"presidents": {
"current": {
"name": "Donald Trump",
"party": "Republican"
}
}
}
I saved this to a mock.json and called the script with a parameter mock_rest.py --data-file="./mock.json".
I hope that gives you a starting point and a good example.

If your load_web_app function uses the requests library to access the REST API, using requests-mock is a convenient way to fake that library's functionality for test purposes.

For those who stumble upon this question, and do not want to end up writing the code to create their own mock server implementations of the API, you can use Mocktastic, which is a downloadable desktop application for Windows, MacOS and Linux, which provides an easy to use GUI to setup your mock API servers.

Reading page's messages with Python Facebook SDK

Basically i need to get all messages of a page using facebook SDK in python.
Following some tutorial i arrived to this point:
import facebook
def main():
cfg = {
"page_id" : "MY PAGE ID",
"access_token" : "LONG LIVE ACCESS TOKEN"
}
api = get_api(cfg)
msg = "Hre"
status = api.put_wall_post(msg) #used to post to wall message Hre
x = api.get_object('/'+str(MY PAGE ID)+"/conversations/") #Give actual conversations
def get_api(cfg):
graph = facebook.GraphAPI(cfg['access_token'])
resp = graph.get_object('me/accounts')
page_access_token = None
for page in resp['data']:
if page['id'] == cfg['page_id']:
page_access_token = page['access_token']
graph = facebook.GraphAPI(page_access_token)
return graph
if __name__ == "__main__":
main()
The first problem is that api.get_object('/'+str(MY PAGE ID)+"/conversations/")returns a dictionary containing many informations, but what i would like to see is the messages they sent to me, while for now it print the user id that sent to me a message.
The output look like the following:
{u'paging': {u'next': u'https://graph.facebook.com/v2.4/571499452991432/conversations?access_token=Token&limit=25&until=1441825848&__paging_token=enc_AdCqaKAP3e1NU9MGSsvSdzDPIIDtB2ZCe2hCYfk7ft5ZAjRhsuVEL7eFYOOCdQ8okvuhZA5iQWaYZBBbrZCRNW8uzWmgnKGl69KKt4catxZAvQYCus7gZDZD', u'previous': u'https://graph.facebook.com/v2.4/571499452991432/conversations?access_token=token&limit=25&since=1441825848&__paging_token=enc_AdCqaKAP3e1NU9MGSsvSdzDPIIDtB2ZCe2hCYfk7ft5ZAjRhsuVEL7eFYOOCdQ8okvuhZA5iQWaYZBBbrZCRNW8uzWmgnKGl69KKt4catxZAvQYCus7gZDZD&__previous=1'}, u'data': [{u'link': u'/communityticino/manager/messages/?mercurythreadid=user%3A1055476438&threadid=mid.1441825847634%3Af2e0247f54f5c4d222&folder=inbox', u'id': u't_mid.1441825847634:f2e0247f54f5c4d222', u'updated_time': u'2015-09-09T19:10:48+0000'}]}
which is basically paging and data.
Given this is there a way to read the conversation?

In order to get the messages content you need first to request the single messages in the conversation, accessible with the 'id' field in the dictionary you copied, result of
x = api.get_object('/'+str(MY PAGE ID)+"/conversations/") #Give actual conversations
you can request the messages in the conversation by calling
msg = api.get_object('/'+<message id>)
Here it gets tricky, because following the graph api documentation you should receive back a dictionary with ALL the possible fields, including the 'message' (content) field. The function however returns only the fields 'created_time' and 'id'.
Thanks to this other question Request fields in Python Facebook SDK I found that you can request for those fields by adding a dict with such fields specified in the arguments of the graph.get_object() function. As far as I know this is undocumented in the facebook sdk reference for python.
The correct code is
args = {'fields' : 'message'}
msg = api.get_object('/'+<message id>, **args)
Similar question: Read facebook messages using python sdk