I'm working on a chat project on the platform Raspberry PI 3, Openelec OS.
Trying to write to the DB and getting unwanted anonymous keys.
Unwanted key marked with yellow. Movie2 and it's keys and values are the wanted result, but I made it manually.
I only ask how can I prevent this anonymous random key to be there and how can I replace it with other key? (string, a movie name for example)
This is my code:
url = 'https://chat-example-97c62.firebaseio.com/Movie1.json'
postdata = {
'date': str(time.asctime( time.localtime(time.time()) )),
'temp': str("Hello from Kodi")
}
req = urllib2.Request(url)
req.add_header('Content-Type','application/json')
data = json.dumps(postdata)
Thanks.
When you send a POST request to Firebase, it automatically generates a Key (Anonymous Key), if you want to use your own key you need to use a PATCH request, this is an example on Python 3:
def update_entry(user, message):
my_data = dict()
my_data["user"] = user
my_data["message"] = message
json_data = json.dumps(my_data).encode()
request = urllib.requests.Request("https://<YOUR-PROJECT-ID>.firebaseio.com/movies/<REPLACE_THIS_WITH_YOUR_DESIRED_KEY>.json", data=json_data, method="PATCH")
try:
loader = urllib.request.urlopen(request)
except urllib.error.URLError as e:
message = json.loads(e.read())
print(message["error"])
else:
print(loader.read())
Related
I keep getting this error message when I run my script on my Djano App. (Object of type WSGIRequest is not JSON serializable) when I have my serializer set to JSON. If i change it to Pickle, I get this error message. (cannot serialize '_io.BufferedReader' object). I have spent days on this trying to figure out how to fix this. I appreciate any help. Thanks
Here is my script im sending to celery.
def ImportSchools(request):
print("Getting school data from SIS")
url = ""
payload = {}
token = APIInformation.objects.get(api_name="PowerSchool")
key = token.key
headers = {'Authorization': 'Bearer {}'.format(key)}
response = requests.request("GET", url, headers=headers, data = payload)
encode_xml = response.text.encode('utf8')
pretty_xml = xml.dom.minidom.parseString(encode_xml)
pretty_xml_str = pretty_xml.toprettyxml()
xml_string = ET.fromstring(encode_xml)
schools = xml_string.findall("school")
for school in schools:
psid = school.find("id").text
name = school.find("name").text
school_number = school.find("school_number").text
low_grade = school.find("low_grade").text
high_grade = school.find("high_grade").text
if not School.objects.filter(schoolpsid=psid):
print("Record doesn't exist in DB, creating record.")
x = School.objects.create(schoolpsid=psid, school_name=name, school_number=school_number, low_grade=low_grade, high_grade=high_grade)
x.save()
elif School.objects.filter(schoolpsid=psid).exists():
print("Record exists in DB, updating record.")
School.objects.filter(schoolpsid=psid).update(school_name=name, school_number=school_number, low_grade=low_grade, high_grade=high_grade)
print("School Data Pull Complete")
return("Done")
So the solution of this was to remove request as an argument and it fixed the issue.
I'm just wondering if there is any way to write a python script to check to see if a twitch.tv stream is live?
I'm not sure why my app engine tag was removed, but this would be using app engine.
Since all answers are actually outdated as of 2020-05-02, i'll give it a shot. You now are required to register a developer application (I believe), and now you must use an endpoint that requires a user-id instead of a username (as they can change).
See https://dev.twitch.tv/docs/v5/reference/users
and https://dev.twitch.tv/docs/v5/reference/streams
First you'll need to Register an application
From that you'll need to get your Client-ID.
The one in this example is not a real
TWITCH_STREAM_API_ENDPOINT_V5 = "https://api.twitch.tv/kraken/streams/{}"
API_HEADERS = {
'Client-ID' : 'tqanfnani3tygk9a9esl8conhnaz6wj',
'Accept' : 'application/vnd.twitchtv.v5+json',
}
reqSession = requests.Session()
def checkUser(userID): #returns true if online, false if not
url = TWITCH_STREAM_API_ENDPOINT_V5.format(userID)
try:
req = reqSession.get(url, headers=API_HEADERS)
jsondata = req.json()
if 'stream' in jsondata:
if jsondata['stream'] is not None: #stream is online
return True
else:
return False
except Exception as e:
print("Error checking user: ", e)
return False
I hated having to go through the process of making an api key and all those things just to check if a channel was live, so i tried to find a workaround:
As of june 2021 if you send a http get request to a url like https://www.twitch.tv/CHANNEL_NAME, in the response there will be a "isLiveBroadcast": true if the stream is live, and if the stream is not live, there will be nothing like that.
So i wrote this code as an example in nodejs:
const fetch = require('node-fetch');
const channelName = '39daph';
async function main(){
let a = await fetch(`https://www.twitch.tv/${channelName}`);
if( (await a.text()).includes('isLiveBroadcast') )
console.log(`${channelName} is live`);
else
console.log(`${channelName} is not live`);
}
main();
here is also an example in python:
import requests
channelName = '39daph'
contents = requests.get('https://www.twitch.tv/' +channelName).content.decode('utf-8')
if 'isLiveBroadcast' in contents:
print(channelName + ' is live')
else:
print(channelName + ' is not live')
It looks like Twitch provides an API (documentation here) that provides a way to get that info. A very simple example of getting the feed would be:
import urllib2
url = 'http://api.justin.tv/api/stream/list.json?channel=FollowGrubby'
contents = urllib2.urlopen(url)
print contents.read()
This will dump all of the info, which you can then parse with a JSON library (XML looks to be available too). Looks like the value returns empty if the stream isn't live (haven't tested this much at all, nor have I read anything :) ). Hope this helps!
RocketDonkey's fine answer seems to be outdated by now, so I'm posting an updated answer for people like me who stumble across this SO-question with google.
You can check the status of the user EXAMPLEUSER by parsing
https://api.twitch.tv/kraken/streams/EXAMPLEUSER
The entry "stream":null will tell you that the user if offline, if that user exists.
Here is a small Python script which you can use on the commandline that will print 0 for user online, 1 for user offline and 2 for user not found.
#!/usr/bin/env python3
# checks whether a twitch.tv userstream is live
import argparse
from urllib.request import urlopen
from urllib.error import URLError
import json
def parse_args():
""" parses commandline, returns args namespace object """
desc = ('Check online status of twitch.tv user.\n'
'Exit prints are 0: online, 1: offline, 2: not found, 3: error.')
parser = argparse.ArgumentParser(description = desc,
formatter_class = argparse.RawTextHelpFormatter)
parser.add_argument('USER', nargs = 1, help = 'twitch.tv username')
args = parser.parse_args()
return args
def check_user(user):
""" returns 0: online, 1: offline, 2: not found, 3: error """
url = 'https://api.twitch.tv/kraken/streams/' + user
try:
info = json.loads(urlopen(url, timeout = 15).read().decode('utf-8'))
if info['stream'] == None:
status = 1
else:
status = 0
except URLError as e:
if e.reason == 'Not Found' or e.reason == 'Unprocessable Entity':
status = 2
else:
status = 3
return status
# main
try:
user = parse_args().USER[0]
print(check_user(user))
except KeyboardInterrupt:
pass
Here is a more up to date answer using the latest version of the Twitch API (helix). (kraken is deprecated and you shouldn't use GQL since it's not documented for third party use).
It works but you should store the token and reuse the token rather than generate a new token every time you run the script.
import requests
client_id = ''
client_secret = ''
streamer_name = ''
body = {
'client_id': client_id,
'client_secret': client_secret,
"grant_type": 'client_credentials'
}
r = requests.post('https://id.twitch.tv/oauth2/token', body)
#data output
keys = r.json();
print(keys)
headers = {
'Client-ID': client_id,
'Authorization': 'Bearer ' + keys['access_token']
}
print(headers)
stream = requests.get('https://api.twitch.tv/helix/streams?user_login=' + streamer_name, headers=headers)
stream_data = stream.json();
print(stream_data);
if len(stream_data['data']) == 1:
print(streamer_name + ' is live: ' + stream_data['data'][0]['title'] + ' playing ' + stream_data['data'][0]['game_name']);
else:
print(streamer_name + ' is not live');
📚 Explanation
Now, the Twitch API v5 is deprecated. The helix API is in place, where an OAuth Authorization Bearer AND client-id is needed. This is pretty annoying, so I went on a search for a viable workaround, and found one.
🌎 GraphQL
When inspecting Twitch's network requests, while not being logged in, I found out the anonymous API relies on GraphQL. GraphQL is a query language for APIs.
query {
user(login: "USERNAME") {
stream {
id
}
}
}
In the graphql query above, we are querying a user by their login name. If they are streaming, the stream's id will be given. If not, None will be returned.
🐍 The Final Code
The finished python code, in a function, is below. The client-id is taken from Twitch's website. Twitch uses the client-id to fetch information for anonymous users. It will always work, without the need of getting your own client-id.
import requests
# ...
def checkIfUserIsStreaming(username):
url = "https://gql.twitch.tv/gql"
query = "query {\n user(login: \""+username+"\") {\n stream {\n id\n }\n }\n}"
return True if requests.request("POST", url, json={"query": query, "variables": {}}, headers={"client-id": "kimne78kx3ncx6brgo4mv6wki5h1ko"}).json()["data"]["user"]["stream"] else False
I've created a website where you can play with Twitch's GraphQL API. Refer to the GraphQL Docs for help on GraphQL syntax! There's also Twitch GraphQL API documentation on my playground.
Use the twitch api with your client_id as a parameter, then parse the json:
https://api.twitch.tv/kraken/streams/massansc?client_id=XXXXXXX
Twitch Client Id is explained here: https://dev.twitch.tv/docs#client-id,
you need to register a developer application: https://www.twitch.tv/kraken/oauth2/clients/new
Example:
import requests
import json
def is_live_stream(streamer_name, client_id):
twitch_api_stream_url = "https://api.twitch.tv/kraken/streams/" \
+ streamer_name + "?client_id=" + client_id
streamer_html = requests.get(twitch_api_stream_url)
streamer = json.loads(streamer_html.content)
return streamer["stream"] is not None
I'll try to shoot my shot, just in case someone still needs an answer to this, so here it goes
import requests
import time
from twitchAPI.twitch import Twitch
client_id = ""
client_secret = ""
twitch = Twitch(client_id, client_secret)
twitch.authenticate_app([])
TWITCH_STREAM_API_ENDPOINT_V5 = "https://api.twitch.tv/kraken/streams/{}"
API_HEADERS = {
'Client-ID' : client_id,
'Accept' : 'application/vnd.twitchtv.v5+json',
}
def checkUser(user): #returns true if online, false if not
userid = twitch.get_users(logins=[user])['data'][0]['id']
url = TWITCH_STREAM_API_ENDPOINT_V5.format(userid)
try:
req = requests.Session().get(url, headers=API_HEADERS)
jsondata = req.json()
if 'stream' in jsondata:
if jsondata['stream'] is not None:
return True
else:
return False
except Exception as e:
print("Error checking user: ", e)
return False
print(checkUser('michaelreeves'))
https://dev.twitch.tv/docs/api/reference#get-streams
import requests
# ================================================================
# your twitch client id
client_id = ''
# your twitch secret
client_secret = ''
# twitch username you want to check if it is streaming online
twitch_user = ''
# ================================================================
#getting auth token
url = 'https://id.twitch.tv/oauth2/token'
params = {
'client_id':client_id,
'client_secret':client_secret,
'grant_type':'client_credentials'}
req = requests.post(url=url,params=params)
token = req.json()['access_token']
print(f'{token=}')
# ================================================================
#getting user data (user id for example)
url = f'https://api.twitch.tv/helix/users?login={twitch_user}'
headers = {
'Authorization':f'Bearer {token}',
'Client-Id':f'{client_id}'}
req = requests.get(url=url,headers=headers)
userdata = req.json()
userid = userdata['data'][0]['id']
print(f'{userid=}')
# ================================================================
#getting stream info (by user id for example)
url = f'https://api.twitch.tv/helix/streams?user_id={userid}'
headers = {
'Authorization':f'Bearer {token}',
'Client-Id':f'{client_id}'}
req = requests.get(url=url,headers=headers)
streaminfo = req.json()
print(f'{streaminfo=}')
# ================================================================
This solution doesn't require registering an application
import requests
HEADERS = { 'client-id' : 'kimne78kx3ncx6brgo4mv6wki5h1ko' }
GQL_QUERY = """
query($login: String) {
user(login: $login) {
stream {
id
}
}
}
"""
def isLive(username):
QUERY = {
'query': GQL_QUERY,
'variables': {
'login': username
}
}
response = requests.post('https://gql.twitch.tv/gql',
json=QUERY, headers=HEADERS)
dict_response = response.json()
return True if dict_response['data']['user']['stream'] is not None else False
if __name__ == '__main__':
USERS = ['forsen', 'offineandy', 'dyrus']
for user in USERS:
IS_LIVE = isLive(user)
print(f'User {user} live: {IS_LIVE}')
Yes.
You can use Twitch API call https://api.twitch.tv/kraken/streams/YOUR_CHANNEL_NAME and parse result to check if it's live.
The below function returns a streamID if the channel is live, else returns -1.
import urllib2, json, sys
TwitchChannel = 'A_Channel_Name'
def IsTwitchLive(): # return the stream Id is streaming else returns -1
url = str('https://api.twitch.tv/kraken/streams/'+TwitchChannel)
streamID = -1
respose = urllib2.urlopen(url)
html = respose.read()
data = json.loads(html)
try:
streamID = data['stream']['_id']
except:
streamID = -1
return int(streamID)
I have many functions that successfully POST a urlencoded body using python. However, I have a body that is a multidimensional dictionary. Using this dictionary, I only get a 400 (bad request). bodytmp (below) is an example of the body that works using Fiddler. (The actual url and body can not be supplied.) I have also included a recursive urlencode function that I found here and am using now but with no success.
Does anyone have experience with this type of POST request with a multidimensional dictionary for the urlencoded body?
Thank you. ( I have abbreviated the code to make it more readable, but this is the gist. )
from httplib2 import Http
import httplib
import urllib
def postAgentRegister():
h = Http(disable_ssl_certificate_validation=True)
headers={'Content-Type':'application/x-www-form-urlencoded'}
bodytmp={"Username": "username",
"Password": "password",
"AccountId": "accountid",
"PublicKey": {
"Value1a": "32ab45cd",
"value1b": "10001"
},
"snId": "SN1",
"IpAddresses": {
"Value2a": ["50.156.54.45"],
"Value2b": "null"
}
}
body=recursive_urlencode(bodytmp)
try:
response,content=h.request('https://server/endpoint','POST',headers=headers,body=body)
conn = httplib.HTTPConnection(testserver)
conn.request('POST', endpoint, body, headers)
r1 = conn.getresponse()
status = r1.status
reason = r1.reason
except httplib.HTTPException, e:
status= e.code
print 'status is: ', status
def recursive_urlencode(d):
def recursion(d, base=None):
pairs = []
for key, value in d.items():
if hasattr(value, 'values'):
pairs += recursion(value, key)
else:
new_pair = None
if base:
new_pair = "%s[%s]=%s" % (base, urllib.quote(unicode(key)), urllib.quote(unicode(value)))
else:
new_pair = "%s=%s" % (urllib.quote(unicode(key)), urllib.quote(unicode(value)))
pairs.append(new_pair)
return pairs
return '&'.join(recursion(d))
Might I suggest that you serialize your body to JSON and de-serialize when your server receives it? This way you just have to do a url string encode rather than using your own recursive url encode.
I'm writing a simple python script that will interface with the AIM servers using the OSCAR protocol. It includes a somewhat complex handshake protocol. You essentially have to send a GET request to a specific URL, receive XML or JSON encoded reply, extract a special session token and secret key, then generate a response using the token and the key.
I tried to follow these steps to a tee, but the process fails in the last one. Here is my code:
class simpleOSCAR:
def __init__(self, username, password):
self.username = username
self.password = password
self.open_aim_key = 'whatever'
self.client_name = 'blah blah blah'
self.client_version = 'yadda yadda yadda'
def authenticate(self):
# STEP 1
url = 'https://api.screenname.aol.com/auth/clientLogin?f=json'
data = urllib.urlencode( [
('k', self.open_aim_key),
('s', self.username),
('pwd', self.password),
('clientVersion', self.client_version),
('clientName', self.client_name)]
)
response = urllib2.urlopen(url, data)
json_response = simplejson.loads(urllib.unquote(response.read()))
session_secret = json_response['response']['data']['sessionSecret']
host_time = json_response['response']['data']['hostTime']
self.token = json_response['response']['data']['token']['a']
# STEP 2
self.session_key = base64.b64encode(hmac.new(self.password, session_secret, sha256).digest())
#STEP 3
uri = "http://api.oscar.aol.com/aim/startOSCARSession?"
data = urllib.urlencode([
('a', self.token),
('clientName', self.client_name),
('clientVersion', self.client_version),
('f', 'json'),
('k', self.open_aim_key),
('ts', host_time),
]
)
urldata = uri+data
hashdata = "GET&" + urllib.quote("http://api.oscar.aol.com/aim/startOSCARSession?") + data
digest = base64.b64encode(hmac.new(self.session_key, hashdata, sha256).digest())
urldata = urldata + "&sig_sha256=" + digest
print urldata + "\n"
response = urllib2.urlopen(urldata)
json_response = urllib.unquote(response.read())
print json_response
if __name__ == '__main__':
so = simpleOSCAR("aimscreenname", "somepassword")
so.authenticate()
I get the following response from the server:
{ "response" : {
"statusCode":401,
"statusText":"Authentication Required. statusDetailCode 1014",
"statusDetailCode":1014,
"data":{
"ts":1235878395
}
}
}
I tried troubleshooting it in various ways, but the URL's I generate look the same as the ones shown in the signon flow example. And yet, it fails.
Any idea what I'm doing wrong here? Am I hashing the values wrong? Am I encoding something improperly? Is my session timing out?
Try using Twisted's OSCAR support instead of writing your own? It hasn't seen a lot of maintenance, but I believe it works.
URI Encode your digest?
-moxford
Is there a better pattern for input validation than I'm using in this function?
https://github.com/nathancahill/clearbit-intercom/blob/133e4df0cfd1a146cedb3c749fc1b4fac85a6e1b/server.py#L71
Here's the same function without any validation. It's much more readable, it's short and to the point (9 LoC vs 53 LoC).
def webhook(clearbitkey, appid, intercomkey):
event = request.get_json()
id = event['data']['item']['id']
email = event['data']['item']['email']
person = requests.get(CLEARBIT_USER_ENDPOINT.format(email=email), auth=(clearbitkey, '')).json()
domain = person['employment']['domain']
company = requests.get(CLEARBIT_COMPANY_ENDPOINT.format(domain=domain), auth=(clearbitkey, '')).json()
note = create_note(person, company)
res = requests.post(INTERCOM_ENDPOINT,
json=dict(user=dict(id=id), body=note),
headers=dict(accept='application/json'),
auth=(appid, intercomkey))
return jsonify(note=res.json())
However, it doesn't handle any of these errors:
dict KeyError's (especially nested dicts)
HTTP errors
Invalid JSON
Unexpected responses
Is there a better pattern to follow? I looked into using a data validation library like voluptous but it seems like I'd still have the same problem of verbosity.
Your original code on github seems fine to me. It's a little over complicated, but also handle all cases of error. You can try to improve readability by abstract things.
Just for demonstration, I may write code like this:
class ValidationError(Exception):
"Raises when data validation fails"
pass
class CallExternalApiError(Exception):
"Raises when calling external api fails"
pass
def get_user_from_event(event):
"""Get user profile from event
:param dict event: request.get_json() result
:returns: A dict of user profile
"""
try:
event_type = event['data']['item']['type']
except KeyError:
raise ValidationError('Unexpected JSON format.')
if event_type != 'user':
return ValidationError('Event type is not supported.')
try:
id = event['data']['item']['id']
email = event['data']['item']['email']
except KeyError:
return ValidationError('User object missing fields.')
return {'id': id, 'email': email}
def call_json_api(request_function, api_name, *args, **kwargs):
"""An simple wrapper for sending request
:param request_function: function used for sending request
:param str api_name: name for this api call
"""
try:
res = request_function(*args, **kwargs)
except:
raise CallExternalApiError('API call failed to %s.' % api_name)
try:
return res.json()
except:
raise CallExternalApiError('Invalid response from %s.' % api_name)
#app.route('/<clearbitkey>+<appid>:<intercomkey>', methods=['POST'])
def webhook(clearbitkey, appid, intercomkey):
"""
Webhook endpoint for Intercom.io events. Uses this format for Clearbit and
Intercom.io keys:
/<clearbitkey>+<appid>:<intercomkey>
:clearbitkey: Clearbit API key.
:appid: Intercom.io app id.
:intercomkey: Intercom.io API key.
Supports User events, specifically designed for the User Created event.
Adds a note to the user with their employment and company metrics.
"""
event = request.get_json()
try:
return handle_event(event, clearbitkey, appid, intercomkey)
except (ValidationError, CallExternalApiError) as e:
# TODO: include **res_objs in response
return jsonify(error=str(e))
def handle_event(event):
"""Handle the incoming event
"""
user = get_user_from_event(event)
res_objs = dict(event=event)
person = call_json_api(
requests.get,
'Clearbit',
CLEARBIT_USER_ENDPOINT.format(email=user['email']),
auth=(clearbitkey, '')
)
res_objs['person'] = person
if 'error' in person:
raise CallExternalApiError('Error response from Clearbit.')
domain = person['employment']['domain']
company = None
if domain:
try:
company = call_json_api(
requests.get,
'Clearbit',
CLEARBIT_COMPANY_ENDPOINT.format(domain=domain),
auth=(clearbitkey, ''))
)
if 'error' in company:
company = None
except:
company = None
res_objs['company'] = company
try:
note = create_note(person, company)
except:
return jsonify(error='Failed to generate note for user.', **res_objs)
result = call_json_api(
requests.post,
'Intercom',
(INTERCOM_ENDPOINT, json=dict(user=dict(id=id), body=note),
headers=dict(accept='application/json'),
auth=(appid, intercomkey)
)
return jsonify(note=result, **res_objs)
I hope it helps.