Develop Reference

PlayerDB API Post Requests bring 404

I made a little script to get the UUID's of people who joined my minecraft server, then run them through the PlayerDB API through a post request to: https://playerdb.co/api/player/minecraft/* where the * is a player UUID, however it returns random 404 errors.
Error runs on the player_name_history line and highlights KeyError at 'player':
def getPlayerData(UUID_list):
baseurl = "https://playerdb.co/api/player/minecraft/"
player_names=[]
icon_list = []
for UUID in UUID_list:
response = requests.post(baseurl+UUID)
print("url posted:",baseurl+UUID)
print(response.status_code)
responseObject = response.json()
with open('responseObject.json','w') as f:
json.dump(responseObject, f)
player_name_history = responseObject['data']['player']['meta']['name_history']
for x in player_name_history:
player_name = x['name'] #iterates through all names, last name will not be overrwritten
player_names.append(player_name)
icon_link = responseObject['data']['player']['avatar']
icon_list.append(icon_link)
return player_names, icon_list
for x in player_name_history:
player_name = x['name'] #iterates through all names, last name will not be overrwritten
player_names.append(player_name)
icon_link = responseObject['data']['player']['avatar']
icon_list.append(icon_link)
return player_names, icon_list
You can pass my UUID into the function as a list to test:
['bf22088f-3d5b-45ef-b7dd-8d5bd3cdc310']
Example of it working:
url posted: https://playerdb.co/api/player/minecraft/bf22088f-3d5b-45ef-b7dd-8d5bd3cdc310
200
(['zonkedzolt'], ['https://crafthead.net/avatar/bf22088f-3d5b-45ef-b7dd-8d5bd3cdc310'])
Example of it not working:
url posted: https://playerdb.co/api/player/minecraft/bf22088f-3d5b-45ef-b7dd-8d5bd3cdc310
404
Traceback (most recent call last):
File "g:\*\getPlayerData.py", line 74, in <module>
print(getPlayerData(UUID_list))
File "g:\*\getPlayerData.py", line 58, in getPlayerData
player_name_history = responseObject['data']['player']['meta']['name_history']
KeyError: 'player'
json Dump: {"message": "", "code": "api.404", "data": {}, "success": false, "error": false}
The reason why i suspect it might be my code is because when i get the error, if i ctrl+left click the link on the "url posted:" line it brings up the correct result.

If you are getting error messages from the API like this:
{"message": "", "code": "api.404", "data": {}, "success": false, "error": false}
try requesting each UUID seperately and once you have gotten a good response, try running it with your program. It seems that the API caches UUIDs that have been asked for the first time, but if you ask again too quickly it will return the error above.
Occasionally I still run into the error above, but a re-request sorts it out. You can make a while loop to keep requesting until you recieve a good response. Here is the while loop I made:
goodResponse = False
while goodResponse == False:
response = requests.post(baseurl+UUID)
print("url posted:",baseurl+UUID)
print(response.status_code)
responseObject = response.json()
if "player" in responseObject['data']: #checks if UUID has recieved a good response, otherwise loops until it does.
goodResponse = True #continue code here

Use string formatting to hide username and password in JSON payload sent in request header

I have a function that has parameters with username and password. I want to use f strings or any sort of string formatting so that I don't have to hard code the username and password in the code and rather substitute those arguments passed to the function as well as it violates security requirements.
the payload has to be in the format as shown below unless there is another possibility to send the payload in JSON. As this needs to be send via JSON only I guess. If I try to remove the backslash it errors on me for indentation.
How can i leverage string formatting to hide username and password so that I can provide these information during runtime.
import requests
import sys, pprint, json
from getpass import getpass
from multiprocessing import Pool
import yaml
from functools import partial
http_header = {}
url_dict = {}
def getCookie(username, password, ip_addr):
url = "https://"+ip_addr+"/api/aaaLogin.json"
# payload = " {\r\n\"aaaUser\":"\
# " {\r\n\"attributes\":"\
# " {\r\n\"name\": \"admin\",\r\n" \
# " \"pwd\":\"Admin_1234!\"\r\n" \
# " }\r\n " \
# " }\r\n }\r\n"
payload = {
# 'aaaUser':'',
# 'attributes':'',
'name': username,
'pwd': password,
}
json_payload = json.dumps(payload)
headers = {
'Content-Type': "application/json",
'Cache-Control': "no-cache",
}
try:
req = requests.request("POST", url=url, data=json_payload, headers=headers, verify=False)
except:
print('Failed to obtain auth cookie: %s' % (e))
sys.exit(1)
else:
cookie=req.headers['Set-Cookie']
# print(cookie)
return cookie
def genericGetRequest(ip_addr, cookie, apiurl, verb):
url = 'https://'+ip_addr+apiurl
http_header["Cookie"]=cookie
http_header["Host"]=ip_addr
try:
req = requests.request(verb, url=url, headers=http_header, verify=False)
except:
print("There is a problem with the {} request!".format(verb))
else:
return(req)
def getResults(username, password, ip):
cookie=getCookie(username, password, ip)
if cookie:
print("User is logged in. Auth-cookie is %s\n" % cookie)
vlan_list = []
trunk_vlans_dict = {}
for i in range(1, 49):
apiurl = f"/api/mo/sys/intf/phys-[eth1/{i}]/.json"
generic = genericGetRequest(ip, cookie, apiurl, 'GET')
generic = generic.json()
imdata = generic['imdata']
vlan = imdata[0]['l1PhysIf']['attributes']
trunk_vlans_dict[f"eth1/{i}"] = vlan['trunkVlans']
vlan_list.append(trunk_vlans_dict)
print(vlan_list)
if __name__ == '__main__':
username = input("Enter username: ")
print("Enter password")
password = getpass()
if password:
deviceListFile = 'nexus_list.yaml'
with open(deviceListFile) as f:
deviceList = yaml.load(f)
num_threads = 5
print("Retreiving Configuration: ")
pool = Pool(num_threads)
partial_getResults = partial(getResults, username, password)
pool.map(partial_getResults, deviceList)
pool.close()
pool.join()
else:
print("Passwords do not match. Exiting...")
After using json dumps I am getting another error as shown below. BTW I have posted entire code for clarity sake.
ssh://vrxxx#werssefsf:22/sdfsdfsdfsdf/Python_Dev/Test1/pyVENV/bin/python -u /NetworkAutomation/Python_Dev/Test1/nxos_test5.py
Enter username: admin
admin
Enter password
Password: Admin_1234!
/NetworkAutomation/Python_Dev/Test1/nxos_test5.py:82: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
deviceList = yaml.load(f)
Retreiving Configuration:
/NetworkAutomation/Python_Dev/Test1/pyVENV/lib/python3.7/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/NetworkAutomation/Python_Dev/Test1/nxos_test5.py", line 58, in getResults
cookie=getCookie(username, password, ip)
File "/NetworkAutomation/Python_Dev/Test1/nxos_test5.py", line 42, in getCookie
cookie=req.headers['Set-Cookie']
File "/NetworkAutomation/Python_Dev/Test1/pyVENV/lib/python3.7/site-packages/requests/structures.py", line 52, in __getitem__
return self._store[key.lower()][1]
KeyError: 'set-cookie'
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/NetworkAutomation/Python_Dev/Test1/nxos_test5.py", line 88, in <module>
pool.map(partial_getResults, deviceList)
File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 268, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 657, in get
raise self._value
KeyError: 'set-cookie'
Process finished with exit code 1

It's difficult to answer your question without a minimal example. It's also not clear what aaaUser, attributes and ip_addr is used for but I think this is what you are trying to achieve:
import json
def getCookie(username, password, ip_addr):
payload = {
'name': username,
'pwd': password
}
return json.dumps(payload)
print(getCookie(username='admin', password='Admin_1234', ip_addr="127.0.0.0"))
# {"name": "admin", "pwd": "Admin_1234"}
There is unlikely a need for all the new lines and string formatting, you just need to pass the Python dictionary through the json dumps method.

Thanks Johnny. Your suggestion worked. I just have to use aaaUser and attributes in a nested dict format as shown below.
payload = {
'aaaUser': {
'attributes': {
'name': username,
'pwd': password,
}}}
json_payload = json.dumps(payload)

How to add exception in a loop?

I'm running a Python script which uses a value list as query parameters for an HTTP request over an API endpoint. Here a snap:
df = pd.read_excel('grp.xlsx', sheet_name='Sheet1', usecols="A")
for item in df.PLACE:
df.PLACE.head()
#1st level request
def wbsearchentities_q(**kwargs):
params = {
'action': 'wbsearchentities',
'format': 'json',
'language': 'en',
'search': item
}
params.update(kwargs)
response = requests.get(API_ENDPOINT, params=params)
return response
r = wbsearchentities_q(ids=item)
item_id = (r.json()['search'][0]['id'])
item_label = (r.json()['search'][0]['label'])
I'm having this error: IndexError: list index out of range which means that some items from my list are not recognized by the API endpoint.
I would just pass over and continue the loop. I tried to fix using this without result.
Thanks in advance.

you can try:
for item in df.PLACE:
try:
... your code ...
except:
pass

In order to be specific only for that error (recommanded in order to avoid not handling other errors), and continue to the next item in the df:
try:
item_id = (r.json()['search'][0]['id'])
item_label = (r.json()['search'][0]['label'])
except IndexError:
continue

Django Testing KeyError When Retrieving Header from Request Object

So I'm trying to confirm the location of a given view during testing. The docs say:
You can also use dictionary syntax on the response object to query the
value of any settings in the HTTP headers. For example, you could
determine the content type of a response using
response['Content-Type'].
However, when I put it to use I'm getting a key error. Help please.
Test:
def test_normal_rewardstore_usage(self):
logged_in = self.client.login(username=self.u1.username, password="psst")
response = self.client.get(reverse('rewards:rewardstore'))
location = "http://testserver%s" % (reverse('rewards:rewardpage', kwargs={'company':self.r1.company.slug, 'slug':self.r1.slug}))
self.assertEqual(response.status_code, 200)
self.assertEqual(response['Location'], location)
Error:
Traceback (most recent call last):
File "/app/rewards/tests/test_views.py", line 58, in test_normal_rewardstore_usage
self.assertEqual(response['Location'], location)
File "/app/.heroku/python/lib/python2.7/site-packages/django/http/response.py", line 189, in __getitem__
return self._headers[header.lower()][1]
KeyError: 'location'
View:
def RewardStore_Index(request, template='rewards/reward-store.html', page_template='rewards/rewards_page.html'):
user = User.objects.get(pk=request.user.pk)
contact = Contact.objects.get(user__pk=request.user.pk, is_active=True)
if request.user.is_authenticated():
a = Member.objects.get(pk=request.user.pk)
a = a.account_verified
rewards = Reward.objects.filter(country__iso=contact.get_country)
else:
a = False
g = GeoIP()
c = g.country(request.user)
c = c['country_code']
rewards = Reward.objects.filter(country__iso=c)
context = {
'targetuser': request.user,
'rewards': rewards,
'page_template': page_template,
'email_authenticated': True if a else False,
'notifications': NotificationMap.objects.filter(user__pk=request.user.pk, read=False).prefetch_related('notification', 'notification__users')
}
if request.is_ajax():
template = page_template
return render_to_response(
template, context, context_instance=RequestContext(request))

The headers made available in the response object will vary depending on the server used (source), so it wouldn't surprise me if the django test runner doesn't send back all the headers that you see in production. Also, Location is generally used with redirecting responses, and you're asserting that you receive a 200 first. Are you sure that you should be expecting a Location header in this circumstance?

Can I use the a dictionary key in an API call? (Python)

I would think there is not problem with that, but I am really having troubles with this piece of code and cannot seem to come up with a solution.
I have a dictionary whose keys are proper name, e.g. John Green, and I am using the API of the Sunlight Foundation to retrieve information about congress members (check here). Now I need to request using name and lastname so my code looks something like this:
for key in my_dict:
query_params2 = { 'apikey': 'xxxxxxxxxxx',
'firstname' : key.split()[0],
'lastname' : key.split()[-1]
}
endpoint2 = "http://services.sunlightlabs.com/api/legislators.get.json"
resp2 = requests.get(endpoint2, params = query_params2)
data2 = resp2.json().decode('utf-8')
print data2['response']['legislator']['bioguide_id']
Which gives some error I can not really interpret:
Traceback (most recent call last):
File "my_program.py", line 102, in <module>
data = resp.json()
File "//anaconda/lib/python2.7/site-packages/requests/models.py", line 741, in json
return json.loads(self.text, **kwargs)
File "//anaconda/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "//anaconda/lib/python2.7/json/decoder.py", line 365, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "//anaconda/lib/python2.7/json/decoder.py", line 383, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
I would guess it has something to do with the encoding, but I am not sure what and how to solve it.
Needless to say, if I put by hand a name and lastname, the request works perfectly.
Anybody can help with this? Thanks a lot!

This has nothing to do with the encoding. The answer simply isn't JSON. When I try your code with 'John' and 'Green' I get a 400 Bad Request and the content of the response is 'No Such Object Exists'.
Trying John Green in the web interface also comes up with an empty answer. Also the URL in the API documentation differs from the URL in your example.
The following works for me (again no John Green):
import requests
LEGISLATORS_URL = 'https://congress.api.sunlightfoundation.com/legislators'
API_KEY = 'xxxx'
def main():
names = [('John', 'Green'), ('John', 'Kerry')]
for first_name, last_name in names:
print 'Checking', first_name, last_name
response = requests.get(
LEGISLATORS_URL,
params={
'apikey': API_KEY,
'first_name': first_name,
'last_name': last_name,
'all_legislators': 'true'
}
).json()
print response['count']
if response['count'] > 0:
print response['results'][0]['bioguide_id']
if __name__ == '__main__':
main()
Output:
Checking John Green
0
Checking John Kerry
1
K000148