Develop Reference

python unable to retrieve InstanceID for loop iteration

I am running a python script that reads the 'data.json' and provide the InstanceIds of the PrivateIP. here's the json file (data.json) that has 2 PrivateIPs with their own separate AWSAccount:
{
"ServerIPList" : [{
"PrivateIP": "17.11.11.11",
"HostName": "ip-17-11-11-11.ec2.internal",
"Region": "us-east-1",
"AccountID": "123456789123"
},
{
"PrivateIP": "18.22.22.22",
"HostName": "ip-18-22-22-22.ec2.internal",
"Region": "us-east-1",
"AccountID": "567891234567"
}],
}
Now, when i run the script using a FOR loop, it supposes to run on iteration where it keeps describing the first PrivateIP and saves the output(InstanceId) to the empty list []. It, then, describes another PrivateIP. However, it successfully convert the InstanceID of first PrivateIP but threw an error while describing the second PrivateIP. Any thought where am i missing the logic. here's my code:
import json, boto3
EC2_list = []
PrivateIP = []
AccountID = []
path = json.loads{'data.json'}
for item in input_data['ServerIPList']:
PrivateIP.append(item['PrivateIP'])
AccountID.append(item['AccountID'])
if "," in instanceIds:
instanceIds_list = instanceIds.split(",")
else:
instanceIds_list = instanceIds
if "," in AccountID:
AccountID_list = AccountID.split(",")
else:
AccountID_list = AccountID
for i in PrivateIP_list:
for account in range(len(AccountID)):
RoleArn = f"arn:aws:iam::{AccountID[account]}:role/Cloudbees"
print(RoleArn)
response = sts.assume_role(RoleArn=RoleArn, RoleSessionName="learnaws-test-session")
ec2_client = boto3.client('ec2', region_name=region_Name,
aws_access_key_id=response['Credentials']['AccessKeyId'],
aws_secret_access_key=response['Credentials']['SecretAccessKey'],
aws_session_token = response['Credentials']['SessionToken'])
ec2 = boto3.resource('ec2', region_name=region_Name,
aws_access_key_id=response['Credentials']['AccessKeyId'],
aws_secret_access_key=response['Credentials']['SecretAccessKey'],
aws_session_token = response['Credentials']['SessionToken'])
EC2Response = ec2_client.describe_instances(Filters=[
{
'Name': 'private-ip-address',
'Values': [
i,
]
}
])
print("This is the actual response :")
#print(EC2Response)
for instance in EC2Response['Reservations'][0]['Instances']:
instanceIds = instance['InstanceId']
print(instanceIds)
EC2_list.append(instanceIds)
output is :
Traceback (most recent call last):
File "ec2InstanceState.py", line 260, in <module>
for instance in EC2Response['Reservations'][0]['Instances']:
IndexError: list index out of range
arn:aws:iam::123456789123:role/Cloudbees
This is the actual response :
"i-xxxxxxxxxxxxx"
arn:aws:iam::567891234567:role/Cloudbees
This is the actual response : <BLANK> NO INSTANCEID -------????

How to solve for Key Error in Complex Nested Json

I keep receiving key error 'Notes'. Notes is in a nested response. How do I solve this? I have included a sample of the json.
Traceback added by request.
Traceback (most recent call last):
File "/Users/xxxxxx/Documents/code/trade show w notes", line 16, in <module>
target = "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s" % (contact["company_name"], contact["location"], contact["summary"], contact["job_title"], contact["name"], contact["job_industry"], contact["email"], contact["first_name"], contact["last_name"], contact["notes"])
KeyError: 'notes'
\
"data": [
{
"team_id": 53806,
"name": "Nicholas Bancroft Cooke",
"first_name": "Nicholas",
],
"email": null,
"metadata": null,
"qualification": [
{
"qualification_id": 17573056,
"qualification": "connected",
"notes": null,
\\
page = 1
url = "https://teams-api.grip.events/1/team/53806/event/123721/member/236388/contact/inbound_lead/reviewed?page=1"
headers = {
'authorization': 'Bearer eaf3bd4b-6861-4ca2-a86e-3a96c73deac0',
}
data = ["company_name", "job_title", "name", "job_industry", "summary", "notes", "location", "first_name", "last_name", "email"]
with open("list31.txt", "w", encoding='utf-8') as f: #added " encoding='utf-8' "
for page in range(1, 1000):
response = requests.get(url=url, headers=headers).json()
contacts = response["data"]
for contact in contacts:
target = "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s" % (contact["company_name"], contact["location"], contact["summary"], contact["job_title"], contact["name"], contact["job_industry"], contact["email"], contact["first_name"], contact["last_name"], contact["notes"])
f.write(target + "\n")
print(target)

If you are referring to the notes key which is under qualification.
Then it should be: contact["qualification"][0]["notes"]
Here contact["qualification"] is a list. if by any chance contact["qualification"] is an empty list it will raise an IndexError. you need to handle it with a try-catch or if-clause

How to parse musixmatch python api JSON response?

How do you parse the JSON response from the musixmatch api?
I'm calling the method
res = artist_api.artist_search_get(format=format, q_artist=artist)
I'm getting a response of
{'message': {'body': {'artist_list': [{'artist': {'artist_alias_list': [],
'artist_comment': '',
'artist_country': '',
'artist_credits': {'artist_list': []},
'artist_edit_url': None,
'artist_id': 26575484.0,
'artist_mbid': None,
'artist_name': 'Illenium',
'artist_name_translation_list': [],
'artist_rating': 55.0, .........
I'm trying to get the artist_id.
I tried getting the artist_id like this:
print(res['message']['body']['artist']['artist_id'])
artist_api = swagger_client.ArtistApi()
format = 'json'
try:
artist_list = ["adele", "lady gaga", "john legend"];
for artist in artist_list:
print(artist)
res = artist_api.artist_search_get(format=format, q_artist=artist)
print(res['message']['body']['artist']['artist_id'])
except ApiException as e:
print "Exception when calling ArtistApi->artist_search_get: %s\n" % e
I'm getting this error message:
Traceback (most recent call last):
File "musixmatch.py", line 39, in <module>
print(res['message']['body']['artist_id'])
TypeError: 'InlineResponse2004' object has no attribute '__getitem__'
Please help I've searched through a bunch of threads but I still can't find the answer to this. I'm new to python so I'm not sure what I'm doing wrong.

You a little messed up with the hierarchy of JSON-document, use this code to get desired values:
[artist['artist']['artist_id'] for artist in res['message']['body']['artist_list']]
As an alternative, can be used JSONPath-expression: $..artist_list..artist_id to get the same result.
Example
from jsonpath_rw import parse
import json
res = json.loads("""{
"message": {
"header": {},
"body": {
"artist_list": [
{
"artist": {
"artist_credits": { },
"artist_country": "string",
"artist_id": 110
}
},
{
"artist": {
"artist_credits": {},
"artist_country": "string",
"artist_id": 220
}
}
]
}
}
}""")
# 1-way: Iterate through JSON-document
print([artist['artist']['artist_id'] for artist in res['message']['body']['artist_list']])
# result: [110, 220]
# 2-way: Use JSONPath-expression
jsonpath_expr = parse('$..artist_list..artist_id')
print([match.value for match in jsonpath_expr.find(res)])
# result: [110, 220]

TypeError: list indices must be integers or slices, not str - get from json

I try to get all lattitudes and longtitudes from this json.
Code:
import urllib.parse
import requests
raw_json = 'http://live.ksmobile.net/live/getreplayvideos?userid='
print()
userid = 735890904669618176
#userid = input('UserID: ')
url = raw_json + urllib.parse.urlencode({'userid=': userid}) + '&page_size=1000'
print(url)
json_data = requests.get(url).json()
print()
for coordinates in json_data['data']['video_info']:
print(coordinates['lat'], coordinates['lnt'])
print()
Error:
/usr/bin/python3.6 /media/anon/3D0B8DD536C9574F/PythonProjects/getLocation/getCoordinates
http://live.ksmobile.net/live/getreplayvideos?userid=userid%3D=735890904669618176&page_size=1000
Traceback (most recent call last):
File "/media/anon/3D0B8DD536C9574F/PythonProjects/getLocation/getCoordinates", line 17, in <module>
for coordinates in json_data['data']['video_info']:
TypeError: list indices must be integers or slices, not str
Process finished with exit code 1
Where do I go wrong?
In advance, thanks for your help and time.
I just post some of the json to show what it looks like.
The json looks like this:
{
"status": "200",
"msg": "",
"data": {
"time": "1499275646",
"video_info": [
{
"vid": "14992026438883533757",
"watchnumber": "38",
"topicid": "0",
"topic": "",
"vtime": "1499202678",
"title": "happy 4th of july",
"userid": "735890904669618176",
"online": "0",
"addr": "",
"isaddr": "2",
"lnt": "-80.1282576",
"lat": "26.2810628",
"area": "A_US",
"countryCode": "US",
"chatSystem": "1",
},
Full json: https://pastebin.com/qJywTqa1

Your URL construction is incorrect. The URL you have built (as shown in the output of your script) is:
http://live.ksmobile.net/live/getreplayvideos?userid=userid%3D=735890904669618176&page_size=1000
Where you actually want this:
http://live.ksmobile.net/live/getreplayvideos?userid=735890904669618176&page_size=1000
So your were actually getting this JSON in your response:
{
"status": "200",
"msg": "",
"data": []
}
Which is why you were seeing that error.
Here is the corrected script:
import urllib.parse
import requests
raw_json = 'http://live.ksmobile.net/live/getreplayvideos?'
print()
userid = 735890904669618176
#userid = input('UserID: ')
url = raw_json + urllib.parse.urlencode({'userid': userid}) + '&page_size=1000'
print(url)
json_data = requests.get(url).json()
print()
for coordinates in json_data['data']['video_info']:
print(coordinates['lat'], coordinates['lnt'])
print()

According to your posted json, you have problem in this statement-
print(coordinates['lat'], coordinates['lnt'])
Here coordinates is a list having only one item which is dictionary. So your statement should be-
print(coordinates[0]['lat'], coordinates[0]['lnt'])

Reading a Json response recursively with python

I'm trying to print all the "keys, values" from a json response without knowing the keys names (without using the syntax json['example'], for example). I'm doing this with a recursively function that uses iteritems(), but I'm having some problems:
This is the Json response that I'm trying to read:
{"servers": [{"id": "a059eccb-d929-43b2-8db3-b32b6201d60f", "links": [{"href": "http://192.168.100.142:8774/v2/2ad1fc162c254e59bea043560b7f73cb/servers/a059eccb-d929-43b2-8db3-b32b6201d60f", "rel": "self"}, {"href": "http://192.168.100.142:8774/2ad1fc162c254e59bea043560b7f73cb/servers/a059eccb-d929-43b2-8db3-b32b6201d60f", "rel": "bookmark"}], "name": "birk"}]}
This is the funcion that I'm using:
def format_main_response(self, json_string):
print "json: " + json_string
content = json.loads(str(json_string))
for key, value in content.iteritems():
print key
if type(value) == type(['']):
strg = str(json.dumps(value))
strg = strg.strip('[]')
self.format_main_response(strg)
else:
print value
I'm using the strip function to take out all the '[ ]' from my json string. If I didn't do that I got an error when trying to load it using 'json.loads()' function.
Traceback (most recent call last):
File "main.py", line 135, in <module>
formatter.format_main_response(nova_API.list_servers())
File "/home/python/jsonformatter.py", line 51, in format_main_response
self.format_main_response(strg, mod)
File "/home/python/jsonformatter.py", line 51, in format_main_response
self.format_main_response(strg, mod)
File "/home/python/jsonformatter.py", line 31, in format_main_response
for key, value in content.iteritems():
AttributeError: 'list' object has no attribute 'iteritems'
My problem is that in some point the json that should be printed looks like this, without the '[ ]':
{"href": "http://192.168.100.142:8774/v2/2ad1fc162c254e59bea043560b7f73cb/servers/a059eccb-d929-43b2-8db3-b32b6201d60f", "rel": "self"}, {"href": "http://192.168.100.142:8774/2ad1fc162c254e59bea043560b7f73cb/servers/a059eccb-d929-43b2-8db3-b32b6201d60f", "rel": "bookmark"}
When the function tries to find the 'key,value' from this json, I got this error:
Traceback (most recent call last): File "main.py", line 135, in <module>
formatter.format_main_response(nova_API.list_servers())
File "/home/python/jsonformatter.py", line 34, in format_main_response
self.format_main_response(strg)
File "/home/python/jsonformatter.py", line 34, in format_main_response
self.format_main_response(strg)
File "/home/python/jsonformatter.py", line 28, in format_main_response
content = json.loads(str(json_string))
File "/usr/lib/python2.7/json/__init__.py", line 326, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 369, in decode
raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 1 column 135 - line 1 column 273 (char 135 - 273)
What should I do in this case? Or any other way to get the same result?

Use that:
def format_main_response(json_string):
print "json: " + json_string
content = json.loads(str(json_string))
for key, value in content.iteritems():
print key
if type(value) == type(['']):
for sub_value in value:
strg = str(json.dumps(sub_value))
format_main_response(strg)
else:
print value
That's the result:
~$ python test_pdb.py
json: {"servers": [{"id": "a059eccb-d929-43b2-8db3-b32b6201d60f", "links": [{"href": "http://192.168.100.142:8774/v2/2ad1fc162c254e59bea043560b7f73cb/servers/a059eccb-d929-43b2-8db3-b32b6201d60f", "rel": "self"}, {"href": "http://192.168.100.142:8774/2ad1fc162c254e59bea043560b7f73cb/servers/a059eccb-d929-43b2-8db3-b32b6201d60f", "rel": "bookmark"}], "name": "birk"}]}
servers
json: {"id": "a059eccb-d929-43b2-8db3-b32b6201d60f", "links": [{"href": "http://192.168.100.142:8774/v2/2ad1fc162c254e59bea043560b7f73cb/servers/a059eccb-d929-43b2-8db3-b32b6201d60f", "rel": "self"}, {"href": "http://192.168.100.142:8774/2ad1fc162c254e59bea043560b7f73cb/servers/a059eccb-d929-43b2-8db3-b32b6201d60f", "rel": "bookmark"}], "name": "birk"}
id
a059eccb-d929-43b2-8db3-b32b6201d60f
links
json: {"href": "http://192.168.100.142:8774/v2/2ad1fc162c254e59bea043560b7f73cb/servers/a059eccb-d929-43b2-8db3-b32b6201d60f", "rel": "self"}
href
http://192.168.100.142:8774/v2/2ad1fc162c254e59bea043560b7f73cb/servers/a059eccb-d929-43b2-8db3-b32b6201d60f
rel
self
json: {"href": "http://192.168.100.142:8774/2ad1fc162c254e59bea043560b7f73cb/servers/a059eccb-d929-43b2-8db3-b32b6201d60f", "rel": "bookmark"}
href
http://192.168.100.142:8774/2ad1fc162c254e59bea043560b7f73cb/servers/a059eccb-d929-43b2-8db3-b32b6201d60f
rel
bookmark
name
birk

How about:
jsonStr = {"href": "http://192.168.100.142:8774/v2/2ad1fc162c254e59bea043560b7f73cb/servers/a059eccb-d929-43b2-8db3-b32b6201d60f", "rel": "self"}, {"href": "http://192.168.100.142:8774/2ad1fc162c254e59bea043560b7f73cb/servers/a059eccb-d929-43b2-8db3-b32b6201d60f", "rel": "bookmark"}
print json.dumps(jsonStr, sort_keys=True, indent=2, separators=(',', ': '))
This should give you the format you want

Code below recursively traverses the json response and prints the key,value pairs:
Trick is to load json response only once in the main and then recursively traverse the response:
def parse_json_response(content):
if len (content.keys()) > 1 :
for key, value in content.iteritems():
print "key : ", key
print "Value", value
if type(value) is dict:
parse_json_response(value)
else:
print value
if __name__ == '__main__':
content = json.loads(str(response))
parse_json_response(content)
Hope it helps.