Develop Reference

AWS Config Python Unicode Mess

I am running into issue with trying to pull out usable items from this output. I am just trying to pull a single value from this string of Unicode and it has been super fun.
my print(response) returns this: FYI this is way longer than this little snippet.
{u'configurationItems': [{u'configurationItemCaptureTime': datetime.datetime(2020, 6, 4, 21, 56, 31, 134000, tzinfo=tzlocal()), u'resourceCreationTime': datetime.datetime(2020, 5, 22, 16, 32, 55, 162000, tzinfo=tzlocal()), u'availabilityZone': u'Not Applicable', u'awsRegion': u'us-east-1', u'tags': {u'brassmonkeynew': u'tomtagnew'}, u'resourceType': u'AWS::DynamoDB::Table', u'resourceId': u'tj-test2', u'configurationStateId': u'1591307791134', u'relatedEvents': [], u'relationships': [], u'arn': u'arn:aws:dynamodb:us-east-1:896911201517:table/tj-test2', u'version': u'1.3', u'configurationItemMD5Hash': u'', u'supplementaryConfiguration': {u'ContinuousBackupsDescription': u'{"continuousBackupsStatus":"ENABLED","pointInTimeRecoveryDescription":{"pointInTimeRecoveryStatus":"DISABLED"}}', u'Tags': u'[{"key":"brassmonkeynew","value":"tomtagnew"}]'}, u'resourceName': u'tj-test2', u'configuration': u'{"attributeDefinitions":[{"attributeName":"tj-test2","attributeType":"S"}],"tableName":"tj-test2","keySchema":[{"attributeName":"tj-test2","keyType":"HASH"}],"tableStatus":"ACTIVE","creationDateTime":1590165175162,"provisionedThroughput":{"numberOfDecreasesToday":0,"readCapacityUnits":5,"writeCapacityUnits":5},"tableArn":"arn:aws:dynamodb:us-east-1:896911201517:table/tj-test2","tableId":"816956d7-95d1-4d31-8d18-f11b18de4643"}', u'configurationItemStatus': u'OK', u'accountId': u'896911201517'}, {u'configurationItemCaptureTime': datetime.datetime(2020, 6, 1, 16, 27, 21, 316000, tzinfo=tzlocal()), u'resourceCreationTime': datetime.datetime(2020, 5, 22, 16, 32, 55, 162000, tzinfo=tzlocal()), u'availabilityZone': u'Not Applicable', u'awsRegion': u'us-east-1', u'tags': {u'brassmonkeynew': u'tomtagnew', u'backup-schedule': u'daily'}, u'resourceType': u'AWS::DynamoDB::Table', u'resourceId': u'tj-test2', u'configurationStateId': u'1591028841316', u'relatedEvents': [], u'relationships': [], u'arn': u'arn:aws:dynamodb:us-east-1:896911201517:table/tj-test2', u'version': u'1.3', u'configurationItemMD5Hash': u'', u'supplementaryConfiguration': {u'ContinuousBackupsDescription': u'{"continuousBackupsStatus":"ENABLED","pointInTimeRecoveryDescription":{"pointInTimeRecoveryStatus":"DISABLED"}}', u'Tags': u'[{"key":"brassmonkeynew","value":"tomtagnew"},{"key":"backup-schedule","value":"daily"}]'}, u'resourceName': u'tj-test2', u'configuration': u'{"attributeDefinitions":[{"attributeName":"tj-test2","attributeType":"S"}],"tableName":"tj-test2","keySchema":[{"attributeName":"tj-
and so on. I have tried a few different ways of getting this info but every time I get a key error:
I also tried converting this into JSON and but since i have Date/time at the top it gives me this error:
“TypeError: [] is not JSON serializable
Failed attempts:
# print(response[0]["tableArn"])
print(response2)
print(response2['tableArn'])
print(response2.arn)
print(response2['configurationItems'][0]['tableArn'])
print(response2['configurationItems']['tableArn'])
print(response.configurationItems[0])
arn = response.configurationItems[0].arn
def lambda_handler(event, context):
# print("Received event: " + json.dumps(event, indent=2))
message = event['Records'][0]['Sns']['Message']
print("From SNS: " + message)
response = client.get_resource_config_history(
resourceType='AWS::DynamoDB::Table',
resourceId = message
)
response2 = dict(response)
print(response)
return message

Here's some Python3 code that shows how to access the elements:
import boto3
import json
import pprint
config_client = boto3.client('config')
response = config_client.get_resource_config_history(
resourceType='AWS::DynamoDB::Table',
resourceId = 'stack-table'
)
for item in response['configurationItems']:
configuration = item['configuration'] # Returns a JSON string
config = json.loads(configuration) # Convert to Python object
pprint.pprint(config) # Show what's in it
print(config['tableArn']) # Access elements in object
The trick is that the configuration field contains a JSON string that needs to be converted into a Python object for easy access.

Flask app returns json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

My flask app keeps bringing up this error when I run the json request to an api, but it runs well from my terminal. What could be wrong? I've loked through the website but none of the solution is applicable. This is my code:
import requests
key = "XpVcEz3pf5hXpJ7psgasaszhMng"
isbns = "2497573957X"
res = requests.get("https://www.goodreads.com/book/review_counts.json",
params={"key": key, "isbns": isbns})
goodreads = res.json()
averageratings = goodreads["books"][0]["average_rating"]
And this is the json result I get from my terminal where I ran the code:
{'books': [{
'id': 29207858,
'isbn': '1632168146',
'isbn13': '9781632168146',
'ratings_count': 0,
'reviews_count': 1,
'text_reviews_count': 0,
'work_ratings_count': 26,
'work_reviews_count': 113,
'work_text_reviews_count': 10,
'average_rating': '4.04'
}]
}

Convert HTML dictionary to PYTHON dictionary

i need to get values only for Czechia country from this website list "https://coronavirus-19-api.herokuapp.com/countries" and store like a variable dictionary in ptyhon.
Like this:
Czechia = {"cases":434,"todayCases":0,"deaths":0,"todayDeaths":0,"recovered":3,"active":431,"critical":2}

You could use requests to GET the JSON data from your server URL, then construct a new dictionary with country as the key:
from requests import get
URL = "https://coronavirus-19-api.herokuapp.com/countries"
req = get(URL).json()
result = {obj['country']: {k: v for k, v in obj.items() if k != 'country'} for obj in req}
print(result)
Output:
{'China': {'cases': 80894, 'todayCases': 13, 'deaths': 3237, 'todayDeaths': 11, 'recovered': 69614, 'active': 8043, 'critical': 2622}, 'Italy': {'cases': 31506, 'todayCases': 0, 'deaths': 2503, 'todayDeaths': 0, 'recovered': 2941, 'active': 26062, 'critical': 2060}...
Now you can access your data in O(1) time instead of doing a O(N) linear scan:
print(result["Czechia"])
# {'cases': 464, 'todayCases': 30, 'deaths': 0, 'todayDeaths': 0, 'recovered': 3, 'active': 461, 'critical': 2}
Note: Its probably also safe to ensure req.status_code is 200 OK or whatever else you expect to receive from the server.

In [1]: import requests
...: import json
...:
...: data = requests.get('https://coronavirus-19-api.herokuapp.com/countries').json()
...: result = next(item for item in data if item["country"] == "Czechia")
...: print(json.dumps(result, indent=4))
{
"country": "Czechia",
"cases": 464,
"todayCases": 30,
"deaths": 0,
"todayDeaths": 0,
"recovered": 3,
"active": 461,
"critical": 2
}
In [2]:

While the RoadRunner 's answers solves your problem, I am just giving you the one other way of doing it using python's urllib module.
from urllib.request import urlopen
##import ast
import json
def Corona_Tracker():
res = urlopen('https://coronavirus-19-api.herokuapp.com/countries')
result = res.read().strip()
result_str = json.loads(result)
return result_str
if __name__ == "__main__":
result_str=Corona_Tracker()
while True:
for data in result_str:
if data['country'] == "India":
print(data)
Just replace India with your country , it gives the below output
>>> {'country': 'India', 'cases': 148, 'todayCases': 5, 'deaths': 3, 'todayDeaths': 0, 'recovered': 14, 'active': 131, 'critical': 0}

Can't fetch an item out of weird json content

I'm trying to get some items from json content. However, the structure of that json content is foreign to me and as a result I can't fetch the value of property out of it.
I've tried so far with:
import json
import requests
from bs4 import BeautifulSoup
link = 'https://www.zillow.com/homedetails/5958-SW-4th-St-Miami-FL-33144/43835884_zpid/'
def fetch_content(link):
content = requests.get(link,headers={"User-Agent":"Mozilla/5.0"})
soup = BeautifulSoup(content.text,"lxml")
item = soup.select_one("script#hdpApolloPreloadedData").text
print(json.loads(item)['apiCache'])
if __name__ == '__main__':
fetch_content(link)
The result I get running the above script is:
{"VariantQuery{\"zpid\":43835884}":{"property":{"zpid":43835884,"streetAddress":"5958 SW 4th St",
Which I can't further process for that weird key in front.
Expected output:
{"zpid":43835884,"streetAddress":"5958 SW 4th St", ----
How can I get the value of that property?

You can get zpid and address by their mangled json with:
json.loads(json.loads(item.text)['apiCache'])['VariantQuery{"zpid":43835884}']['property']['zpid']
Out[1889]: 43835884
json.loads(json.loads(item.text)['apiCache'])['VariantQuery{"zpid":43835884}']['property']['streetAddress']
Out[1890]: '5958 SW 4th St'
I noticed you can always get the zpid like this:
link = 'https://www.zillow.com/homedetails/5958-SW-4th-St-Miami-FL-33144/43835884_zpid/'
content = requests.get(link,headers={"User-Agent":"Mozilla/5.0"})
soup = BeautifulSoup(content.text,"lxml")
item = soup.select_one("script#hdpApolloPreloadedData").text
print(json.loads(item)['zpid'])

Just modify your function to the following. I also added another function (process_fetched_content()) to give you some more freedom. You could simply run it and it will take care of situations even when you have multiple keys that start with 'VariantQuery{"zpid":'. The final output is a dict with the keys being your zpid and the values being what you are looking for.
If you have a lot of zpid values, then this will let you accumulate them all together and then process them. The benefit is the list of keys is then the list of zpids you have.
Here's how you could use this code.
results = process_fetched_content(raw_dictionary = fetch_content(link, verbose=False))
print(results)
output:
{'43835884': {'zpid': 43835884, 'streetAddress': '5958 SW 4th St', 'zipcode': '33144', 'city': 'Miami', 'state': 'FL', 'latitude': 25.76661, 'longitude': -80.292801, 'price': 340000, 'dateSold': 1576875600000, 'bathrooms': 2, 'bedrooms': 3, 'livingArea': 1757, 'yearBuilt': 1973, 'lotSize': 4331, 'homeType': 'SINGLE_FAMILY', 'homeStatus': 'RECENTLY_SOLD', 'photoCount': 19, 'imageLink': 'https://photos.zillowstatic.com/p_g/IS7yxihwtuqmlq1000000000.jpg', 'daysOnZillow': 0, 'isFeatured': False, 'shouldHighlight': False, 'brokerId': 0, 'zestimate': 341336, 'rentZestimate': 2200, 'listing_sub_type': {}, 'priceReduction': '', 'isUnmappable': False, 'rentalPetsFlags': 128, 'mediumImageLink': 'https://photos.zillowstatic.com/p_c/IS7yxihwtuqmlq1000000000.jpg', 'isPreforeclosureAuction': False, 'homeStatusForHDP': 'RECENTLY_SOLD', 'priceForHDP': 340000, 'festimate': 341336, 'isListingOwnedByCurrentSignedInAgent': False, 'isListingClaimedByCurrentSignedInUser': False, 'hiResImageLink': 'https://photos.zillowstatic.com/p_f/IS7yxihwtuqmlq1000000000.jpg', 'watchImageLink': 'https://photos.zillowstatic.com/p_j/IS7yxihwtuqmlq1000000000.jpg', 'tvImageLink': 'https://photos.zillowstatic.com/p_m/IS7yxihwtuqmlq1000000000.jpg', 'tvCollectionImageLink': 'https://photos.zillowstatic.com/p_l/IS7yxihwtuqmlq1000000000.jpg', 'tvHighResImageLink': 'https://photos.zillowstatic.com/p_n/IS7yxihwtuqmlq1000000000.jpg', 'zillowHasRightsToImages': True, 'desktopWebHdpImageLink': 'https://photos.zillowstatic.com/p_h/IS7yxihwtuqmlq1000000000.jpg', 'isNonOwnerOccupied': False, 'hideZestimate': False, 'isPremierBuilder': False, 'isZillowOwned': False, 'currency': 'USD', 'country': 'USA', 'taxAssessedValue': 224131, 'streetAddressOnly': '5958 SW 4th St', 'unit': ' '}}
Code
import json
import requests
from bs4 import BeautifulSoup
link = 'https://www.zillow.com/homedetails/5958-SW-4th-St-Miami-FL-33144/43835884_zpid/'
def fetch_content(link, verbose=False):
content = requests.get(link,headers={"User-Agent":"Mozilla/5.0"})
soup = BeautifulSoup(content.text,"lxml")
item = soup.select_one("script#hdpApolloPreloadedData").text
d = json.loads(item)['apiCache']
d = json.loads(d)
if verbose:
print(d)
return d
def process_fetched_content(raw_dictionary=None):
if raw_dictionary is not None:
keys = [k for k in raw_dictionary.keys() if k.startswith('VariantQuery{"zpid":')]
results = dict((k.split(':')[-1].replace('}',''), d.get(k).get('property', None)) for k in keys)
return results
else:
return None

How to extract numbers from JSON API

I want to extract numbers and calculate the sum of these numbers from JSON API. The format is
{
comments: [
{
name: "Matthias"
count: 97
},
{
name: "Geomer"
count: 97
}
...
]
}
And my code is
import json
import urllib
url = 'http://python-data.dr-chuck.net/comments_204529.json'
print 'Retrieving', url
uh = urllib.urlopen(url)
data = uh.read()
print 'Retrieved',len(data),'characters'
result = json.loads(url)
print result
I can get the result of how many characters in this data but cannot continue with the code because it's said JSON object cannot be decoded.
Does anyone know how to finish this code? Much appreciated!

First of all, I suggest you study the built-in Python Data Structures to get a better understanding about what you are dealing with.
result is a dictionary, result["comments"] is a list of dictionaries - you can make a list comprehension to get all the comments counts:
>>> import json
>>> import urllib
>>>
>>> url = 'http://python-data.dr-chuck.net/comments_204529.json'
>>> uh = urllib.urlopen(url)
>>> result = json.load(uh)
>>>
>>> [comment["count"] for comment in result["comments"]]
[100, 96, 95, 93, 85, 85, 77, 73, 73, 70, 65, 65, 65, 62, 62, 62, 61, 57, 50, 49, 46, 46, 43, 42, 39, 38, 37, 36, 34, 33, 31, 28, 28, 26, 26, 25, 22, 20, 20, 18, 17, 15, 14, 12, 10, 9, 8, 6, 5, 3]