I am running into issue with trying to pull out usable items from this output. I am just trying to pull a single value from this string of Unicode and it has been super fun.
my print(response) returns this: FYI this is way longer than this little snippet.
{u'configurationItems': [{u'configurationItemCaptureTime': datetime.datetime(2020, 6, 4, 21, 56, 31, 134000, tzinfo=tzlocal()), u'resourceCreationTime': datetime.datetime(2020, 5, 22, 16, 32, 55, 162000, tzinfo=tzlocal()), u'availabilityZone': u'Not Applicable', u'awsRegion': u'us-east-1', u'tags': {u'brassmonkeynew': u'tomtagnew'}, u'resourceType': u'AWS::DynamoDB::Table', u'resourceId': u'tj-test2', u'configurationStateId': u'1591307791134', u'relatedEvents': [], u'relationships': [], u'arn': u'arn:aws:dynamodb:us-east-1:896911201517:table/tj-test2', u'version': u'1.3', u'configurationItemMD5Hash': u'', u'supplementaryConfiguration': {u'ContinuousBackupsDescription': u'{"continuousBackupsStatus":"ENABLED","pointInTimeRecoveryDescription":{"pointInTimeRecoveryStatus":"DISABLED"}}', u'Tags': u'[{"key":"brassmonkeynew","value":"tomtagnew"}]'}, u'resourceName': u'tj-test2', u'configuration': u'{"attributeDefinitions":[{"attributeName":"tj-test2","attributeType":"S"}],"tableName":"tj-test2","keySchema":[{"attributeName":"tj-test2","keyType":"HASH"}],"tableStatus":"ACTIVE","creationDateTime":1590165175162,"provisionedThroughput":{"numberOfDecreasesToday":0,"readCapacityUnits":5,"writeCapacityUnits":5},"tableArn":"arn:aws:dynamodb:us-east-1:896911201517:table/tj-test2","tableId":"816956d7-95d1-4d31-8d18-f11b18de4643"}', u'configurationItemStatus': u'OK', u'accountId': u'896911201517'}, {u'configurationItemCaptureTime': datetime.datetime(2020, 6, 1, 16, 27, 21, 316000, tzinfo=tzlocal()), u'resourceCreationTime': datetime.datetime(2020, 5, 22, 16, 32, 55, 162000, tzinfo=tzlocal()), u'availabilityZone': u'Not Applicable', u'awsRegion': u'us-east-1', u'tags': {u'brassmonkeynew': u'tomtagnew', u'backup-schedule': u'daily'}, u'resourceType': u'AWS::DynamoDB::Table', u'resourceId': u'tj-test2', u'configurationStateId': u'1591028841316', u'relatedEvents': [], u'relationships': [], u'arn': u'arn:aws:dynamodb:us-east-1:896911201517:table/tj-test2', u'version': u'1.3', u'configurationItemMD5Hash': u'', u'supplementaryConfiguration': {u'ContinuousBackupsDescription': u'{"continuousBackupsStatus":"ENABLED","pointInTimeRecoveryDescription":{"pointInTimeRecoveryStatus":"DISABLED"}}', u'Tags': u'[{"key":"brassmonkeynew","value":"tomtagnew"},{"key":"backup-schedule","value":"daily"}]'}, u'resourceName': u'tj-test2', u'configuration': u'{"attributeDefinitions":[{"attributeName":"tj-test2","attributeType":"S"}],"tableName":"tj-test2","keySchema":[{"attributeName":"tj-
and so on. I have tried a few different ways of getting this info but every time I get a key error:
I also tried converting this into JSON and but since i have Date/time at the top it gives me this error:
“TypeError: [] is not JSON serializable
Failed attempts:
# print(response[0]["tableArn"])
print(response2)
print(response2['tableArn'])
print(response2.arn)
print(response2['configurationItems'][0]['tableArn'])
print(response2['configurationItems']['tableArn'])
print(response.configurationItems[0])
arn = response.configurationItems[0].arn
def lambda_handler(event, context):
# print("Received event: " + json.dumps(event, indent=2))
message = event['Records'][0]['Sns']['Message']
print("From SNS: " + message)
response = client.get_resource_config_history(
resourceType='AWS::DynamoDB::Table',
resourceId = message
)
response2 = dict(response)
print(response)
return message
Here's some Python3 code that shows how to access the elements:
import boto3
import json
import pprint
config_client = boto3.client('config')
response = config_client.get_resource_config_history(
resourceType='AWS::DynamoDB::Table',
resourceId = 'stack-table'
)
for item in response['configurationItems']:
configuration = item['configuration'] # Returns a JSON string
config = json.loads(configuration) # Convert to Python object
pprint.pprint(config) # Show what's in it
print(config['tableArn']) # Access elements in object
The trick is that the configuration field contains a JSON string that needs to be converted into a Python object for easy access.
Related
aws ec2 describe-snapshots --owner-ids $AWS_ACCOUNT_ID --query
"Snapshots[?(StartTime<='$dtt')].[SnapshotId]" --output text | tr '\t'
'\n' | sort
I have this shell script which I want to convert to python.
I tried looking at the boto3 documentation and came up with this
client = boto3.client('ec2')
client.describe_snapshots(OwnerIds = [os.environ['AWS_ACCOUNT_ID']], )
But I can't figure out how to change that --query tag in python.
I couldn't find it in the documentation.
What am I missing here?
You should ignore the --query portion and everything after it, and process that within Python instead.
First, store the result of the call in a variable:
ec2_client = boto3.client('ec2')
response = ec2_client.describe_snapshots(OwnerIds = ['self'])
It will return something like:
{
'NextToken': '',
'Snapshots': [
{
'Description': 'This is my snapshot.',
'OwnerId': '012345678910',
'Progress': '100%',
'SnapshotId': 'snap-1234567890abcdef0',
'StartTime': datetime(2014, 2, 28, 21, 28, 32, 4, 59, 0),
'State': 'completed',
'VolumeId': 'vol-049df61146c4d7901',
'VolumeSize': 8,
},
],
'ResponseMetadata': {
'...': '...',
},
}
Therefore, you can use response['Snapshots'] to extract your desired results, for example:
for snapshot in response['Snapshots']:
if snapshot['StartTime'] < datetime(2022, 6, 1):
print(snapshot['SnapshotId'])
It's really all Python at that point.
Given code like:
def do_stuff():
session = boto3.session.Session()
client = session.client(service_name="secretsmanager", region_name=region_name)
client.get_secret_value(SecretId=sendgrid_api_key_arn)
how do I mock
client.get_secret_value("some-value") to return some value
and how do I mock it to throw an exception
#patch("boto3.session")
def test_get_sendgrid_api_key_secret_when_client_error(mock_session):
session = mock_session.Session();
client = session.client()
client.get_secret_value().return_value =
{"SecretString": "my-secret"} <- this is wrapped in a MagicMock which is useless.
You need to set a tree of return values:
boto3.session.Session needs to return a mock object
That mock object needs a client method that returns another mock object
And that mock object needs a get_secret_value method that returns a fake value
If I assume that target.py exists and contains:
import boto3.session
def do_stuff():
session = boto3.session.Session()
client = session.client(service_name="secretsmanager", region_name='myregion')
return client.get_secret_value(SecretId='some-secret-id')
Then I can test it like this:
from unittest import mock
import target
#mock.patch("boto3.session.Session")
def test_do_stuff(mock_session_class):
mock_session_object = mock.Mock()
mock_client = mock.Mock()
mock_client.get_secret_value.return_value = {'SecretString': 'my-secret'}
mock_session_object.client.return_value = mock_client
mock_session_class.return_value = mock_session_object
res = target.do_stuff()
assert res['SecretString'] == 'my-secret'
This works, although I suspect there's a more graceful way to set it
up.
You'll need to patch client.get_secret_value, by adding one more decorator. Once this is done you an set the return_value to exception you need.
#patch("boto3.session")
#patch("boto3.session.Session.client.get_secret_value")
def test_get_sendgrid_api_key_secret_when_client_error(mock_session,second_fn):
with pytest.raises('CustomException')
session = mock_session.Session();
second_fn.return_value = lambda : Exception('CustomException')
client = session.client()
client.get_secret_value()
When client.get_secret_value() is executed, test case will pass for raising the intended exception. Similarly instead of an lambda we can assign any other value to mock the return of function.
One problem with patching boto3.session manually is that if you are using other AWS services aside from secretsmanager, then the mocked patch would be applied to everything, and this may not be what you want.
I propose here 2 solutions that are dedicated for patching/wrapping the secretsmanager functionality:
See ./test_secret_manual_amend.py below. Intercept the botocore.client.BaseClient._make_api_call.GetSecretValue and control how the response would be. This is the more flexible solution as you can freely check the accessed secret-name, raise exceptions, and return anything you like.
See ./test_secret_using_moto.py below. Use moto.mock_secretsmanager. This is the cleanest approach since this imitates how you would normally interact with the actual AWs secretsmanager. But probably a bit less flexible than solution-1.
./src.py
import boto3
sendgrid_api_key_arn = "some_name"
region_name = "ap-southeast-1"
credentials = {}
def do_stuff():
session = boto3.session.Session(**credentials)
client = session.client(service_name="secretsmanager", region_name=region_name)
return client.get_secret_value(SecretId=sendgrid_api_key_arn)
./test_secret_manual_amend.py
import botocore
import json
import pytest
from src import do_stuff, sendgrid_api_key_arn
def _amend_get_secret_value(secret_name, secret_value, mocker):
orig = botocore.client.BaseClient._make_api_call
def amend_make_api_call(self, operation_name, kwargs):
# Intercept boto3 operations for <secretsmanager.get_secret_value>. Optionally, you can also
# check on the argument <SecretId> and control how you want the response would be. This is
# a very flexible solution as you have full control over the whole process of fetching a
# secret.
if operation_name == 'GetSecretValue' and kwargs["SecretId"] == secret_name:
if isinstance(secret_value, Exception):
raise secret_value
return {
'Name': secret_name,
'SecretString': secret_value,
}
return orig(self, operation_name, kwargs)
mocker.patch('botocore.client.BaseClient._make_api_call', new=amend_make_api_call)
#pytest.mark.parametrize(
'secret_value',
[
"some value",
str(1993),
json.dumps({"SecretString": "my-secret"}),
json.dumps([2, 3, 5, 7, 11, 13, 17, 19]),
KeyError("How dare you touch my secret!"),
ValueError("Oh my goodness you even have the guts to repeat it!!!"),
],
)
def test_secret_manual_amend(secret_value, mocker):
_amend_get_secret_value(sendgrid_api_key_arn, secret_value, mocker)
if isinstance(secret_value, Exception):
with pytest.raises(type(secret_value)) as error:
do_stuff()
result = error
else:
result = do_stuff()
print("Result:", result)
./test_secret_using_moto.py
import boto3
import json
from moto import mock_secretsmanager
import pytest
from src import do_stuff, sendgrid_api_key_arn
def _setup_secrets_manager(secret_name, secret_value):
secret_manager = boto3.client('secretsmanager')
secret_manager.create_secret(
Name=secret_name,
SecretString=secret_value,
)
#mock_secretsmanager
#pytest.mark.parametrize(
'secret_value',
[
"some value",
str(1993),
json.dumps({"SecretString": "my-secret"}),
json.dumps([2, 3, 5, 7, 11, 13, 17, 19]),
],
)
def test_secret_using_moto(secret_value):
_setup_secrets_manager(sendgrid_api_key_arn, secret_value)
result = do_stuff()
print("Result:", result)
Output:
$ pytest -rP
====================================================================================== test session starts ======================================================================================
test_secret_manual_amend.py ...... [ 60%]
test_secret_using_moto.py .... [100%]
============================================================================================ PASSES =============================================================================================
_____________________________________________________________________________ test_secret_manual_amend[some value] ______________________________________________________________________________
------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------
Result: {'Name': 'some_name', 'SecretString': 'some value'}
________________________________________________________________________________ test_secret_manual_amend[1993] _________________________________________________________________________________
------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------
Result: {'Name': 'some_name', 'SecretString': '1993'}
____________________________________________________________________ test_secret_manual_amend[{"SecretString": "my-secret"}] ____________________________________________________________________
------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------
Result: {'Name': 'some_name', 'SecretString': '{"SecretString": "my-secret"}'}
____________________________________________________________________ test_secret_manual_amend[[2, 3, 5, 7, 11, 13, 17, 19]] _____________________________________________________________________
------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------
Result: {'Name': 'some_name', 'SecretString': '[2, 3, 5, 7, 11, 13, 17, 19]'}
____________________________________________________________________________ test_secret_manual_amend[secret_value4] ____________________________________________________________________________
------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------
Result: <ExceptionInfo KeyError('How dare you touch my secret!') tblen=4>
____________________________________________________________________________ test_secret_manual_amend[secret_value5] ____________________________________________________________________________
------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------
Result: <ExceptionInfo ValueError('Oh my goodness you even have the guts to repeat it!!!') tblen=4>
______________________________________________________________________________ test_secret_using_moto[some value] _______________________________________________________________________________
------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------
Result: {'ARN': 'arn:aws:secretsmanager:ap-southeast-1:1234567890:secret:some_name-ghtZn', 'Name': 'some_name', 'VersionId': '42a45aad-8764-49fb-b1ad-67759063f804', 'SecretString': 'some value', 'VersionStages': ['AWSCURRENT'], 'CreatedDate': datetime.datetime(2021, 5, 13, 15, 3, 43, tzinfo=tzlocal()), 'ResponseMetadata': {'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'amazon.com'}, 'RetryAttempts': 0}}
_________________________________________________________________________________ test_secret_using_moto[1993] __________________________________________________________________________________
------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------
Result: {'ARN': 'arn:aws:secretsmanager:ap-southeast-1:1234567890:secret:some_name-JdOXC', 'Name': 'some_name', 'VersionId': '098b2878-4368-4f25-b75d-942e82745257', 'SecretString': '1993', 'VersionStages': ['AWSCURRENT'], 'CreatedDate': datetime.datetime(2021, 5, 13, 15, 3, 44, tzinfo=tzlocal()), 'ResponseMetadata': {'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'amazon.com'}, 'RetryAttempts': 0}}
_____________________________________________________________________ test_secret_using_moto[{"SecretString": "my-secret"}] _____________________________________________________________________
------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------
Result: {'ARN': 'arn:aws:secretsmanager:ap-southeast-1:1234567890:secret:some_name-jQgDK', 'Name': 'some_name', 'VersionId': '04bd6ceb-6ec9-427b-817c-c90360abfcd9', 'SecretString': '{"SecretString": "my-secret"}', 'VersionStages': ['AWSCURRENT'], 'CreatedDate': datetime.datetime(2021, 5, 13, 15, 3, 44, tzinfo=tzlocal()), 'ResponseMetadata': {'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'amazon.com'}, 'RetryAttempts': 0}}
_____________________________________________________________________ test_secret_using_moto[[2, 3, 5, 7, 11, 13, 17, 19]] ______________________________________________________________________
------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------
Result: {'ARN': 'arn:aws:secretsmanager:ap-southeast-1:1234567890:secret:some_name-dQpLq', 'Name': 'some_name', 'VersionId': '94b7a169-c32b-4424-b47c-2dee2802c211', 'SecretString': '[2, 3, 5, 7, 11, 13, 17, 19]', 'VersionStages': ['AWSCURRENT'], 'CreatedDate': datetime.datetime(2021, 5, 13, 15, 3, 44, tzinfo=tzlocal()), 'ResponseMetadata': {'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'amazon.com'}, 'RetryAttempts': 0}}
====================================================================================== 10 passed in 1.26s =======================================================================================
I am using the python requests module to query the influxdb on my localhost using the influx API. The code is as follows.
import requests
INFLUX_DATABASE_NAME = "mydb"
influx_write_url = f'http://localhost:8086/write?db={INFLUX_DATABASE_NAME}'
influx_query_url = f'http://localhost:8086/query?db={INFLUX_DATABASE_NAME}'
pid = "CTR2257"
time_range = "7d"
pid_mod = "\'"+ pid +"\'"
query_string = f'SELECT "temp1"::field,"temp2"::field,"hum1"::field,"hum2"::field,"co2"::field,"light"::field,"fan"::field FROM "stat" WHERE productID = {pid_mod} AND time > now() - {time_range}'
payload = {'q': query_string}
r = requests.get(url=influx_query_url, params=payload)
# Printing the response content(JSON data)
print(r.content)
The print statement produces the below output. The output contains the desired dataset but the formatting is not preferred.
b'{"results":[{"statement_id":0,"series":[{"name":"stat","columns":["time","temp1","temp2","hum1","hum2","co2","light","fan"],"values":[["2020-11-04T22:09:26.4960419Z",32,34,65,68,9,true,true],["2020-11-04T23:31:33.8177588Z",30,32,90,85,5,true,true],["2020-11-04T23:31:46.5988965Z",30,32,90,85,5,true,true],["2020-11-04T23:31:49.9271554Z",30,32,90,85,5,true,true]]}]}]}\n'
When I parse this data using .json() method from requests module
request_data = r.json()
print(request_data)
I get the below python dictionary. Again the formatting is same and is not desired.
{'results': [{'statement_id': 0, 'series': [{'name': 'stat', 'columns': ['time', 'temp1', 'temp2', 'hum1', 'hum2', 'co2', 'light', 'fan'], 'values': [['2020-11-04T22:09:26.4960419Z', 32, 34, 65, 68, 9, True, True], ['2020-11-04T23:31:33.8177588Z', 30, 32, 90, 85, 5, True, True], ['2020-11-04T23:31:46.5988965Z', 30, 32, 90, 85, 5, True, True], ['2020-11-04T23:31:49.9271554Z', 30, 32, 90, 85, 5, True, True]]}]}]}
What I am looking for is to get the point data for each field in long format instead of wide format. For ex.
temp1 = [point1, point2, point3,....pointn]
temp2 = [point1, point2, point3,....pointn]
hum1 = [point1, point2, point3,....pointn]
hum2 = [point1, point2, point3,....pointn]
and so own.
Any ideas or help is highly appreciated.
i need to get values only for Czechia country from this website list "https://coronavirus-19-api.herokuapp.com/countries" and store like a variable dictionary in ptyhon.
Like this:
Czechia = {"cases":434,"todayCases":0,"deaths":0,"todayDeaths":0,"recovered":3,"active":431,"critical":2}
You could use requests to GET the JSON data from your server URL, then construct a new dictionary with country as the key:
from requests import get
URL = "https://coronavirus-19-api.herokuapp.com/countries"
req = get(URL).json()
result = {obj['country']: {k: v for k, v in obj.items() if k != 'country'} for obj in req}
print(result)
Output:
{'China': {'cases': 80894, 'todayCases': 13, 'deaths': 3237, 'todayDeaths': 11, 'recovered': 69614, 'active': 8043, 'critical': 2622}, 'Italy': {'cases': 31506, 'todayCases': 0, 'deaths': 2503, 'todayDeaths': 0, 'recovered': 2941, 'active': 26062, 'critical': 2060}...
Now you can access your data in O(1) time instead of doing a O(N) linear scan:
print(result["Czechia"])
# {'cases': 464, 'todayCases': 30, 'deaths': 0, 'todayDeaths': 0, 'recovered': 3, 'active': 461, 'critical': 2}
Note: Its probably also safe to ensure req.status_code is 200 OK or whatever else you expect to receive from the server.
In [1]: import requests
...: import json
...:
...: data = requests.get('https://coronavirus-19-api.herokuapp.com/countries').json()
...: result = next(item for item in data if item["country"] == "Czechia")
...: print(json.dumps(result, indent=4))
{
"country": "Czechia",
"cases": 464,
"todayCases": 30,
"deaths": 0,
"todayDeaths": 0,
"recovered": 3,
"active": 461,
"critical": 2
}
In [2]:
While the RoadRunner 's answers solves your problem, I am just giving you the one other way of doing it using python's urllib module.
from urllib.request import urlopen
##import ast
import json
def Corona_Tracker():
res = urlopen('https://coronavirus-19-api.herokuapp.com/countries')
result = res.read().strip()
result_str = json.loads(result)
return result_str
if __name__ == "__main__":
result_str=Corona_Tracker()
while True:
for data in result_str:
if data['country'] == "India":
print(data)
Just replace India with your country , it gives the below output
>>> {'country': 'India', 'cases': 148, 'todayCases': 5, 'deaths': 3, 'todayDeaths': 0, 'recovered': 14, 'active': 131, 'critical': 0}
i am new in Python language. I need to get all Amazon-Web-Services Identity and Access Management (Amazon-IAM) policy details using Boto 3 and Python.
I tried to parse JSON output from Boto 3 client and also need to save key-value pair into a map (policyName, Arn). Sample JSON output is like this:
{
'ResponseMetadata': {
'HTTPStatusCode': 200,
'HTTPHeaders': {
'vary': 'Accept-Encoding',
'content-length': '19143',
'content-type': 'text/xml',
'date': 'Thu, 23 Feb 2017 06:39:25 GMT'
}
},
u 'Books': [ {
u 'PolicyName': 'book1',
u 'Arn': '002dfgdfgdfgdfgvdfxgdfgdfgdfgfdg',
u 'CreateDate': datetime.datetime(2017, 2, 22, 13, 10, 55, tzinfo = tzutc()),
u 'UpdateDate': datetime.datetime(2017, 2, 22, 13, 10, 55, tzinfo = tzutc())
}, {
u 'PolicyName': 'book2','
u 'Arn': '002dfgdfgdfgdfgvdfxgdfgdfgdfgfdg',
u 'CreateDate': datetime.datetime(2017, 2, 22, 13, 10, 55, tzinfo = tzutc()),
u 'UpdateDate': datetime.datetime(2017, 2, 22, 13, 10, 55, tzinfo = tzutc())
}]
}
I have following code
iampolicylist_response = iamClient.list_policies(
Scope='Local',
MaxItems=150
)
print iampolicylist_response
res=json.dumps(iampolicylist_response)
print res
ret={}
for i in res["PolicyName"]:
ret[i["PolicyName"]]=i["Arn"]
return ret
Using json.loads, it shows error like this
TypeError: expected string or buffer
Using json.dumps, it shows error like this
TypeError: datetime.datetime(2017, 2, 22, 13, 10, 55, tzinfo=tzutc()) is not JSON serializable
What is actual issue?
The result iampolicylist_response is already a dictionary
You do not need to parse it .
See http://boto3.readthedocs.io/en/latest/reference/services/iam.html#IAM.Client.list_policies
The response is a dictionary object
Remove res=json.dumps(iampolicylist_response)