i am new in Python language. I need to get all Amazon-Web-Services Identity and Access Management (Amazon-IAM) policy details using Boto 3 and Python.
I tried to parse JSON output from Boto 3 client and also need to save key-value pair into a map (policyName, Arn). Sample JSON output is like this:
{
'ResponseMetadata': {
'HTTPStatusCode': 200,
'HTTPHeaders': {
'vary': 'Accept-Encoding',
'content-length': '19143',
'content-type': 'text/xml',
'date': 'Thu, 23 Feb 2017 06:39:25 GMT'
}
},
u 'Books': [ {
u 'PolicyName': 'book1',
u 'Arn': '002dfgdfgdfgdfgvdfxgdfgdfgdfgfdg',
u 'CreateDate': datetime.datetime(2017, 2, 22, 13, 10, 55, tzinfo = tzutc()),
u 'UpdateDate': datetime.datetime(2017, 2, 22, 13, 10, 55, tzinfo = tzutc())
}, {
u 'PolicyName': 'book2','
u 'Arn': '002dfgdfgdfgdfgvdfxgdfgdfgdfgfdg',
u 'CreateDate': datetime.datetime(2017, 2, 22, 13, 10, 55, tzinfo = tzutc()),
u 'UpdateDate': datetime.datetime(2017, 2, 22, 13, 10, 55, tzinfo = tzutc())
}]
}
I have following code
iampolicylist_response = iamClient.list_policies(
Scope='Local',
MaxItems=150
)
print iampolicylist_response
res=json.dumps(iampolicylist_response)
print res
ret={}
for i in res["PolicyName"]:
ret[i["PolicyName"]]=i["Arn"]
return ret
Using json.loads, it shows error like this
TypeError: expected string or buffer
Using json.dumps, it shows error like this
TypeError: datetime.datetime(2017, 2, 22, 13, 10, 55, tzinfo=tzutc()) is not JSON serializable
What is actual issue?
The result iampolicylist_response is already a dictionary
You do not need to parse it .
See http://boto3.readthedocs.io/en/latest/reference/services/iam.html#IAM.Client.list_policies
The response is a dictionary object
Remove res=json.dumps(iampolicylist_response)
Related
aws ec2 describe-snapshots --owner-ids $AWS_ACCOUNT_ID --query
"Snapshots[?(StartTime<='$dtt')].[SnapshotId]" --output text | tr '\t'
'\n' | sort
I have this shell script which I want to convert to python.
I tried looking at the boto3 documentation and came up with this
client = boto3.client('ec2')
client.describe_snapshots(OwnerIds = [os.environ['AWS_ACCOUNT_ID']], )
But I can't figure out how to change that --query tag in python.
I couldn't find it in the documentation.
What am I missing here?
You should ignore the --query portion and everything after it, and process that within Python instead.
First, store the result of the call in a variable:
ec2_client = boto3.client('ec2')
response = ec2_client.describe_snapshots(OwnerIds = ['self'])
It will return something like:
{
'NextToken': '',
'Snapshots': [
{
'Description': 'This is my snapshot.',
'OwnerId': '012345678910',
'Progress': '100%',
'SnapshotId': 'snap-1234567890abcdef0',
'StartTime': datetime(2014, 2, 28, 21, 28, 32, 4, 59, 0),
'State': 'completed',
'VolumeId': 'vol-049df61146c4d7901',
'VolumeSize': 8,
},
],
'ResponseMetadata': {
'...': '...',
},
}
Therefore, you can use response['Snapshots'] to extract your desired results, for example:
for snapshot in response['Snapshots']:
if snapshot['StartTime'] < datetime(2022, 6, 1):
print(snapshot['SnapshotId'])
It's really all Python at that point.
I do have a json format which is generated from docanno annotation tool. I want to convert the json into another format. Please check below for the format
Docanno json format :
{"id": 2, "data": "My name is Nithin Reddy and i'm working as a Data Scientist.", "label": [[3, 8, "Misc"], [11, 23, "Person"], [32, 39, "Activity"], [45, 59, "Designation"]]}
{"id": 3, "data": "I live in Hyderabad.", "label": [[2, 6, "Misc"], [10, 19, "Location"]]}
{"id": 4, "data": "I'm pusring my master's from Bits Pilani.", "label": [[15, 24, "Education"], [29, 40, "Organization"]]}
Required json format :
("My name is Nithin Reddy and i'm working as a Data Scientist.", {"entities": [(3, 8, "Misc"), (11, 23, "Person"), (32, 39, "Activity"), (45, 59, "Designation")]}),
("I live in Hyderabad.", {"entities": [(2, 6, "Misc"), (10, 19, "Location")]}),
("I'm pusring my master's from Bits Pilani.", {"entities": [(15, 24, "Education"), (29, 40, "Organization")]})
I tried the below code, but it's not working
import json
with open('data.json') as f:
data = json.load(f)
new_data = []
for i in data:
new_data.append((i['data'], {"entities": i['label']}))
with open('data_new.json', 'w') as f:
json.dump(new_data, f)
Can anyone help me with the python code which will change the json to required format?
Given code like:
def do_stuff():
session = boto3.session.Session()
client = session.client(service_name="secretsmanager", region_name=region_name)
client.get_secret_value(SecretId=sendgrid_api_key_arn)
how do I mock
client.get_secret_value("some-value") to return some value
and how do I mock it to throw an exception
#patch("boto3.session")
def test_get_sendgrid_api_key_secret_when_client_error(mock_session):
session = mock_session.Session();
client = session.client()
client.get_secret_value().return_value =
{"SecretString": "my-secret"} <- this is wrapped in a MagicMock which is useless.
You need to set a tree of return values:
boto3.session.Session needs to return a mock object
That mock object needs a client method that returns another mock object
And that mock object needs a get_secret_value method that returns a fake value
If I assume that target.py exists and contains:
import boto3.session
def do_stuff():
session = boto3.session.Session()
client = session.client(service_name="secretsmanager", region_name='myregion')
return client.get_secret_value(SecretId='some-secret-id')
Then I can test it like this:
from unittest import mock
import target
#mock.patch("boto3.session.Session")
def test_do_stuff(mock_session_class):
mock_session_object = mock.Mock()
mock_client = mock.Mock()
mock_client.get_secret_value.return_value = {'SecretString': 'my-secret'}
mock_session_object.client.return_value = mock_client
mock_session_class.return_value = mock_session_object
res = target.do_stuff()
assert res['SecretString'] == 'my-secret'
This works, although I suspect there's a more graceful way to set it
up.
You'll need to patch client.get_secret_value, by adding one more decorator. Once this is done you an set the return_value to exception you need.
#patch("boto3.session")
#patch("boto3.session.Session.client.get_secret_value")
def test_get_sendgrid_api_key_secret_when_client_error(mock_session,second_fn):
with pytest.raises('CustomException')
session = mock_session.Session();
second_fn.return_value = lambda : Exception('CustomException')
client = session.client()
client.get_secret_value()
When client.get_secret_value() is executed, test case will pass for raising the intended exception. Similarly instead of an lambda we can assign any other value to mock the return of function.
One problem with patching boto3.session manually is that if you are using other AWS services aside from secretsmanager, then the mocked patch would be applied to everything, and this may not be what you want.
I propose here 2 solutions that are dedicated for patching/wrapping the secretsmanager functionality:
See ./test_secret_manual_amend.py below. Intercept the botocore.client.BaseClient._make_api_call.GetSecretValue and control how the response would be. This is the more flexible solution as you can freely check the accessed secret-name, raise exceptions, and return anything you like.
See ./test_secret_using_moto.py below. Use moto.mock_secretsmanager. This is the cleanest approach since this imitates how you would normally interact with the actual AWs secretsmanager. But probably a bit less flexible than solution-1.
./src.py
import boto3
sendgrid_api_key_arn = "some_name"
region_name = "ap-southeast-1"
credentials = {}
def do_stuff():
session = boto3.session.Session(**credentials)
client = session.client(service_name="secretsmanager", region_name=region_name)
return client.get_secret_value(SecretId=sendgrid_api_key_arn)
./test_secret_manual_amend.py
import botocore
import json
import pytest
from src import do_stuff, sendgrid_api_key_arn
def _amend_get_secret_value(secret_name, secret_value, mocker):
orig = botocore.client.BaseClient._make_api_call
def amend_make_api_call(self, operation_name, kwargs):
# Intercept boto3 operations for <secretsmanager.get_secret_value>. Optionally, you can also
# check on the argument <SecretId> and control how you want the response would be. This is
# a very flexible solution as you have full control over the whole process of fetching a
# secret.
if operation_name == 'GetSecretValue' and kwargs["SecretId"] == secret_name:
if isinstance(secret_value, Exception):
raise secret_value
return {
'Name': secret_name,
'SecretString': secret_value,
}
return orig(self, operation_name, kwargs)
mocker.patch('botocore.client.BaseClient._make_api_call', new=amend_make_api_call)
#pytest.mark.parametrize(
'secret_value',
[
"some value",
str(1993),
json.dumps({"SecretString": "my-secret"}),
json.dumps([2, 3, 5, 7, 11, 13, 17, 19]),
KeyError("How dare you touch my secret!"),
ValueError("Oh my goodness you even have the guts to repeat it!!!"),
],
)
def test_secret_manual_amend(secret_value, mocker):
_amend_get_secret_value(sendgrid_api_key_arn, secret_value, mocker)
if isinstance(secret_value, Exception):
with pytest.raises(type(secret_value)) as error:
do_stuff()
result = error
else:
result = do_stuff()
print("Result:", result)
./test_secret_using_moto.py
import boto3
import json
from moto import mock_secretsmanager
import pytest
from src import do_stuff, sendgrid_api_key_arn
def _setup_secrets_manager(secret_name, secret_value):
secret_manager = boto3.client('secretsmanager')
secret_manager.create_secret(
Name=secret_name,
SecretString=secret_value,
)
#mock_secretsmanager
#pytest.mark.parametrize(
'secret_value',
[
"some value",
str(1993),
json.dumps({"SecretString": "my-secret"}),
json.dumps([2, 3, 5, 7, 11, 13, 17, 19]),
],
)
def test_secret_using_moto(secret_value):
_setup_secrets_manager(sendgrid_api_key_arn, secret_value)
result = do_stuff()
print("Result:", result)
Output:
$ pytest -rP
====================================================================================== test session starts ======================================================================================
test_secret_manual_amend.py ...... [ 60%]
test_secret_using_moto.py .... [100%]
============================================================================================ PASSES =============================================================================================
_____________________________________________________________________________ test_secret_manual_amend[some value] ______________________________________________________________________________
------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------
Result: {'Name': 'some_name', 'SecretString': 'some value'}
________________________________________________________________________________ test_secret_manual_amend[1993] _________________________________________________________________________________
------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------
Result: {'Name': 'some_name', 'SecretString': '1993'}
____________________________________________________________________ test_secret_manual_amend[{"SecretString": "my-secret"}] ____________________________________________________________________
------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------
Result: {'Name': 'some_name', 'SecretString': '{"SecretString": "my-secret"}'}
____________________________________________________________________ test_secret_manual_amend[[2, 3, 5, 7, 11, 13, 17, 19]] _____________________________________________________________________
------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------
Result: {'Name': 'some_name', 'SecretString': '[2, 3, 5, 7, 11, 13, 17, 19]'}
____________________________________________________________________________ test_secret_manual_amend[secret_value4] ____________________________________________________________________________
------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------
Result: <ExceptionInfo KeyError('How dare you touch my secret!') tblen=4>
____________________________________________________________________________ test_secret_manual_amend[secret_value5] ____________________________________________________________________________
------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------
Result: <ExceptionInfo ValueError('Oh my goodness you even have the guts to repeat it!!!') tblen=4>
______________________________________________________________________________ test_secret_using_moto[some value] _______________________________________________________________________________
------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------
Result: {'ARN': 'arn:aws:secretsmanager:ap-southeast-1:1234567890:secret:some_name-ghtZn', 'Name': 'some_name', 'VersionId': '42a45aad-8764-49fb-b1ad-67759063f804', 'SecretString': 'some value', 'VersionStages': ['AWSCURRENT'], 'CreatedDate': datetime.datetime(2021, 5, 13, 15, 3, 43, tzinfo=tzlocal()), 'ResponseMetadata': {'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'amazon.com'}, 'RetryAttempts': 0}}
_________________________________________________________________________________ test_secret_using_moto[1993] __________________________________________________________________________________
------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------
Result: {'ARN': 'arn:aws:secretsmanager:ap-southeast-1:1234567890:secret:some_name-JdOXC', 'Name': 'some_name', 'VersionId': '098b2878-4368-4f25-b75d-942e82745257', 'SecretString': '1993', 'VersionStages': ['AWSCURRENT'], 'CreatedDate': datetime.datetime(2021, 5, 13, 15, 3, 44, tzinfo=tzlocal()), 'ResponseMetadata': {'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'amazon.com'}, 'RetryAttempts': 0}}
_____________________________________________________________________ test_secret_using_moto[{"SecretString": "my-secret"}] _____________________________________________________________________
------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------
Result: {'ARN': 'arn:aws:secretsmanager:ap-southeast-1:1234567890:secret:some_name-jQgDK', 'Name': 'some_name', 'VersionId': '04bd6ceb-6ec9-427b-817c-c90360abfcd9', 'SecretString': '{"SecretString": "my-secret"}', 'VersionStages': ['AWSCURRENT'], 'CreatedDate': datetime.datetime(2021, 5, 13, 15, 3, 44, tzinfo=tzlocal()), 'ResponseMetadata': {'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'amazon.com'}, 'RetryAttempts': 0}}
_____________________________________________________________________ test_secret_using_moto[[2, 3, 5, 7, 11, 13, 17, 19]] ______________________________________________________________________
------------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------------
Result: {'ARN': 'arn:aws:secretsmanager:ap-southeast-1:1234567890:secret:some_name-dQpLq', 'Name': 'some_name', 'VersionId': '94b7a169-c32b-4424-b47c-2dee2802c211', 'SecretString': '[2, 3, 5, 7, 11, 13, 17, 19]', 'VersionStages': ['AWSCURRENT'], 'CreatedDate': datetime.datetime(2021, 5, 13, 15, 3, 44, tzinfo=tzlocal()), 'ResponseMetadata': {'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'amazon.com'}, 'RetryAttempts': 0}}
====================================================================================== 10 passed in 1.26s =======================================================================================
I am running into issue with trying to pull out usable items from this output. I am just trying to pull a single value from this string of Unicode and it has been super fun.
my print(response) returns this: FYI this is way longer than this little snippet.
{u'configurationItems': [{u'configurationItemCaptureTime': datetime.datetime(2020, 6, 4, 21, 56, 31, 134000, tzinfo=tzlocal()), u'resourceCreationTime': datetime.datetime(2020, 5, 22, 16, 32, 55, 162000, tzinfo=tzlocal()), u'availabilityZone': u'Not Applicable', u'awsRegion': u'us-east-1', u'tags': {u'brassmonkeynew': u'tomtagnew'}, u'resourceType': u'AWS::DynamoDB::Table', u'resourceId': u'tj-test2', u'configurationStateId': u'1591307791134', u'relatedEvents': [], u'relationships': [], u'arn': u'arn:aws:dynamodb:us-east-1:896911201517:table/tj-test2', u'version': u'1.3', u'configurationItemMD5Hash': u'', u'supplementaryConfiguration': {u'ContinuousBackupsDescription': u'{"continuousBackupsStatus":"ENABLED","pointInTimeRecoveryDescription":{"pointInTimeRecoveryStatus":"DISABLED"}}', u'Tags': u'[{"key":"brassmonkeynew","value":"tomtagnew"}]'}, u'resourceName': u'tj-test2', u'configuration': u'{"attributeDefinitions":[{"attributeName":"tj-test2","attributeType":"S"}],"tableName":"tj-test2","keySchema":[{"attributeName":"tj-test2","keyType":"HASH"}],"tableStatus":"ACTIVE","creationDateTime":1590165175162,"provisionedThroughput":{"numberOfDecreasesToday":0,"readCapacityUnits":5,"writeCapacityUnits":5},"tableArn":"arn:aws:dynamodb:us-east-1:896911201517:table/tj-test2","tableId":"816956d7-95d1-4d31-8d18-f11b18de4643"}', u'configurationItemStatus': u'OK', u'accountId': u'896911201517'}, {u'configurationItemCaptureTime': datetime.datetime(2020, 6, 1, 16, 27, 21, 316000, tzinfo=tzlocal()), u'resourceCreationTime': datetime.datetime(2020, 5, 22, 16, 32, 55, 162000, tzinfo=tzlocal()), u'availabilityZone': u'Not Applicable', u'awsRegion': u'us-east-1', u'tags': {u'brassmonkeynew': u'tomtagnew', u'backup-schedule': u'daily'}, u'resourceType': u'AWS::DynamoDB::Table', u'resourceId': u'tj-test2', u'configurationStateId': u'1591028841316', u'relatedEvents': [], u'relationships': [], u'arn': u'arn:aws:dynamodb:us-east-1:896911201517:table/tj-test2', u'version': u'1.3', u'configurationItemMD5Hash': u'', u'supplementaryConfiguration': {u'ContinuousBackupsDescription': u'{"continuousBackupsStatus":"ENABLED","pointInTimeRecoveryDescription":{"pointInTimeRecoveryStatus":"DISABLED"}}', u'Tags': u'[{"key":"brassmonkeynew","value":"tomtagnew"},{"key":"backup-schedule","value":"daily"}]'}, u'resourceName': u'tj-test2', u'configuration': u'{"attributeDefinitions":[{"attributeName":"tj-test2","attributeType":"S"}],"tableName":"tj-test2","keySchema":[{"attributeName":"tj-
and so on. I have tried a few different ways of getting this info but every time I get a key error:
I also tried converting this into JSON and but since i have Date/time at the top it gives me this error:
“TypeError: [] is not JSON serializable
Failed attempts:
# print(response[0]["tableArn"])
print(response2)
print(response2['tableArn'])
print(response2.arn)
print(response2['configurationItems'][0]['tableArn'])
print(response2['configurationItems']['tableArn'])
print(response.configurationItems[0])
arn = response.configurationItems[0].arn
def lambda_handler(event, context):
# print("Received event: " + json.dumps(event, indent=2))
message = event['Records'][0]['Sns']['Message']
print("From SNS: " + message)
response = client.get_resource_config_history(
resourceType='AWS::DynamoDB::Table',
resourceId = message
)
response2 = dict(response)
print(response)
return message
Here's some Python3 code that shows how to access the elements:
import boto3
import json
import pprint
config_client = boto3.client('config')
response = config_client.get_resource_config_history(
resourceType='AWS::DynamoDB::Table',
resourceId = 'stack-table'
)
for item in response['configurationItems']:
configuration = item['configuration'] # Returns a JSON string
config = json.loads(configuration) # Convert to Python object
pprint.pprint(config) # Show what's in it
print(config['tableArn']) # Access elements in object
The trick is that the configuration field contains a JSON string that needs to be converted into a Python object for easy access.
I have the following structure as a list (originally returned as a result set from django ORM):
{'valuex': 65.92, 'day': datetime.date(2014, 10, 15), 'valuey': 74}{'valuex': 58.55, 'day': datetime.date(2014, 10, 16), 'valuey': 61}
And I'd like to transform it (and output it) as the following json:
{
'2014-10-15' : { 'valuex': 65.92, 'day': datetime.date(2014, 10, 15), 'valuey': 74 },
'2014-10-16' : { 'valuex': 58.55, 'day': datetime.date(2014, 10, 16), 'valuey': 61 }
}
I assumed that it would be something trivial like:
ret = {}
for r in results:
ret[r.get('day')] = r
and then outputing to json could be done by Django (#render_to_json)
But that really only returns the dates with no corresponding dictionary.
I'm new to Python and I'm lost here.
UPDATE:
I found out that the actual issue is that Django is not outputing my structure as json. the script above actually does create the structure I'm trying to achieve. but django fails to convert it to json. is it safe to do:
return HttpResponse(str(ret))
instead? Or is there some way to make it render to proper json (json.dumps has the same issue).
UPDATE2:
Okay, json.dumps(ret) acutally did work for me, only needed to get rid of the datetime object as it's not json seralizeable.
import datetime
results = [{'valuex': 65.92, 'day': datetime.date(2014, 10, 15), 'valuey': 74}, {'valuex': 58.55, 'day': datetime.date(2014, 10, 16), 'valuey': 61}]
ret = {}
for r in results:
ret.update({str(r['day']): r})
print ret
{'2014-10-16': {'day': datetime.date(2014, 10, 16), 'valuey': 61, 'valuex': 58.55}, '2014-10-15': {'day': datetime.date(2014, 10, 15), 'valuey': 74, 'valuex': 65.92}}
Solution #1
return HttpResponse(json.dumps({'result': 'done'}), content_type='application/json')
Solution #2
class JSONResponse(HttpResponse):
"""
An HttpResponse that renders its content into JSON.
"""
def __init__(self, data, **kwargs):
content = simplejson.dumps(data)
kwargs['content_type'] = 'application/json'
super(JSONResponse, self).__init__(content, **kwargs)
in view instead of HttpResponse write
data = {'success': True}
return JSONResponse(data)
Regarding datetime, you've to convert the datetime object to string using
my_datetime.strftime('<format>')
and use the same in your dict.
import datetime
d1 = [{'valuex': 65.92, 'day': datetime.date(2014, 10, 15), 'valuey': 74},{'valuex': 58.55, 'day': datetime.date(2014, 10, 16), 'valuey': 61}]
for i in d1:
d11 = dict([[str(i['day']),i]])
print d11
{'2014-10-15': {'day': datetime.date(2014, 10, 15), 'valuey': 74, 'valuex': 65.92}}
{'2014-10-16': {'day': datetime.date(2014, 10, 16), 'valuey': 61, 'valuex': 58.55}}
OR
new_dict = {}
for i in di:
new_dict[str(i['day'])] = i
Update:
Try this
>>> from django.http import JsonResponse
>>> response = JsonResponse({'foo': 'bar'})
>>> response.content
'{"foo": "bar"}'