I am trying to get a policy from boto3 client but there is no method to do so using policy name. By wrapping the create_policy method in a try-except block i can check whether a policy exists or not. Is there any way to get a policy-arn by name using boto3 except for listing all policies and iterating over it.
The ARN should be deterministic given the prefix (if any, and the name).
iam = session.client('iam')
sts = session.client('sts')
# Slow and costly if you have many pages
paginator = iam.get_paginator('list_policies')
all_policies = [policy for page in paginator.paginate() for policy in page['Policies']]
[policy_1] = [p for p in all_policies if p['PolicyName'] == policy_name]
# Fast and direct
account_id = sts.get_caller_identity()['Account']
policy_arn = f'arn:aws:iam::{account_id}:policy/{policy_name}'
policy_2 = iam.get_policy(PolicyArn=policy_arn)['Policy']
# They're equal except with the direct method you'll also get description field
all(policy_1[k] == policy_2[k] for k in policy_1.keys() & policy_2.keys())
You will need to iterate over the policies to get policy names. I am not aware of a get-policy type api that uses policy names only policy ARNs.
Is there a reason that you do not want to get a list of policies? Other than to not download the list.
Related
I am defining a method which fetches all accountIDs from an organization.
If I am using get_paginator('list_accounts'), then am I okay if I do not check the NextToken?
Code to get the list of all AWS account IDs in the organization:
def get_all_account_ids():
org_client = boto3.client('organizations')
paginator = org_client.get_paginator('list_accounts')
page_iterator = paginator.paginate()
account_ids = []
for page in page_iterator:
for acct in page['Accounts']:
print(acct['Id']) # print the account id
# add to account_ids list
account_ids.append(acct['Id'])
return account_ids
I have seen examples of using either get_paginator() call or while loop checking for NextToken. But I have not seen example using both paginator and NextToken?
No you don't have to check NextToken. That's the point of paginators:
Paginators are a feature of boto3 that act as an abstraction over the process of iterating over an entire result set of a truncated API operation.
I'm using the Python ibm-cloud-sdk in an attempt to iterate all resources in a particular IBM Cloud account. My trouble has been that pagination doesn't appear to "work for me". When I pass in the "next_url" I still get the same list coming back from the call.
Here is my test code. I successfully print many of my COS instances, but I only seem to be able to print the first page....maybe I've been looking at this too long and just missed something obvious...anyone have any clue why I can't retrieve the next page?
try:
####### authenticate and set the service url
auth = IAMAuthenticator(RESOURCE_CONTROLLER_APIKEY)
service = ResourceControllerV2(authenticator=auth)
service.set_service_url(RESOURCE_CONTROLLER_URL)
####### Retrieve the resource instance listing
r = service.list_resource_instances().get_result()
####### get the row count and resources list
rows_count = r['rows_count']
resources = r['resources']
while rows_count > 0:
print('Number of rows_count {}'.format(rows_count))
next_url = r['next_url']
for i, resource in enumerate(resources):
type = resource['id'].split(':')[4]
if type == 'cloud-object-storage':
instance_name = resource['name']
instance_id = resource['guid']
crn = resource['crn']
print('Found instance id : name - {} : {}'.format(instance_id, instance_name))
############### this is SUPPOSED to get the next page
r = service.list_resource_instances(start=next_url).get_result()
rows_count = r['rows_count']
resources = r['resources']
except Exception as e:
Error = 'Error : {}'.format(e)
print(Error)
exit(1)
From looking at the API documentation for listing resource instances, the value of next_url includes the URL path and the start parameter including its token for start.
To retrieve the next page, you would only need to pass in the parameter start with the token as value. IMHO this is not ideal.
I typically do not use the SDK, but a simply Python request. Then, I can use the endpoint (base) URI + next_url as full URI.
If you stick with the SDK, use urllib.parse to extract the query parameter. Not tested, but something like:
from urllib.parse import urlparse,parse_qs
o=urlparse(next_url)
q=parse_qs(o.query)
r = service.list_resource_instances(start=q['start'][0]).get_result()
Could you use the Search API for listing the resources in your account rather than the resource controller? The search index is set up for exactly that operation, whereas paginating results from the resource controller seems much more brute force.
https://cloud.ibm.com/apidocs/search#search
InvalidS3ObjectException when calling the AnalyzeDocument operation: Unable to get object metadata from S3. Check object key, region and/or access permissions."
I keep getting this error. Over. And. Over. This program worked with my test cases of what I'm bringing in, the json with a {"body":"imagename.jpg"}. But the very moment I try to utilize the actual code my JS brings in, I get this error. The thing that confuses me is that I've checked the regions and they are fine. I went into my account and created users with full access to all AWS and S3 features, and utilized those logins, I've used my root account, everything. All I'm trying to do is access an image from my s3 bucket. Why won't it work? Below is my code. It works if I utilize the test case I provided above, but the moment I try and use the website it's connected to, it doesn't work.
def main(event, context):
key_map, value_map, block_map = get_kv_map(event) #Take map variables in to get the key and value map we need.
It goes to this function...
def get_kv_map(event):
filePath = event
fileExt = filePath.get('body')
s3 = boto3.resource('s3', region_name='us-east-1')
bucket = s3.Bucket('react-images-ex')
obj = bucket.Object(bucket)
client = boto3.client('textract') #We utilize boto3's textract lib
response = client.analyze_document(Document={'S3Object': {'Bucket': 'react-images-ex', 'Name': fileExt}}, FeatureTypes=['FORMS'])
# Get the text blocks
blocks=response['Blocks'] #We make a blocks variable that will be the blocks we find in the document
# get key and value maps
key_map = {}
value_map = {}
block_map = {}
for block in blocks: #Traverse the blocks found in the document
block_id = block['Id'] #Set variable for blockId to the Id's found on that block location
block_map[block_id] = block #Make the block map at that ID be the block variable
if block['BlockType'] == "KEY_VALUE_SET": #if we see that the type of block we're on is a key and value set pair, we check if it's a key or not. If it's not a key, we know it's a value. We send it to the respective map.
if 'KEY' in block['EntityTypes']:
key_map[block_id] = block
else:
value_map[block_id] = block
return key_map, value_map, block_map #Return the maps we need after they're filled.
I have been told before this code is fine, and it should work. So why exactly is it that I get this error?
Based on the comments.
The issue with body was that it was json string, not actual json object.
The solution was to parse the string into json:
fileExt = json.loads(filePath.get('body'))
Try awscli to see if you can access the image in s3:
aws s3 ls s3://react-images-ex/<some-fileExt>
Either you are parsing the fileExt wrongly, or you don't have S3 permission to access the file. The awscli command will help to verify this.
How can the number of blobs returned from ContainerClient.list_blobs() method can be limited?
The Azure Blob service RESP API docs mentions a maxresults parameter, but it seems it is not honored by list_blobs(maxresults=123).
A combination of itertools.islice and the results_per_page parameter (which translates to the REST maxresults parameter) will do the trick:
import itertools
service: BlobServiceClient = BlobServiceClient.from_connection_string(cstr)
cc = service.get_container_client("foo")
n = 42
for b in itertools.islice(cc.list_blobs(results_per_page=n), n):
print(b.name)
Please use by_page() on the ItemPaged class
pages = ContainerClient.list_blobs(maxresults=123).by_page()
first_page = next(pages)
items_in_page = list(a_page) #this will give you 123 results on the first page
second_page = next(pages) # it will throw exception if there's no second page
items_in_page = list(a_page) #this will give you 123 results on the second page
There's no way to do this currently with the SDK. The maxresults parameter really means "max results per page"; if you have more blobs than this, list_blobs will make multiple calls to the REST API until the listing is complete.
You could call the API directly and ignore pages after the first, but that would require you to handle the details of authentication, parsing the response, etc.
Unable to filter the instance by tags and get the list of instances.Please help how can i proceed further.
import boto3
ec2 = boto3.resource('ec2')
def lambda_handler(event, context):
# Use the filter() method of the instances collection to retrieve
# all running EC2 instances.
filters = [{'Name':'OS_Name', 'Values':['Rstudio']}]
#filter the instances
instances = ec2.instances.filter(Filters=filters)
#locate all running instances
RunningInstances = [instance.id for instance in instances]
#print the instances for logging purposes
#print RunningInstances
#make sure there are actually instances to shut down.
if len(RunningInstances) > 0:
#perform the shutdown
shuttingDown = ec2.instances.filter(InstanceIds=RunningInstances).stop()
print (shuttingDown)
else:
print ("Nothing to see here")
you need to specify the type of filter so in this case it would be tag. filters = [{'Name':'tag:OS_Name', 'Values':['Rstudio']}]
from boto3 docs
tag : - The key/value combination of a tag assigned to the resource. Use the tag key in the filter name and the tag value as the filter value. For example, to find all resources that have a tag with the key Owner and the value TeamA , specify tag:Owner for the filter name and TeamA for the filter value.