I'm using the Python ibm-cloud-sdk in an attempt to iterate all resources in a particular IBM Cloud account. My trouble has been that pagination doesn't appear to "work for me". When I pass in the "next_url" I still get the same list coming back from the call.
Here is my test code. I successfully print many of my COS instances, but I only seem to be able to print the first page....maybe I've been looking at this too long and just missed something obvious...anyone have any clue why I can't retrieve the next page?
try:
####### authenticate and set the service url
auth = IAMAuthenticator(RESOURCE_CONTROLLER_APIKEY)
service = ResourceControllerV2(authenticator=auth)
service.set_service_url(RESOURCE_CONTROLLER_URL)
####### Retrieve the resource instance listing
r = service.list_resource_instances().get_result()
####### get the row count and resources list
rows_count = r['rows_count']
resources = r['resources']
while rows_count > 0:
print('Number of rows_count {}'.format(rows_count))
next_url = r['next_url']
for i, resource in enumerate(resources):
type = resource['id'].split(':')[4]
if type == 'cloud-object-storage':
instance_name = resource['name']
instance_id = resource['guid']
crn = resource['crn']
print('Found instance id : name - {} : {}'.format(instance_id, instance_name))
############### this is SUPPOSED to get the next page
r = service.list_resource_instances(start=next_url).get_result()
rows_count = r['rows_count']
resources = r['resources']
except Exception as e:
Error = 'Error : {}'.format(e)
print(Error)
exit(1)
From looking at the API documentation for listing resource instances, the value of next_url includes the URL path and the start parameter including its token for start.
To retrieve the next page, you would only need to pass in the parameter start with the token as value. IMHO this is not ideal.
I typically do not use the SDK, but a simply Python request. Then, I can use the endpoint (base) URI + next_url as full URI.
If you stick with the SDK, use urllib.parse to extract the query parameter. Not tested, but something like:
from urllib.parse import urlparse,parse_qs
o=urlparse(next_url)
q=parse_qs(o.query)
r = service.list_resource_instances(start=q['start'][0]).get_result()
Could you use the Search API for listing the resources in your account rather than the resource controller? The search index is set up for exactly that operation, whereas paginating results from the resource controller seems much more brute force.
https://cloud.ibm.com/apidocs/search#search
Related
I am defining a method which fetches all accountIDs from an organization.
If I am using get_paginator('list_accounts'), then am I okay if I do not check the NextToken?
Code to get the list of all AWS account IDs in the organization:
def get_all_account_ids():
org_client = boto3.client('organizations')
paginator = org_client.get_paginator('list_accounts')
page_iterator = paginator.paginate()
account_ids = []
for page in page_iterator:
for acct in page['Accounts']:
print(acct['Id']) # print the account id
# add to account_ids list
account_ids.append(acct['Id'])
return account_ids
I have seen examples of using either get_paginator() call or while loop checking for NextToken. But I have not seen example using both paginator and NextToken?
No you don't have to check NextToken. That's the point of paginators:
Paginators are a feature of boto3 that act as an abstraction over the process of iterating over an entire result set of a truncated API operation.
I am using an API from this site https://dev.whatismymmr.com, and I want to specifically request for the closestRank but I just get a KeyError: 'ranked.closestRank'. but I can get the entire ['ranked'] object (which contains the closestRank) but I just end up with a lot of information I don't need.
How can I end up with just the Closest rank?
My code
import requests
LeagueName = input ("Summoner name")
base = ("https://eune.whatismymmr.com/api/v1/summoner?name=")
Thething = base + LeagueName
print (Thething)
response = requests.get(Thething)
print(response.status_code)
MMR = response.json()
print (MMR['ranked.closestRank'])
The API command
<queue>.closestRank (the queue is the game mode, it can be normal or ranked)
you can use the summoner name babada27 for testing.
Hope this is what you are looking for -
change The last line to
print (MMR["ranked"]["closestRank"])
I've been working with boto3 for a while in order to gather some values from the Parameter Store SSM, this is the code I use, which is very simple:
def get_raw_parameters_group_by_namespace(namespace_path):
raw_params_response = None
try:
if not namespace_path:
raise Exception('Namespace path should be specified to get data')
raw_params_response = ssm_ps.get_parameters_by_path(Path = namespace_path)
except Exception as e:
raise Exception('An error ocurred while trying to get parameters group: ' + str(e))
return raw_params_response
I used to have around 7 to 10 parameters in SSM and that method worked fine, however, we needed the add some additional parameters these days and the number of them increased to 14, so I tried adding a property in the boto3 ssm method called "MaxResults" and set it to 50:
ssm_ps.get_parameters_by_path(Path = namespace_path, MaxResults = 50)
but I get the following:
"error": "An error ocurred while trying to get parameters group: An error occurred (ValidationException) when calling the GetParametersByPath operation: 1 validation error detected: Value '50' at 'maxResults' failed to satisfy constraint: Member must have value less than or equal to 10."
After talking with the team, increasing the quota in the account is not an option, so I wonder to know if probably using the "NextToken" property would be a good option.
I am not sure on how this can be used, I have searched for examples, but I could not find something useful. Does anyone know how to use NextToken please? Or an example on how is it supposed to work?
I tried something like:
raw_params_response = ssm_ps.get_parameters_by_path(Path = namespace_path, NextToken = 'Token')
But I am not sure on the usage of this.
Thanks in advance.
I remember running into this at some point.
You want to use a paginator - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm.html#SSM.Paginator.GetParametersByPath
This is how I used it:
import boto3
client = boto3.client('ssm',region_name='eu-central-1')
paginator = client.get_paginator('get_parameters_by_path')
response_iterator = paginator.paginate(
Path='/some/path'
)
parameters=[]
for page in response_iterator:
for entry in page['Parameters']:
parameters.append(entry)
And you would get a list like [{"Name": "/some/path/param, "Value": "something"}] in parameters with all the parameters under the path.
*edit: response would be much richer than just the Name, Value keys. check the paginator docs!
Let me suggest using this library (I'm the author): AWStanding
You can achieve this easily, without worrying about pagination:
import os
from awstanding.parameter_store import load_path
load_path('/stripe', '/spotify')
STRIPE_PRICE = os.environ.get('STRIPE_PRICE', 'fallback_value')
STRIPE_WEBHOOK = os.environ.get('STRIPE_WEBHOOK', 'fallback_value')
SPOTIFY_API_KEY = os.environ.get('SPOTIFY_API_KEY', 'fallback_value')
print(f'price: {STRIPE_PRICE}, webhook: {STRIPE_WEBHOOK}, spotify: {SPOTIFY_API_KEY}')
>>> price: price_1xxxxxxxxxxxxxxxxxxxxxxx, webhook: fallback_value, spotify: fallback_value
How can the number of blobs returned from ContainerClient.list_blobs() method can be limited?
The Azure Blob service RESP API docs mentions a maxresults parameter, but it seems it is not honored by list_blobs(maxresults=123).
A combination of itertools.islice and the results_per_page parameter (which translates to the REST maxresults parameter) will do the trick:
import itertools
service: BlobServiceClient = BlobServiceClient.from_connection_string(cstr)
cc = service.get_container_client("foo")
n = 42
for b in itertools.islice(cc.list_blobs(results_per_page=n), n):
print(b.name)
Please use by_page() on the ItemPaged class
pages = ContainerClient.list_blobs(maxresults=123).by_page()
first_page = next(pages)
items_in_page = list(a_page) #this will give you 123 results on the first page
second_page = next(pages) # it will throw exception if there's no second page
items_in_page = list(a_page) #this will give you 123 results on the second page
There's no way to do this currently with the SDK. The maxresults parameter really means "max results per page"; if you have more blobs than this, list_blobs will make multiple calls to the REST API until the listing is complete.
You could call the API directly and ignore pages after the first, but that would require you to handle the details of authentication, parsing the response, etc.
I am trying to get a policy from boto3 client but there is no method to do so using policy name. By wrapping the create_policy method in a try-except block i can check whether a policy exists or not. Is there any way to get a policy-arn by name using boto3 except for listing all policies and iterating over it.
The ARN should be deterministic given the prefix (if any, and the name).
iam = session.client('iam')
sts = session.client('sts')
# Slow and costly if you have many pages
paginator = iam.get_paginator('list_policies')
all_policies = [policy for page in paginator.paginate() for policy in page['Policies']]
[policy_1] = [p for p in all_policies if p['PolicyName'] == policy_name]
# Fast and direct
account_id = sts.get_caller_identity()['Account']
policy_arn = f'arn:aws:iam::{account_id}:policy/{policy_name}'
policy_2 = iam.get_policy(PolicyArn=policy_arn)['Policy']
# They're equal except with the direct method you'll also get description field
all(policy_1[k] == policy_2[k] for k in policy_1.keys() & policy_2.keys())
You will need to iterate over the policies to get policy names. I am not aware of a get-policy type api that uses policy names only policy ARNs.
Is there a reason that you do not want to get a list of policies? Other than to not download the list.