Develop Reference

Python Boto3 Get Parameters from SSM by Path using NextToken

I've been working with boto3 for a while in order to gather some values from the Parameter Store SSM, this is the code I use, which is very simple:
def get_raw_parameters_group_by_namespace(namespace_path):
raw_params_response = None
try:
if not namespace_path:
raise Exception('Namespace path should be specified to get data')
raw_params_response = ssm_ps.get_parameters_by_path(Path = namespace_path)
except Exception as e:
raise Exception('An error ocurred while trying to get parameters group: ' + str(e))
return raw_params_response
I used to have around 7 to 10 parameters in SSM and that method worked fine, however, we needed the add some additional parameters these days and the number of them increased to 14, so I tried adding a property in the boto3 ssm method called "MaxResults" and set it to 50:
ssm_ps.get_parameters_by_path(Path = namespace_path, MaxResults = 50)
but I get the following:
"error": "An error ocurred while trying to get parameters group: An error occurred (ValidationException) when calling the GetParametersByPath operation: 1 validation error detected: Value '50' at 'maxResults' failed to satisfy constraint: Member must have value less than or equal to 10."
After talking with the team, increasing the quota in the account is not an option, so I wonder to know if probably using the "NextToken" property would be a good option.
I am not sure on how this can be used, I have searched for examples, but I could not find something useful. Does anyone know how to use NextToken please? Or an example on how is it supposed to work?
I tried something like:
raw_params_response = ssm_ps.get_parameters_by_path(Path = namespace_path, NextToken = 'Token')
But I am not sure on the usage of this.
Thanks in advance.

I remember running into this at some point.
You want to use a paginator - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ssm.html#SSM.Paginator.GetParametersByPath
This is how I used it:
import boto3
client = boto3.client('ssm',region_name='eu-central-1')
paginator = client.get_paginator('get_parameters_by_path')
response_iterator = paginator.paginate(
Path='/some/path'
)
parameters=[]
for page in response_iterator:
for entry in page['Parameters']:
parameters.append(entry)
And you would get a list like [{"Name": "/some/path/param, "Value": "something"}] in parameters with all the parameters under the path.
*edit: response would be much richer than just the Name, Value keys. check the paginator docs!

Let me suggest using this library (I'm the author): AWStanding
You can achieve this easily, without worrying about pagination:
import os
from awstanding.parameter_store import load_path
load_path('/stripe', '/spotify')
STRIPE_PRICE = os.environ.get('STRIPE_PRICE', 'fallback_value')
STRIPE_WEBHOOK = os.environ.get('STRIPE_WEBHOOK', 'fallback_value')
SPOTIFY_API_KEY = os.environ.get('SPOTIFY_API_KEY', 'fallback_value')
print(f'price: {STRIPE_PRICE}, webhook: {STRIPE_WEBHOOK}, spotify: {SPOTIFY_API_KEY}')
>>> price: price_1xxxxxxxxxxxxxxxxxxxxxxx, webhook: fallback_value, spotify: fallback_value

How can I filter ports by country?

This is my code:
I import the modules
import shodan
import json
I create my key,
SHODAN_API_KEY = ('xxxxxxxxxxxxxxxxxxxx')
api = shodan.Shodan(SHODAN_API_KEY)
I open my json file,
with open('Ports.json', 'r') as f:
Ports_dict = json.load(f)
#I loop through my dict,
for Port in Ports_dict:
print(Port['Port_name'])
try:
results = api.search(Port['Port_name']) # how can I filter ports by country??
#and I print the content.
print('Results found: {}'.format(results['total']))
for result in results['matches']:
print('IP: {}'.format(result['ip_str']))
print(result['data'])
print('')
print ('Country_code: %s' % result['location']['country_code'])
except shodan.APIError as e:
print(' Error: %s' % e)
But how can I filter ports by country?

In order to filter the results you need to use a search filter. The following article explains the general search query syntax of Shodan:
https://help.shodan.io/the-basics/search-query-fundamentals
Here is a list of all available search filters:
https://beta.shodan.io/search/filters
And here is a page full of example search queries:
https://beta.shodan.io/search/examples
In your case, you would want to use the port and country filter. For example, the following search query returns the MySQL and PostgreSQL servers in the US:
https://beta.shodan.io/search?query=port%3A3306%2C5432+country%3AUS
I would also recommend using the Shodan CLI for downloading data as it will handle paging through results for you:
https://help.shodan.io/guides/how-to-download-data-with-api
If you need to do it yourself within Python then you would also need to loop through the search results either by providing a page parameter or by simply using the Shodan.search_cursor() method (instead of Shodan.search() as you did in your code). The above article also shows how to use the search_cursor() method.

KeyError with Youtube API using python

This is my first time to ask something here. I've been trying to access the Youtube API to get something for an experiment I'm doing. Everything's working so far. I just wanted to ask about this very inconsistent error that I'm getting.
-----------
1
Title: All Movie Trailers of New York Comic-Con (2016) Power Rangers, John Wick 2...
Uploaded by: KinoCheck International
Uploaded on: 2016-10-12T14:43:42.000Z
Video ID: pWOH-OZQUj0
2
Title: Movieclips Trailers
Uploaded by: Movieclips Trailers
Uploaded on: 2011-04-01T18:43:14.000Z
Video ID: Traceback (most recent call last):
File "scrapeyoutube.py", line 24, in <module>
print "Video ID:\t", search_result['id']['videoId']
KeyError: 'videoId'
I tried getting the video ID ('videoID' as per documentation). But for some reason, the code works for the 1st query, and then totally flops for the 2nd one. It's weird because it's only happening for this particular element. Everything else ('description','publishedAt', etc.) is working. Here's my code:
from apiclient.discovery import build
import json
import pprint
import sys
APINAME = 'youtube'
APIVERSION = 'v3'
APIKEY = 'secret teehee'
service = build(APINAME, APIVERSION, developerKey = APIKEY)
#volumes source ('public'), search query ('androide')
searchrequest = service.search().list(q ='movie trailers', part ='id, snippet', maxResults = 25).execute()
searchcount = 0
print "-----------"
for search_result in searchrequest.get("items", []):
searchcount +=1
print searchcount
print "Title:\t", search_result['snippet']['title']
# print "Description:\t", search_result['snippet']['description']
print "Uploaded by:\t", search_result['snippet']['channelTitle']
print "Uploaded on:\t", search_result['snippet']['publishedAt']
print "Video ID:\t", search_result['id']['videoId']
Hope you guys can help me. Thanks!

Use 'get' method for result.
result['id'].get('videoId')
there are in some element no this key.
if you use square parenteces, python throw exeption keyError, but if you use 'get' method, python return None for element whitch have not key videoId

Using search() method returns channels, playlists as well together with videos in search. That might be why your problem.
I use their interactive playgrounds to learn the structure of returned JSON, functions, etc. For your question, I suggest to visit https://developers.google.com/youtube/v3/docs/search/list .
Make sure if a kind of an item is "youtube#video", then access videoId of that item.
Sample of code:
...
for index in response["items"]: # response is a JSON file I have got from API
tmp = {} # temporary dict to assert into my custom JSON
if index["id"]["kind"] == "youtube#video":
tmp["videoID"] = index["id"]["videoId"]
...
This is a part of code from my personal project I am currently working on.

Because some results to Key "ID", return:
{u'kind': u'youtube#playlist', u'playlistId': u'PLd0_QArxznVHnlvJp0ki5bpmBj4f64J7P'}
You can see, there is no key "videoId".

I keep getting [u'error']

Why am I getting this error when trying to print keys from dictionary?
import urllib
import urllib2
import json
ret = urllib2.urlopen(urllib2.Request('https://poloniex.com/public?command=returnTicker'))
a = json.loads(ret.read())
print a.keys()
ret = urllib2.urlopen(urllib2.Request('https://poloniex.com/public?command=return24Volume'))
b = json.loads(ret.read())
print b.keys()

The error is produced by the website - your code as such is ok, it produces a json-object which has apparently the structure '{ "error" : ""}'. Try printing it out & figure out what's wrong, you probably need some authentication tokens or similar stuff.
There seems to be a API wrapper available, you should consider using or at least understanding it: http://pastebin.com/8fBVpjaj
It's directly featured on polniex website, and it clearly shows the need of API secret and key.

The error is originating from the site. The dict is loaded, and expresses the error as it's only key.
Try this:
ret = urllib2.urlopen(urllib2.Request('https://poloniex.com/public?command=return24hVolume'))
b = json.loads(ret.read())
print b.keys()
Notice the 'h' in return24hVolume.

The second URL returns:
{"error":"Invalid command."}
So as Reut Sharabani points out, you have to use 'h' in the URL:
https://poloniex.com/public?command=return24hVolume

Error Handling When Searching The Shodan API

I'm trying to write a Python script that will search the Shodan API and return ID, CVE and Description. As some of my search results ('java' for example) do not have an established CVE (or CVE key), my script chokes. I'm aware that I need to wrap the search in try/except error handling, but I've not had any luck with what I've been able to find researching the web. Here is the error I get and below that is the code. Thanks very much in advance.
--------Error-------
print '%s: %s: %s' % (exploit['id'], exploit['cve'], exploit['description'])
KeyError: 'cve
--------My Code------
from shodan import WebAPI
api = WebAPI("my shodan key")
user_selection = raw_input("Enter something: ")
print "searching for", (user_selection),"....."
results = api.exploitdb.search(user_selection)
for exploit in results['matches']:
print '%s: %s: %s' % (exploit['id'], exploit['cve'], exploit['description'])

It looks like you want to use dict.get and provide it a default value to be returned where the key doesn't exist:
print '%s: %s: %s' % (exploit.get('id', '[blank]'), exploit.get('cve', '[blank]'), exploit.get('description', '[blank]'))