Develop Reference

How to calculate EC2 instance price using Python script [duplicate]

Now that AWS have a Pricing API, how could one use Boto3 to fetch the current hourly price for a given on-demand EC2 instance type (e.g. t2.micro), region (e.g. eu-west-1) and operating system (e.g. Linux)? I only want the price returned. Based on my understanding, having those four pieces of information should be enough to filter down to a singular result.
However, all the examples I've seen fetch huge lists of data from the API that would have to be post-processed in order to get what I want. I would like to filter the data on the API side, before it's being returned.

Here is the solution I ended up with. Using Boto3's own Pricing API with a filter for the instance type, region and operating system. The API still returns a lot of information, so I needed to do a bit of post-processing.
import boto3
import json
from pkg_resources import resource_filename
# Search product filter. This will reduce the amount of data returned by the
# get_products function of the Pricing API
FLT = '[{{"Field": "tenancy", "Value": "shared", "Type": "TERM_MATCH"}},'\
'{{"Field": "operatingSystem", "Value": "{o}", "Type": "TERM_MATCH"}},'\
'{{"Field": "preInstalledSw", "Value": "NA", "Type": "TERM_MATCH"}},'\
'{{"Field": "instanceType", "Value": "{t}", "Type": "TERM_MATCH"}},'\
'{{"Field": "location", "Value": "{r}", "Type": "TERM_MATCH"}},'\
'{{"Field": "capacitystatus", "Value": "Used", "Type": "TERM_MATCH"}}]'
# Get current AWS price for an on-demand instance
def get_price(region, instance, os):
f = FLT.format(r=region, t=instance, o=os)
data = client.get_products(ServiceCode='AmazonEC2', Filters=json.loads(f))
od = json.loads(data['PriceList'][0])['terms']['OnDemand']
id1 = list(od)[0]
id2 = list(od[id1]['priceDimensions'])[0]
return od[id1]['priceDimensions'][id2]['pricePerUnit']['USD']
# Translate region code to region name. Even though the API data contains
# regionCode field, it will not return accurate data. However using the location
# field will, but then we need to translate the region code into a region name.
# You could skip this by using the region names in your code directly, but most
# other APIs are using the region code.
def get_region_name(region_code):
default_region = 'US East (N. Virginia)'
endpoint_file = resource_filename('botocore', 'data/endpoints.json')
try:
with open(endpoint_file, 'r') as f:
data = json.load(f)
# Botocore is using Europe while Pricing API using EU...sigh...
return data['partitions'][0]['regions'][region_code]['description'].replace('Europe', 'EU')
except IOError:
return default_region
# Use AWS Pricing API through Boto3
# API only has us-east-1 and ap-south-1 as valid endpoints.
# It doesn't have any impact on your selected region for your instance.
client = boto3.client('pricing', region_name='us-east-1')
# Get current price for a given instance, region and os
price = get_price(get_region_name('eu-west-1'), 't3.micro', 'Linux')
print(price)
This example outputs 0.0114000000 (hourly price in USD) fairly quickly. (This number was verified to match the current value listed here at the date of this writing)

If you don't like the native function, then look at Lyft's awspricing library for Python. Here's an example:
import awspricing
ec2_offer = awspricing.offer('AmazonEC2')
p = ec2_offer.ondemand_hourly(
't2.micro',
operating_system='Linux',
region='eu-west-1'
)
print(p) # 0.0126
I'd recommend enabling caching (see AWSPRICING_USE_CACHE) otherwise it will be slow.

I have updated toringe's solution a bit to handle different key errors
def price_information(self, instance_type, os, region):
# Search product filter
FLT = '[{{"Field": "operatingSystem", "Value": "{o}", "Type": "TERM_MATCH"}},' \
'{{"Field": "instanceType", "Value": "{t}", "Type": "TERM_MATCH"}}]'
f = FLT.format(t=instance_type, o=os)
try:
data = self.pricing_client.get_products(ServiceCode='AmazonEC2', Filters=json.loads(f))
instance_price = 0
for price in data['PriceList']:
try:
first_id = list(eval(price)['terms']['OnDemand'].keys())[0]
price_data = eval(price)['terms']['OnDemand'][first_id]
second_id = list(price_data['priceDimensions'].keys())[0]
instance_price = price_data['priceDimensions'][second_id]['pricePerUnit']['USD']
if float(price) > 0:
break
except Exception as e:
print(e)
print(instance_price)
return instance_price
except Exception as e:
print(e)
return 0

Based on other answers, here's some code that returns the On Demand prices for all instance types (or for a given instance type, if you add the search filter), gets some relevant attributes for each instance type, and pretty-prints the data.
It assumes pricing is the AWS Pricing client.
import json
def ec2_get_ondemand_prices(Filters):
data = []
reply = pricing.get_products(ServiceCode='AmazonEC2', Filters=Filters, MaxResults=100)
data.extend([json.loads(r) for r in reply['PriceList']])
while 'NextToken' in reply.keys():
reply = pricing.get_products(ServiceCode='AmazonEC2', Filters=Filters, MaxResults=100, NextToken=reply['NextToken'])
data.extend([json.loads(r) for r in reply['PriceList']])
print(f"\x1b[33mGET \x1b[0m{len(reply['PriceList']):3} \x1b[94m{len(data):4}\x1b[0m")
instances = {}
for d in data:
attr = d['product']['attributes']
type = attr['instanceType']
if type in data: continue
region = attr.get('location', '')
clock = attr.get('clockSpeed', '')
type = attr.get('instanceType', '')
market = attr.get('marketoption', '')
ram = attr.get('memory', '')
os = attr.get('operatingSystem', '')
arch = attr.get('processorArchitecture', '')
region = attr.get('regionCode', '')
storage = attr.get('storage', '')
tenancy = attr.get('tenancy', '')
usage = attr.get('usagetype', '')
vcpu = attr.get('vcpu', '')
terms = d['terms']
ondemand = terms['OnDemand']
ins = ondemand[next(iter(ondemand))]
pricedim = ins['priceDimensions']
price = pricedim[next(iter(pricedim))]
desc = price['description']
p = float(price['pricePerUnit']['USD'])
unit = price['unit'].lower()
if 'GiB' not in ram: print('\x1b[31mWARN\x1b[0m')
if 'hrs'!=unit: print('\x1b[31mWARN\x1b[0m')
if p==0.: continue
instances[type] = {'type':type, 'market':market, 'vcpu':vcpu, 'ram':float(ram.replace('GiB','')), 'ondm':p, 'unit':unit, 'terms':list(terms.keys()), 'desc':desc}
instances = {k:v for k,v in sorted(instances.items(), key=lambda e: e[1]['ondm'])}
for ins in instances.values():
p = ins['ondm']
print(f"{ins['type']:32} {ins['market'].lower()}\x1b[91m: \x1b[0m{ins['vcpu']:3} vcores\x1b[91m, \x1b[0m{ins['ram']:7.1f} GB, \x1b[0m{p:7.4f} \x1b[95m$/h\x1b[0m, \x1b[0m\x1b[0m{p*720:8,.1f} \x1b[95m$/m\x1b[0m, \x1b[0m\x1b[0m{p*720*12:7,.0f} \x1b[95m$/y\x1b[0m, \x1b[0m{ins['unit']}\x1b[91m, \x1b[0m{ins['terms']}\x1b[0m")
# print(desc, , sep='\n')
print(f'\x1b[92m{len(instances)}\x1b[0m')
flt = [
# {'Field': 'instanceType', 'Value': 't4g.nano', 'Type': 'TERM_MATCH'}, # enable this filter to select only 1 instance type
{'Field': 'regionCode', 'Value': 'us-east-2', 'Type': 'TERM_MATCH'}, # alternative notation?: {'Field': 'location', 'Value': 'US East (Ohio)', 'Type': 'TERM_MATCH'},
{'Field': 'operatingSystem', 'Value': 'Linux', 'Type': 'TERM_MATCH'},
{'Field': 'tenancy', 'Value': 'shared', 'Type': 'TERM_MATCH'},
{'Field': 'capacitystatus', 'Value': 'Used', 'Type': 'TERM_MATCH'},
]
ec2_get_ondemand_prices(Filters=flt)

How do I only return a specific value in a set of values in an API JSON response?

I'm trying to return only a specific value from the "data" key in this response that I'm currently working with:
{
"dataset": {
"id": 49333506,
"dataset_code": "YMAB",
"database_code": "QOR",
"name": "Y-mAbs Therapeutics Inc. (YMAB) Option Earnings Crush, Liquidity, and Volatility Ratings",
"description": "Option Earnings Crush, Liquidity, and Volatility Ratings for Y-mAbs Therapeutics Inc. (YMAB). All time periods are measured in calendar days. See documentation for methodology.",
"refreshed_at": "2022-08-05 21:20:34 UTC",
"newest_available_date": "2022-08-05",
"oldest_available_date": "2020-02-12",
"column_names": [
"Date",
"EarningsCrushRate",
"CalendarDaysUntilEarnings",
"TradingDaysUntilEarnings",
"LiquidityRating",
"HasLeapOptions",
"HasWeeklyOptions",
"Iv30Rank",
"Iv30Percentile",
"Iv30Rating",
"Iv60Rank",
"Iv60Percentile",
"Iv60Rating",
"Iv90Rank",
"Iv90Percentile",
"Iv90Rating",
"Iv360Rank",
"Iv360Percentile",
"Iv360Rating"
],
"frequency": "daily",
"type": "Time Series",
"premium": true,
"limit": null,
"transform": null,
"column_index": null,
"start_date": "2020-02-12",
"end_date": "2022-08-05",
"data": [
[
"2022-08-05",
null,
null,
null,
2.0,
0.0,
0.0,
0.1437,
0.4286,
0.3706,
0.1686,
0.4762,
0.3936,
0.1379,
0.4502,
0.4129,
0.107,
0.5152,
0.4657
],
I only want to return the date, and a single value at a time from the "data": [ key that's within "dataset": {.
Here's the code I have so far, but am stuck as to make this happen:
r = requests.get(url=f"https://data.nasdaq.com/api/v3/datasets/QOR/{symbol}/data.json?api_key={apikey}")
d = r.json()
dataset = d['dataset_data']
data = dataset['data']
column_names = dataset['column_names']
date = column_names[0]
ercrush = column_names[1]
calendar = column_names[2]
tradingdays = column_names[3]
liquidity = column_names[4]
leaps = column_names[5]
weeklies = column_names[6]
ivrank30 = column_names[7]
ivper30 = column_names[8]
ivrate30 = column_names[9]
ivrank60 = column_names[10]
ivper60 = column_names[11]
ivrate60 =column_names[12]
ivrank90 = column_names[13]
ivper90 = column_names[14]
ivrank90 = column_names[15]
ivrank360= column_names[16]
ivper360 = column_names[17]
ivrank360 = column_names[18]
values = data[0]
For example - I'm only trying to return the Date, defined as column_names[0] paired with the value of "2022-08-05" that's within "data": [ , etc.
How would I go about doing this?
Thanks so much for any help.

I figured out the issue!
I created another variable called results = values and now I can pick the values I want and easily match them with the column_names!
Awesome!
The finished code that works:
r = requests.get(url=f"https://data.nasdaq.com/api/v3/datasets/QOR/{symbol}/data.json?api_key=KyVWdRX_o26L5XNUkgqN")
d = r.json()
dataset = d['dataset_data']
data = dataset['data']
column_names = dataset['column_names']
Date = column_names[0]
ercrush = column_names[1]
calendar = column_names[2]
tradingdays = column_names[3]
liquidity = column_names[4]
leaps = column_names[5]
weeklies = column_names[6]
ivrank30 = column_names[7]
ivper30 = column_names[8]
ivrate30 = column_names[9]
ivrank60 = column_names[10]
ivper60 = column_names[11]
ivrate60 =column_names[12]
ivrank90 = column_names[13]
ivper90 = column_names[14]
ivrank90 = column_names[15]
ivrank360= column_names[16]
ivper360 = column_names[17]
ivrank360 = column_names[18]
values = data[0]
results = values[2] #the correction
print(results)

Python: An append() is causing an infinite loop

I have some campaigns that I want to modify but I've to call the server and export all the campaign settings modify or add the information I want and send it back to the server.
My modifications are stored in an SQLite db, so first, I query the db looking for the account IDs (I can have many campaigns that are in different accounts), then I look for campaigns per account.
theCursor.execute('SELECT * FROM SandboxPubTB')
rows = theCursor.fetchall()
for row in rows:
accounts.append(row[7])
for account in set(accounts):
theCursor.execute('SELECT CAMPAIGNNAME, CAMPAIGNID FROM SandboxPubTB WHERE ACCOUNTID =?', (account,) )
campaignRows = theCursor.fetchall()
for campaign in campaignRows:
campId= campaign[0] + "," + campaign[1] + "," + account
campaigns.append(campId)
My campId list contains the campaign name, campaign ID and the campaigns account ID in each entry.
After that, I iterate through the list and assign the values to a variable at the campaign level:
for campaignNameId in set(campaigns):
campaignNameId = campaignNameId.split(',')
campaignName = campaignNameId[0]
campaignId = campaignNameId[1]
campaignAccount = campaignNameId[2]
Now it's time to call the server to export the campaign settings:
targetUrl = "https://sites.com/api/1.0/" + campaignAccount + "/campaigns/"+ campaignId
resp = requests.get(url=targetUrl, headers=header)
r = resp.json()
temp = r['publisher_bid_modifier']['values']
If I print temp this is what I get:
[{
'target': 'msn-can',
'cpc_modification': 0.5
}, {
'target': 'msn-can-home',
'cpc_modification': 0.5
}, {
'target': 'smartify-journalistatecom',
'cpc_modification': 0.5
}, {
'target': 'foxnews-iosapp',
'cpc_modification': 1.22
}]
And finally, here is where I'm stuck. What I'm trying to do is iterate through the dictionary list above and if "target" value exists and "cpc_modification" is different than what I've in the db I'll change the cpc value. If "target" doesn't exist I want to append "target" and "cpc_modification" to the dictionary list.
I succeeded with the first part, but the append part is different. In the elif, even if I use an else, the temp.append triggers an infinite loop and I have no idea why.
for dt in r['publisher_bid_modifier']['values']:
#print(dt['target'])
#if dt['target']:
theCursor.execute('SELECT ADJUST, SITE FROM SandboxPubTB WHERE CAMPAIGNNAME =?', (campaignName,) )
campaignRows = theCursor.fetchall()
for t in campaignRows:
if t[1] == dt['target'] :
dt['cpc_modification'] = "{:.2f}".format((int(t[0]) / 100) + 1)
elif dt['target'] not in temp:
temp.append("{'target': "'" + t[1] + "'", 'cpc_modification': "'" + str(t[0]) + "'"}")
That's weird because I've tried to emulate the same behavior with a local variable and it seems to work without problem.
data = [{
"target": "publisher1",
"cpc_modification": 1.5
},
{
"target": "publisher2",
"cpc_modification": 0.9
}
]
for t in data:
if t['target'] == "publisher10":
t['cpc_modification'] = 1.9
else:
data.append({'target': 'publisher10', 'cpc_modification': 12})
print(t['target'], t['cpc_modification'])
print(data)
I've tried many things, and I can't figure out what is wrong.

I believe your problem is in here. You are iterating over 'data' but you are also appending stuff to 'data' as you are iterating over it. You should create a new variable to append to.
temp = data
for t in data:
if t['target'] == "publisher10":
t['cpc_modification'] = 1.9
else:
temp.append({'target': 'publisher10', 'cpc_modification': 12})
print(t['target'], t['cpc_modification'])
print(temp)
I made an example here of what I believe is happending

Calling action function wit api in python

I want to get the stock price of any item. I have a function getPrice which returns the price of the item. I am trying to use wit ai.
This is what I have tried.
from wit import Wit
def getPrice(request):
#statements
return price
def send(request, response):
print request['text']
print('Sending to user...', response['text'])
actions = {
'send': send,
'getPrice':getPrice
}
client = Wit(access_token="<token>", actions=actions)
resp = client.message('what is the stock price of xyz')
print('Result: ' + str(resp))
and the result I am getting is:
Result: {
u 'entities': {
u 'search_query': [{
u 'suggested': True,
u 'confidence': 0.578445451443443,
u 'type': u 'value',
u 'value': u 'stock of xyz'
}],
u 'item': [{
u 'confidence': 0.9613219630126943,
u 'value': u 'xyz'
}]
},
u 'msg_id': u '5ac1a450-7c5d-3190-8666-5d88a1884f94',
u '_text': u 'what is the stock of xyz?'
}
I want the Bot to display the price. How can I call this action function?

You can do
print resp['entities']['item'][0]['value']

Instead of:
print('Result: ' + str(resp))
try:
stock = resp['entities']['item'][0]['value']
print('Price: {}'.format(getPrice(stock)))

Retrieve the country from the geographical locations in Python

I am trying to get the country name from the latitude and longitude points from my pandas dataframe.
Currently I have used geolocator.reverse(latitude,longitude) to get the full address of the geographic location. But there is no option to retrieve the country name from the full address as it returns a list.
Method used:
def get_country(row):
pos = str(row['StartLat']) + ', ' + str(row['StartLong'])
locations = geolocator.reverse(pos)
return locations
Call to get_country by passing the dataframe:
df4['country'] = df4.apply(lambda row: get_country(row), axis = 1)
Current output:
StartLat StartLong Address
52.509669 13.376294 Potsdamer Platz, Mitte, Berlin, Deutschland, Europe
Just wondering whether there is some Python library to retrieve the country when we pass the geographic points.
Any help would be appreciated.

In your get_country function, your return value location will have an attribute raw, which is a dict that looks like this:
{
'address': {
'attraction': 'Potsdamer Platz',
'city': 'Berlin',
'city_district': 'Mitte',
'country': 'Deutschland',
'country_code': 'de',
'postcode': '10117',
'road': 'Potsdamer Platz',
'state': 'Berlin'
},
'boundingbox': ['52.5093982', '52.5095982', '13.3764983', '13.3766983'],
'display_name': 'Potsdamer Platz, Mitte, Berlin, 10117, Deutschland',
... and so one ...
}
so location.raw['address']['country'] gives 'Deutschland'
If I read your question correctly, a possible solution could be:
def get_country(row):
pos = str(row['StartLat']) + ', ' + str(row['StartLong'])
locations = geolocator.reverse(pos)
return location.raw['address']['country']
EDIT: The format of the location.raw object will differ depending on which geolocator service you are using. My example uses geopy.geocoders.Nominatim, from the example on geopy's documentation site, so your results might differ.

My code,hopefully that helps:
from geopy.geocoders import Nominatim
nm = Nominatim()
place, (lat, lng) = nm.geocode("3995 23rd st, San Francisco,CA 94114")
print('Country' + ": " + place.split()[-1])

I'm not sure what service you're using with geopy, but as a small plug which I'm probably biased towards, this I think could be a simpler solution for you.
https://github.com/Ziptastic/ziptastic-python
from ziptastic import Ziptastic
# Set API key.
api = Ziptastic('<your api key>')
result = api.get_from_coordinates('42.9934', '-84.1595')
Which will return a list of dictionaries like so:
[
{
"city": "Owosso",
"geohash": "dpshsfsytw8k",
"country": "US",
"county": "Shiawassee",
"state": "Michigan",
"state_short": "MI",
"postal_code": "48867",
"latitude": 42.9934,
"longitude": -84.1595,
"timezone": "America/Detroit"
}
]