Develop Reference

Convert shell script to Python

aws ec2 describe-snapshots --owner-ids $AWS_ACCOUNT_ID --query
"Snapshots[?(StartTime<='$dtt')].[SnapshotId]" --output text | tr '\t'
'\n' | sort
I have this shell script which I want to convert to python.
I tried looking at the boto3 documentation and came up with this
client = boto3.client('ec2')
client.describe_snapshots(OwnerIds = [os.environ['AWS_ACCOUNT_ID']], )
But I can't figure out how to change that --query tag in python.
I couldn't find it in the documentation.
What am I missing here?

You should ignore the --query portion and everything after it, and process that within Python instead.
First, store the result of the call in a variable:
ec2_client = boto3.client('ec2')
response = ec2_client.describe_snapshots(OwnerIds = ['self'])
It will return something like:
{
'NextToken': '',
'Snapshots': [
{
'Description': 'This is my snapshot.',
'OwnerId': '012345678910',
'Progress': '100%',
'SnapshotId': 'snap-1234567890abcdef0',
'StartTime': datetime(2014, 2, 28, 21, 28, 32, 4, 59, 0),
'State': 'completed',
'VolumeId': 'vol-049df61146c4d7901',
'VolumeSize': 8,
},
],
'ResponseMetadata': {
'...': '...',
},
}
Therefore, you can use response['Snapshots'] to extract your desired results, for example:
for snapshot in response['Snapshots']:
if snapshot['StartTime'] < datetime(2022, 6, 1):
print(snapshot['SnapshotId'])
It's really all Python at that point.

Django: How to search through django.template.context.RequestContext

I'm working over tests in Django and faced <class 'django.template.context.RequestContext'>, which I'm trying to iterate through and find <class 'ecom.models.Product'> object inside.
test.py
def test_ProductDetail_object_in_context(self):
response = self.client.get(reverse('product_detail', args=[1]))
# assertEqual - test passes
self.assertEqual(response.context[0]['object'], Product.objects.get(id=1))
# assertIn - test fails
self.assertIn(Product.objects.get(id=1), response.context[0])
views.py
class ProductDetailView(DetailView):
model = Product
def get_context_data(self, **kwargs):
context = super().get_context_data(**kwargs)
data = cartData(self.request)
cartItems = data['cartItems']
context['cartItems'] = cartItems
return context
What's inside response.context:
[
[
{'True': True, 'False': False, 'None': None},
{'csrf_token': <SimpleLazyObject: <function csrf.<locals>._get_val at 0x7fd80>>,
'request': <WSGIRequest: GET '/1/'>,
'user': <SimpleLazyObject: <django.contrib.auth.models.AnonymousUser object at 0x7fd820>>, '
perms': <django.contrib.auth.context_processors.PermWrapper object at 0x7fd80>,
'messages': <django.contrib.messages.storage.fallback.FallbackStorage object at 0x7fd8290>,
'DEFAULT_MESSAGE_LEVELS': {'DEBUG': 10, 'INFO': 20, 'SUCCESS': 25, 'WARNING': 30, 'ERROR': 40}
},
{},
{'object': <Product: Pen>,
'product': <Product: Pen>,
'view': <ecom.views.ProductDetailView object at 0x7fd8210>,
'cartItems': 0}
],
[
{'True': True, 'False': False, 'None': None},
{'csrf_token': <SimpleLazyObject: <function csrf.<locals>._get_val at 0x7fd8240>>,
'request': <WSGIRequest: GET '/1/'>,
'user': <SimpleLazyObject: <django.contrib.auth.models.AnonymousUser object at 0x7fd8250>>,
'perms': <django.contrib.auth.context_processors.PermWrapper object at 0x7fd8250>,
'messages': <django.contrib.messages.storage.fallback.FallbackStorage object at 0x7fd8290>,
'DEFAULT_MESSAGE_LEVELS': {'DEBUG': 10, 'INFO': 20, 'SUCCESS': 25, 'WARNING': 30, 'ERROR': 40}
},
{},
{'object': <Product: Pen>,
'product': <Product: Pen>,
'view': <ecom.views.ProductDetailView object at 0x7fd8210>,
'cartItems': 0}
]
]
Type of response.context:
<class 'django.template.context.RequestContext'>
What's inside Product.objects.get(id=1) is: Pen
Type of Product.objects.get(id=1) is: <class 'ecom.models.Product'>
I don't undestand why:
it found Product object in self.assertEqual(response.context[0]['object'], Product.objects.get(id=1)), but not in self.assertIn(Product.objects.get(id=1), response.context[0]['object']) - says TypeError: argument of type 'Product' is not iterable
it also didn't find it in self.assertIn(Product.objects.get(id=1), response.context[0]) - says "AssertionError: <Product: Pen> not found in [....here goes contents of response.context[0]....]"
it also didn't find it in self.assertIn(Product.objects.get(id=1), response.context[0][3]) - says "in getitem raise KeyError(key), KeyError: 3"
how to work with RequestContext class? JSON like?
Sorry for a bit mixed up question, just trying to understand how to work with RequestContext.
Thank you in advance!

I think your test is failing because assertIn looks through the KEYS not the values. Solution would be:
self.assertIn(Product.objects.get(id=1), response.context[0].values())
A little more explanation: response.context[0] seems like it's some key-value storage, i.e. a dict. When you do response.context[0]["object"], you've just accessed the value at the key "object" where response.context[0] is the dict. Doing some in query on the dictionary only looks up the keys of the dictionary.

Using python to get a track from the spotify API by using search Endpoint

So I'm trying to get a track from the spotify API by searching for it by using the search endpoint of the API (See documentation). First, I authorize myself so I can send GET requests. This happens without issues, I added the code for reproducibility.
import requests
CLIENT_ID = "your_id_here"
CLIENT_SECRET = "your_secret_here"
AUTH_URL = "https://accounts.spotify.com/api/token"
auth_response = requests.post(AUTH_URL, {
'grant_type': 'client_credentials',
'client_id': CLIENT_ID,
'client_secret': CLIENT_SECRET,
})
#Convert response to JSON
auth_response_data = auth_response.json()
#Save the access token
access_token = auth_response_data['access_token']
#Need to pass access token into header to send properly formed GET request to API server
headers = {
'Authorization': 'Bearer {token}'.format(token=access_token)
}
Then, I want to use the search endpoint of the API to find a track by using the track name + artist (I need the track ID later on). When I use the example provided in the documentation, everything works fine and an artist object is returned by using the following query:
BASE_URL = 'https://api.spotify.com/v1/'
r = requests.get(BASE_URL + 'search?q=tania%20bowra&type=artist', headers=headers)
r = r.json()
This is the response, which looks exactly like the one in documentation:
{'artists': {'href': 'https://api.spotify.com/v1/search?query=tania+bowra&type=artist&offset=0&limit=20',
'items': [{'external_urls': {'spotify': 'https://open.spotify.com/artist/08td7MxkoHQkXnWAYD8d6Q'},
'followers': {'href': None, 'total': 235},
'genres': [],
'href': 'https://api.spotify.com/v1/artists/08td7MxkoHQkXnWAYD8d6Q',
'id': '08td7MxkoHQkXnWAYD8d6Q',
'images': [{'height': 640,
'url': 'https://i.scdn.co/image/ab67616d0000b2731ae2bdc1378da1b440e1f610',
'width': 640},
{'height': 300,
'url': 'https://i.scdn.co/image/ab67616d00001e021ae2bdc1378da1b440e1f610',
'width': 300},
{'height': 64,
'url': 'https://i.scdn.co/image/ab67616d000048511ae2bdc1378da1b440e1f610',
'width': 64}],
'name': 'Tania Bowra',
'popularity': 1,
'type': 'artist',
'uri': 'spotify:artist:08td7MxkoHQkXnWAYD8d6Q'}],
'limit': 20,
'next': None,
'offset': 0,
'previous': None,
'total': 1}}
Applying the same logic, I tried to get a track object from the api by using an artist and a track name, like so:
BASE_URL = 'https://api.spotify.com/v1/'
r = requests.get(BASE_URL + 'search?q=artist:queen%20track:bohemian%20rapsody&type=track', headers=headers)
r = r.json()
Even though I do get a valid response (statuscode==200), it seems to be empty:
{'tracks': {'href': 'https://api.spotify.com/v1/search?query=artist%3Aqueen+track%3Abohemian+rapsody&type=track&offset=0&limit=20',
'items': [],
'limit': 20,
'next': None,
'offset': 0,
'previous': None,
'total': 0}}
My question is: why is this happening?

You are now searching for the query: artist:queen%20track:bohemian%20rapsody while this should just be queen%20bohemian%20rapsody instead. the type afterwards shows what items you want to return. You dont have to determine the artist and track name seperately in the query. Interpret the query just like typing something into the spotify search bar.

Problem solved. It was rhapsody instead of rapsody... Sucks be a non-native english speaker sometimes =)

How to properly patch boto3 calls in unit test

I'm new to Python unit testing, and I want to mock calls to the boto3 3rd party library. Here's my stripped down code:
real_code.py:
import boto3
def temp_get_variable(var_name):
return boto3.client('ssm').get_parameter(Name=var_name)['Parameter']['Value']
test_real_code.py:
import unittest
from datetime import datetime
from unittest.mock import patch
import real_code
class TestRealCode(unittest.TestCase):
#patch('patching_config.boto3.client')
def test_get_variable(self, mock_boto_client):
response = {
'Parameter': {
'Name': 'MyTestParameterName',
'Type': 'String',
'Value': 'myValue',
'Version': 123,
'Selector': 'asdf',
'SourceResult': 'asdf',
'LastModifiedDate': datetime(2019, 7, 16),
'ARN': 'asdf'
}
}
mock_boto_client.get_variable.return_value = response
result_value = real_code.get_variable("MyTestParameterName")
self.assertEqual("myValue", result_value)
When I run it the test fails with
Expected :myValue
Actual :<MagicMock name='client().get_parameter().__getitem__().__getitem__()' id='2040071816528'>
What am I doing wrong? I thought by setting mock_boto_client.get_variable.return_value = response it would mock out the call and return my canned response instead. I don't understand why I am getting a MagicMock object instead of the return value I tried to set. I'd like to set up my test so that when the call to get_parameter is made with specific parameters, the mock returns the canned response I specified in the test.

There are two issues with your test code. The first is that when your mock object mock_boto_client called, it returns a new mock object. This means that the object that get_parameter() is being called on is different than the one you are attempting to set a return value on. You can have it return itself with the following:
mock_boto_client.return_value = mock_boto_client
You can also use a different mock object:
foo = MagicMock()
mock_boto_client.return_value = foo
The second issue that you have is that you are mocking the wrong method call. mock_boto_client.get_variable.return_value should be mock_boto_client.get_parameter.return_value. Here is the test updated and working:
import unittest
from datetime import datetime
from unittest.mock import patch
import real_code
class TestRealCode(unittest.TestCase):
#patch('boto3.client')
def test_get_variable(self, mock_boto_client):
response = {
'Parameter': {
'Name': 'MyTestParameterName',
'Type': 'String',
'Value': 'myValue',
'Version': 123,
'Selector': 'asdf',
'SourceResult': 'asdf',
'LastModifiedDate': datetime(2019, 7, 16),
'ARN': 'asdf'
}
}
mock_boto_client.return_value = mock_boto_client
mock_boto_client.get_parameter.return_value = response
result_value = real_code.get_variable("MyTestParameterName")
self.assertEqual("myValue", result_value)

String indices must be integers - Django

I have a pretty big dictionary which looks like this:
{
'startIndex': 1,
'username': 'myemail#gmail.com',
'items': [{
'id': '67022006',
'name': 'Adopt-a-Hydrant',
'kind': 'analytics#accountSummary',
'webProperties': [{
'id': 'UA-67522226-1',
'name': 'Adopt-a-Hydrant',
'websiteUrl': 'https://www.udemy.com/,
'internalWebPropertyId': '104343473',
'profiles': [{
'id': '108333146',
'name': 'Adopt a Hydrant (Udemy)',
'type': 'WEB',
'kind': 'analytics#profileSummary'
}, {
'id': '132099908',
'name': 'Unfiltered view',
'type': 'WEB',
'kind': 'analytics#profileSummary'
}],
'level': 'STANDARD',
'kind': 'analytics#webPropertySummary'
}]
}, {
'id': '44222959',
'name': 'A223n',
'kind': 'analytics#accountSummary',
And so on....
When I copy this dictionary on my Jupyter notebook and I run the exact same function I run on my django code it runs as expected, everything is literarily the same, in my django code I'm even printing the dictionary out then I copy it to the notebook and run it and I get what I'm expecting.
Just for more info this is the function:
google_profile = gp.google_profile # Get google_profile from DB
print(google_profile)
all_properties = []
for properties in google_profile['items']:
all_properties.append(properties)
site_selection=[]
for single_property in all_properties:
single_propery_name=single_property['name']
for single_view in single_property['webProperties'][0]['profiles']:
single_view_id = single_view['id']
single_view_name = (single_view['name'])
selections = single_propery_name + ' (View: '+single_view_name+' ID: '+single_view_id+')'
site_selection.append(selections)
print (site_selection)
So my guess is that my notebook has some sort of json parser installed or something like that? Is that possible? Why in django I can't access dictionaries the same way I can on my ipython notebooks?
EDITS
More info:
The error is at the line: for properties in google_profile['items']:
Django debug is: TypeError at /gconnect/ string indices must be integers
Local Vars are:
all_properties =[]
current_user = 'myemail#gmail.com'
google_profile = `the above dictionary`

So just to make it clear for who finds this question:
If you save a dictionary in a database django will save it as a string, so you won't be able to access it after.
To solve this you can re-convert it to a dictionary:
The answer from this post worked perfectly for me, in other words:
import json
s = "{'muffin' : 'lolz', 'foo' : 'kitty'}"
json_acceptable_string = s.replace("'", "\"")
d = json.loads(json_acceptable_string)
# d = {u'muffin': u'lolz', u'foo': u'kitty'}
There are many ways to convert a string to a dictionary, this is only one. If you stumbled in this problem you can quickly check if it's a string instead of a dictionary with:
print(type(var))
In my case I had:
<class 'str'>
before converting it with the above method and then I got
<class 'dict'>
and everything worked as supposed to