Develop Reference

python subprocess.run() wrongly parses command

I use python to execute the shell commands. The command call make with some extra flags:
version_info = r"CXXOPTS+='-DVERSION=" + r'\"' + args.version + r'\"' + r"'" + \
" -DSOFTWARE_FLAVOR=" + r'\"' + args.software_flavor + r'\"' + \
" -DSOFTWARE_DECORATOR=" + r'\"' + args.software_decorator + r'\"' + r"'"
where args is the arguments parsed by ArgumentParser from the command line that called this python script.
Then I execute:
command = f"make {version_info} all"
subprocess.run(command, shell=True, encoding="iso-8859-1", check=True)
It reports error: unknown output-sync type FTWARE_FLAVOR.
It seems it cannot recognize -DSOFTWARE_FLAVOR.
Then I tried to split this command line as follows:
version = r"CXXOPTS+='-DVERSION=" + r'\"' + args.version + r'\"' + r"'"
flavor = r"CXXOPTS+='-DSOFTWARE_FLAVOR=" + r'\"' + args.software_flavor + r'\"' + r"'"
.....
Then it works correctly.
What is the reason of this error?
Thank you in advance.

I think your life becomes simpler if you drop the shell=True and pass a list to subprocess.run, because this allows you to drop much of the elaborate quoting you're using to create your command line. For example:
from collections import namedtuple
version_info = (
f'CXXOPTS+=-DVERSION="{args.version}" '
f'-DSOFTWARE_FLAVOR="{args.software_flavor}" '
f'-DSOFTWARE_DECORATOR="{args.software_decorator}"'
)
command=['make', version_info, 'all']
subprocess.run(command)
If I have a Makefile that looks like this:
all:
echo CXXOPTS: "$(CXXOPTS)"
And I add some code to populate args:
import argparse
import subprocess
p = argparse.ArgumentParser()
p.add_argument('--version', default='1')
p.add_argument('--software-flavor', default='vanilla')
p.add_argument('--software-decorator', default='martha.stewaret')
args = p.parse_args()
version_info = (
f'CXXOPTS+=-DVERSION="{args.version}" -DSOFTWARE_FLAVOR="{args.software_flavor}" '
f'-DSOFTWARE_DECORATOR="{args.software_decorator}"'
)
command=['make', version_info, 'all']
subprocess.run(command)
Then I get as output:
echo CXXOPTS: -DVERSION="1" -DSOFTWARE_FLAVOR="vanilla" -DSOFTWARE_DECORATOR="martha.stewaret"
CXXOPTS: -DVERSION=1 -DSOFTWARE_FLAVOR=vanilla -DSOFTWARE_DECORATOR=martha.stewaret

How to clean up the leftovers using pytest fixtures yield?

Currently if test will fail from any reason the objects which were created in AWS service catalog(SC) can stay there after test is finished due all failed asserts stop script, so clean few lines after cant be invoked.
Example of code:
product_name, result = launch_product(role, product, storagerole)
logger.info("Creating pod with storagerole: {}".format(storagerole))
assert result, 'Product ' + product + ' could not be launched by role ' + role + ' assuming role ' + storagerole
# Get part of pod unique name
for key in client.get_provisioned_product_outputs(ProvisionedProductName=product_name)["Outputs"]:
if key["OutputKey"] == 'SSH':
pod_unique_id = key["OutputValue"].split('-')[1]
# Pick up pod with selected unique name
querypod = "kubectl get po -n rstudio | grep " + pod_unique_id + " | awk 'END {print $1}'| tr -d '\n'"
launched_pod = subprocess.check_output(querypod, shell=True).decode()
logger.info("Checking pod: {}".format(launched_pod))
cmd = "kubectl -n rstudio exec " + launched_pod + " -- aws sts get-caller-identity"
try:
output = subprocess.check_output(cmd, shell=True).decode()
except subprocess.CalledProcessError as error:
logger.error("error: {}".format(error))
assert delete_product(role, product_name), 'Product ' + product_name + ' could not be deleted by role ' + role
assert False, error
try:
assert "assumed-role/" + storagerole + "/kiam-kiam" in output, 'Expected role ' + storagerole + ' was not assumed within container'
except AssertionError as error:
logger.error("error: {}".format(error))
assert delete_product(role, product_name), 'Product ' + product_name + ' could not be deleted by role ' + role
assert False, error
logger.info("All steps passed, deleting pod: {}".format(launched_pod))
assert delete_product(role, product_name), 'Product ' + product_name + ' could not be deleted by role ' + role
how can we make a solution to clean remains even if any assertion is failed using pytest fixtures?

How to use Get File Properties REST API for Azure Files Storage using Python

I am trying to create a Python script which will utilize both the Python SDK for Azure and REST API's in order to extract information for files in my Azure Files Storage Account.
I am using the SDK to access the files in storage and get there names. Then using the name I want to be able to have a REST API call to get the file properties, specifically the Last-Modified property. I try to access the last modified property using the SDK but it always returns None for some reason.
I want to use the last modified date to determine if it has been more than 24 hours and if it has then I want to delete the file. I am not sure if it possible to set some sort of auto delete after a certain period property on the file when i first create and upload it to azure. If there is then this will solve my problems anyhow.
I have posted the code I am using below. When i try to make the HTTP request I get the error "Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature."
import datetime
import requests
import json
import base64
import hmac
import hashlib
import urllib
from azure.storage.file import *
StorageAccountConnectionString = ""
fileshareName = "testFileShare"
storage_account_name = "testStorage"
storage_account_key = ""
api_version = "2018-03-28"
file_service = FileService(connection_string=StorageAccountConnectionString)
listOfStateDirectories = file_service.list_directories_and_files(fileshareName)
for state_directory in listOfStateDirectories:
print("Cleaning up State Directory: " + state_directory.name)
if(isinstance(state_directory, Directory)):
listOfBridgeDirectories = file_service.list_directories_and_files(fileshareName, state_directory.name)
for bridge_directory in listOfBridgeDirectories:
if(isinstance(bridge_directory, Directory)):
print("Cleaning up Bridge Directory: " + bridge_directory.name)
path_to_bridge_directory = state_directory.name + "/" + bridge_directory.name
listOfFilesAndFolders = file_service.list_directories_and_files(fileshareName, path_to_bridge_directory)
for file_or_folder in listOfFilesAndFolders:
if isinstance(file_or_folder, File):
name_of_file = file_or_folder.name
# Get the time of the current request
request_time = datetime.datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')
string_to_append_to_url = fileshareName + '/' + path_to_bridge_directory + '/' + name_of_file
# Parse the url to make sure everything is good
# string_to_append_to_url = urllib.parse.quote(string_to_append_to_url)
string_params = {
'verb': 'HEAD',
'Content-Encoding': '',
'Content-Language': '',
'Content-Length': '',
'Content-MD5': '',
'Content-Type': '',
'Date': '',
'If-Modified-Since': '',
'If-Match': '',
'If-None-Match': '',
'If-Unmodified-Since': '',
'Range': '',
'CanonicalizedHeaders': 'x-ms-date:' + request_time + '\nx-ms-version:' + api_version + '\n',
'CanonicalizedResource': '/' + storage_account_name + '/' + string_to_append_to_url
}
string_to_sign = (string_params['verb'] + '\n'
+ string_params['Content-Encoding'] + '\n'
+ string_params['Content-Language'] + '\n'
+ string_params['Content-Length'] + '\n'
+ string_params['Content-MD5'] + '\n'
+ string_params['Content-Type'] + '\n'
+ string_params['Date'] + '\n'
+ string_params['If-Modified-Since'] + '\n'
+ string_params['If-Match'] + '\n'
+ string_params['If-None-Match'] + '\n'
+ string_params['If-Unmodified-Since'] + '\n'
+ string_params['Range'] + '\n'
+ string_params['CanonicalizedHeaders']
+ string_params['CanonicalizedResource'])
signed_string = base64.b64encode(hmac.new(base64.b64decode(storage_account_key), msg=string_to_sign.encode('utf-8'), digestmod=hashlib.sha256).digest()).decode()
headers = {
'x-ms-date': request_time,
'x-ms-version': api_version,
'Authorization': ('SharedKey ' + storage_account_name + ':' + signed_string)
}
url = ('https://' + storage_account_name + '.file.core.windows.net/' + string_to_append_to_url)
print(url)
r = requests.get(url, headers=headers)
print(r.content)
NOTE: Some of the directories will have white spaces so I am not sure if this is effecting the REST API call because the URL will also have spaces. If it does effect it then how would i go about accessing those files whose URL's will contain spaces

I try to access the last modified property using the SDK but it always returns None for some reason.
Not all of SDK API and REST API will return the Last-Modified property in the headers of the response, which include REST API List Directories and Files and Python SDK API list_directories_and_files.
I tried to reproduce your issue using SDK, as the code below.
generator = file_service.list_directories_and_files(share_name, directory_name)
for file_or_dir in generator:
if isinstance(file_or_dir, File):
print(file_or_dir.name, file_or_dir.properties.last_modified)
Due to the list_directories_and_files method will not return any properties in the File object, so the file_or_dir.properties.last_modified value of the code above is None.
The REST APIs Get File, Get File Properties, Get File Metadata and the Python SDK APIs get_file_properties, get_file_metadata will return Last-Modified property in the headers of the response, so to change the code as below to get the last_modified property to make it works.
generator = file_service.list_directories_and_files(share_name, directory_name)
for file_or_dir in generator:
if isinstance(file_or_dir, File):
file_name = file_or_dir.name
file = file_service.get_file_properties(share_name, directory_name, file_name, timeout=None, snapshot=None)
print(file_or_dir.name, file.properties.last_modified)
Ofcouse, to call the REST API is as same as to use SDK API. However, to build a SAS signature string is easy to make mistakes and not friendly for reading code.

Python when using "def" API - return nothing (or error)

I do not know how exactly to ask this question, as I can't share API access. If someone could help with the correct way of asking, will appreciate it.
I have a code which works perfectly fine and executes what I need, BUT when I use this API request code inside the DEF function, it returns with nothing or error...
The error is simple that I cannot get the output, or return with 0 values
This is the code:
def sm_main_data():
#DATA PROCESSING - Impression|Engagements|VideoViews
urlMain = "https://api.simplymeasured.com/v1/analytics/" + key.accountId + "/posts/metrics?\
filter=analytics.timeseries_key.gte(" + config.start + ").lte(" + config.end + ")&\
filter=channel.eq(" + config.which_social_media + ")&\
metrics=analytics.engagement_total,analytics.video.views_count,analytics.impressions&\
dimensions=data_source_id,channel,analytics.timeseries_key.by(" + config.per_what + ")"
headers = {'content-type': 'application/json',
'Authorization': 'Bearer ' + key.token}
#Receive data from SM (main data) / modified it and save as JSON file
responseMain = requests.get(urlMain, headers=headers).json()
pprint.pprint(responseMain)
pass
sm_main_data()
I have tried to print variables inside def:
print(key.accountId)
print(config.start)
print(config.end)
print(config.which_social_media)
print(config.per_what)
Which all printed correctly.
Currently I'm lost... and cannot get an even theoretical idea about what could go wrong. Why this code does not work inside def function???
Edit 1.0
Error
{u'errors': [{u'detail': u'` filter` parameter is not supported.,` metrics` parameter is not supported.,` dimensions` parameter is not supported.',
u'status': u'422',
u'title': u'Unprocessable Entity'}]}
Edit 1.1
Code without def
#DATA PROCESSING - Impression|Engagements|VideoViews
urlMain = "https://api.simplymeasured.com/v1/analytics/" + key.accountId + "/posts/metrics?\
filter=analytics.timeseries_key.gte(" + config.start + ").lte(" + config.end + ")&\
filter=channel.eq(" + config.which_social_media + ")&\
metrics=analytics.engagement_total,analytics.video.views_count,analytics.impressions&\
dimensions=data_source_id,channel,analytics.timeseries_key.by(" + config.per_what + ")"
headers = {'content-type': 'application/json',
'Authorization': 'Bearer ' + key.token}
#Receive data from SM (main data) / modified it and save as JSON file
responseMain = requests.get(urlMain, headers=headers).json()
pprint.pprint(responseMain)

The error shows you are trying to send parameters with extra spaces before them to the server:
` filter` parameter is not supported.
` metrics` parameter is not supported.
Those extra spaces before the names are part of the parameter name, because you included those in your string:
def sm_main_data():
# ...
urlMain = "https://api.simplymeasured.com/v1/analytics/" + key.accountId + "/posts/metrics?\
filter=analytics.timeseries_key.gte(" + config.start + ").lte(" + config.end + ")&\
filter=channel.eq(" + config.which_social_media + ")&\
metrics=analytics.engagement_total,analytics.video.views_count,analytics.impressions&\
dimensions=data_source_id,channel,analytics.timeseries_key.by(" + config.per_what + ")"
# ^^^ those lines are indented but the whitespace is part of the string
You would get the same problem if you had indented the urlMain string definition for any other reason, like for a if statement or a try...except statement, not just a function. You'd have to not indent those parts that are inside a string literal.
Rather than use \ continuations in the string, you could use separate string literals to create one long string, or end the string literal with a closing " followed by a + and a " opening quote on the next line:
urlMain = (
"https://api.simplymeasured.com/v1/analytics/" +
key.accountId + "/posts/metrics?" +
"filter=analytics.timeseries_key.gte(" + config.start + ").lte(" + config.end + ")&" +
"filter=channel.eq(" + config.which_social_media + ")&" +
"metrics=analytics.engagement_total,analytics.video.views_count,analytics.impressions&" +
"dimensions=data_source_id,channel,analytics.timeseries_key.by(" + config.per_what + ")"
)
All those + concatenations are not very readable, you should really use string formatting to insert values into a string.
However, you do not need to build a string like that anyway, as requests can do this for you when you give it a dictionary as the params argument. Use lists to pass in multiple values for a given parameter name:
url = "https://api.simplymeasured.com/v1/analytics/{k.accountId}/posts/metrics".format(
k=key)
params = {
'filter': [ # a list for multiple entries: filter=...&filter=...
'analytics.timeseries_key.gte({c.start}).lte({c.end})'.format(c=config),
'channel.eq({c.which_social_media})'.format(c=config),
],
'metrics': (
'analytics.engagement_total,analytics.video.views_count,'
'analytics.impressions'),
'dimensions':
'data_source_id,channel,'
'analytics.timeseries_key.by({c.per_what})'.format(c=config),
}
headers = {'Authorization': 'Bearer {k.token}'.format(k=key)}
responseMain = requests.get(urlMain, params=params, headers=headers).json()
Here I used str.format() to insert values from the config and key objects; note that the placeholders pull out the attributes
Note: I removed the Content-Type header, as that header doesn't apply to a GET request (which doesn't have content, the request body is always empty).

401: Authorization Required when retrieving Request Token from Twitter API

I'm working on a Django application which retrieves some data from Twitter, and I need to work with requests manually since it's for an exam whose main focus is for us to learn how to work with network protocols such as HTTP. So please don't suggest me to use a wrapper.
My issue is that whenever I try to request the initial request token (with a POST request that is referenced in this doc and also here), I receive a 401: Authorization Required and I have no clue what could be causing it, since I'm including the Authorization header in exactly the format required by the docs I've linked. It may be that while working with urllib I don't manage to make the POST request as I wanted, or perhaps there's something wrongin how I made the signature/where I've placed it in the header, even though in my opinion it should be alright.
Here's my code to handle the request to "sign in with Twitter":
def sign(request):
url = 'https://api.twitter.com/oauth/request_token'
#I store consumer key, consumer secret and callback url in a txt
dir_path = path.dirname(path.realpath(__file__))
with open(path.join(dir_path, 'credentials.txt'), 'r') as TweetCred:
creds = TweetCred.read().split(linesep)
oauth_consumer_key = creds[0].split()[1]
oauth_callback = creds[1].split()[1]
consumer_secret = creds[2].split()[1]
oauth_nonce = ''.join(SystemRandom().choice(string.ascii_letters + string.digits) for _ in range(32))
#Desperate to understand what could be causing the error, I tried to use an integer timestamp sinceit was integer in the examples
oauth_timestamp = int(time.time())
parameter_string = ( 'oauth_nonce=' + urllib.parse.quote(oauth_nonce, safe='') +
'&oauth_callback=' + urllib.parse.quote(oauth_callback, safe='') +
'&oauth_signature_method=' + urllib.parse.quote('HMAC-SHA1', safe='') +
'&oauth_timestamp=' + urllib.parse.quote(str(oauth_timestamp), safe='') +
'&oauth_consumer_key=' + urllib.parse.quote(oauth_consumer_key, safe='') +
'&oauth_version=' + urllib.parse.quote('1.0', safe='')
)
sigbase = 'POST' + '&' + urllib.parse.quote(url, safe='') + '&' + urllib.parse.quote(parameter_string, safe='')
signkey = urllib.parse.quote(consumer_secret, safe='') + '&'
key = bytes(signkey, 'UTF-8')
base = bytes(sigbase, 'UTF-8')
digester = hmac.new(key, base, hashlib.sha1)
binsignature = digester.digest()
signature = urllib.parse.quote(base64.urlsafe_b64encode(binsignature), safe='')
oauth_header = ('Oauth ' + 'oauth_nonce="%s", ' % urllib.parse.quote(oauth_nonce, safe='') +
'oauth_callback="%s", ' % urllib.parse.quote(oauth_callback, safe='') +
'oauth_signature_method="%s", ' % urllib.parse.quote('HMAC-SHA1', safe='') +
'oauth_timestamp="%s", ' %urllib.parse.quote(str(oauth_timestamp), safe='') +
'oauth_consumer_key="%s", ' % urllib.parse.quote(oauth_consumer_key, safe='') +
'oauth_version="%s", ' % urllib.parse.quote('1.0', safe='') +
'oauth_signature="%s"' % signature
)
headers = {'Authorization': oauth_header}
#urllib wants a body to know that it's a POST, at least that's what the docs said, so I'm providing an empty one
values = {}
data = urllib.parse.urlencode(values).encode('ascii')
TokenRequest = urllib.request.Request(url, data, headers)
try:
print('opening request token url...')
with urllib.request.urlopen(TokenRequest) as response:
if response.getcode() != 200:
print('Response is %s' % response.getcode())
return HttpResponse('Error in getting the token from Twitter, please try again...')
body = loads(response.read())
if body['oauth_callback_confirmed'] != 'true':
print('oauth_callback not confirmed')
return HttpResponse('Error in getting the token from Twitter, please try again...')
oauth_token = body['oauth_token']
oauth_token_secret = body['oauth_token_secret']
except urllib.error.HTTPError as err:
#my program always ends up here, catches the exception and returns the line below
return HttpResponse('Error %s in getting the token from Twitter, please try again...' % err)
print('Successfully retrieved request token! Redirecting...')
loggee = User.objects.Create()
loggee.oauth_token = oauth_token
loggee.oauth_token_secret = oauth_token_secret
return HttpResponseRedirect('https://api.twitter.com/oauth/authenticate?oauth_token='+oauth_token)
Any help will be appreciated! I'm quite in a hurry and I really can't wrap my head around this!

Your signature base string needs to be sorted lexigraphically (which means alphabetically for all intents), before you calculate your signature (ref: Creating a signature)
Therefore your parameter string needs to be:
parameter_string = (
'oauth_callback=' + urllib.parse.quote(oauth_callback, safe='') +
'&oauth_consumer_key=' + urllib.parse.quote(oauth_consumer_key, safe='') +
'&oauth_nonce=' + urllib.parse.quote(oauth_nonce, safe='') +
'&oauth_signature_method=' + urllib.parse.quote('HMAC-SHA1', safe='') +
'&oauth_timestamp=' + urllib.parse.quote(str(oauth_timestamp), safe='') +
'&oauth_version=' + urllib.parse.quote('1.0', safe='')
)