I'm trying to use a Python script to turn on/off a CloudSQL Instance in GoogleCloud. I've finally found a way to do it, using the GoogleCloudAPI, in Shell:
### shell script
ACCESS_TOKEN="$(gcloud auth print-access-token)"
ACTIVATION_POLICY="ALWAYS/NEVER" # Use 'ALWAYS' to turn on, 'NEVER' to turn off
curl --header "Authorization: Bearer ${ACCESS_TOKEN}" --header 'Content-Type: application/json' --data '{"settings" : {"activationPolicy" : "${ACTIVATION_POLICY}"}}' -X PATCH https://www.googleapis.com/sql/v1beta4/projects/${PROJECT_ID}/instances/${INSTANCE_ID}
So, great, problem solved... except I cannot use 'gcloud auth print-access-token' on the machine I'm running the script in, so that solves nothing. I found a question from 2017 trying to generate this 'access-token' using Python as well, which apparently didn't work either.
I need to be able to generate this 'access-token' using Python itself. I've been looking around in the Google's documentation but I still haven't managed to find anything related to that, closest I found was using oauth2 and googleapiclient to get the list of instances running, but can't seem to change activation policies from there:
### python3 script
from google.oauth2 import service_account
import googleapiclient.discovery
SCOPES = ['https://www.googleapis.com/auth/sqlservice.admin']
SERVICE_ACCOUNT_FILE = '/path/to/service.json'
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
sqladmin = googleapiclient.discovery.build('sqladmin', 'v1beta4', credentials=credentials)
response = sqladmin.instances().get(project=PROJECT_ID, instance=INSTANCE_ID).execute()
The documentation doesn't make it clear how to use either tool to turn off the CloudSQL instance though, or at least none that I could find do. The above code returns me a JSON file and I can see the 'activationPolicy' there, under settings. I can't find a way to 'change it' though.
Managed to follow #norbjd suggestion and find a 'patch' method, and gave 'SQL Admin' permission to my credentials, so it can now use the sqladmin API. Tried to patch it using the following code:
instance = sqladmin.instances().patch(project=PROJECT_ID, instance=INSTANCE_ID)
instance.data = {"settings" : {"activationPolicy" : "NEVER"}} #also tried with it in a string, like this: instance.data = {"settings" : {"activationPolicy" : "NEVER"}}
instance.headers['Content-Type'] = 'application/json'
instance.execute()
Considering instance.data didn't exist prior to that, but instance.headers did:
{'accept': 'application/json', 'accept-encoding': 'gzip, deflate', 'user-agent': '(gzip)', 'x-goog-api-client': 'gdcl/1.7.11 gl-python/3.6.9'}
After the execute, though, nothing seemed to happen. It did not change the actual activationPolicy.
In the end, the problem was solved using the ACCESS_TOKEN and making a Python request using the requests module. If you try to get an ACCESS_TOKEN from your credentials just after generating them, you won't be getting any, but if you actually use your credentials with the googleapiclient.discovery, this updates that object with a valid access token, that can then be used in a Python request, as follows:
from google.oauth2 import service_account
import googleapiclient.discovery
import json
import requests
PROJECT_ID = '{PROJECT_ID_HERE}'
INSTANCE_ID = '{INSTANCE_ID_HERE}'
SCOPES = ['https://www.googleapis.com/auth/sqlservice.admin']
SERVICE_ACCOUNT_FILE = '/path/to/service.json'
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
sqladmin = googleapiclient.discovery.build('sqladmin', 'v1beta4', credentials=credentials)
response = sqladmin.instances().get(project=PROJECT_ID, instance=INSTANCE_ID).execute()
print(json.dumps(response))
access_token = credentials.token # Now that the credentials were used, they have a valid access_token associated with them!
activation_policy = 'NEVER' # or 'ALWAYS'
url = "https://www.googleapis.com/sql/v1beta4/projects/{PROJECT_ID}/instances/{INSTANCE_ID}".format(PROJECT_ID=PROJECT_ID, INSTANCE_ID=INSTANCE_ID)
header = {
"Authorization" : "Bearer {}".format(access_token),
"Content-Type" : "application/json"
}
data = {
"settings" : {
"activationPolicy" : activation_policy
}
}
response = requests.patch(url, headers=header, data=json.dumps(data))
print(response.text)
All it took was to actually use the credentials for something before attempting to retrieve the ACCESS_TOKEN
Related
I am trying to list unused service accounts in a gcp project
Working fine when using gcloud command
gcloud recommender insights list \
--insight-type=google.iam.serviceAccount.Insight \
--location=global \
--filter=insightSubtype=SERVICE_ACCOUNT_USAGE --project
Getting an error when i am trying to list the unused service accounts using python sdk. Below is the error
import requests
import json
import re
import sys
import subprocess
import os
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
from google.oauth2 import service_account
credentials = service_account.Credentials.from_service_account_file("")
service = discovery.build('cloudresourcemanager', 'v1', credentials=credentials)
request = service.projects().list()
token1 = subprocess.Popen("gcloud auth print-access-token", stdout=subprocess.PIPE, shell = True)
token, error = token1.communicate()
token = str(token.decode("utf-8"))
token = token.rstrip("\n")
token = token.rstrip("\r")
while request is not None:
response = request.execute()
print(response)
for project in response.get('projects', []):
projectid = project['projectId']
projectname = project['name']
headers = {
'Authorization': 'Bearer ' + token,
'x-goog-user-project': projectname
}
post_url= "https://recommender.googleapis.com/v1/projects/" + projectid + "/locations/"+ "global" +"/insightTypes/google.iam.serviceAccount.Insight/insights?filter=insightSubtype=SERVICE_ACCOUNT_USAGE"
post_url_data = requests.get(post_url, headers = headers)
get_api_json = json.loads(post_url_data.text)
print(get_api_json)
I am iterating through all projects, for some projects i am getting below error, I have checked in the console, the projects exist..
Error: project not found or deleted, status; INVALID_ARGUMENT, details: [{'#type': 'type.googleapis.com/google.rpc/ErrorInfo', 'reason':'USER_PROJECT_DENIED'
Any idea what's missing here?
the error means that you have insufficient permissions on the user project or it is deleted or not found.
A User Project is for quota and billing purposes. The caller must have the IAM permission serviceusage.services.use permission on the project. The user/quota project is set with this command:
gcloud auth application-default set-quota-project
You should always use project_id in your requests unless otherwise specified in documentation.
Notice the change in your code:
for project in response.get('projects', []):
projectid = project['projectId']
projectname = project['name']
headers = {
'Authorization': 'Bearer ' + token,
'x-goog-user-project': projectid # >>> HERE <<<, use projectid instead of projectname and make sure you have the required permission/s
}
post_url= "https://recommender.googleapis.com/v1/projects/" + projectid + "/locations/"+ "global" +"/insightTypes/google.iam.serviceAccount.Insight/insights?filter=insightSubtype=SERVICE_ACCOUNT_USAGE"
post_url_data = requests.get(post_url, headers = headers)
get_api_json = json.loads(post_url_data.text)
print(get_api_json)
I am trying to reach my automL model prediction endpoint that I have setup, I have created a service account and added the correct role to it, now I am trying to call the endpoint in Python, but I am not sure how to call it: Here is what I have tried. The key file is downloaded from google so it's good.
from google.oauth2 import service_account
project_id = 'aaa'
endpoint_id = 'bbb'
with open('./ServiceAccountKey.json') as source:
info = json.load(source)
credentials = service_account.Credentials.from_service_account_info(info)
scoped_credentials = credentials.with_scopes(
['https://www.googleapis.com/auth/cloud-platform'])
access_token = scoped_credentials.get_access_token()
endpoint = f"https://us-central1-aiplatform.googleapis.com/v1alpha1/projects/{project_id}/locations/us-central1/endpoints/{endpoint_id}:predict"
headers = {
"Content-Type": "application/json",
"Authorization": "Bearer " + access_token}
payload = {}
x = requests.post(endpoint, data = payload, header=headers)
print(x)
The error I get is this:
AttributeError: 'Credentials' object has no attribute 'get_access_token'
Permissions I have given the service account:
The best way is to use the AI Platform Python client library. Set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path to your service account file and you are good to go.
Better to not deal with access token and refresh tokens yourself if there is a ready-made library available.
I have a Python function using the preview option of sending custom metrics to Azure using the REST API https://learn.microsoft.com/en-us/azure/azure-monitor/platform/metrics-store-custom-rest-api, previously this was a C# function where authorisation and getting a bearer token was handled automagically by:
var azureServiceTokenProvider = new AzureServiceTokenProvider();
string bearerToken = await azureServiceTokenProvider.GetAccessTokenAsync("https://monitoring.azure.com/").ConfigureAwait(false);
This worked in VS Code using the logged in user and in Azure when a Managed Identity was assigned to the Function.
I needed to convert this to Python but so far the best (working) I've been able to come up with is:
import logging, requests, os, adal
import azure.functions as func
def main(req: func.HttpRequest) -> func.HttpResponse:
regional_monitoring_url = "https://eastus.monitoring.azure.com"
monitored_resource_id = os.environ['RESOURCE_ID']
full_endpoint = f"{regional_monitoring_url}{monitored_resource_id}/metrics"
tenant_id = os.environ['AZURE_TENANT_ID']
context = adal.AuthenticationContext(f'https://login.microsoftonline.com/{tenant_id}')
token = context.acquire_token_with_client_credentials("https://monitoring.azure.com/", os.environ['AZURE_CLIENT_ID'], os.environ['AZURE_CLIENT_SECRET'] )
bearer_token = token['accessToken']
json = req.get_json()
headers = {"Authorization": 'Bearer ' + bearer_token}
result = requests.post(url = full_endpoint, headers = headers, json = json)
return func.HttpResponse(f"Done - {result.status_code} {result.text}", status_code=200)
This obviously relies on me creating a Service Principal with the relevant permissions. I'm trying to work out how to use the automatic Managed Identity authorisation that the C# libraries have.
I know ADAL should be replaced by MSAL but I can't work out how/if that automagically handles Managed Identities so I tried azure-identity:
from azure.identity import DefaultAzureCredential
credential = DefaultAzureCredential()
token = credential.get_token("https://monitoring.azure.com/.default")
bearer_token = token.token
This gets me a token but because it requires a scope rather than a resource, which means adding .default to the resource URL, when I send the bearer token to the monitoring endpoint it complains the resource doesn't match and must be exactly "https://monitoring.azure.com/"
Is this just not currently possible or am I missing something with either azure-identity or the MSAL Python modules?
According to my research, when werequest an Azure AD token to emit custom metrics, ensure that the audience the token is requested for is https://monitoring.azure.com/. For more details, please refer to here. So we should update scope as https://monitoring.azure.com//.default
For example
def main(req: func.HttpRequest) -> func.HttpResponse:
logging.info('Python HTTP trigger function processed a request.')
credential = DefaultAzureCredential()
token = credential.get_token("https://monitoring.azure.com//.default")
bearer_token = token.token
#full_endpoint=""
json = req.get_json()
headers = {"Authorization": 'Bearer ' + bearer_token}
#result = requests.post(url = full_endpoint, headers = headers, json = json)
return func.HttpResponse(f"Done - {bearer_token}", status_code=200)
I'm trying to access GetDealItems API and i have a nightmare to get this working. Even though I use the valid client_id','client_secret','ruName' i keep getting
{'error': 'invalid_client', 'error_description': 'client authentication failed'}
below is the ebay doc
https://developer.ebay.com/api-docs/buy/deal/resources/deal_item/methods/getDealItems
I guess i need to use this scope and url in my request
scopes:'https://api.ebay.com/oauth/api_scope/buy.deal' and the
url='https://api.ebay.com/buy/deal/v1/deal_item?limit=1000'
Please see below my Python code.
import requests, urllib, base64
def getAuthToken():
AppSettings = {
'client_id':'xxxx7c8ec878c-c80c4c69',
'client_secret':'xxxx56db-4b4a-97b4-fad2',
'ruName':'xxxxx-gscrcsrtj'}
authHeaderData = AppSettings['client_id'] + ':' + AppSettings['client_secret']
encodedAuthHeader = base64.b64encode(str.encode(authHeaderData))
headers = {
"Content-Type" : "application/x-www-form-urlencoded",
"Authorization" : "Bearer " + str(encodedAuthHeader)
}
body= {
"grant_type" : "client_credentials",
"redirect_uri" : AppSettings['ruName'],
"scope" : "https://api.ebay.com/oauth/api_scope/buy.deal"
}
data = urllib.parse.urlencode(body)
tokenURL = "https://api.ebay.com/identity/v1/oauth2/token"
response = requests.post(tokenURL, headers=headers, data=data)
return response.json()
response = getAuthToken()
print(response)
response['access_token'] #access keys as required
response['error_description'] #if errors
The most obvious problem I see is that you are using Bearer when you should be using Basic in your Authorization header.
Also, You are urlencoding your redirect_url when you pass the entire dictionary into urlencode. The docs say you are supposed to urlencode the scope parameter, but honestly, I never encode the scope and it still works for me.
Here is your modified code, with a few formatting changes:
import requests, urllib, base64
client_id='xxxx7c8ec878c-c80c4c69'
client_secret='xxxx56db-4b4a-97b4-fad2'
ruName='xxxxx-gscrcsrtj'
scope = urllib.parse.quote('https://api.ebay.com/oauth/api_scope/buy.deal')
def basic_token(key, secret):
return 'Basic ' + base64.b64encode((key + ':' + secret).encode()).decode()
def getAuthToken():
headers = {
"Content-Type" : "application/x-www-form-urlencoded",
"Authorization" : basic_token(client_id, client_secret)
}
data = (
'grant_type=client_credentials&'
f'redirect_uri={ruName}&'
f'scope={scope}'
)
tokenURL = "https://api.ebay.com/identity/v1/oauth2/token"
response = requests.post(tokenURL, headers=headers, data=data)
return response.json()
Update:
I think you need to use the authorization_code grant instead of client_credentials.
To use the authorization_code grant, modify your body to look like this:
data = (
'grant_type=authorization_code&'
f'code={authorization_code}&'
f'redirect_uri={ruName}&'
f'scope={scope}'
)
Also, you will need to follow your "redirect url" to get the actual authorization code. Execute the following:
redirect_url = (
'https://auth.ebay.com/oauth2/authorize?'
f'client_id={client_id}&'
f'response_type=code&'
f'redirect_uri={ruName}&'
f'scope={scope}'
)
print(redirect_url)
Copy/paste the url from stdout, follow the link, and click "accept", then you will be redirected to a url that looks like this:
https://signin.ebay.com/ws/eBayISAPI.dll?ThirdPartyAuthSucessFailure&isAuthSuccessful=true&code=<authorization code here>&expires_in=299
Copy/paste the authorization code into your code, then see if it works.
Realistically, eBay expects you to automate this within your application using a server, but it doesn't make sense for you to go through the trouble if you are building an app for personal use.
GetDealItems API uses client_credentials grant as evident from the docs
The authorization should be using client_id and secret as described in getting access tokens
curl -X POST 'https://api.ebay.com/identity/v1/oauth2/token' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-H 'Authorization: Basic UkVTVFRlc3...wZi1hOGZhLTI4MmY=' \
-d 'grant_type=client_credentials&scope=https%3A%2F%2Fapi.ebay.com%2Foauth%2Fapi_scope%2Fbuy.deal'
Note: if the error is client_authorization_failed, ensure that the correct Keyset for production is used for production. Also ensure that the keyset is also enabled for Oauth
Finally, you can use/refer to the official python SDK as well here
A simple way to check if the particular scope, in this case https://api.ebay.com/oauth/api_scope/buy.deal is even allowed for this app, is to navigate to the keyset page under Keys link and click on "Oauth scopes" under the keyset which details the scopes allowed and their purpose. If the application is once authorized for buy.deal, then the scope will appear there.
UPDATE
GetDeals API is restricted in Production for authorized applications only. Please reach out to the eBay developer program as provided in the link on the page below.
https://developer.ebay.com/api-docs/buy/deal/overview.html#API
I was trying to use the chunk upload feature of twitter.
Reference: https://developer.twitter.com/en/docs/media/upload-media/api-reference/post-media-upload-init
Since this feature is not available on tweepy, I was trying to implement that from scratch. However, since the user already login with the application using tweepy, tweepy saved the access token and secret of the user obtained there.
I tried to use these access token as following, but I got the error of "code":32,"message":"Could not authenticate you."
I want to explore a way that will not ask user to type their email and password again to upload video.
from requests_oauthlib import OAuth1, OAuth1Session
import os
def upload_video(video_file):
video_file = os.path.join(MEDIA_ROOT, 'test_video.mp4')
oauth_token = OAUTH_TOKEN
oauth_token_secret = OAUTH_TOKEN_SECRET
consumer_key = TWITTER_CONSUMER_TOKEN
consumer_secret = TWITTER_CONSUMER_TOKEN_SECRET
oauth = OAuth1Session(consumer_key,
client_secret=consumer_secret,
resource_owner_key=oauth_token,
resource_owner_secret=oauth_token_secret,
signature_method='HMAC-SHA1')
headers = requests.utils.default_headers()
headers.update(
{
'User-Agent': 'OAuth gem v0.4.4',
}
)
video_url = 'https://upload.twitter.com/1.1/media/upload.json?command=INIT&total_bytes={size}&media_type={type}'.format(size=os.path.getsize(video_file), type='video/mp4')
response = oauth.post(video_url,
headers=headers,
files=dict(foo='bar')) # to make it in multipart/form-data
print(response.content)
print(response.request.body)
print(response.request.headers)
I also printed out the request headers which was something like the following:
{'User-Agent': b'OAuth gem v0.4.4', 'Accept-Encoding': b'gzip, deflate', 'Accept': b'*/*', 'Connection': b'keep-alive', 'Content-Length': '141', 'Content-Type': b'multipart/form-data; boundary=f5c7d61e8ab8a14b2a22ced4171b723e',
'Authorization': b'OAuth oauth_nonce="147920959083366377161589749583", oauth_timestamp="1589749583", oauth_version="1.0", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="consumer_key", oauth_token="oauth_token", oauth_signature="mcEaLBsANVu%2B7lavaNfrOiHZbgs%3D"'}
I also tried twurl command which did get a proper response back, but it required me to type in username and password and which generated a different set of oauth_token and secret from that of tweepy.
It turned out that the reason I got this was because I needed to put the params in the data instead of like queries.
This repository really helped me on how to upload videos to Twitter.
https://github.com/twitterdev/large-video-upload-python