Develop Reference

get secrets from enterprise vault using python

I am trying to get secrets(user id/password) from enterprise vault. When manually I try to read user id and password, I log in to vault by okta and then I select a namespace and inside that, I can get the secrets by going into the proper path.
I want to do that programmatically but I am not understanding from where to start. I found some packages "HVAC" which is useful for vault login.
Can anyone here post the way to login into the vault and then fetching secrets from the vault? Consider the application that will be running on the AWS ec2 machine. The application has access to AWS sts service and AWS Cognito.
I am using the below code and running it from ec2 instance:
import logging
import requests
from requests.exceptions import RequestException
import hvac
logger = logging.getLogger(__name__)
EC2_METADATA_URL_BASE = 'http://169.254.169.254'
def load_aws_ec2_role_iam_credentials(role_name, metadata_url_base=EC2_METADATA_URL_BASE):
metadata_pkcs7_url = '{base}/latest/meta-data/iam/security-credentials/{role}'.format(
base=metadata_url_base,
role=role_name,
)
logger.debug("load_aws_ec2_role_iam_credentials connecting to %s" % metadata_pkcs7_url)
response = requests.get(url=metadata_pkcs7_url)
response.raise_for_status()
security_credentials = response.json()
return security_credentials
credentials = load_aws_ec2_role_iam_credentials('my_ec2_role')
a = credentials['AccessKeyId']
b = credentials['SecretAccessKey']
c = credentials['Token']
client = hvac.Client(
url='http://vault.mycompany.net/ui/vault/secrets?namespace=namespace1',
token = c
)
print(client.is_authenticated())
list_response = client.secrets.kv.v2.list_secrets(
path='path'
)
print(list_response['data'])
I am getting response "true" and then this error
getting this error:
Traceback (most recent call last):
File "3.py", line 44, in <module>
print(list_response['data'])
TypeError: 'Response' object is not subscriptable
.Can anyone tell me what wrong I am doing?what will be the url if in my enterprise vault there is namespace called "namespace1"

Create Google Document with Python

I'm trying to use Google API to create new google document (for automation of requests review)
I'm using the following code from "Example" from Google, and it doesn't work for me:
https://github.com/gsuitedevs/python-samples/blob/master/docs/quickstart/quickstart.py
When I executing this script I'm getting error:
Traceback (most recent call last):
File "google_docs.py", line 46, in <module>
main()
File "google_docs.py", line 31, in main
'credentials.json', SCOPES)
File "/usr/local/lib/python3.7/site-packages/google_auth_oauthlib/flow.py", line 199, in from_client_secrets_file
return cls.from_client_config(client_config, scopes=scopes, **kwargs)
File "/usr/local/lib/python3.7/site-packages/google_auth_oauthlib/flow.py", line 159, in from_client_config
'Client secrets must be for a web or installed app.')
ValueError: Client secrets must be for a web or installed app.
Note: I've create a Service account, and the following code works just fine with the same credentials:
import gspread
client = gspread.service_account(filename='credentials.json')
sheet = client.open('Test').sheet1
print(sheet.get_all_records())
So, I guess it means that credentials file is correct, but I wonder what can I change to be able to create Google Documents, not only spreadsheets...

After several hours of experiments and frustration I've found one dirty workaround:
import gspread
from googleapiclient.discovery import build
# The ID of a sample document.
DOCUMENT_ID = '195j9eDD3ccgjQRttHhJPymLJUCOUjs-jmwTrekvdjFE'
client = gspread.service_account(filename='credentials.json')
service = build('docs', 'v1', credentials=client.auth)
document = service.documents().get(documentId=DOCUMENT_ID).execute()
print('The title of the document is: {}'.format(document.get('title')))
so, I will put it here just in case if somebody will face the same issue I had today :)

How can I retrieve data from Google Cloud Storage into the SQL Server

Is there a way to retrieve data from Google Cloud Storage without third party apps?
I tried with python but I'm getting error code below.
import json
from httplib2 import Http
from oauth2client.client import SignedJwtAssertionCredentials
from apiclient.discovery import build
# Change these variables to fit your case
client_email = *******.iam.gserviceaccount.com
json_file = C:\******
cloud_storage_bucket = pubsite_prod_rev_********
report_to_download = installs_********_201901_app_version
private_key = json.loads(open(json_file).read())[my private key here]
credentials = SignedJwtAssertionCredentials(*******#gmail.com,my_private_key here),
storage = build('storage', 'v1', http=credentials.authorize(Http()))
print storage.objects().get()
   bucket=cloud_storage_bucket
   object=report_to_download).execute()
Python throws this error:
multiple statements found while compiling a single statement

The error message comes from your incorrect Python format, syntax or indentation.
Look at the last statement you want to execute, the bucket and object should be inside the get().
print storage.objects().get(
bucket=cloud_storage_bucket,
object=report_to_download).execute()

Trying to access an account's Youtube data via the Youtube Reporting API results in HttpError 403: "The caller does not have permission"

I'm attempting to use Youtube's Reporting API to access my company's Youtube data and analyze it. However, when I run an example script that Google has supplied, I receive the follow error:
HttpError: <HttpError 403 when requesting https://youtubereporting.googleapis.com/v1/media/%20?alt=json returned "The caller does not have permission">
I've created an Oauth 2.0 client ID, downloaded the client secrets file, and pointed the script to that. I am also being redirected to Google's authorization page, where I log in to the Youtube account that I'm trying get data from and authorize the script. For reference, the scope I'm using is:
https://www.googleapis.com/auth/yt-analytics-monetary.readonly
This is my first time trying to use Google's API so I'm sure I'm missing something, but I can't seem to figure it out. If anyone could offer some guidance I'd really appreciate it.
Edit: I've included the code that I'm using from Google's sample script below.
import urllib
import json
import os
from io import FileIO
import google.oauth2.credentials
import google_auth_oauthlib.flow
from oauth2client import client
from googleapiclient.discovery import build
from googleapiclient.errors import HttpError
from google_auth_oauthlib.flow import InstalledAppFlow
from googleapiclient.http import MediaIoBaseDownload
def get_service():
flow = InstalledAppFlow.from_client_secrets_file(client_secrets_file=client_secrets, scopes=scope)
credentials = flow.run_console()
return build(api_service_name, api_version, credentials = credentials)
def execute_api_request(client_library_function, **kwargs):
response = client_library_function(**kwargs).execute()
print(response)
scope = ['https://www.googleapis.com/auth/yt-analytics-monetary.readonly']
api_service_name = 'youtubereporting'
api_version = 'v1'
client_secrets = 'client_secret_397754690264-ba1rtbpi731qkveqb11361q4ggui2bmd.apps.googleusercontent.com.json'
youtube_analytics = get_service()
youtube_analytics.jobs().create(body={'reportTypeId':'channel_combined_a2', 'name':'test_job'}).execute()
# this lists all the reports that are generated after you've created a job
# so it might take a while before you're able to get list a report
youtube_analytics.jobs().reports().list(jobId = job_id).execute()
# this uses a reportId from one of the reports that are listed above
report_url = youtube_analytics.jobs().reports().get(jobId = job_id, reportId = 'REPORTIDGOESHERE').execute()['downloadUrl']
request = youtube_analytics.media().download(resourceName = " ")
request.uri = report_url
fh = FileIO('yt_test.txt', mode='wb')
downloader = MediaIoBaseDownload(fh, request, chunksize=-1)
done = False
while done is False:
status, done = downloader.next_chunk()
if status:
print('Download %d%%.' % int(status.progress() * 100))
print('Download Complete!')
Edit #2: Just wanted to update this with a solution that I found for anyone that might stumble upon it.
The problem was that I wasn't setting the request's uri with a report url that's created when you create a job that in turn generates a report. I was under the assumption that this was its own report, but actually this is just the method in which you download said reports. So, I went ahead and created a job which generated a report. Then, I grabbed the "downloadUrl" from the reports metadata, set the request's uri as the url, and ran the downloader as usual. I've updated the code above to reflect this.

Adding this so I can mark it as answered. My solution is below:
The problem was that I wasn't setting the request's uri with a report url that's created when you create a job that in turn generates a report. I was under the assumption that this was its own report, but actually this is just the method in which you download said reports. So, I went ahead and created a job which generated a report. Then, I grabbed the "downloadUrl" from the reports metadata, set the request's uri as the url, and ran the downloader as usual. I've updated the code above to reflect this.

GSpread pass credentials from Python not JSON

Im using GSpread trying to pass the content on my JSON file (Google API Service Application credentials) as a python Dictionary on my script. Im trying to not to carry a json file wherever I take my script.
I get the following error when I tried to pass a dictionary instead of a json file on the following line:
credentials = ServiceAccountCredentials.from_json_keyfile_name(auth_gdrive(), scope)
TypeError: expected str, bytes or os.PathLike object, not set
### auth_gdrive() returns a dictionary like this:
def auth_gdrive():
dic = {
"type": "miauuuuuu",
"pass": "miauuuu"
}
Im not allow to show whats really in the dic.

Since I wanted to pass the credentials details from within my application , and not from a json file I couldn't use:
ServiceAccountCredentials.from_json_keyfile_name()
from_json_keyfile_name() expects a json file. But looking into the docs I found the following:
ServiceAccountCredentials.from_json_keyfile_dict()
This will expect an dict object , this is all I needed.
Link:
https://oauth2client.readthedocs.io/en/latest/source/oauth2client.service_account.html
Thank you everyone again

Additional tip: I am using Google API to read Google Drive files, but also using AWS. I stored the service account credentials in AWS Secrets Manager, so that I did not need a file. I copy-pasted each key-value pair from the downloaded JSON file into AWS Secrets Manager. But I kept getting the error:
Traceback (most recent call last):
File "./copy_from_google_drive_to_s3.py", line 301, in <module>
sys.exit(main())
File "./copy_from_google_drive_to_s3.py", line 96, in main
keyfile_dict=keyDict, scopes=scopes,
File "/usr/local/lib/python3.7/site-packages/oauth2client/service_account.py", line 253, in from_json_keyfile_dict
revoke_uri=revoke_uri)
File "/usr/local/lib/python3.7/site-packages/oauth2client/service_account.py", line 185, in _from_parsed_json_keyfile
signer = crypt.Signer.from_string(private_key_pkcs8_pem)
File "/usr/local/lib/python3.7/site-packages/oauth2client/_pure_python_crypt.py", line 182, in from_string
raise ValueError('No key could be detected.')
ValueError: No key could be detected.
I had to convert the string representation of newline back into newline:
# Last part of using AWS Secrets Manager, returns json string.
sa_creds = get_secret_value_response['SecretString']
# Convert JSON string to dict.
sa_creds = json.loads(sa_creds)
# In the private key, 1-char newline got replaced with 2-char '\n'
sa_creds['private_key'] = sa_creds['private_key'].replace('\\n', '\n')
credentials = ServiceAccountCredentials.from_json_keyfile_dict(
keyfile_dict=sa_creds,
scopes=['https://www.googleapis.com/auth/drive.readonly',]
)

My solution is close to Bob McCormick. The difference is that it's using the credentials method for using service account info instead of JSON file.
Here i'm using Googles Secret Manager to import service account information so that my code can connect to a different GCP project:
from google.cloud import secretmanager
from google.oauth2 import service_account
# Create the Secret Manager client.
secret_client = secretmanager.SecretManagerServiceClient()
# Build the resource name of the secret version.
name = f"projects/{project-id}/secrets/{very-secret-secret-name}/versions/latest"
# Access the secret version.
secret_response = secret_client.access_secret_version(request={"name": name})
# Getting the secret data
secret_payload = json.loads(secret_response.payload.data.decode("UTF-8"))
# Applying the credentials as INFO instead of JSON
credentials = service_account.Credentials.from_service_account_info(
secret_payload,
scopes=["https://www.googleapis.com/auth/cloud-platform"],
)

Since you're using ServiceAccountCredentials, I'm assuming you're using OAuth2 for authorization. You can skip the json file by using oauth2client.SignedJwtAssertionCredentials to create the appropriate credentials object and pass that to gspread.authorize.
import gspread
from oauth2client.client import SignedJwtAssertionCredentials
credentials = SignedJwtAssertionCredentials(service_account_name, private_key.encode(),
['https://spreadsheets.google.com/feeds'])
gclient = gspread.authorize(credentials)
UPDATE: It appears that oauth2client.SignedJwtAssertionCredentials has been deprecated in favor of oauth2client.service_account.ServiceAccountCredentials, which only supports json and p12 keyfiles.