I am trying to get secrets(user id/password) from enterprise vault. When manually I try to read user id and password, I log in to vault by okta and then I select a namespace and inside that, I can get the secrets by going into the proper path.
I want to do that programmatically but I am not understanding from where to start. I found some packages "HVAC" which is useful for vault login.
Can anyone here post the way to login into the vault and then fetching secrets from the vault? Consider the application that will be running on the AWS ec2 machine. The application has access to AWS sts service and AWS Cognito.
I am using the below code and running it from ec2 instance:
import logging
import requests
from requests.exceptions import RequestException
import hvac
logger = logging.getLogger(__name__)
EC2_METADATA_URL_BASE = 'http://169.254.169.254'
def load_aws_ec2_role_iam_credentials(role_name, metadata_url_base=EC2_METADATA_URL_BASE):
metadata_pkcs7_url = '{base}/latest/meta-data/iam/security-credentials/{role}'.format(
base=metadata_url_base,
role=role_name,
)
logger.debug("load_aws_ec2_role_iam_credentials connecting to %s" % metadata_pkcs7_url)
response = requests.get(url=metadata_pkcs7_url)
response.raise_for_status()
security_credentials = response.json()
return security_credentials
credentials = load_aws_ec2_role_iam_credentials('my_ec2_role')
a = credentials['AccessKeyId']
b = credentials['SecretAccessKey']
c = credentials['Token']
client = hvac.Client(
url='http://vault.mycompany.net/ui/vault/secrets?namespace=namespace1',
token = c
)
print(client.is_authenticated())
list_response = client.secrets.kv.v2.list_secrets(
path='path'
)
print(list_response['data'])
I am getting response "true" and then this error
getting this error:
Traceback (most recent call last):
File "3.py", line 44, in <module>
print(list_response['data'])
TypeError: 'Response' object is not subscriptable
.Can anyone tell me what wrong I am doing?what will be the url if in my enterprise vault there is namespace called "namespace1"
Related
I am trying to authenticate with the python SDK to pull Azure VNet data.
As a first step to verify that I can authenticate I am trying to use the subscription client to list subscriptions. I am creating a certificate credential to use for authentication.
When I make the call to list the subscriptions from the subscription client the call hangs seemingly indefinitely with no error returned. I am trying to authenticate to azure_gov. Here is the code:
import logging
import os
import boto3
from msrestazure.azure_cloud import AZURE_US_GOV_CLOUD as CLOUD
from azure.identity import CertificateCredential
from azure.mgmt.subscription import SubscriptionClient
# Setup logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
logging.basicConfig(level=logging.INFO)
# Constants
CERT_PATH = '/tmp/cert.pem'
AZURE_CERT_PATH = '/tmp/cert.pem'
AZURE_TENANT_ID = os.environ['AZURE_TENANT_ID']
AZURE_CLIENT_ID = os.environ['AZURE_CLIENT_ID']
AZURE_SDK_S3_BUCKET = os.environ['AZURE_SDK_S3_BUCKET']
s3 = boto3.client('s3')
s3.download_file(AZURE_SDK_S3_BUCKET, 'certs/cert.pem', CERT_PATH)
# Setup Azure credentials
credential = CertificateCredential(
tenant_id=AZURE_TENANT_ID,
client_id=AZURE_CLIENT_ID,
certificate_path=AZURE_CERT_PATH,
authority=CLOUD.endpoints.active_directory)
logger.info(f'tenant_id = {AZURE_TENANT_ID}, client_id = {AZURE_CLIENT_ID}')
logger.info(f'CLOUD: {CLOUD}')
sub_client = SubscriptionClient(
credential=credential,
base_url=CLOUD.endpoints.resource_manager)
#Code times out here
subscription = next(sub_client.subscriptions.list())
logger.info(f'Fetched subscription {subscription.subscription_id}')
I have verified multiple times that the cert, tenant_id, and client_id all match what I see in active directory.
I've found the following posts from Microsoft: first post and second post, which both use the azure.mgmt.resource SubscriptionClient which gives no attribute 'signed_session' in the CertificateCredential when trying to use a CertificateCredential to setup the client.
I have found the following adapter for using the CertificateCredential class with this client and tried using it but it also gives me the same timeout issue on the next(sub_client.subscriptions.list) call.
EDIT:
I am still seeing issues with this, when things completely time out after the max number of retries I get the following error:
Attempted credentials:
EnvironmentCredential: Authentication failed: <urllib3.connection.HTTPSConnection object at 0x7fad94f116d8>: Failed to establish a new connection: [Errno 110] Connection timed out
I don't think it is an environment issue as I can log into the Azure CLI from the same instance.
I have set up my web app to receive Firebase Cloud Messages, but in order to send them I understand that I need an OAuth2.0 Token, which is generated from my Service Account private key. I have downloaded the key and installed Google API Client Library for Python. This is my code:
from oauth2client import *
def _get_access_token():
"""Retrieve a valid access token that can be used to authorize requests.
:return: Access token.
"""
credentials = ServiceAccountCredentials.from_json_keyfile_name(
'service-account.json', FCM_SCOPE)
access_token_info = credentials.get_access_token()
return access_token_info.access_token
_get_access_token()
When I run that, I get
Traceback (most recent call last):
File "get-oauth-token.py", line 11, in <module>
_get_access_token()
File "get-oauth-token.py", line 7, in _get_access_token
credentials = ServiceAccountCredentials.from_json_keyfile_name(
NameError: name 'ServiceAccountCredentials' is not defined
I assume that I'm missing an import, but I'm not sure what.
This should solve:
from oauth2client.service_account import ServiceAccountCredentials
fsm_scope = 'https://www.googleapis.com/auth/firebase.messaging'
def _get_access_token():
"""Retrieve a valid access token that can be used to authorize requests.
:return: Access token.
"""
credentials = ServiceAccountCredentials.from_json_keyfile_name(
'service-account.json', fsm_scope)
access_token_info = credentials.get_access_token()
return access_token_info.access_token
_get_access_token()
I was able to piece together a Python script to interact with the Google API Library using information from here and here. The code below is working and I'm able to list all accounts within a particular Project. See below:
Code:
import os
from googleapiclient import discovery
from gcp import get_key
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = get_key()
service = discovery.build('cloudresourcemanager', 'v1')
project_id = 'test-buckets'
get_iam = {}
request = service.projects().getIamPolicy(resource=project_id, body=get_iam)
response = request.execute()
bindings = response['bindings']
for binding in bindings:
for member in binding['members']:
print(member)
Output:
ssh://xxxx#x.x.x.x:22/home/xxxx/venv/bin/python3.4 -u /home/xxxx/scratch.py
serviceAccount:xxxxxxxxxxx-compute#developer.gserviceaccount.com
serviceAccount:xxxxxxxxxxx#xxxxxxxxxxx.iam.gserviceaccount.com
user:xxxx#.xxxxx.com
Now, I'm trying to leverage the API to delete specific accounts that are not needed (script below), however, I can't seem to get the the API to work. I'm using the Google API Library again here, but it doesn't seem to be working. I'm not sure what I'm doing wrong. Any help you can offer is greatly appreciated.
Code:
import os
from pprint import pprint
from googleapiclient import discovery
from gcp import get_key
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = get_key()
service = discovery.build('cloudresourcemanager', 'v1')
project_id = 'test-buckets'
service_account = 'projects/xxxxxxx/serviceAccounts/test-delete#test-buckets.iam.gserviceaccount.com'
request = service.projects().delete(name=service_account, body={})
response = request.execute()
pprint(response)
Output:
ssh://xxxx#x.x.x.x:22/home/xxxx/venv/bin/python3.4 -u /home/xxxx/scratch2.py
Traceback (most recent call last):
File "/home/xxxx/scratch2.py", line 13, in <module>
request = service.projects().delete(name=service_account, body={})
File "/home/xxxx/venv/lib/python3.4/site-packages/googleapiclient/discovery.py", line 722, in method
raise TypeError('Got an unexpected keyword argument "%s"' % name)
TypeError: Got an unexpected keyword argument "body"
Process finished with exit code 1
Maybe it's a little too late for an answer, but I think the only error in your code is that this line of code:
request = service.projects().delete(name=service_account, body={})
should in fact be like this:
request = service.projects().delete(name=service_account)
As you can see in the error you shared, TypeError: Got an unexpected keyword argument "body", so body is not an accepted argument here. In any case, printing the response in pprint(response) will probably output nothing, as the response to a DELETE operation has no content.
I figured out that I was using the wrong API when building the service too. Below is the corrected code:
service = discovery.build('iam', 'v1')
request = service.projects().serviceAccounts().delete(name=project_name)
response = request.execute()
I'm following this tutorial to upload a file to a bucket I've manually created:
https://cloud.google.com/storage/docs/xml-api/gspythonlibrary
I seem to have trouble with setting up the credentials both as service account or user account. I want to use this in a web server so ideally it should be setup with service account.
I created an API using API Manager in Console and downloaded the JSON. Meanwhile my gcloud auth is setup with my OAUTH login. I did try gsutil config -e and got error:
CommandException: OAuth2 is the preferred authentication mechanism with the Cloud SDK. Run "gcloud auth login" to configure authentication, unless you want to authenticate with an HMAC access key and secret, in which case run "gsutil config -a".
I also tried to authenticate the service account using:
gcloud auth activate-service-account --key-file <json file>
but still no luck with enabling access with python boto. I also copied the ID and Key from ~/.config/gcloud/ to ~/.boto but that didn't work either. I'm not sure how am I supposed to setup the authentication for the python server to access cloud storage. I'm not using App Engine but Cloud Compute to setup the webserver.
Here's my source code:
import boto
import gcs_oauth2_boto_plugin
import os
import shutil
import StringIO
import tempfile
import time
CLIENT_ID = 'my client id from ~/.config/gcloud/credentials'
CLIENT_SECRET = 'my client secret from ~/.config/gcloud/credentials'
gcs_oauth2_boto_plugin.SetFallbackClientIdAndSecret(CLIENT_ID, CLIENT_SECRET)
uri = boto.storage_uri('', 'gs')
project_id = 'my-test-project-id'
header_values = {"x-test-project-id": project_id}
# If the default project is defined, call get_all_buckets() without arguments.
for bucket in uri.get_all_buckets(headers=header_values):
print bucket.name
Most recent error:
Traceback (most recent call last):
File "upload/uploader.py", line 14, in <module>
for bucket in uri.get_all_buckets(headers=header_values):
File "/Users/ankitjain/dev/metax/venv/lib/python2.7/site-packages/boto/storage_uri.py", line 574, in get_all_buckets
return conn.get_all_buckets(headers)
File "/Users/ankitjain/dev/metax/venv/lib/python2.7/site-packages/boto/s3/connection.py", line 444, in get_all_buckets
response.status, response.reason, body)
boto.exception.GSResponseError: GSResponseError: 403 Forbidden
<?xml version='1.0' encoding='UTF-8'?><Error><Code>InvalidSecurity</Code><Message>The provided security credentials are not valid.</Message><Details>Incorrect Authorization header</Details></Error>
Okay after some more experiments I gave up on using GCE library for uploading data to Cloud Storage. I actually found using AWS boto to upload to cloud storage works much better. All I had to do was specify Google's host in s3 library:
conn = boto.connect_s3(app.config['S3_KEY'], app.config['S3_SECRET'], "c.storage.googleapis.com")
bucket = conn.get_bucket(app.config['S3_BUCKET'], validate=False)
I used the HMAC credentials generated the way described in the Google Docs. Reference:
https://cloud.google.com/storage/docs/migrating
I'm trying to access a Google Spreadsheet via the GData API using oAuth 2.0 service-account credentials created for a Python 2.7 app hosted on Google App Engine.
The app uses the recent gdata-python-client from Google, v. 2.0.18 (gdata and atom).
The app uses the recent google-api-python-client-gae, v. 1.2.
In the Google Developer Console for this project (in this example referred to as "my-gae-app"), I have created a Service Account and delegated domain-wide authority to the service account as described here
The desired spreadsheet in Google Drive belongs to a Google Apps for Work domain, here referred to as "mygoogleappsdomain.com".
I have granted read+write access for the spreadsheet to my-gae-app#appspot.gserviceaccount.com and to the email address shown for this service account and which is assigned to clientEmail variable in the code below. Not sure which of the two email addresses would be actually needed, so I assigned both. The user with the impersonateUser email address also has read+write access to this spreadsheet.
With Google API Python Client's AppAssertionCredentials I can access the meta-data of the desired spreadsheet via the Google Drive API. However, if I try to access the spreadsheet's content using gdata, I'm getting errors. The best result I could get so far with the service account is using SignedJwtAssertionCredentials, as suggested here. However, I'm stuck with this AccessRefreshTokenError: access denied and I don't understand what's going wrong.
import os
import httplib2
from google.appengine.api import memcache
from apiclient.discovery import build
from oauth2client.client import SignedJwtAssertionCredentials
import gdata.spreadsheets.client
import gdata.spreadsheet.service
# AppAssertionCredentials is not supported in gdata python client library,
# so we use SignedJwtAssertionCredentials with the credential
# file of this service account.
# Load the key in PKCS 12 format that you downloaded from the Google API
# Console when you created your Service account.
clientEmail = '10346........-g3dp......................3m1em8#developer.gserviceaccount.com'
p12File = 'app.p12'
path = os.path.join(ROOT_DIR, 'data', 'oAuth2', p12File)
impersonateUser = 'user#mygoogleappsdomain.com'
spreadsheetKey = '1mcJHJ...................................juQMw' # ID copied from URL of desired spreadsheet in Google Drive
with open(path) as f:
privateKey = f.read()
f.close()
# Getting credentials with AppAssertionCredentials only worked successfully
# for Google API Client Library for Python, e.g. accessing file's meta-data.
# So we use SignedJwtAssertionCredentials, as suggested in
# https://stackoverflow.com/questions/16026286/using-oauth2-with-service-account-on-gdata-in-python
credentials = SignedJwtAssertionCredentials(
clientEmail,
privateKey,
scope=(
'https://www.googleapis.com/auth/drive.file ',
# added the scope above as suggested somewhere else,
# but error occurs with and with-out this scope
'https://www.googleapis.com/auth/drive',
'https://spreadsheets.google.com/feeds',
'https://docs.google.com/feeds'
),
sub=impersonateUser
)
http = httplib2.Http()
http = credentials.authorize(http)
auth2token = gdata.gauth.OAuth2TokenFromCredentials(credentials)
# error will occur, wether using SpreadsheetsService() or SpreadsheetsClient()
#srv = gdata.spreadsheet.service.SpreadsheetsService()
#srv = auth2token.authorize(srv)
clt = gdata.spreadsheets.client.SpreadsheetsClient()
clt = auth2token.authorize(clt)
# Until here no errors
wks = clt.get_worksheets(spreadsheetKey)
# AccessTokenRefreshError: access_denied
This is the error I get in the remote shell:
s~my-gae-app> wks = clt.get_worksheets(spreadsheetKey)
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "gdata/spreadsheets/client.py", line 108, in get_worksheets
**kwargs)
File "gdata/client.py", line 640, in get_feed
**kwargs)
File "gdata/client.py", line 267, in request
uri=uri, auth_token=auth_token, http_request=http_request, **kwargs)
File "atom/client.py", line 122, in request
return self.http_client.request(http_request)
File "gdata/gauth.py", line 1344, in new_request
refresh_response = self._refresh(request_orig)
File "gdata/gauth.py", line 1485, in _refresh
self.credentials._refresh(httplib2.Http().request)
File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 653, in _refresh
self._do_refresh_request(http_request)
File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 710, in _do_refresh_request
raise AccessTokenRefreshError(error_msg)
AccessTokenRefreshError: access_denied
I'm not sure if this indicates that this service-account is denied access to the spreadsheet, or if there was an error with refreshing the access token. Do you know what's wrong with this code or setup?
I've figured out that calling SignedJwtAssertionCredentials with-out the sub parameter (for the "impersonated" user) will not produce AccessTokenRefreshError: access_denied
import os
import httplib2
from google.appengine.api import memcache
from apiclient.discovery import build
from oauth2client.client import SignedJwtAssertionCredentials
import gdata.spreadsheets.client
import gdata.spreadsheet.service
# AppAssertionCredentials is not supported in gdata python client library,
# so we use SignedJwtAssertionCredentials with the credential
# file of this service account.
# Load the key in PKCS 12 format that you downloaded from the Google API
# Console when you created your Service account.
clientEmail = '10346........-g3dp......................3m1em8#developer.gserviceaccount.com'
p12File = 'app.p12'
path = os.path.join(ROOT_DIR, 'data', 'oAuth2', p12File)
impersonateUser = 'user#mygoogleappsdomain.com'
spreadsheetKey = '1mcJHJ...................................juQMw' # ID copied from URL of desired spreadsheet in Google Drive
with open(path) as f:
privateKey = f.read()
f.close()
# Getting credentials with AppAssertionCredentials only worked successfully
# for Google API Client Library for Python, e.g. accessing file's meta-data.
# So we use SignedJwtAssertionCredentials, as suggested in
# http://stackoverflow.com/questions/16026286/using-oauth2-with-service-account-on-gdata-in-python
# but with-out the sub parameter!
credentials = SignedJwtAssertionCredentials(
clientEmail,
privateKey,
scope=(
'https://www.googleapis.com/auth/drive.file ',
# added the scope above as suggested somewhere else,
# but error occurs with and with-out this scope
'https://www.googleapis.com/auth/drive',
'https://spreadsheets.google.com/feeds',
'https://docs.google.com/feeds'
),
# sub=impersonateUser
)
http = httplib2.Http()
http = credentials.authorize(http)
auth2token = gdata.gauth.OAuth2TokenFromCredentials(credentials)
# this pattern would eventually also work using SpreadsheetsService()
# SpreadsheetsService methods are different from SpreadsheetsClient, though
#srv = gdata.spreadsheet.service.SpreadsheetsService()
#srv = auth2token.authorize(srv)
clt = gdata.spreadsheets.client.SpreadsheetsClient()
clt = auth2token.authorize(clt)
wks = clt.get_worksheets(spreadsheetKey)
# works now!