I want to transfer the ETL scripts (I used to run them on a local machine) to the google cloud, using the Google Functions + Google Scheduler.
Scripts move data from/to Google Analytics (this is an example).
I have a problem with the location of the key file (.p12).
I would like to put it in Google Storage and point the way to it.
Currently KEY_FILE_LOCATION = r'c:/local_path/file.p12'.
Connect to google analytics:
def initialize_analyticsreporting():
credentials = ServiceAccountCredentials.from_p12_keyfile(
SERVICE_ACCOUNT_EMAIL, KEY_FILE_LOCATION, scopes=SCOPES)
http = credentials.authorize(httplib2.Http())
analytics = build('analytics', 'v4', http=http, discoveryServiceUrl=DISCOVERY_URI)
return analytics
I would like to use
from google.cloud import storage
client = storage.Client ()
bucket = client.get_bucket ('utk_owox')
.....
But I can not understand how to do it correctly.
Please be aware that Cloud Functions API client libraries that use application default credentials automatically obtain the built-in service account credentials from the Cloud Functions host at runtime. By default, the client authenticates using the YOUR_PROJECT_ID#appspot.gserviceaccount.com service account.
So if you can configure this service account to have appropriate permissions for the resources that you need to access, it is recommended to relay on it.
Example how to upload file to the bucket in python.
Example how to download file from the bucket in python:
def download_blob(bucket_name, source_blob_name, destination_file_name):
"""Downloads a blob from the bucket."""
storage_client = storage.Client()
bucket = storage_client.get_bucket(bucket_name)
blob = bucket.blob(source_blob_name)
blob.download_to_filename(destination_file_name)
print('Blob {} downloaded to {}.'.format(
source_blob_name,
destination_file_name))
Create credentials from the file as described here:
def get_ga_service(bucket):
download_gcs_file('auth.json', '/tmp/auth.json', bucket)
credentials = ServiceAccountCredentials.from_json_keyfile_name(
'/tmp/auth.json',
scopes=['https://www.googleapis.com/auth/analytics',
'https://www.googleapis.com/auth/analytics.edit'])
# Build the service object.
return build('analytics', 'v3', credentials=credentials, cache_discovery=False)
Related
Requirement :
I need to execute a script from a remote server which download a particular file from google cloud storage.
Script should use a service account where key should be in hashicorp vault.
i have already a hashicorp vault setup established
but not sure how to invoke it from shell/python script
so if some can help me in how to download file from GCS by using service account key in hashicorp vault
either python/shell script would be fine
you can follow the google documentation to download an objects using:
Console
gsutil
Programming Language Script
curl + rest api
for more information visit this Link
For a python utility you can use this code
from google.cloud import storage
def download_blob(bucket_name, source_blob_name, destination_file_name):
"""Downloads a blob from the bucket."""
# bucket_name = "your-bucket-name"
# source_blob_name = "storage-object-name"
# destination_file_name = "local/path/to/file"
storage_client = storage.Client()
bucket = storage_client.bucket(bucket_name)
# Construct a client side representation of a blob.
# Note `Bucket.blob` differs from `Bucket.get_blob` as it doesn't retrieve
# any content from Google Cloud Storage. As we don't need additional data,
# using `Bucket.blob` is preferred here.
blob = bucket.blob(source_blob_name)
blob.download_to_filename(destination_file_name)
print(
"Blob {} downloaded to {}.".format(
source_blob_name, destination_file_name
)
)
Google Drive API docs are not super great at helping determine best way to authenticate using a service account that I can then upload a .png file to the Drive. My end goal it so upload a .png file, copy a template doc, batch update that doc using text replace, and insert the newly uploaded .png image into that doc.
Sample code below:
from dotenv import load_dotenv
from google.oauth2 import service_account
from googleapiclient.discovery import build
from googleapiclient.http import MediaFileUpload
def credentials_from_file():
credentials = service_account.Credentials.from_service_account_file(
os.getenv('SERVICE_ACCOUNT_FILE'),
scopes=os.getenv('SCOPES')
)
drive_service = build('drive', 'v3', credentials=credentials)
return drive_service
def google_upload(drive_service, metadata_name, parents, permissions, file_path, mime_type):
file_metadata = {'kind':'drive#file', 'name':metadata_name, 'parents':parents, 'permissions':permissions}
media = MediaFileUpload(file_path, mimetype=mime_type)
file = drive_service.files().create(body=file_metadata, media_body=media, fields='id', supportsAllDrives=True).execute()
print('File ID: %s' % file.get('id'))
Implementation of Code
credentials = credentials_from_file()
drive_service = build('drive', 'v3', credentials=credentials)
metadata_name = custom_variables_png_table_img
parents = ['xxxx']
permissions = [{'kind':'drive#permission', 'emailAddress':os.getenv('EMAIL_ACCOUNT'), 'role':'owner'}]
file_path = custom_variables_png_table_img
mime_type = 'image/png'
google_upload(drive_service, metadata_name, parents, permissions, file_path, mime_type)
EDIT:
Looks like I forgot to actually write was the problem is. It's two fold.
I keep getting 2 errors when trying to run the google_upload() function which looks like an authentication error with the service account.
Error #1: jwt_grant access_token = response_data["access_token"] KeyError: 'access_token'
The above exception was the direct cause of the following exception:
Error #2: google.auth.exceptions.RefreshError: ('No access token in response.', {'id_token': 'xxx'})
Permissions being properly set on the recently uploaded image file.
The code you are using currently seams to be the same as what I have seen before.
from apiclient.discovery import build
from oauth2client.service_account import ServiceAccountCredentials
SCOPES = ['https://www.googleapis.com/auth/drive.readonly']
KEY_FILE_LOCATION = '<REPLACE_WITH_JSON_FILE_PATH_TO_FILE>'
def initialize_drive():
"""Initializes an Google Drive API V3 service object.
Returns:
An authorized Google Drive API V3 service object.
"""
credentials = ServiceAccountCredentials.from_json_keyfile_name(
KEY_FILE_LOCATION, SCOPES)
# Build the service object.
driveService = build('drive', 'v4', credentials=credentials)
return driveService
You haven't mentioned what is wrong with your code however i can make a few guesses.
The thing is that you mention you want to upload an image and the insert it into a document. You need to remember that the Google drive api is just a file storage api it can do more then that store files.
When you upload the file using the service account you need to remember that the service account is not you. So when you are uploading this file to this directory parents = ['xxxx'] where ever that directory is, either on the service accounts drive account or if this directory is one of your persona directories which you have shared with the service account. You may not have permissions to see this file.
By calling permissions create after uploading your file you can grant your own personal account permissions to access the file as well.
As for adding the image to a document. well the only way google can help you with that is if it is a Google doc type document. Then you would need to go though the Google docs api which would then give you access to add things programmaticlly to a document. I haven't used this API much so im not sure if it has the ability to add images to a document.
You should be able to use the google docs api with your service account you will just need to create a docs service using the same creds you already have from google drive.
service = build('docs', 'v1', credentials=creds)
I am trying to create a simple web application that automatically uploads my database & media backup to a designated google drive. I have followed the official document and created a service account credential, gave it the owner role, and extracted a key(json file) from Google cloud platform. I enabled the Google Drive API on my account and wrote this code, but the credentials.valid returns False and my file would not upload to my drive.
from google.oauth2 import service_account
import googleapiclient as google
from googleapiclient.http import MediaFileUpload, HttpRequest
from googleapiclient.discovery import build
SCOPES = ['https://www.googleapis.com/auth/drive']
credentials = service_account.Credentials.from_service_account_file('./service-credentials.json', scopes=SCOPES)
print(credentials.valid)
service = build('drive', 'v3', credentials=credentials)
file_metadata = {'name' : 'python.png'}
media = MediaFileUpload('./python.png', mimetype='image/png')
file_up = service.files().create(body=file_metadata, media_body=media, fields='id').execute()
file_back = service.files().get(fileId=file_up['id']).execute()
print(file_back.get('WebContentLink'))
How about this modification?
Modification points:
I think that in your script, service of service = build('drive', 'v3', credentials=credentials) can be used for uploading the file.
In my environment, I could confirm that the file can be uploaded using your script.
From my file would not upload to my drive., I thought that you might misunderstand about the service account. The file uploaded with the service account is created to the Drive of the service account. This Drive is different from your Google Drive of your account. I thought that this might be the reason of my file would not upload to my drive..
If you want to see the file uploaded with the service account at your Google Drive, it is required to share the uploaded file with your Google account. Or, it is required to upload the file to the folder in your Google Drive shared with the service account.
And also, in your script, file_back.get('WebContentLink') is used. In this case, None is always returned because WebContentLink is required to be WebContentLink. And also, in Drive API v3, the default returned values don't include webContentLink. So it is required to set fields.
When above points are reflected to your script, your script becomes as follows.
Modified script:
from google.oauth2 import service_account
import googleapiclient as google
from googleapiclient.http import MediaFileUpload, HttpRequest
from googleapiclient.discovery import build
SCOPES = ['https://www.googleapis.com/auth/drive']
credentials = service_account.Credentials.from_service_account_file('./service-credentials.json', scopes=SCOPES)
service = build('drive', 'v3', credentials=credentials)
file_metadata = {'name': 'python.png'}
media = MediaFileUpload('./python.png', mimetype='image/png')
file_up = service.files().create(body=file_metadata, media_body=media, fields='id').execute()
# Create a permission. Here, your Google account is shared with the uploaded file.
yourEmailOfGoogleAccount = '###' # <--- Please set your Email address of Google account.
permission = {
'type': 'user',
'role': 'writer',
'emailAddress': yourEmailOfGoogleAccount,
}
service.permissions().create(fileId=file_up['id'], body=permission).execute()
file_back = service.files().get(fileId=file_up['id'], fields='webContentLink').execute() # or fields='*'
print(file_back.get('webContentLink'))
When you run above script, the uploaded file can be seen at "Shared with me" in your Google Drive.
If you want to put the specific folder of your Google Drive, please use the following script. In this case, before you run the script, please share the folder with the email of the service account. Please be careful this.
from google.oauth2 import service_account
import googleapiclient as google
from googleapiclient.http import MediaFileUpload, HttpRequest
from googleapiclient.discovery import build
SCOPES = ['https://www.googleapis.com/auth/drive']
credentials = service_account.Credentials.from_service_account_file('./service-credentials.json', scopes=SCOPES)
service = build('drive', 'v3', credentials=credentials)
file_metadata = {'name': 'python.png', 'parents': ['###']} # <--- Please set the folder ID shared with the service account.
media = MediaFileUpload('./python.png', mimetype='image/png')
file_up = service.files().create(body=file_metadata, media_body=media, fields='id').execute()
file_back = service.files().get(fileId=file_up['id'], fields='webContentLink').execute() # or fields='*'
print(file_back.get('webContentLink'))
Note:
In the current stage, when the owner of file uploaded with the service account is changed, an error like You can't yet change the owner of this item. (We're working on it.). So I proposed above modified script.
References:
Files: create
Files: get
Permissions: create
I don't know what this is so confusing for me. I have the following code to access the gmail api from a cloud function. It works. The issue is that I am using a json file of credentials for the app engine service account. But, this is the default service account for cloud functions, so shouldn't I be able to somehow accomplish the same task without needing to use the credential files?
credentials = service_account.Credentials.from_service_account_info(SERVICE_ACCOUNT_FILE, scopes=SCOPES)
delegated_credentials = credentials.with_subject("my.email#email.com")
service = build('gmail', 'v1', credentials=delegated_credentials, cache_discovery=False)
I have tried
from google.auth import compute_engine
credentials = compute_engine.Credentials()
service = build('gmail', 'v1', credentials=credentials, cache_discovery=False)
We are creating files using the following authentication
self.GOOGLESHEETS_SCOPES =
['https://www.googleapis.com/auth/drive.file',
'https://www.googleapis.com/auth/drive',
'https://www.googleapis.com/auth/spreadsheets']
self.credentials = GoogleCredentials.get_application_default()
self.credentials = GoogleCredentials.get_application_default().create_scoped(self.GOOGLESHEETS_SCOPES)
self.credentials = AppAssertionCredentials(self.GOOGLESHEETS_SCOPES)
self.service = build('sheets', 'v4', credentials=self.credentials)
Can we access these files using the Google Drive GUI? I cannot find any documentation that confirms or denies this possibility
If you're using the regular Sheets service you can create the file with the create method of the API and access it in the Drive UI.