So I was following a tutorial to create the credentials using Oauth2 and connect python to GCP, I've made the following code that successfully calls the auth window in google.
import os
import pickle
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
from google.cloud import bigquery
credentials = None
#token.pickle stores user credentials from prev successful login
if os.path.exists("token.pickle"):
print("Loading Credentials from file...")
with open("token.pickle","rb") as token:
credentials = pickle.load(token)
# if theres not a valid token either refresh or login
if not credentials or not credentials.valid:
if credentials and credentials.expired and credentials.refresh_token:
print("Refreshing Access Token...")
credentials.refresh(Request())
else:
print("Fetching New Tokens...")
flow = InstalledAppFlow.from_client_secrets_file(
"client_secrets.json",
scopes=["https://www.googleapis.com/auth/cloud-platform"],
)
flow.run_local_server(
port=8080, prompt="consent", authorization_prompt_message=""
)
credentials = flow.credentials
# saving credentials for future runs
with open("token.pickle","rb") as f:
print("Saving Credentials for future...")
pickle.dump(credentials,f)
However when I try to connect to Bigquery using this:
client = bigquery.Client(credentials=credentials)
Its telling me the following, the 3 dots have all the data from the token however I deleted it due to security reason:
google.auth.exceptions.DefaultCredentialsError: File {...} was not found.
Related
[Updated]
I am trying to use Google Spreadsheet API to automate our reporting process (which is being handled manually). I have created Service Account and downloaded a json file.
from __future__ import print_function
import google.auth
from googleapiclient.discovery import build
from googleapiclient.errors import HttpError
def create(title):
"""
Creates the Sheet the user has access to.
Load pre-authorized user credentials from the environment.
TODO(developer) - See https://developers.google.com/identity
for guides on implementing OAuth2 for the application.
"""
creds, _ = google.auth.default()
# pylint: disable=maybe-no-member
try:
service = build('sheets', 'v4', credentials=creds)
spreadsheet = {
'properties': {
'title': title
}
}
spreadsheet = service.spreadsheets().create(body=spreadsheet,
fields='spreadsheetId') \
.execute()
print(f"Spreadsheet ID: {(spreadsheet.get('spreadsheetId'))}")
return spreadsheet.get('spreadsheetId')
except HttpError as error:
print(f"An error occurred: {error}")
return error
if __name__ == '__main__':
# Pass: title
create("mysheet1")
Here is the result:
TransportError: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Enginemetadata service. Status: 404 Response:\nb''", <google_auth_httplib2._Response object at 0x7f262a34fd90>)
It's strange that there is no code lines to receive access to Google Work Space (like connect to API using keys?)
AFTER SETTING UP OAUTH CLIENTS ID
Now I am facing new problem. After running the code, It asked me to "Please visit this URL to authorize this application: with a link", I clicked and got this message: "This site can’t be reachedlocalhost refused to connect".(I am running the code on Colab)
Have you enabled the spreadsheet API ?
https://support.google.com/googleapi/answer/6158841?hl=en
I suspect it's because you have not given it the scope:
from google.oauth2.credentials import Credentials
from google_auth_oauthlib.flow import InstalledAppFlow
SCOPES = ['https://www.googleapis.com/auth/spreadsheets.readonly']
def main():
flow = InstalledAppFlow.from_client_secrets_file(
'credentials.json', SCOPES)
creds = flow.run_local_server(port=0)
service = build('sheets', 'v4', credentials=creds)
Where credentials.json is your file
Scopes: https://developers.google.com/identity/protocols/oauth2/scopes#sheets
You will also need to download the gcloud cli:
If you ever try to run it locally, on your windows/linux/mac machine.
default credentials are the credentials stored by either the default service account in google cloud or your local machine
https://cloud.google.com/sdk/gcloud
And run
gcloud auth login
What we want to solve
access the shared drive with a user account by OAuth authentication
retrieve spreadsheet -> convert to parquet type 3. save to GCS
save to GCS
These processes are written in the main() function below, and I would like to apply them to periodic processing every day using CloudFunction and CloudScheduler.
However, as it is, the code below requires the user to manually log in to his/her Google account by going to the browser.
I would like to rewrite the code so that this login can be done automatically, but I am having trouble understanding it...
I would appreciate it if someone could help me...
Translated with www.DeepL.com/Translator (free version)
### ※※Authentication is required by browser※※
creds = flow.run_local_server(port=0)
### Result
Please visit this URL to authorize this application:
https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=132987612861-
4j24afrouontpeiv5ryy7sn64inhr.apps.googleusercontent.com&redirect_uri=
http%yyy%2Flocalhost%3yy6%2F&scope=httpsyyF%2Fwww.googleapis.com%2Fauth%2Fdrive.
readonly&state=XXXXXXXXXXXXXXXXXXXXXXXXXXX&access_type=offline
The readonly&state=XXXXXXXXXXXXXXXXXXXXX part changes with each execution.
Browser screen that transitions when the above code section is executed
The entire relevant source code
from __future__ import print_function
import io
import os
import key
import json
import os.path
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
from pprint import pprint
from webbrowser import Konqueror
from google.cloud import storage as gcs
from google.oauth2 import service_account
from google.auth.transport.requests import Request
from google.oauth2.credentials import Credentials
from google_auth_oauthlib.flow import InstalledAppFlow
from googleapiclient.http import MediaIoBaseDownload, MediaIoBaseUpload, MediaFileUpload
from googleapiclient.discovery import build
from googleapiclient.errors import HttpError
SCOPES = ['https://www.googleapis.com/auth/drive.readonly']
def main(event, context):
"""Drive v3 API
Function to access shared Drive→get Spreadsheet→convert to parquet→upload to GCS """
creds = None
file_id = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxx' #Unedited data in shared drive
mime_type = 'text/csv'
# OAuth authentication to access shared drives
if os.path.exists('token.json'):
creds = Credentials.from_authorized_user_file('token.json', SCOPES)
# Allow users to log in if there are no (valid) credentials available if not creds or not creds.valid:
if creds and creds.expired and creds.refresh_token:
creds.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file(
'credentials.json', SCOPES)
### ※※Browser authentication required※※
creds = flow.run_local_server(port=0)##Currently, we need a manual login here!
with open('token.json', 'w') as token:
token.write(creds.to_json())
try:
# Retrieve spreadsheets from shared drives
service = build('drive', 'v3', credentials=creds)
request = service.files().export_media(fileId=file_id, mimeType=mime_type)
fh = io.BytesIO()
downloader = MediaIoBaseDownload(fh, request)
done = False
print(io.StringIO(fh.getvalue().decode()))
while done is False:
status, done = downloader.next_chunk()
# Read "Shared Drive/SpreadSheet" -> convert to parquet
df = pd.read_csv(io.StringIO(fh.getvalue().decode()))
table = pa.Table.from_pandas(df)
buf = pa.BufferOutputStream()
pq.write_table(table, buf,compression=None)
# service_account for save to GCS
key_path = 'service_account_file.json'
service_account_info = json.load(open(key_path))
credentials = service_account.Credentials.from_service_account_info(service_account_info)
client = gcs.Client(
credentials=credentials,
project=credentials.project_id,
)
# GCS information to be saved
bucket_name = 'bucket-name'
blob_name = 'sample-folder/daily-data.parquet'#save_path
bucket = client.get_bucket(bucket_name)
blob = bucket.blob(blob_name)
# parquet save to GCS
blob.upload_from_string(data=buf.getvalue().to_pybytes())
# ↓If a print appears, the data has been saved.
print("Blob '{}' created to '{}'!".format(blob_name, bucket_name))
except HttpError as error:
# TODO(developer) - Handle errors from drive API.
print(f'An error occurred: {error}')
What I tried by myself
I tried to use selenium to run the browser, but could not implement it well because the browser login URL is different each time. ←I may be able to find a way to do it.
Try this approach. Worked for me!
The solution consists in create a service account and share your data folder with the SA e-mail.
Drive API
Service account
this is my first contribution here.
I'm trying to access Gmail through a python script. To do so, I've created a Google Apps Script function and used the Apps Script API between the 2.
(This doc displays what I'm trying to do)
So the python script correctly accesses the API, but fails to execute the function.
While it works in the Script Editor, in Python it raises a permissions issue:
'errorMessage': 'Exception: The script does not have permission to perform that action.
Required permissions: (
https://www.googleapis.com/auth/gmail.labels ||
https://www.googleapis.com/auth/gmail.metadata ||
https://www.googleapis.com/auth/gmail.readonly ||
https://www.googleapis.com/auth/gmail.modify ||
https://mail.google.com/
)',
'errorType': 'ScriptError'
I guess it is related to the Client ID OAuth, since I was not able to find where to grant it permissions. I've just :
created the credentials in Google Cloud Platform,
exported it as creds.json in my python script folder.
Here is my code, almost copy pasted from a tutorial:
import pickle
import os.path
from googleapiclient import errors
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
# Here I've edited the scopes of authorizations required
SCOPES = [
"https://www.googleapis.com/auth/gmail.labels",
"https://www.googleapis.com/auth/gmail.metadata",
"https://www.googleapis.com/auth/gmail.readonly",
"https://www.googleapis.com/auth/gmail.modify",
"https://mail.google.com/"
]
def get_scripts_service():
creds = None
# The file token.pickle stores the user's access and refresh tokens, and is
# created automatically when the authorization flow completes for the first
# time.
if os.path.exists('token.pickle'):
with open('token.pickle', 'rb') as token:
creds = pickle.load(token)
if not creds or not creds.valid:
if creds and creds.expired and creds.refresh_token:
creds.refresh(Request())
else:
# Here I've placed the downloaded credentials .json file
flow = InstalledAppFlow.from_client_secrets_file(
'creds.json', SCOPES)
creds = flow.run_local_server(port=0)
with open('token.pickle', 'wb') as token:
pickle.dump(creds, token)
return build('script', 'v1', credentials=creds)
service = get_scripts_service()
API_ID = # Here i've pasted my API_ID
request = {"function": # Here i've pasted my functionName}
try:
response = service.scripts().run(body=request, scriptId=API_ID).execute()
print (response)
except errors.HttpError as error:
# The API encountered a problem.
print(error.content)
How do I grant permissions to my script?
Simple as Aerials said! Thanks.
It was because the Client ID was created before I edited the scopes. I've deleted the token and created a new one.
Trying to get OAuth2 Google login working, this is the raw request that my app makes:
Method: POST
URL: https://www.googleapis.com/oauth2/v3/token
Headers: Content-Type: application/x-www-form-urlencoded
Values:
client_id: XXX-0123456789abcdef0123456789abcdef.apps.googleusercontent.com
client_secret: A1b2C3d4E5f6G7h8I9j0K1l2M
code: 1/A1b2C3d4E5f6G7h8I9j0K1l2M3n4O5p6Q7r8S9t0U1v
grant_type: authorization_code
redirect_uri: http://localhost:5000/callback/google/
And this is the response:
Status: 401 Unauthorized
Body:
{
"error": "invalid_client",
"error_description": "Unauthorized"
}
Have verified that this is the exact request / response that my app is making (a Python app using Flask and rauth), and have verified that I can reproduce the exact same request / response using Postman.
Per instructions in other threads, I have done all of the following in the Google APIs console:
In "OAuth consent screen" settings, set "Product name" to something different than "Project name"
Also in "OAuth consent screen" settings, double-check that email is set
Enable the Google+ API
Enable the Gmail API
Recreate the client ID / secret
Double-check that there are no leading or trailing spaces in the client ID / secret values, I have copied them correctly from the API console
No matter what I do, still getting the same response of "invalid_client": "Unauthorized".
Help with this would be appreciated. Am trying to set up OAuth2-powered "Log in with X" functionality in my app, have gotten Facebook and Twitter working without issues, would like to get Google working too, but if I can't resolve this then I'm afraid I'll have to ditch Google auth.
Invalid client means that the client id or the client secret that you are using are not valid. They must be the ones you have downloaded from Google Developer console.
Tip: You might want to consider using the Google python client library it does all the heavy lifting for you.
from __future__ import print_function
import pickle
import os.path
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
# If modifying these scopes, delete the file token.pickle.
SCOPES = ['https://www.googleapis.com/auth/drive.metadata.readonly']
def main():
"""Shows basic usage of the Drive v3 API.
Prints the names and ids of the first 10 files the user has access to.
"""
creds = None
# The file token.pickle stores the user's access and refresh tokens, and is
# created automatically when the authorization flow completes for the first
# time.
if os.path.exists('token.pickle'):
with open('token.pickle', 'rb') as token:
creds = pickle.load(token)
# If there are no (valid) credentials available, let the user log in.
if not creds or not creds.valid:
if creds and creds.expired and creds.refresh_token:
creds.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file(
'credentials.json', SCOPES)
creds = flow.run_local_server(port=0)
# Save the credentials for the next run
with open('token.pickle', 'wb') as token:
pickle.dump(creds, token)
service = build('drive', 'v3', credentials=creds)
# Call the Drive v3 API
results = service.files().list(
pageSize=10, fields="nextPageToken, files(id, name)").execute()
items = results.get('files', [])
if not items:
print('No files found.')
else:
print('Files:')
for item in items:
print(u'{0} ({1})'.format(item['name'], item['id']))
if __name__ == '__main__':
main()
Refresh Cliente Secret File on https://console.cloud.google.com/apis/credentials and delete token pickle folder
I am trying to use the following code to insert videos using youtube's API. I have generated my client secret file, however, could not understand how the oauth2.json file is generated.
import httplib2
import os
import sys
from apiclient.discovery import build
from apiclient.errors import HttpError
from oauth2client.client import flow_from_clientsecrets
from oauth2client.file import Storage
from oauth2client.tools import argparser
from oauth2client import tools
# The CLIENT_SECRETS_FILE variable specifies the name of a file that contains
# the OAuth 2.0 information for this application, including its client_id and
# client_secret. You can acquire an OAuth 2.0 client ID and client secret from
CLIENT_SECRETS_FILE = "my_client_secret.json"
# This variable defines a message to display if the CLIENT_SECRETS_FILE is
# missing.
MISSING_CLIENT_SECRETS_MESSAGE = """
WARNING: Please configure OAuth 2.0
To make this sample run you will need to populate the client_secrets.json
file
found at:
%s
with information from the Cloud Console
https://cloud.google.com/console
For more information about the client_secrets.json file format, please
visit:
https://developers.google.com/api-client-
library/python/guide/aaa_client_secrets
""" % os.path.abspath(os.path.join(os.path.dirname(__file__),
CLIENT_SECRETS_FILE))
# This OAuth 2.0 access scope allows for full read/write access to the
# authenticated user's account.
YOUTUBE_SCOPE = "https://www.googleapis.com/auth/youtube"
YOUTUBE_API_SERVICE_NAME = "youtube"
YOUTUBE_API_VERSION = "v3"
def get_authenticated_service():
flow = flow_from_clientsecrets(CLIENT_SECRETS_FILE, scope=YOUTUBE_SCOPE,
message=MISSING_CLIENT_SECRETS_MESSAGE)
storage = Storage("%s-oauth2.json" % sys.argv[0])
credentials = storage.get()
if credentials is None or credentials.invalid:
credentials = tools.run_flow(flow, storage)
return build(YOUTUBE_API_SERVICE_NAME, YOUTUBE_API_VERSION,
http=credentials.authorize(httplib2.Http()))
def add_video_to_playlist(youtube,videoID,playlistID):
add_video_request=youtube.playlistItem().insert(
part="snippet",
body={
'snippet': {
'playlistId': playlistID,
'resourceId': {
'kind': 'youtube#video',
'videoId': videoID
}
#'position': 0
}
}).execute()
if __name__ == '__main__':
youtube = get_authenticated_service()
add_video_to_playlist(youtube,"yszl2oxi8IY","PL2JW1S4IMwYubm06iDKfDsmWVB-J8funQ")
How do I generate the OATUH2.JSON file?
The file <yourscript>-oauth2.json is used to store credentials (access token, refresh token, id token, token expiration date etc...)
From google API client guide :
The oauth2client.file.Storage class stores and retrieves a single
Credentials object. The class supports locking such that multiple
processes and threads can operate on a single store.
If the file doesn't exist or the credentials in it are invalid the authentication flow is run :
if credentials is None or credentials.invalid:
credentials = tools.run_flow(flow, storage)
Thus, if this file already exists and contains valid credentials, it won't be necessary to request an authorization code (through user registration and user accepting scope) to get a first access token since it will use the existing access token/refresh token in this file to hit Google API.
You can read more about oauth2client.tools.run_flow here