I've written the following code to upload an image to my own Google Drive using a service account.
My code is returning successfully, giving me an ID back but there's nothing appearing on my actual Google Drive.
from django.conf import settings
import os
from apiclient.discovery import build
from apiclient.http import MediaFileUpload
from oauth2client.service_account import ServiceAccountCredentials
def get_service(api_name, api_version, scope, key_file_location):
credentials = ServiceAccountCredentials.from_json_keyfile_name(key_file_location, scopes=scope)
service = build(api_name, api_version, credentials=credentials)
return service
def setup_upload():
scope = ['https://www.googleapis.com/auth/drive']
key_file_location = os.path.join(os.path.dirname(settings.BASE_DIR), 'common/my-json-file.json')
service = get_service('drive', 'v3', scope, key_file_location)
file_path = os.path.join(os.path.dirname(settings.BASE_DIR), 'common/apple.png')
file_metadata = {'name': 'apple.png'}
media = MediaFileUpload(file_path, mimetype="image/png")
file = service.files().create(body=file_metadata, media_body=media, fields='id').execute()
print(file.get('id')) #this returns an actual ID
setup_upload()
I do get a long ID string back from the last line of setup_upload(). But nothing is appearing on my actual Google Drive. I'm expecting to see the apple.png file pop up in my home directory.
What am I missing here?
Quoting from the tutorial you linked to:
"You can take the service account email address and give it access to a directory on your Google drive. It will then be allowed to upload to that directory, but you wont have access to the files. You will need to complete a second step and give yourself personally permission to access those files by updating or patching the file permissions."
Have you given the service account access to the place where you want to write the file? If you haven't specified where to upload the file to, the service account may just be uploading the file into its own Drive.
In addition to #user2705223's answer, if you want to be able to access the files it uploads, then you must grant yourself access to them through the service account. Check if you had a successful login credentials and authorize the service account with right scope. You can try following this documentation to help you do the authorization to make API requests.
Related
I want to find a free cloud storage service with free API, that could help me back up some files automatically.
I want to write some script (for example python) to upload files automatically.
I investigated OneDrive and GoogleDrive. OneDrive API is not free, GoogleDrive API is free while it need human interactive authorization before using API.
For now I'm simply using email SMTP protocol to send files as email attachments, but there's a max file size limition, which will fail me in the future, as my file size is growing.
Is there any other recommendations ?
I believe your goal as follows.
You want to upload a file using Drive API with the service account.
You want to achieve your goal using python.
At first, in your situation, how about using google-api-python-client? In this answer, I would like to explain the following flow and the sample script using google-api-python-client.
Usage:
1. Create service account.
Please create the service account and download a JSON file. Ref
2. Install google-api-python-client.
In order to use the sample script, please install google-api-python-client.
$ pip install google-api-python-client
3. Prepare a folder.
Please create a new folder in your Google Drive. And, please share the created folder with the email of your service account. Because the Google Drive of your account is different from the Drive of service account. By sharing the folder with the service account, the file can be uploaded to the folder in your Google Drive using the service account. By this, you can see the uploaded file on your Google Drive by your browser.
4. Prepare sample script.
Please set the filename of credentials of service account, the filename of the file you want to upload and the folder ID of folder you shared your folder with the service account to the variables of SERVICE_ACCOUNT, UPLOAD_FILE and FOLDER_ID, respectively.
from oauth2client.service_account import ServiceAccountCredentials
from googleapiclient.discovery import build
from googleapiclient.http import MediaFileUpload
SERVICE_ACCOUNT = '###' # Please set the file of your credentials of service account.
UPLOAD_FILE = 'sampleFilename' # Please set the filename with the path you want to upload.
FOLDER_ID = '###' # Please set the folder ID that you shared your folder with the service account.
FILENAME = 'sampleFilename' # You can set the filename of the uploaded file on Google Drive.
SCOPES = ['https://www.googleapis.com/auth/drive']
credentials = ServiceAccountCredentials.from_json_keyfile_name(SERVICE_ACCOUNT, SCOPES)
drive = build('drive', 'v3', credentials=credentials)
metadata = {'name': FILENAME, "parents": [FOLDER_ID]}
file = MediaFileUpload(UPLOAD_FILE, resumable=True)
response = drive.files().create(body=metadata, media_body=file).execute()
fileId = response.get('id')
print(fileId) # You can see the file ID of the uploaded file.
When you run this script, the file is uploaded to the shared folder in your Google Drive.
When you set the mimeType of the file you want to use, please modify file = MediaFileUpload(UPLOAD_FILE, resumable=True) to file = MediaFileUpload(UPLOAD_FILE, mimeType='###', resumable=True).
References:
google-api-python-client
Creating a service account
Upload file data
gdownload.py using Python3
from apiclient.http import MediaIoBaseDownload
from apiclient.discovery import build
from httplib2 import Http
from oauth2client import file, client, tools
import io,os
CLIENT_SECRET = 'client_secrets.json'
SCOPES = ['https://www.googleapis.com/auth/admin.datatransfer','https://www.googleapis.com/auth/drive.appfolder','https://www.googleapis.com/auth/drive']
store = file.Storage('tokenWrite.json')
creds = store.get()
if not creds or creds.invalid:
flow = client.flow_from_clientsecrets(CLIENT_SECRET, SCOPES)
flags = tools.argparser.parse_args(args=[])
creds = tools.run_flow(flow, store, flags)
DRIVE = build('drive', 'v2', http=creds.authorize(Http()))
files = DRIVE.files().list().execute().get('items', [])
def download_file(filename,file_id):
#request = DRIVE.files().get(fileId=file_id)
request = DRIVE.files().get_media(fileId=file_id)
fh = io.BytesIO()
downloader = MediaIoBaseDownload(fh, request,chunksize=-1)
done = False
while done is False:
status, done = downloader.next_chunk()
print("Download %d%%." % int(status.progress() * 100))
fh.seek(0)
f=open(filename,'wb')
f.write(fh.read())
f.close()
rinput = vars(__builtins__).get('raw_input',input)
fname=rinput('enter file name: ')
for f in files:
if f['title'].encode('utf-8')==fname:
print('downloading...',f['title'])
download_file(f['title'],f['id'])
os._exit(0)
I have a script to export text from a GDrive file using an OAuth client, which works perfectly well -
import googleapiclient.discovery as google
from apiclient.http import MediaIoBaseDownload
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
import datetime, io, os, pickle
Scopes=" ".join(['https://www.googleapis.com/auth/drive.file',
'https://www.googleapis.com/auth/drive.metadata',
'https://www.googleapis.com/auth/drive.readonly'])
TokenFile="token.pickle"
def init_creds(clientfile,
scopes,
tokenfile=TokenFile):
token=None
if os.path.exists(tokenfile):
with open(tokenfile, 'rb') as f:
token=pickle.load(f)
if (not token or
not token.valid or
token.expiry < datetime.datetime.utcnow()):
if (token and
token.expired and
token.refresh_token):
token.refresh(Request())
else:
flow=InstalledAppFlow.from_client_secrets_file(clientfile, scopes)
token=flow.run_local_server(port=0)
with open(tokenfile, 'wb') as f:
pickle.dump(token, f)
return token
def export_text(id,
clientfile,
scopes=Scopes):
creds=init_creds(clientfile=clientfile,
scopes=scopes)
service=google.build('drive', 'v3', credentials=creds)
request=service.files().export_media(fileId=id,
mimeType='text/plain')
buf=io.BytesIO()
downloader, done = MediaIoBaseDownload(buf, request), False
while done is False:
status, done = downloader.next_chunk()
destfilename="tmp/%s.txt" % id
return buf.getvalue().decode("utf-8")
if __name__=='__main__':
print (export_text(id="#{redacted}"
clientfile="/path/to/oath/client.json"))
But it's a pain to have to go through the OAuth flow every time, and since it's only me using the script I want to simplify things and use a Service Account instead, following on from this post -
Google Drive API Python Service Account Example
My new Service Account script, doing exactly the same thing, is as follows -
import googleapiclient.discovery as google
from oauth2client.service_account import ServiceAccountCredentials
from apiclient.http import MediaIoBaseDownload
import io
Scopes=" ".join(['https://www.googleapis.com/auth/drive.file',
'https://www.googleapis.com/auth/drive.metadata',
'https://www.googleapis.com/auth/drive.readonly'])
def export_text(id,
clientfile,
scopes=Scopes):
creds=ServiceAccountCredentials.from_json_keyfile_name(clientfile,
scopes)
service=google.build('drive', 'v3', credentials=creds)
request=service.files().export_media(fileId=id,
mimeType='text/plain')
buf=io.BytesIO()
downloader, done = MediaIoBaseDownload(buf, request), False
while done is False:
status, done = downloader.next_chunk()
destfilename="tmp/%s.txt" % id
return buf.getvalue().decode("utf-8")
if __name__=='__main__':
print (export_text(id="#{redacted}",
clientfile="path/to/service/account.json"))
but when I run it for the same id, I get the following -
googleapiclient.errors.HttpError: <HttpError 404 when requesting https://www.googleapis.com/drive/v3/files/#{redacted}/export?mimeType=text%2Fplain&alt=media returned "File not found: #{redacted}.">
It feels like the Service Account script is passing the authentication step (ie Service Account creds are okay) but then failing when trying to fetch the file - weird as I can fetch it fine using the OAuth version :/
Any thoughts on what might be causing this 404 error in the Service Account version, given the OAuth client version clearly works for the same id?
Answer:
You need to share your file with the service account.
More Information:
As you would with any file, you need to give a user explicit permissions to be able to see it. As a service account is a separate entitiy to you, this goes for them as well.
Using the file sharing settings (you can just do this in the Drive UI by right-clicking the file and hitting Share), give the email address of the service account the correct permission (read/write). The email address of the service account is in the form:
service-account-name#project-id.iam.gserviceaccount.com
Before making your call do a File.list to see which files the service account has access to. Doing a file.get on a file that the service account doesn't have access to will result in a file not found error. Remember that the service account is not you, it has its own google drive account. Any files you want to access need to be uploaded to its account or shared with the service account.
If the file.list fails then it would suggest to me that there is something wrong with the authorization and you should ensure that the service account has access to client file maybe its that file it cant find.
Granting service account acccess
Create a directory on your personal google drive account. Take the service account email address, it can be found in the key file you downloaded it has a # in it. Then share that directory on your drive account with the service account like you would share with any other user.
Adding files to that directory may or may not give the service account access to them automatically permissions is a pain you may need to also share the file with the service account.
Remember to have the service account grant your personal account permissions to the file when it uploads it its going to be the owner.
Google Drive API docs are not super great at helping determine best way to authenticate using a service account that I can then upload a .png file to the Drive. My end goal it so upload a .png file, copy a template doc, batch update that doc using text replace, and insert the newly uploaded .png image into that doc.
Sample code below:
from dotenv import load_dotenv
from google.oauth2 import service_account
from googleapiclient.discovery import build
from googleapiclient.http import MediaFileUpload
def credentials_from_file():
credentials = service_account.Credentials.from_service_account_file(
os.getenv('SERVICE_ACCOUNT_FILE'),
scopes=os.getenv('SCOPES')
)
drive_service = build('drive', 'v3', credentials=credentials)
return drive_service
def google_upload(drive_service, metadata_name, parents, permissions, file_path, mime_type):
file_metadata = {'kind':'drive#file', 'name':metadata_name, 'parents':parents, 'permissions':permissions}
media = MediaFileUpload(file_path, mimetype=mime_type)
file = drive_service.files().create(body=file_metadata, media_body=media, fields='id', supportsAllDrives=True).execute()
print('File ID: %s' % file.get('id'))
Implementation of Code
credentials = credentials_from_file()
drive_service = build('drive', 'v3', credentials=credentials)
metadata_name = custom_variables_png_table_img
parents = ['xxxx']
permissions = [{'kind':'drive#permission', 'emailAddress':os.getenv('EMAIL_ACCOUNT'), 'role':'owner'}]
file_path = custom_variables_png_table_img
mime_type = 'image/png'
google_upload(drive_service, metadata_name, parents, permissions, file_path, mime_type)
EDIT:
Looks like I forgot to actually write was the problem is. It's two fold.
I keep getting 2 errors when trying to run the google_upload() function which looks like an authentication error with the service account.
Error #1: jwt_grant access_token = response_data["access_token"] KeyError: 'access_token'
The above exception was the direct cause of the following exception:
Error #2: google.auth.exceptions.RefreshError: ('No access token in response.', {'id_token': 'xxx'})
Permissions being properly set on the recently uploaded image file.
The code you are using currently seams to be the same as what I have seen before.
from apiclient.discovery import build
from oauth2client.service_account import ServiceAccountCredentials
SCOPES = ['https://www.googleapis.com/auth/drive.readonly']
KEY_FILE_LOCATION = '<REPLACE_WITH_JSON_FILE_PATH_TO_FILE>'
def initialize_drive():
"""Initializes an Google Drive API V3 service object.
Returns:
An authorized Google Drive API V3 service object.
"""
credentials = ServiceAccountCredentials.from_json_keyfile_name(
KEY_FILE_LOCATION, SCOPES)
# Build the service object.
driveService = build('drive', 'v4', credentials=credentials)
return driveService
You haven't mentioned what is wrong with your code however i can make a few guesses.
The thing is that you mention you want to upload an image and the insert it into a document. You need to remember that the Google drive api is just a file storage api it can do more then that store files.
When you upload the file using the service account you need to remember that the service account is not you. So when you are uploading this file to this directory parents = ['xxxx'] where ever that directory is, either on the service accounts drive account or if this directory is one of your persona directories which you have shared with the service account. You may not have permissions to see this file.
By calling permissions create after uploading your file you can grant your own personal account permissions to access the file as well.
As for adding the image to a document. well the only way google can help you with that is if it is a Google doc type document. Then you would need to go though the Google docs api which would then give you access to add things programmaticlly to a document. I haven't used this API much so im not sure if it has the ability to add images to a document.
You should be able to use the google docs api with your service account you will just need to create a docs service using the same creds you already have from google drive.
service = build('docs', 'v1', credentials=creds)
I am trying to create a simple web application that automatically uploads my database & media backup to a designated google drive. I have followed the official document and created a service account credential, gave it the owner role, and extracted a key(json file) from Google cloud platform. I enabled the Google Drive API on my account and wrote this code, but the credentials.valid returns False and my file would not upload to my drive.
from google.oauth2 import service_account
import googleapiclient as google
from googleapiclient.http import MediaFileUpload, HttpRequest
from googleapiclient.discovery import build
SCOPES = ['https://www.googleapis.com/auth/drive']
credentials = service_account.Credentials.from_service_account_file('./service-credentials.json', scopes=SCOPES)
print(credentials.valid)
service = build('drive', 'v3', credentials=credentials)
file_metadata = {'name' : 'python.png'}
media = MediaFileUpload('./python.png', mimetype='image/png')
file_up = service.files().create(body=file_metadata, media_body=media, fields='id').execute()
file_back = service.files().get(fileId=file_up['id']).execute()
print(file_back.get('WebContentLink'))
How about this modification?
Modification points:
I think that in your script, service of service = build('drive', 'v3', credentials=credentials) can be used for uploading the file.
In my environment, I could confirm that the file can be uploaded using your script.
From my file would not upload to my drive., I thought that you might misunderstand about the service account. The file uploaded with the service account is created to the Drive of the service account. This Drive is different from your Google Drive of your account. I thought that this might be the reason of my file would not upload to my drive..
If you want to see the file uploaded with the service account at your Google Drive, it is required to share the uploaded file with your Google account. Or, it is required to upload the file to the folder in your Google Drive shared with the service account.
And also, in your script, file_back.get('WebContentLink') is used. In this case, None is always returned because WebContentLink is required to be WebContentLink. And also, in Drive API v3, the default returned values don't include webContentLink. So it is required to set fields.
When above points are reflected to your script, your script becomes as follows.
Modified script:
from google.oauth2 import service_account
import googleapiclient as google
from googleapiclient.http import MediaFileUpload, HttpRequest
from googleapiclient.discovery import build
SCOPES = ['https://www.googleapis.com/auth/drive']
credentials = service_account.Credentials.from_service_account_file('./service-credentials.json', scopes=SCOPES)
service = build('drive', 'v3', credentials=credentials)
file_metadata = {'name': 'python.png'}
media = MediaFileUpload('./python.png', mimetype='image/png')
file_up = service.files().create(body=file_metadata, media_body=media, fields='id').execute()
# Create a permission. Here, your Google account is shared with the uploaded file.
yourEmailOfGoogleAccount = '###' # <--- Please set your Email address of Google account.
permission = {
'type': 'user',
'role': 'writer',
'emailAddress': yourEmailOfGoogleAccount,
}
service.permissions().create(fileId=file_up['id'], body=permission).execute()
file_back = service.files().get(fileId=file_up['id'], fields='webContentLink').execute() # or fields='*'
print(file_back.get('webContentLink'))
When you run above script, the uploaded file can be seen at "Shared with me" in your Google Drive.
If you want to put the specific folder of your Google Drive, please use the following script. In this case, before you run the script, please share the folder with the email of the service account. Please be careful this.
from google.oauth2 import service_account
import googleapiclient as google
from googleapiclient.http import MediaFileUpload, HttpRequest
from googleapiclient.discovery import build
SCOPES = ['https://www.googleapis.com/auth/drive']
credentials = service_account.Credentials.from_service_account_file('./service-credentials.json', scopes=SCOPES)
service = build('drive', 'v3', credentials=credentials)
file_metadata = {'name': 'python.png', 'parents': ['###']} # <--- Please set the folder ID shared with the service account.
media = MediaFileUpload('./python.png', mimetype='image/png')
file_up = service.files().create(body=file_metadata, media_body=media, fields='id').execute()
file_back = service.files().get(fileId=file_up['id'], fields='webContentLink').execute() # or fields='*'
print(file_back.get('webContentLink'))
Note:
In the current stage, when the owner of file uploaded with the service account is changed, an error like You can't yet change the owner of this item. (We're working on it.). So I proposed above modified script.
References:
Files: create
Files: get
Permissions: create
I am stuck with a problem.
#!/usr/bin/python
import httplib2
import pprint
from apiclient.discovery import build
from apiclient.http import MediaFileUpload
from oauth2client.client import OAuth2WebServerFlow
# Copy your credentials from the console
CLIENT_ID = 'YOUR_CLIENT_ID'
CLIENT_SECRET = 'YOUR_CLIENT_SECRET'
# Check https://developers.google.com/drive/scopes for all available scopes
OAUTH_SCOPE = 'https://www.googleapis.com/auth/drive'
# Redirect URI for installed apps
REDIRECT_URI = 'urn:ietf:wg:oauth:2.0:oob'
# Path to the file to upload
FILENAME = 'document.txt'
# Run through the OAuth flow and retrieve credentials
flow = OAuth2WebServerFlow(CLIENT_ID, CLIENT_SECRET, OAUTH_SCOPE, REDIRECT_URI)
authorize_url = flow.step1_get_authorize_url()
print 'Go to the following link in your browser: ' + authorize_url
code = raw_input('Enter verification code: ').strip()
credentials = flow.step2_exchange(code)
# Create an httplib2.Http object and authorize it with our credentials
http = httplib2.Http()
http = credentials.authorize(http)
drive_service = build('drive', 'v2', http=http)
# Insert a file
media_body = MediaFileUpload(FILENAME, mimetype='text/plain', resumable=True)
body = {
'title': 'My document',
'description': 'A test document',
'mimeType': 'text/plain'
}
file = drive_service.files().insert(body=body, media_body=media_body).execute()
pprint.pprint(file)
The above code asks user to copy the url to the browser, and then authorize their account, and then again, copy-paste the code and paste it on the terminal. I know storing the credentials and using the refresh tokens, users will have to do this just for once.
But, I don't want so much of user-interactions. Is it possible that user authorizes by just logging in to their gmail account? As in, from my code itself, the authorization link should get open in a web browser without user doing it, and just signs in to his/her account, and that's it, authorization is done, and this login should also happen for just one time, as in, one time authorization, so that whatever is uploaded, gets uploaded on his Google Drive account and maintained. The authorization code should be directly retrieved, and these credentials should be stored as usual and be used and tokens should also be refreshed.
I came across Google Drive Service Account, good thing is that user-intervention is gone completely, but bad thing is that, it doesn't allow the account where the file is to be uploaded. It uploads the file on that drive who has created the app.
Can anyone pls help me out with this? If going with the above code, then what should I be doing to automate the task? If going with the Service Account, what should I be doing to make the app upload the data to the user's own drive account?
Any help would be appreciated.
Thanks!
I'm guessing that you have a local Python app (not web app) that you want to be able to access user's Drive. There is no reason why the authorisation needs to be done in your Python app. So for example you could write a small web app that walks the user through the authentication process, then emails you or the user the appropriate strings to paste into the python app.
Did the same for DropBox. You can integrate PyQt's or PySide QWebView (Webkit) where in you can put the auto open the URL. A mix of javascript and python can get the job done.