I'm using a Google Service Account to upload videos using Resumable Method to Google Drive. The python code works well but I'm running into Google Service Account storage issue.
It seems like Google Service Account can only have 15 GB of storage. Even though I upload the video to a regular Google Drive folder, the video is still owned by the Service Account. Therefore, I tried to transfer the owner of the videos to a different account but it didn't work, the error is bad request. User message: \"You can't yet change the owner of this item. (We're working on it.)
Below is my python code that generate an access token from the service account and perform the Resumable Upload
credentials = ServiceAccountCredentials.from_json_keyfile_name(
'creds.json',
scopes='https://www.googleapis.com/auth/drive'
)
delegated_credentials = credentials.create_delegated('service_account_email')
access_token = delegated_credentials.get_access_token().access_token
filesize = os.path.getsize(file_location)
# Retrieve session for resumable upload.
headers1 = {"Authorization": "Bearer " + access_token, "Content-Type": "application/json"}
params = {
"name": file_name,
"mimeType": "video/mp4",
"parents": [folder_id]
}
r = requests.post(
"https://www.googleapis.com/upload/drive/v3/files?uploadType=resumable",
headers=headers1,
data=json.dumps(params)
)
location = r.headers['Location']
# Upload the file.
headers2 = {"Content-Range": "bytes 0-" + str(filesize - 1) + "/" + str(filesize)}
r = requests.put(
location,
headers=headers2,
data=open(file_location, 'rb')
)
Is there a workaround or increase the storage limit of the Google Service Accounts?
Any advice would be very appreciated. Thank you!
You want to use the Service Account to make a Resumable Upload to Drive.
You want the owner of the video not to be the Service Account, but a regular account which has enough Drive storage capacity.
If that's correct, then you can just have to delegate domain-wide authority to the Service Account, so that it can act on behalf of any user in the domain and, when uploading the file, impersonate the account you want to be the owner of the file.
Delegating domain-wide authority:
The process of granting domain-wide authority is explained here:
On the Service accounts page, select your Service Account, and while editing the SA, click SHOW DOMAIN-WIDE DELEGATION and, on the content that was just displayed, check the option Enable G Suite Domain-wide Delegation.
Once you've done this, go to the Admin console and then go to Main menu > Security > API Controls.
Select Manage Domain Wide Delegation in the Domain wide delegation pane, and click Add new.
Fill up the corresponding fields: (1) in Client ID, enter the SA's Client ID, which you can find both in the credentials JSON file and on the Service Account page, and (2) in OAuth scopes, add the scopes corresponding to the resources you want the SA to access on behalf of users in the domain. In this case, I guess that's just https://www.googleapis.com/auth/drive.
After clicking Authorize, you have conferred the Service Account the ability to access resources on behalf of any user in the domain.
Impersonating another user:
Now the Service Account impersonate any user in the domain, but you have to specify which user you want it to impersonate. In order to do that, you just have to do a small change in your code. Right now, you're setting the service_account_email when delegating credentials via create_delegated:
delegated_credentials = credentials.create_delegated('service_account_email')
That is, the Service Account is acting on behalf of the Service Account. If you didn't want to impersonate another account, there would be no real need for this line of code (it doesn't have any effect, since credentials and delegated_credentials both refer to the same account (the Service Account).
But since you want to use the Service Account to act on behalf of another account, you have to specify this other account's email address on this line:
delegated_credentials = credentials.create_delegated('user_account_email')
That's the only change you need to do in your code. If you have granted domain-wide delegation, the Service Account will act as if it was this other user. It will be like it was this other user who uploaded the file, so this user will be the owner of the file.
Note:
You are using a deprecated library (oauth2client). Since this is still working, there is no real need to do it now, but please consider changing your code to google-auth.
Reference:
Delegating domain-wide authority to the service account
Related
I have a script which modifies my work supplied GCal. For authentication I use an access/refresh token (like this: https://developers.google.com/people/quickstart/python).
I want to run the script in Docker now. For authentication I have decided to use a service account.
I have created the service account, shared my calendar with it and accepted the calendar. In the Google Console where you create the service account, I set the permission to "owner".
When I try to run the script using the service account (not in Docker yet) it returns only a subset of attributes for each calendar event. I can see that accessRole = freeBusyReader.
How do I grant write access to this service account? I have tried:
rule = service.acl().get(calendarId="myId", ruleId='user:service#myApp-351310.iam.gserviceaccount.com').execute() # Get this from acl_items
rule["role"] = "owner"
service.acl().update(calendarId="myId", ruleId="user:service#myApp-351310.iam.gserviceaccount.com", body=rule).execute()
I have read about firmwide delegation and impersonation of users. I'm not sure if this is requred or not. Does anyone know how to do this?
The code to authenticate a service account is slightly different then the sample you were using for an installed application it is as follows.
credentials = ServiceAccountCredentials.from_json_keyfile_name(
key_file_location, scopes=scopes)
credentials = credentials.create_delegated(user_email)
This video shows How to create Google Oauth2 Service account credentials. just make sure to enable the google calendar api.
Remember service accounts are only supported via domain wide delegation to users on your google workspace domain. You cant use a standard google gmail user.
I recommend following the Delegate domain-wide authority to your service account sample it shows how to set up the delegation to a service account from your workspace domain. Just change out the section about admin sdk to that of google calendar as this is the api you are trying to connect to.
You add the user_Email being the user on your domain you want the service account to impersonate.
I want to create an automatic deployment of GCP for clients.
In order to do that, I have opened a page for them to login with google, and then enabled the IAM API and the Service Usage API.
Then I have created a service account that I want to use from this point forward in order to enable other required APIs on demand and not all at once.
When I try to enable the cloudkms API, I get
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://serviceusage.googleapis.com/v1/projects/x-y-z/services/cloudkms.googleapis.com?alt=json returned "The caller does not have permission"
I tried using the service account credentials (google.auth.jwt.Credentials) that I have created from the response of creating the service account, and I have added all the required permissions. I don't want to grant the role owner to the service account, because I want the account to have as less permissions as possible.
When I try to get the status of cloudkms API using the user's permissions, it works.
I have seen some solutions addressing me needing to create credentials for the service account here : https://console.developers.google.com/apis/credentials but I really need to do this programatically as well.
My code:
credentials = jwt.Credentials.from_service_account_file(service_account_info['email'] + '.json', audience="https://www.googleapis.com/auth/cloud-platform")
# credentials = GoogleCredentials.get_application_default() - it works with this
service_usage = googleapiclient.discovery.build('serviceusage', 'v1', credentials=credentials)
service_usage.services().get(name="projects/<project_id>/services/cloudkms.googleapis.com").execute()
The error was mentioned above.
You need the Cloud IAM permission serviceusage.services.enable to enable services. Depending on what features your require, such as listing services, you need serviceusage.services.list.
Typically you add the role roles/serviceusage.serviceUsageAdmin which includes the following permissions:
serviceusage.services.get
serviceusage.services.list
serviceusage.services.enable
serviceusage.services.disable
Goto IAM
Edit user selected
Add new rol
Type Service Usage Admin
Save
In a django application, I try to have RW access to a google calendar which I own myself.
Tried several ways with a service account & client secrets, but all resulting in authentication errors.
The API explorer works, but it requests consent in a popup window, which is obviously not acceptable.
Documentation on google OAuth2 describes several scenarios. Probably "web server application" applies here? It says:
"The authorization sequence begins when your application redirects a
browser to a Google URL; the URL includes query parameters that
indicate the type of access being requested. Google handles the user
authentication, session selection, and user consent. The result is an
authorization code, which the application can exchange for an access
token and a refresh token."
Again, we do not want a browser redirection, we want direct access to the google calendar.
So question is: how can a django server access a google calendar, on which I have full rights, view events and add events using a simple server stored key or similar mechanism?
With help of DalmTo and this great article, I got RW access to a google calendar working from python code. I will summarize the solution here.
Here are the steps:
First of all register for a google service account: Service accounts are pre-authorized accounts that avoid you need to get consent or refresh keys every time:
https://developers.google.com/identity/protocols/OAuth2ServiceAccount
(The part on G-suite can be ignored)
Download the service account credentials and store them safely. Your python code will need access to this file.
Go to your google calendar you want to get access to.
e.g. https://calendar.google.com/calendar/r/month
On the right side you see your calendars. Create an additional one for testing (since we'll write to it soon). Then point to this new calendar: click the 3 dots next to it and edit the sharing settings. Add the service account email address to the share under "share with specific people". (you can find the service account email address in the file downloaded previously under "client_email")
In the same screen, note the "calendar ID", you'll need it in below code.
Now you service account has the RW rights to the calendar.
Add at least one event to the calendar using the web UI (https://calendar.google.com/calendar/r/month) so we can read and change it from below code.
Then use following python code to read the calendar and change an event.
from google.oauth2 import service_account
import googleapiclient.discovery
SCOPES = ['https://www.googleapis.com/auth/calendar']
SERVICE_ACCOUNT_FILE = '<path to your service account file>'
CAL_ID = '<your calendar ID>'
credentials = service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_FILE, scopes=SCOPES)
service = googleapiclient.discovery.build('calendar', 'v3', credentials=credentials)
events_result = service.events().list(calendarId=CAL_ID).execute()
events = events_result.get('items', [])
event_id = events[0]['id']
event = events[0]
service.events().update(calendarId=CAL_ID, eventId=event_id, body={"end":{"date":"2018-03-25"},"start":{"date":"2018-03-25"},"summary":"Kilroy was here"}).execute()
And there you go... read an event and updated the event.
I'm trying to list all Google Drive files Domain-wide, both users that still work here, and those that have moved on. With that, we can grep the output for certain terms (former customers) to delete customer-related files.
I believe I have a successful way to list all users using the Admin SDK Quickstart, since we have only about 200 total users (max is 500). I also have a way to list all files for a user using the Drive REST API's files.list() method. What I need to know is how to impersonate each user iteratively, in order to run the file listing script.
I have found the blurb .setServiceAccountUser(someone#domain.com) but I'm not really sure where to implement this, either in the service account authorization step, or in a separate middle-man script.
Have a look at https://github.com/pinoyyid/googleDriveTransferOwnership/blob/master/src/couk/cleverthinking/tof/Main.java
Specifically lines 285-299 which deal with generating a credential for an impersonated user.
GoogleCredential.Builder builder = new GoogleCredential.Builder()
.setTransport(HTTP_TRANSPORT)
.setJsonFactory(JSON_FACTORY)
.setServiceAccountId(serviceAccountEmailAddress)
.setServiceAccountPrivateKeyFromP12File(f)
.setServiceAccountScopes(Collections.singleton(SCOPE));
// if requested, impresonate a domain user
if (!"ServiceAccount".equals(impersonatedAccountEmailAddress)) {
builder.setServiceAccountUser(impersonatedAccountEmailAddress);
}
// build the Drive service
Drive service = new Drive.Builder(HTTP_TRANSPORT, JSON_FACTORY, null)
.setApplicationName("TOF")
.setHttpRequestInitializer(builder.build()).build();
This is Java, but should at least tell you what the steps are.
You need to implement the authorization flow for Service Accounts.
Once you create a service account in a GCP project (console.developers.google.com), enable DWD (domain-wide delegation), then authorize that service account in your G Suite admin console, that key can then be used to "impersonate" any account in the G Suite instance:
Create the credentials object from the json file
from oauth2client.service_account import ServiceAccountCredentials
scopes = ['https://www.googleapis.com/auth/gmail.readonly']
credentials = ServiceAccountCredentials.from_json_keyfile_name(
'/path/to/keyfile.json', scopes=scopes)
Create a credential that can impersonate user#example.org (could be any user in the domain though)
delegated_credentials = credentials.create_delegated('user#example.org')
Authorize the credential object (i.e. get an access_token)
from httplib2 import Http
http_auth = credentials.authorize(Http())
Call the Gmail API:
from apiclient import discovery
service = discovery.build('gmail', 'v1', http=http)
response = service.users().messages().list(userId='user#example.org').execute()
I am trying to list all the files in my drive (about 10) but the following will only list 1 (and that isn't even a real file of mine)....
the code:
from httplib2 import Http
from oauth2client.client import SignedJwtAssertionCredentials
client='my_client_id'
client_email = 'my_client_email'
with open("/path/to/file.p12") as f:
private_key = f.read()
credentials = SignedJwtAssertionCredentials(client_email, private_key, 'https://www.googleapis.com/auth/drive')
http_auth = credentials.authorize(Http())
drive_service = build('drive', 'v2', http=http_auth)
r = drive_service.files().list().execute()
files = r['items']
for f in files:
print f['id'], f['title']
result:
"<file_id> How to get started with Drive"
EDIT:
This question is similar but the answer is to have the correct oauth scope, which I have above.
EDIT #2:
I thought it might be a timing issue so I gave it a few hours and still no goose.
EDIT #3:
If I try to copy a file from another user then list my files then I'll get 2 files:
" How to get started with Drive"
" My New File"
So, this is just listing files created by that app? How do I get the rest of my files???
You use a service account to authenticate. A service account does not have by default the right to access your Drive data, but only files that it owns by itself.
You have three options to work around this :
Create a folder in your Drive account, and share it (read/write) with the service account. Any file you place in that folder will be readable and writable both by you and your service account.
If you use Google Apps For Business, setup domain wide delegation to allow your service account to impersonate all users in your domain. That way you will be able to get your service account to behave as if it were your actual Google Apps account.
Whether you use or not Google Apps For Business : do not use a service account but rather 3-legged OAuth. With 3-legged OAuth you will be able to generate an access token and a refresh token that will allow your application to act in Drive on behalf of your actual Google account. Note that this last options does not use service accounts at all.
The simplest is obviously option (1). If it is not acceptable then I would go with option (3), unless you actually want to be able to impersonate all the users in your domain.