Develop Reference

How to directly upload files from google drive to amazon s3?

I want to directly upload some files from google drive to amazon s3 but unable to do so.
I dont know, how can i directly upload the files from google drive to amazon s3.
I tried getting the download link using python and google api.
but when I try to upload to amazon s3 I'm getting errors:
axios
.get("https://drive.google.com/u/0/uc?id="+id+"&export=download", {
responseType: 'arraybuffer',
withCredentials: false,
headers:{
'Authorization':'Bearer: <access_token>',
'Content-Type': 'text/plain'
}
})
.then((response:any) => Buffer.from(response.data, 'binary'))
.then((data:any)=>{
console.log(data)
})
EROR
has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
Can anyone please tell me, how can i resolve this error?

you got the data from drive then;
with aws-sdk package you can create
s3 = new AWS.S3(....)
then use s3.putObject method

Manage to find the answer and posting this as answer for anyone who is looking for same:
First you need a downloadable link which you can construct using file id:
Here it is:
"https://www.googleapis.com/drive/v3/files/"+file.id+'?alt=media'
Then you can make a GET request with your google access token:
'Authorization','Bearer '+accessToken
This way you can download file from google drive which you can upload to s3.

Download google drive attachments of an email using Gmail API in python

I currently use this solution to download attachments from Gmail using Gmail API via python.
However, every time an attachment exceeds 25MB, the attachments automatically get uploaded to Google Drive and the files are linked in the mail. In such cases, there is no attachmentId in the message.
I can only see the file names in 'snippet' section of the message file.
Is there any way I can download the Google dive attachments from mail?
There is a similar question posted here, but there's no solution provided to it yet

How to download a Drive "attachment"
The "attachment" referred to is actually just a link to a Drive file, so confusingly it is not an attachment at all, but just text or HTML.
The issue here is that since it's not an attachment as such, you won't be able to fetch this with the Gmail API by itself. You'll need to use the Drive API.
To use the Drive API you'll need to get the file ID. Which will be within the HTML content part among others.
You can use the re module to perform a findall on the HTML content, I used the following regex pattern to recognize drive links:
(?<=https:\/\/drive\.google\.com\/file\/d\/).+(?=\/view\?usp=drive_web)
Here is a sample python function to get the file IDs. It will return a list.
def get_file_ids(service, user_id, msg_id):
message = service.users().messages().get(userId=user_id, id=msg_id).execute()
for part in message['payload']['parts']:
if part["mimeType"] == "text/html":
b64 = part["body"]["data"].encode('UTF-8')
unencoded_data = str(base64.urlsafe_b64decode(b64))
results = re.findall(
'(?<=https:\/\/drive\.google\.com\/file\/d\/).+(?=\/view\?usp=drive_web)',
unencoded_data
)
return results
Once you have the IDs then you will need to make a call to the Drive API.
You could follow the example in the docs:
file_ids = get_file_ids(service, "me", "[YOUR_MSG_ID]"
for id in file_ids:
request = service.files().get_media(fileId=id)
fh = io.BytesIO()
downloader = MediaIoBaseDownload(fh, request)
done = False
while done is False:
status, done = downloader.next_chunk()
print "Download %d%%." % int(status.progress() * 100)
Remember, seeing as you will now be using the Drive API as well as the Gmail API, you'll need to change the scopes in your project. Also remember to activate the Drive API in the developers console, update your OAuth consent screen, credentials and delete the local token.pickle file.
References
Drive API Docs
Managing Downloads Guide
Gmail API Docs

Drive API has also limtitation of downloading 10MBs only

How to upload file into specific folder with Google Drive API and Python?

I'm uploading a jpg file into my google drive account. It works fine, but I need it to upload to a specific folder but am not sure how to set the parents parameter in the metadata.
Here's my code:
data = {"file": open(filedirectory, 'rb').read(), "title" : filename, "parents" : [{"id": "<folderid>"}]}
drive_url = "https://www.googleapis.com/upload/drive/v3/files?uploadType=media"
drive_r = requests.post(drive_url, data=data, headers={"Authorization": "Bearer " + access_token, "Content-type": "image/jpeg"})

I believe your goal as follows.
You want to upload a file to the specific folder in Google Drive.
You want to achieve this using requests of python.
Modification points:
In the case of uploadType=media, the official document says as follows.
Simple upload (uploadType=media). Use this upload type to quickly transfer a small media file (5 MB or less) without supplying metadata. To perform a simple upload, refer to Perform a simple upload.
So in order to upload the file content and file metadata, please use uploadType=multipart.
And also, in your endpoint, Drive API v3 is used. But "parents" : [{"id": "<folderid>"}] is for Drive API v2. It is required to also modify it.
When your script is modified for uploadType=multipart, it becomes as follows.
Modified script:
When you use this script, please set the variables of filedirectory, filename, folderid, access_token.
import json
import requests
filedirectory = '###'
filename = '###'
folderid = '###'
access_token = '###'
metadata = {
"name": filename,
"parents": [folderid]
}
files = {
'data': ('metadata', json.dumps(metadata), 'application/json'),
'file': open(filedirectory, "rb").read() # or open(filedirectory, "rb")
}
r = requests.post(
"https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart",
headers={"Authorization": "Bearer " + access_token},
files=files
)
print(r.text)
Note:
This modified script supposes that your access token can be used for uploading the file to Google Drive.
References:
Upload file data
Create files

it seems I have been able to get the Javascript SDK to react, with this code
gapi.client.drive.files.create({
name: 'multapart.jpg', //OPTIONAL
uploadType: 'multipart',
},{body:'your content here',content:'your content here'})
where media is bytes representation for an image
in chronium edge it complained that
Access to XMLHttpRequest at 'https://content.googleapis.com/drive/v3/files?name=resumable.jpg&body=%EF%BF%BD%EF%....%BD&uploadType=multipart&alt=json&key=[you cant see this haha]'
from origin 'https://content.googleapis.com'
has been blocked by CORS policy: Cross origin requests are only supported for protocol schemes: http, data, chrome-extension, edge, https, chrome-untrusted.
in edge legacy it said it was a malformed and when I did uploadType=media, it said is was using https://content.googleapis.com/drive/v3/files instead of https://content.googleapis.com/upload/drive/v3/files, so the JS SDK is unreliable and riddled with bugs, glad I got it to react, if only someone can find the logic to give it the right URL because I believe the js SDK is not buggy, google doesnt want ppl to use it

How do I properly open an html document in Google Drive?

Im using Google Drive API for creating and opening html file. But the problem is that the document opens with the technical content (links to css, js files, html tags ...) like this
How to make it so that it would open correctly, in a user-friendly form?
part of my google-api code
def file_to_drive(import_file=None):
service = build('drive', 'v3', credentials=creds)
file_name = import_file
media_body = MediaFileUpload(file_name, resumable=True, mimetype='text/html')
body = {
'title': file_name,
'description': 'Uploaded By You'}
file = service.files().create(body=body, media_body=media_body, fields='id')

The google drive API is a file store api. It allows you to upload and download files. It does not have the ability to open files. You could share a link to the file with someone that has access then when they click on the link it will open for them in the google drive web application.
The only api able to open files for editing would be the Google docs api which gives you limited ability to open google doc files. that however would require that you covert your html file to a google docs format. Even if this was an option you would need to create your own "user friendly form" Google apis return data as json and not user friendly options thats not what APIs are for.

How do you get Google App Engine to gunzip during download?

I am trying to get Google App Engine to gunzip my .gz blob file (single file compressed) automatically by setting the response headers as follows:
class download(blobstore_handlers.BlobstoreDownloadHandler):
def get(self, resource):
resource = str(urllib.unquote(resource))
blob_info = blobstore.BlobInfo.get(resource)
self.response.headers['Content-Encoding'] = str('gzip')
# self.response.headers['Content-type'] = str('application/x-gzip')
self.response.headers['Content-type'] = str(blob_info.content_type)
self.response.headers['Content-Length'] = str(blob_info.size)
cd = 'attachment; filename=%s' % (blob_info.filename)
self.response.headers['Content-Disposition'] = str(cd)
self.response.headers['Cache-Control'] = str('must-revalidate, post-check=0, pre-check=0')
self.response.headers['Pragma'] = str(' public')
self.send_blob(blob_info)
When this runs, the file is downloaded without the .gz extension. However, the downloaded file is still gzipped. The file size of the downloaded data match the .gz file size on the server. Also, I can confirm this by manually gunzipping the downloaded file. I am trying to avoid the manual gunzip step.
I am trying to get the blob file to automatically gunzip during the download. What am I doing wrong?
By the way, the gzip file contains only a single file. On my self-hosted (non Google) server, I could accomplish the automatic gunzip by setting same response headers; albeit, my code there is written in PHP.
UPDATE:
I rewrote the handler to serve data from the bucket. However, this generates HTML 500 error. The file is partially downloaded before the failure. The rewrite is as follows:
class download(blobstore_handlers.BlobstoreDownloadHandler):
def get(self, resource):
resource = str(urllib.unquote(resource))
blob_info = blobstore.BlobInfo.get(resource)
file = '/gs/mydatabucket/%s' % blob_info.filename
print file
self.response.headers['Content-Encoding'] = str('gzip')
self.response.headers['Content-Type'] = str('application/x-gzip')
# self.response.headers['Content-Length'] = str(blob_info.size)
cd = 'filename=%s' % (file)
self.response.headers['Content-Disposition'] = str(cd)
self.response.headers['Cache-Control'] = str('must-revalidate, post-check=0, pre-check=0')
self.response.headers['Pragma'] = str(' public')
self.send_blob(file)
This downloads 540,672 bytes of the 6,094,848 bytes file to the client before the server terminate and issued a 500 error. When I issue 'file' on the partially downloaded file from the command line, Mac OS seems to correctly identify the file format as 'SQLite 3.x database' file. Any idea of why the 500 error on the server? How can I fix the problem?

You should first check to see if your requesting client supports gzipped content. If it does support gzip content encoding, then you may pass the gzipped blob as is with the proper content-encoding and content-type headers, otherwise you need to decompress the blob for the client. You should also verify that your blob's content_type isn't gzip (this depends on how you created your blob to begin with!)
You may also want to look at Google Cloud Storage as this automatically handles gzip transportation so long as you properly compress the data before storing it with the proper content-encoding and content-type metadata.
See this SO question: Google cloud storage console Content-Encoding to gzip
Or the GCS Docs: https://cloud.google.com/storage/docs/gsutil/addlhelp/WorkingWithObjectMetadata#content-encoding
You may use GCS as easily (if not more easily) as you use the blobstore in AppEngine and it seems to be the preferred storage layer to use going forward. I say this because the File API has been deprecated which made blobstore interaction easier and great efforts and advancements have been made to the GCS libraries making the API similar to the base python file interaction API
UPDATE:
Since the objects are stored in GCS, you can use 302 redirects to point users to files rather than relying on the Blobstore API. This eliminates any unknown behavior of the Blobstore API and GAE delivering your stored objects with the content-type and content-encoding you intended to use. For objects with a public-read ACL, you may simply direct them to either storage.googleapis.com/<bucket>/<object> or <bucket>.storage.googleapis.com/<object>. Alternatively, if you'd like to have application logic dictate access, you should keep the ACL to the objects private and can use GCS Signed URLs to create short lived URLs to use when doing a 302 redirect.
Its worth noting that if you want users to be able to upload objects via GAE, you'd still use the Blobstore API to handle storing the file in GCS, but you'd have to modify the object after it was uploaded to ensure proper gzip compressing and content-encoding meta data is used.
class legacy_download(blobstore_handlers.BlobstoreDownloadHandler):
def get(self, resource):
filename = str(urllib.unquote(resource))
url = 'https://storage.googleapis.com/mybucket/' + filename
self.redirect(url)

GAE already serves everything using gzip if the client supports it.
So I think what's happening after your update is that the browser expects there to be more of the file, but GAE thinks it's already at the end of the file since it's already gzipped. That's why you get the 500.
(if that makes sense)
Anyway, since GAE already handles compression for you, the easiest way is probably to put non compressed files in GCS and let the Google infrastructure handle the compression automatically for you when you serve them.