I want to directly upload some files from google drive to amazon s3 but unable to do so.
I dont know, how can i directly upload the files from google drive to amazon s3.
I tried getting the download link using python and google api.
but when I try to upload to amazon s3 I'm getting errors:
axios
.get("https://drive.google.com/u/0/uc?id="+id+"&export=download", {
responseType: 'arraybuffer',
withCredentials: false,
headers:{
'Authorization':'Bearer: <access_token>',
'Content-Type': 'text/plain'
}
})
.then((response:any) => Buffer.from(response.data, 'binary'))
.then((data:any)=>{
console.log(data)
})
EROR
has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.
Can anyone please tell me, how can i resolve this error?
you got the data from drive then;
with aws-sdk package you can create
s3 = new AWS.S3(....)
then use s3.putObject method
Manage to find the answer and posting this as answer for anyone who is looking for same:
First you need a downloadable link which you can construct using file id:
Here it is:
"https://www.googleapis.com/drive/v3/files/"+file.id+'?alt=media'
Then you can make a GET request with your google access token:
'Authorization','Bearer '+accessToken
This way you can download file from google drive which you can upload to s3.
Related
I am trying to upload the file to the custom Google Cloud Storage bucket with a Flutter web app.
final _storage = FirebaseStorage.instanceFor(bucket: bucketName);
Reference documentRef = _storage.ref().child(filename);
await documentRef.putData(await data);
The code works fine for a default bucket but fails with a new custom GCP bucket.
Error: FirebaseError: Firebase Storage: An unknown error occurred, please check the error payload for server response. (storage/unknown)
The HTTP POST response causing this error says:
{
"error": {
"code": 400,
"message": "Your bucket has not been set up properly for Firebase Storage. Please visit 'https://console.firebase.google.com/project/{my_project_name}/storage/rules' to set up security rules."
}
}
So apparently, I need to add a new bucket to Firestore and set up access rules before I can upload the file there.
Since these buckets are created automatically by my backend microservice, is there a way to add them to Firestore and set up the rules with Python SDK? Alternatively, is there any other way to upload data to GCP buckets with Flutter besides Firebase Storage?
Thank you.
So far I think this is not possible to be done using an SDK, nevertheless this can done making a request to the Firebase API. Here's how it would be done using curl:
curl -X POST \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
https://firebasestorage.clients6.google.com/v1alpha/projects/[PROJECT_NUMBER]/buckets/[BUCKET_NAME]:addFirebase
So you can make a similar request using requests and creating the token as explained here:
import google.auth
import google.auth.transport.requests
creds, project = google.auth.default()
# creds.valid is False, and creds.token is None
# Need to refresh credentials to populate those
auth_req = google.auth.transport.requests.Request()
creds.refresh(auth_req)
# Now you can use creds.token
I'm uploading a jpg file into my google drive account. It works fine, but I need it to upload to a specific folder but am not sure how to set the parents parameter in the metadata.
Here's my code:
data = {"file": open(filedirectory, 'rb').read(), "title" : filename, "parents" : [{"id": "<folderid>"}]}
drive_url = "https://www.googleapis.com/upload/drive/v3/files?uploadType=media"
drive_r = requests.post(drive_url, data=data, headers={"Authorization": "Bearer " + access_token, "Content-type": "image/jpeg"})
I believe your goal as follows.
You want to upload a file to the specific folder in Google Drive.
You want to achieve this using requests of python.
Modification points:
In the case of uploadType=media, the official document says as follows.
Simple upload (uploadType=media). Use this upload type to quickly transfer a small media file (5 MB or less) without supplying metadata. To perform a simple upload, refer to Perform a simple upload.
So in order to upload the file content and file metadata, please use uploadType=multipart.
And also, in your endpoint, Drive API v3 is used. But "parents" : [{"id": "<folderid>"}] is for Drive API v2. It is required to also modify it.
When your script is modified for uploadType=multipart, it becomes as follows.
Modified script:
When you use this script, please set the variables of filedirectory, filename, folderid, access_token.
import json
import requests
filedirectory = '###'
filename = '###'
folderid = '###'
access_token = '###'
metadata = {
"name": filename,
"parents": [folderid]
}
files = {
'data': ('metadata', json.dumps(metadata), 'application/json'),
'file': open(filedirectory, "rb").read() # or open(filedirectory, "rb")
}
r = requests.post(
"https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart",
headers={"Authorization": "Bearer " + access_token},
files=files
)
print(r.text)
Note:
This modified script supposes that your access token can be used for uploading the file to Google Drive.
References:
Upload file data
Create files
it seems I have been able to get the Javascript SDK to react, with this code
gapi.client.drive.files.create({
name: 'multapart.jpg', //OPTIONAL
uploadType: 'multipart',
},{body:'your content here',content:'your content here'})
where media is bytes representation for an image
in chronium edge it complained that
Access to XMLHttpRequest at 'https://content.googleapis.com/drive/v3/files?name=resumable.jpg&body=%EF%BF%BD%EF%....%BD&uploadType=multipart&alt=json&key=[you cant see this haha]'
from origin 'https://content.googleapis.com'
has been blocked by CORS policy: Cross origin requests are only supported for protocol schemes: http, data, chrome-extension, edge, https, chrome-untrusted.
in edge legacy it said it was a malformed and when I did uploadType=media, it said is was using https://content.googleapis.com/drive/v3/files instead of https://content.googleapis.com/upload/drive/v3/files, so the JS SDK is unreliable and riddled with bugs, glad I got it to react, if only someone can find the logic to give it the right URL because I believe the js SDK is not buggy, google doesnt want ppl to use it
I am connecting user on frontend which is in ReactJS and backend is in python.
Now when I connect the user, I get following data:
{
"aud": "some token",
"scope": "https://www.googleapis.com/auth/drive",
"exp": "1544798733",
"expires_in": "3598",
"access_type": "online"
}
Now, when I am connecting through python,to upload a file to google drive, I need many more fields as the user credentials to successfully upload the file. How can I connect/upload file to drive? is there any other solution?
I am referring to this doc for drive access using python.
In the doc provided by google drive, there are several steps that is related to the user credential in step 1. Did you choose them as what it provided in the doc?
Here is the documentation that will guide you to upload a file in the drive.
You can send upload requests in any of the following ways:
Simple upload: uploadType=media. For quick transfer of a small file (5 MB or less). To perform a simple upload, refer to
Performing a Simple Upload.
Multipart upload: uploadType=multipart. For quick transfer of a small file (5 MB or less) and metadata describing the file, all in
a single request. To perform a multipart upload, refer to
Performing a Multipart Upload.
Resumable upload: uploadType=resumable. For more reliable transfer, especially important with large files. Resumable uploads
are a good choice for most applications, since they also work for
small files at the cost of one additional HTTP request per upload.
To perform a resumable upload, refer to Performing a Resumable
Upload.
If you are using python, here is a sample code of basic upload.
file_metadata = {'name': 'photo.jpg'}
media = MediaFileUpload('files/photo.jpg',
mimetype='image/jpeg')
file = drive_service.files().create(body=file_metadata,
media_body=media,
fields='id').execute()
print 'File ID: %s' % file.get('id')
For the rest of details, you can refer to the docs.
What I want to accomplish is to generate a link to view the file (ex.image or pdf). The item is not accessible by URL (https://[bucket].s3.amazonaws.com/img_name.jpg), I think because its private and not public? (I'm not the owner of the bucket, but he gave me the access_key and secret_key?)
For now, all I can do is to download a file with this code.
s3.Bucket('mybucket').download_file('upload/nocturnes.png', 'dropzone/static/pdf/download_nocturnes.png')
I want to access an image on s3 so I can put it on an HTML, can I view it using the access and secret key?. Thank you for those who can help!
You can accomplish this using a pre-signed URL using the generate_presigned_url function. The caveat is that pre-signed URLs must have an expiration date. I was unable to find documentation on the maximum allowed duration. Here is an example:
url = s3.generate_presigned_url('get_object',
Params={
'Bucket': 'mybucket',
'Key': 'upload/nocturnes.png',
},
ExpiresIn=3600)
print url
For people who want to use generate_presigned_url for a public object and therefore don't want to do the signing part that appends credentials, the best solution I found is to still to use the generate_presigned_url, just that the Client.Config.signature_version needs to be set to botocore.UNSIGNED.
The following returns the public link without the signing stuff.
config.signature_version = botocore.UNSIGNED
boto3.client('s3', config=config).generate_presigned_url('get_object', ExpiresIn=0, Params={'Bucket': bucket, 'Key': key})
The relevant discussions on the boto3 repository are:
https://github.com/boto/boto3/issues/110
https://github.com/boto/boto3/issues/169
https://github.com/boto/boto3/issues/1415
I created a form and a simple server with google appengine with which to upload arbitrary file types to my google drive. The form fails to work for certain file types and just gives this error instead:
HttpError: <HttpError 400 when requesting https://www.googleapis.com/upload/drive/v1/files?alt=json returned "Unsupported content with type: application/pdf">
Aren't pdf files supported?
The appengine code that does the upload goes somewhat like this:
def upload_to_drive(self, filestruct):
resource = {
'title': filestruct.filename,
'mimeType': filestruct.type,
}
resource = self.service.files().insert(
body=resource,
media_body=MediaInMemoryUpload(filestruct.value,
filestruct.type),
).execute()
def post(self):
creds = StorageByKeyName(Credentials, my_user_id, 'credentials').get()
self.service = CreateService('drive', 'v1', creds)
post_dict = self.request.POST
for key in post_dict.keys():
if isinstance(post_dict[key], FieldStorage):#might need to import from cgi
#upload to drive and return link
self.upload_to_drive(post_dict[key]) #TODO: there should be error handling here
I've successfully used it for MS Office documents and images. It doesn't work for textfiles too and gives this error:
HttpError: <HttpError 400 when requesting https://www.googleapis.com/upload/drive/v1/files?alt=json returned "Multipart content has too many non-media parts">
I've tried unsetting the 'mimeType' value in the resource dict to let google drive set it automatically. I also tried unsetting the mime type value in the MediaInMemoryUpload constructor. Sadly, none of both worked.
It seems to me that you are using an old version of the Python client library and referring to Drive API v1, while Drive API v2 has been available since the end of June.
Please try updating your library and check the complete Python sample at https://developers.google.com/drive/examples/python.