I am trying to serve files securely (images in this case) to my users. I would like to do this using flask and preferably amazon s3 however I would be open to another cloud storage solution if required.
I have managed to get my flask static files like css and such on s3 however this is all non-secure. So everyone who has the link can open the static files. This is obviously not what I want for secure content. I can't seems to figure out how I can make a file available to just authenticated user that 'owns' the file.
For example: When I log into my dropbox account and copy a random file's download link. Then go over to anther computer and use this link it will denie me access. Even though I am still logged in and the download link is available to user on the latter pc.
Make the request to your Flask application, which will authenticate the user and then issue a redirect to the S3 object. The trick is that the redirect should be to a signed temporary URL that expires in a minute or so, so it can't be saved and used later or by others.
You can use boto.s3.key.generate_url function in your Flask app to create the temporary URL.
Related
I'm building a small website that involves users uploading images that will be displayed later. The images are stored in an S3 bucket.
Sometimes, I need to display a lot of these images at once, and I'm not sure how best to accommodate that, without allowing public access to S3.
Currently, when there's a request to the server, the server downloads the object from S3, and then returns the file to the client- This is understandably slow. I would love to just be able to return the S3 URL and have the client load from there (So the traffic doesn't have to pass through my server and I don't have to wait for the image to download from S3->Server->Client, but I also don't want S3 bucket urls that are just unsecured and that anyone can go to.
What is the best architecture to solve this? Is there a way of giving people very brief temporary permission to a bucket? Is it possible to scope that to a specific url?
I looked around on stackoverflow and github for similar questions, but most of them seem to have to do with how the files are uploaded and not accessing them securely.
As suggested by #jarmod, you can pre-sign your objects' URL.
In this case, once you need to share an image, you need to create a pre-sign URL for the object and share this URL.
Your server will only provide the URL. The user will access the image directly, without your server in the middle of the request.
The AWS site explains how to use pre-sign URLs:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html
https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/s3-example-presigned-urls.html
I have a python app (specifically a dash plotly dashboard) that I have deployed on Heroku. I have static files (csv/maps in the form of html etc.) that are input files for my app. However I am unable to get my python script to read these files when the heroku app starts.
I have already done the initial authentication piece of allowing heroku to access my aws bucket and set permissions.
Ok the steps are like this. This has to be a serverless application. Upon clicking the submit button on your website, an api should be called. Get/Post depending on your need.
(1)
An API will invoke a Lambda function that will take the csv file and store in s3.
Create a rest api using apigateway, connect with lamdba then store in s3.
You can use boto3 library if you pick python for lambda.
(2) Another way, if you don't need to manipulate the data on backend. You can create an API that takes a file (less than 6mb) and stores directly to s3 bucket.
If you are familiar with terraform this might help.
Best wishes.
I'm developing an Android application, which communicates with backend Google App Engine written in Python. User is uploading and downloading files to Google Cloud Storage. So far, the files where being sent to the GAE backend by POST request, and then saved in GCS. I want user to do it directly to GCS (to avoid sending large files over POST). And (on download request) I would like to send user only public URL to file. There is a nice tutorial for it in PHP:
https://cloud.google.com/appengine/docs/php/googlestorage/user_upload and
https://cloud.google.com/appengine/docs/php/googlestorage/public_access and key sentence there: "Once the file is written to Cloud Storage as publically readable, you need to get the public URL for the file, using CloudStorageTools::getPublicUrl." How to do the same in python?
The public URL of a file in GCS looks like this:
https://storage.googleapis.com/<appname>.appspot.com/<filename>
When I store files in GCS, I explicitly give the file a filename, so I can create a serving URL using the template above.
Are you giving a filename when you store files in GCS? If not, are you able to do so? Maybe provided details of how you are saving the files to GCS in your question to get a better answer.
I have a Flask app that lets users download MP3 files. How can I make it so the URL for the download is only valid for a certain time period?
For example, instead of letting anyone simply go to example.com/static/sound.mp3 and access the file, I want to validate each request to prevent an unnecessary amount of bandwidth.
I am using an Apache server although I may consider switching to another if it's easier to implement this. Also, I don't want to use Flask to serve the file because this would cause a performance overhead by forcing Flask to directly serve the file to the user. Rather, it should use Apache to serve the file.
You could use S3 to host the file with a temporary URL. Use Flask to upload the file to S3 (using boto3), but use a dynamically-generated temporary key.
Example URL: http://yourbucket.s3.amazon.com/static/c258d53d-bfa4-453a-8af1-f069d278732c/sound.mp3
Then, when you tell the user where to download the file, give them that URL. You can then delete the S3 file at the end of the time period using a cron job.
This way, Amazon S3 is serving the file directly, resulting in a complete bypass of your Flask server.
I would like for a user, without having to have an Amazon account, to be able to upload mutli-gigabyte files to an S3 bucket of mine.
How can I go about this? I want to enable a user to do this by giving them a key or perhaps through an upload form rather than making a bucket world-writeable obviously.
I'd prefer to use Python on my serverside, but the idea is that a user would need nothing more than their web browser or perhaps opening up their terminal and using built-in executables.
Any thoughts?
You are attempting to proxy the file thorough your python backend to S3, that too large files. Instead you can configure S3 to accept files from user directly (without proxying through your backend code).
It is explained here: Browser Uploads to S3 using HTML POST Forms. This way your server need not handle any upload load at all.
If you also want your users to use their elsewhere ID (google/FB etc) to achieve this workflow, that too is possible. They will be able to upload these files to a sub-folder (path) in your bucket without exposing other parts of your bucket. This is detailed here: Web Identity Federation with Mobile Applications. Though it says mobile, you can apply the same to webapps.
Having said all that, as #Ratan points out, large file uploads could break in between when you try from a browser and it cant retry "only the failed parts". This is where a dedicated app's need come in. Another option is to ask your users to keep the files in their Dropbox/BOX.com account and your server can read from there - these services already take care of large file upload with all retries etc using their apps.
This answer is relevant to .Net as language.
We had such requirement, where we had created an executable. The executable internally called a web method, which validated the app authenticated to upload files to AWS S3 or NOT.
You can do this using a web browser too, but I would not suggest this, if you are targeting big files.