I'd like to securely display a grid of thumbnail images to an authenticated user on our site. All the images will be stored in Amazon S3.
One way, I suppose, is to implement "security by obscurity" by uploading these images with public read access, and making the keys long and random.
I also could set up ACLs, but then I'd have to disclose the access key in the url (I think), or pull the image into my application via the API and display it securely through the web server.
Is there a preferred way to do this? And to be able to display the images quickly without requiring tremendous requests to S3 from the server every time a page is generated?
Thanks in advance
You can generate urls to s3 with an expiry date. Generating such a URL does not require a request to S3 and does not result in the disclosure of your secret key: you use your secret key to generate a signature that is appended to the URL (the access key id is in that URL but that's ok)
See the docs on query string authorization
Related
I'm building a small website that involves users uploading images that will be displayed later. The images are stored in an S3 bucket.
Sometimes, I need to display a lot of these images at once, and I'm not sure how best to accommodate that, without allowing public access to S3.
Currently, when there's a request to the server, the server downloads the object from S3, and then returns the file to the client- This is understandably slow. I would love to just be able to return the S3 URL and have the client load from there (So the traffic doesn't have to pass through my server and I don't have to wait for the image to download from S3->Server->Client, but I also don't want S3 bucket urls that are just unsecured and that anyone can go to.
What is the best architecture to solve this? Is there a way of giving people very brief temporary permission to a bucket? Is it possible to scope that to a specific url?
I looked around on stackoverflow and github for similar questions, but most of them seem to have to do with how the files are uploaded and not accessing them securely.
As suggested by #jarmod, you can pre-sign your objects' URL.
In this case, once you need to share an image, you need to create a pre-sign URL for the object and share this URL.
Your server will only provide the URL. The user will access the image directly, without your server in the middle of the request.
The AWS site explains how to use pre-sign URLs:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html
https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/s3-example-presigned-urls.html
What is the recommended way to let users access files that are stored on AWS S3?
I've currently found two ways:
using the access key ID and secret access key
making the whole bucket public and then accessing the files thru a public link.
Both options aren't really satisfying. Either I reveal the access/secret keys or all files are publicly available. Is there another third more secure way?
The software, that needs access to S3, will be running on Raspberry Pis. I was thinking of encrypting the credentials file, so a user wouldn't be able to read it easily.
The first thing is, to create a new individual AWS user with access key ID and secret for your project with and give the individual S3 permission to that user. Don't use admin credentials for a project. Every project should have a particular AWS user with its credentials and permissions. The second thing is, to rotate keys. So you create a new key with an old one and delete old key. More about rotation you can refer to Managing Access Keys for IAM Users documentation. Indeed, you can also encrypt it and AWS has the service AWS KMS. You can make research in AWS KMS service, it is a great tool for encryption. You can even encrypt your application vulnerable secret keys or passwords.
I am trying to serve files securely (images in this case) to my users. I would like to do this using flask and preferably amazon s3 however I would be open to another cloud storage solution if required.
I have managed to get my flask static files like css and such on s3 however this is all non-secure. So everyone who has the link can open the static files. This is obviously not what I want for secure content. I can't seems to figure out how I can make a file available to just authenticated user that 'owns' the file.
For example: When I log into my dropbox account and copy a random file's download link. Then go over to anther computer and use this link it will denie me access. Even though I am still logged in and the download link is available to user on the latter pc.
Make the request to your Flask application, which will authenticate the user and then issue a redirect to the S3 object. The trick is that the redirect should be to a signed temporary URL that expires in a minute or so, so it can't be saved and used later or by others.
You can use boto.s3.key.generate_url function in your Flask app to create the temporary URL.
I would like for a user, without having to have an Amazon account, to be able to upload mutli-gigabyte files to an S3 bucket of mine.
How can I go about this? I want to enable a user to do this by giving them a key or perhaps through an upload form rather than making a bucket world-writeable obviously.
I'd prefer to use Python on my serverside, but the idea is that a user would need nothing more than their web browser or perhaps opening up their terminal and using built-in executables.
Any thoughts?
You are attempting to proxy the file thorough your python backend to S3, that too large files. Instead you can configure S3 to accept files from user directly (without proxying through your backend code).
It is explained here: Browser Uploads to S3 using HTML POST Forms. This way your server need not handle any upload load at all.
If you also want your users to use their elsewhere ID (google/FB etc) to achieve this workflow, that too is possible. They will be able to upload these files to a sub-folder (path) in your bucket without exposing other parts of your bucket. This is detailed here: Web Identity Federation with Mobile Applications. Though it says mobile, you can apply the same to webapps.
Having said all that, as #Ratan points out, large file uploads could break in between when you try from a browser and it cant retry "only the failed parts". This is where a dedicated app's need come in. Another option is to ask your users to keep the files in their Dropbox/BOX.com account and your server can read from there - these services already take care of large file upload with all retries etc using their apps.
This answer is relevant to .Net as language.
We had such requirement, where we had created an executable. The executable internally called a web method, which validated the app authenticated to upload files to AWS S3 or NOT.
You can do this using a web browser too, but I would not suggest this, if you are targeting big files.
I have a post form that calls Amazon's S3 service. I am doing a direct from browser upload to amazon, and thus have to pass some additional information to Amazon, that I will not know until the file is uploaded (file-type, name, 'signature' which references both). http://aws.amazon.com/articles/1434
It is a large video file, which I don't want to upload to my server first, which (to the best of my knowledge) rules out using urllib2.open(...) to pass the additional variables to amazon as a POST request. I've been working at it for a couple days now, and haven't had any success with it. What would be the best way to do this?
Note: I will probably be using the uploadify plugin to upload the file. Thank you.
You're probably looking for Query String Request Authentication Alternative.
You can authenticate certain types of requests by passing the required
information as query-string parameters instead of using the
Authorization HTTP header. This is useful for enabling direct
third-party browser access to your private Amazon S3 data, without
proxying the request. The idea is to construct a "pre-signed" request
and encode it as a URL that an end-user's browser can retrieve.
Additionally, you can limit a pre-signed request by specifying an
expiration time.