What is the recommended way to let users access files that are stored on AWS S3?
I've currently found two ways:
using the access key ID and secret access key
making the whole bucket public and then accessing the files thru a public link.
Both options aren't really satisfying. Either I reveal the access/secret keys or all files are publicly available. Is there another third more secure way?
The software, that needs access to S3, will be running on Raspberry Pis. I was thinking of encrypting the credentials file, so a user wouldn't be able to read it easily.
The first thing is, to create a new individual AWS user with access key ID and secret for your project with and give the individual S3 permission to that user. Don't use admin credentials for a project. Every project should have a particular AWS user with its credentials and permissions. The second thing is, to rotate keys. So you create a new key with an old one and delete old key. More about rotation you can refer to Managing Access Keys for IAM Users documentation. Indeed, you can also encrypt it and AWS has the service AWS KMS. You can make research in AWS KMS service, it is a great tool for encryption. You can even encrypt your application vulnerable secret keys or passwords.
Related
Goal:
Using python, I want to create a service account in a project on the Google Cloud Platform and grant that service account one role.
Problem:
The docs explain here how to grant a single role to the service account. However, it seems to be only possible by using the Console or the gcloud tool, not with python. The alternative for python is to update the whole IAM policy of the project to grant the role for the single service account and overwrite it (described here). However, overwriting the whole policy seems quite risky because in case of an error the policy of the whole project could be lost. Therefore I want to avoid that.
Question:
I'm creating a service account using the python code provided here in the docs. Is it possible to grant the role already while creating the service account with this code or in any other way?
Creating a service account, creating a service account key, downloading a service account JSON key file, and granting a role are separate steps. There is no single API to create a service account and grant a role at the same time.
Anytime you update a project's IAM bindings is a risk. Google prevents multiple applications from updating IAM at the same time. It is possible to lock everyone (users and services) out of a project by overwriting the policy with no members.
I recommend that you create a test project and develop and debug your code against that project. Use credentials that have no permissions to your other projects. Otherwise use the CLI or Terraform to minimize your risks.
The API is very easy to use provided that you understand the API, IAM bindings, and JSON data structures.
As mentioned in John’s answer, you should be very careful when manipulating the IAM module, if something goes wrong it could end in services completely inoperable.
Here is a Google’s document which manipulates the IAM resources using the REST API.
The owner role can be granted to a user, serviceAccount, or a group that is part of an organization. For example, group#myownpersonaldomain.com could be added as an owner to a project in the myownpersonaldomain.com organization, but not the examplepetstore.com organization.
I have a Django app that will invoke an API call (a Python function) to create a folder with a private S3 bucket. To invoke the function, a user will have to be logged in first.
The S3 bucket will look like this:
s3://somebucket/user1
s3://somebucket/user2
and so on
API call: https://somedomain.com/createfolder/user1
I don't want to allow user2 to call the API by substituting the user1 string with his username.
Assuming I can secure this API call, my Python function will then just go ahead and create a folder with the username given to it.
user1 will also have the ability to download only his/her files.
users can log in to my app via OAuth2 or username/password.
The questions:
a. How do I secure my API so that user2 cannot commit actions on user1?
b. Similarly, how do I create pre-signed URLs pointing to user1's file (a specific file, which could be a zipped form of a folder?
My research: I have heard of JWT and pre-signed URLs, but I'm having difficulties in understanding how to practically implement them in Django.
I have several cloud functions (in Python) that require a modulair package auth in which there is a subfolder with credentials (containing mostly json files of Google Service Accounts files or Firebase configurations).
From a security perspective, I have obviously not included these files on the .git by adding the folder in the .gitignore file (auth/credentials).
However, I am now stuck with what to do when deploying the Google Cloud Function (.gcloudignore). If I deploy it with the credentials then I imagine that these keys are exposed on the server? How could I overcome this?
I have heard some speaking of environmental variables, but I am not sure if this is more secure than just deploying it?
What is the Google Way of doing it?
You have two primary solutions available to you. The first is that the Cloud Function can run with the identity of a custom Service Account. This service account can then be associated with all the roles necessary for your logic to achieve its task. The value of this is that no credentials need be explicitly known to your logic. The environment in which your calls are being made "implicitly" has all that it needs.
See: Per-function identity
The second mechanism which is more in line with what you are currently doing uses the concept of the Compute Metadata Server. This metadata can be configured with the tokens necessary to make on-ward calls. The metadata is configured separately from your Cloud Function logic which merely retrieves the data as needed.
See: Fetching identity and access tokens.
I have developed a Google Cloud Function (GCF) in python, which i want to access from a web service deployed on AWS (written in python). While in the development phase of the GCF, It had Cloud Function Invoker permission set to allUsers. I assume that is why it didn't ask for an Authorization Token when called.
I want to revoke this public access and make it so that i can only call this function from the web service code and it is not accessible public-ally.
Possible Approach :In my research i have found out that this can be done using the following steps:
Removing all the unnecessary members who have permissions to the GCF.
Creating a new service account which has restricted access to only use GCF.
Download the service account key (json) and use it in the AWS web application
Set environment variable GOOGLE_APPLICATION_CREDENTIALS equal to the path of that service account key (json) file.
Questions
How to generate the Access token using the service account, which may then be appended as Authorization Bearer within the HTTP call made to the GCF? Without this token the GCF should throw error.
The docs say not to put the service account key in the source code. Then what is the best way to go about it. They suggest to use KMS which seems like an overkill.
Do not embed secrets related to authentication in source code, such as API keys, OAuth tokens, and service account credentials. You can use an environment variable pointing to credentials outside of the application's source code, such as Cloud Key Management Service.
What are the bare minimum permissions i will require for the service account?
Please feel free to correct me if you think my understanding is wrong and there is a better and preferable way to do it.
UPDATE: The web service on AWS will call the GCF in a server-to-server fashion. There is no need to propagate the client-end (end-user) credentials.
In your description, you don't mention who/what will call your GCF. A user? A Compute? Another GCF? However, this page can help you to find code example
Yes, secret in plain text and pushed on GIT is not yet a secret! Here again, I don't know what performing the call. If it's a compute, functions, cloud run, or any service of GCP, don't use JSON file key, but the component identity. I would say, create a service account and set it to this component. Tell me more on where are you deploying if you want more help!
Related to 2: if you have a service account, what the minimal role: cloudfunctions.Invoker. It's the minimal role to invoke function
gcloud beta functions add-iam-policy-binding RECEIVING_FUNCTION \
--member='serviceAccount:CALLING_FUNCTION_IDENTITY' \
--role='roles/cloudfunctions.invoker'
I have a bunch of authentication files and certificates I'm using with Django (mostly around push notifications). Where should I store these?
Is it inadvisable to store these in my private repo? If yes, how and where should I keep them?
You should not store your secret in version control software even when you use private repository, that's a big risk.
An option to keep them safe is using secret service such as Hashicorp Vault.