I would like to know about a more efficient way than renewing sts role for a cross account role when it run on lambda. By definition those roles last for 1h per default, but so far i'm doing it this way:
def aws_session(role_arn, session_name):
_ = boto3.client('sts')
resp = _.assume_role(RoleArn=role_arn, RoleSessionName=session_name)
session = boto3.Session(
aws_access_key_id=response['Credentials']['AccessKeyId'],
aws_secret_access_key=response['Credentials']['SecretAccessKey'],
aws_session_token=response['Credentials']['SessionToken'],
region_name='us-east-1')
return session
def lambda_handler(event, context):
session = aws_session(role_arn=ARN, session_name='CrossAccountLambdaRole')
s3_sts = session.resource('s3')
But it terribly inefficient because instead of ~300ms, renewing credentials take more than ~1500 ms each time and as we all know, we are charged on the duration execution. Anyone could help me on how to refresh this only when the token expire ? Coz between execution, we are not sure to endup using the same "container", so how to make global variable?
Thx a lot
Remove AssumeRole
I think your problem stems from the fact that your code is picking the role it needs on each run. Your assume role code should indeed be generating a new token on each call. I'm not familiar with the Python Boto library but in Node I only call AssumeRole when I'm testing locally and want to pull down new credentials, I save those credentials and never call assume role again until I want new creds. Every time I call assume role, I get new credentials as expected. You don't need STS directly to run your lambda functions.
An Alternate Approach:
For the production application my Lambda code does not pick its role. The automation scripts that build the Lambda function assign it a role and the lambda function will use that role for ever, with AWS managing the refresh of credentials on the back-end as they expire. You can do this by building your Lambda function in CloudFormation specifying what role you want it to use.
Lambda via CloudFormation
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-function.html
If you then want to view what credentials your function is operating with you can print out environment variables. Lambda will pass in temporary credentials to your function and the credentials will be associated with the role you've defined.
Simpler Approach
If you don't want to deal with CloudFormation deploy your function manually into the AWS console and in the console specify the role it should run with. But the bottom line is you don't need to use STS inside your lambda code. Assign the role externally.
Since your going across accounts, you obviously can't follow the advice many people say of attaching directly to the lambda.
Your best option is parameter store which is covered in detail here:
https://aws.amazon.com/blogs/compute/sharing-secrets-with-aws-lambda-using-aws-systems-manager-parameter-store/
Simply have lambda request the credentials from there instead.
That said, it's probably not going to save much time compared to STS requests... But I've not timed either process.
A perhaps less-good way, that's fairly simple, is to store the credentials in /TMP and build a process around enduring the credentials remain valid -- perhaps assume role with 65 minute duration, and save to a time stamped file with the minutes/seconds dropped. If the file exists, read it in by file I/O.
Keep in mind your saving credentials in a way that can be compromised if your code allows access to read the file in some way... Though as a lambda and with shared responsibility security, it's reasonably secure compared to doing this strategy on a persistent server.
Always use least privilege roles. Only allow your trusted account to assume this role... I think your can even lock trust policies down to a specific incoming lambda role as allowed to assume role in. This way leaked credentials by somehow reading/outputting the file require a malicious user to compromise some other aspect of your account (if locked down by account number only), or execute remote code execution inside your lambda itself (if locked to lambda).... Though, at that point, your credentials are already available to the malicious user to use anyways.
Related
I'm building a flask server in python with Cloud Run, for a chatbot to call.
Sometimes if user wants to do something with the chatbot, the bot need ask the user to login to a 3rd party server before doing the things.
I have two routes:
Route 1 is "/login", it returns a simple iframe which will open a login page in a 3rd party server, generate a "session_id", and save some info I already get to a global variable dict called "runtimes" with the "session_id" as key, so that I can use it later when visitor successfully logged in.
Route 2 is "/callback/<session_id>". After user successfully login to its account, the 3rd party server will call this route with a token in url parameters. Then I will use the "session_id" to read the saved info from "runtimes", and do later things.
It works well in my local machine. But in Google Cloud Run, because it support multiple instances, sometimes it will trigger a new instance when server calls "callback", so it cannot get the "runtime" because they are in different instances.
I know that I can save the runtimes dict to a database to solve this problem, but it looks too overkill...Just not seem right.
Is there any easy way that I can make the "runtimes" be shared between instances?
The solution here is to use a central point of storage: database, memorystore, firestore,... something out of Cloud Run itself.
You can also try the Cloud Run execution runtime v2 that allow you to mount a network disk, such as Cloud Storage or Filestore. You can imagine to store the session data in a file which has the name of the session ID.
Note: On Cloud Run side, something is cooking, but it's not 100% safe, it will be a best effort. A database backup will be required even with that new feature
I have a process that is supposed to run forever and needs to updates data on a S3 bucket on AWS. I am initializing the session using boto3:
session = boto3.session.Session()
my_s3 = session.resource(
"s3",
region_name=my_region_name,
aws_access_key_id=my_aws_access_key_id,
aws_secret_access_key=my_aws_secret_access_key,
aws_session_token=my_aws_session_token,
)
Since the process is supposed to run for days, I am wondering how I can make sure that the session is kept alive and working. Do I need to re-initialize the session sometimes?
Note: not sure if it is useful, but I have actually multiple threads each using its own session.
Thanks!
There is no concept of a 'session'. The Session() is simply an in-memory object that contains information about how to connect to AWS.
It does not actually involve any calls to AWS until an action is performed (eg ListBuckets). Actions are RESTful, and return results immediately. They do not keep open a connection.
A Session is not normally required. If you have stored your AWS credentials in a config file using the AWS CLI aws configure command, you can simply use:
import boto3
s3_resource = boto3.resource('s3')
The Session, however, is useful if you are using temporary credentials returned by an AssumeRole() command, rather than permanent credentials. In such a case, please note that credentials returned by AWS Security Token Service (STS) such as AssumeRole() have time limitations. This, however, is unrelated to the concept of a boto3 Session.
I have my .aws/credentials set as
[default]
aws_access_key_id = [key]
aws_secret_access_key = [secret! Shh!]
and .aws/config
[profile elevated]
role_arn = [elevated role arn]
source_profile = default
mfa_serial = [my device arn]
With the credentials and config files set up like that, boto3 will
automatically make the corresponding AssumeRole calls to AWS STS on your behalf. It will handle in
memory caching as well as refreshing credentials as needed
so that when I use something like
session = boto3.Session(profile_name = "elevated")
in a longer function, all I have to do is input my MFA code immediately after hitting "enter" and everything runs and credentials are managed independent of my input. This is great. I like that when I need to assume a role in another AWS account, boto3 handles all of the calls to sts and all I have to do is babysit.
What about when I don't want to assume another role? If I want to do things directly as my user as a member of the group to which my user is assigned? Is there a way to let boto3 automatically handle the credentials aspect of that?
I see that I can hard-code into a fx my aws_access_key_id and ..._secret_... , but is there a way to force boto3 into handling the session tokens by just using the config and credentials files?
Method 2 in this answer looked promising but it also seems to rely on using the AWS CLI to input and store the keys/session token prior to running a Python script and still requires hard-coding variables into a CLI.
Is there a way to make this automatic by using the config and credentials files that doesn't require having to manually input AWS access keys and handle session tokens?
If you are running the application on EC2, you can attach roles via EC2 Roles.
On your code, you may dynamically get the credentials depending on which role you attach.
session = boto3.
credentials = session.get_credentials().get_frozen_credentials()
access_key = credentials.access_key
secret_key = credentials.secret_key
token = credentials.token
you may also want to use botocore.credentials.RefreshableCredentials to refresh your token once in a while
I want to programatically get all the actions a user is allowed to do across aws services.
I've tried to fiddle with simulate_principal_policy but it seems this method expects a list of all actions, and I don't want to maintain a hard-coded list.
I also tried to call it with iam:* for example and got a generic 'implicitDeny' response so I know the user is not permitted all the actions but I require a higher granularity of actions.
Any ideas as to how do I get the action list dynamically?
Thanks!
To start with, there is no programmatic way to retrieve all possible actions (regardless of whether they are permitted to use an action).
You would need to construct a list of possible actions before checking the security. As an example, the boto3 SDK for Python contains an internal list of commands that it uses to validate commands before sending them to AWS.
Once you have a particular action, you could use Access the Policy Simulator API to validate whether a given user would be allowed to make a particular API call. This is much easier than attempting to parse the various Allow and Deny permissions associated with a given user.
However, a call might be denied based upon the specific parameters of the call. For example, a user might have permissions to terminate any Amazon EC2 instance that has a particular tag, but cannot terminate all instances. To correctly test this, an InstanceId would need to be provided to the simulation.
Also, permissions might be restricted by IP Address and even Time of Day. Thus, while a user would have permission to call an Action, where and when they do it will have an impact on whether the Action is permitted.
Bottom line: It ain't easy! AWS will validate permissions at the time of the call. Use the Policy Simulator to obtain similar validation results.
I am surprised no one has answered this question correctly. Here is code that uses boto3 that addresses the OP's question directly:
import boto3
session = boto3.Session('us-east-1')
for service in session.get_available_services ():
service_client = session.client (service)
print (service)
print (service_client.meta.service_model.operation_names)
IAM, however, is a special case as it won't be listed in the get_available_services() call above:
IAM = session.client ('iam')
print ('iam')
print (IAM.meta.service_model.operation_names)
I'm just starting exploring IAM Roles. So far I launched an instance, created an IAM Role. Everything seems to work as expected. Currently I'm using boto (Python sdk).
What I don't understand :
Does the boto takes care of credential rotation? (For example, imagine I have an instance that should be up for a long time, and it constantly have to upload keys to s3 bucket. In case if credentials are expired, do I need to 'catch' an exception and reconnect? or boto will silently do this for me?)
Is it possible to manually trigger IAM to change credentials on the Role? (I want to do this, because I want to test above example. Or if there is there an alternative to this testcase? )
The boto library does handle credential rotation. Or, rather, AWS rotates the credentials and boto automatically picks up the new credentials. Currently, boto does this by checking the expiration timestamp of the temporary credentials. If the expiration is within 5 minutes of the current time, it will query the metadata service on the instance for the IAM role credentials. The service is responsible for rotating the credentials.
I'm not aware of a way to force the service to rotate the credentials but you could probably force boto to look for updated credentials by manually adjusting the expiration timestamp of the current credentials.