Looking for some guidance with regards to uploading files into AWS S3 bucket via a python script and an IAM role. I am able to upload files using BOTO3 and an aws_access_key_id & aws_secret_access_key for other scripts.
However, I have now been given an IAM role to login to a certain account. I have no issue using AWS CLI to authenticate and query the S3 data so I do believe that my .aws/credential and .aws/config files are correct. However I am not sure how to use the ARN value within my python code.
This is what I have put together so far, but get a variety of errors which all lead to denied access:
session = boto3.Session(profile_name='randomName')
session.client('sts').get_caller_identity()
assumed_role_session = boto3.Session(profile_name='randomNameAccount')
print(assumed_role_session.client('sts').get_caller_identity())
credentials = session.get_credentials()
aws_access_key_id = credentials.access_key
aws_secret_access_key = credentials.secret_key
s3 = boto3.client('s3',
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key)
bucket_name = 'bucketName'
This is a sample of what my credential and config files looks like as a referal.
.aws/config file:
[profile randomNameAccount]
role_arn = arn:aws:iam::12345678910:role/roleName
source_profile = randomName
aws/credentials file:
[randomName]
aws_access_key_id = 12345678910
aws_secret_access_key = 1234567-abcdefghijk
My question is help around the python code to be able to authenticate against AWS and navigate around a S3 bucket using an IAM role and then upload files when I call an upload function.
Thank you in advance.
You should create an entry for the IAM Role in ~/.aws/credentials that refers to a set of IAM User credentials that have permission to assume the role:
[my-user]
aws_access_key_id = AKIAxxx
aws_secret_access_key = xxx
[my-role]
source_profile = my-user
role_arn = arn:aws:iam::123456789012:role/the-role
Add an entry to ~/.aws/config to provide a default region:
[profile my-role]
region = ap-southeast-2
Then you can assume the IAM Role with this code:
import boto3
# Create a session by assuming the role in the named profile
session = boto3.Session(profile_name='my-role')
# Use the session to access resources via the role
s3_client = session.client('s3')
response = s3_client.list_objects(Bucket=...)
Related
I am new to AWS S3. I did a lot of googling around how to connect S3 using python and found everyone is using Boto so its what I am using as the client. I used powershell to login and create the .aws/credentials . In that file, I was able to get the aws_access_key_id, aws_secret_access_key AND aws_session_token needed to establish the session. I understand that session is only about 8 hours so the next day when my python script runs to connect to S3 obviously the session is expired. How can i overcome this and how can I establish a new session daily? Below is my code.
s3_client = boto3.client(
"s3",
aws_access_key_id=id_,
aws_secret_access_key=secret,
aws_session_token=token,
region_name='r'
)
# Test it on a service (yours may be different)
# s3 = session.resource('s3')
# Print out bucket names
for bucket in s3.buckets.all():
# print(bucket.name)
bucket = 'automated-reports' # already created on S3
csv_buffer = StringIO()
all_active_scraper_counts_df.to_csv(csv_buffer, index=False)
# s3_resource = boto3.resource('s3')
put_response = s3_client.put_object(Bucket=bucket, Key="all_active_scrapers.csv", Body=csv_buffer.getvalue())
status = put_response.get("ResponseMetadata", {}).get("HTTPStatusCode")
if status == 200:
print(f"Successful S3 put_object response. Status - {status}")
else:
print(f"Unsuccessful S3 put_object response. Status - {status}")
You have three options:
Run a script that updates the credentials before you run your Python script. You can use AWS CLI sts assume-role to get a new set of credentials.
Add a try/catch statement inside your code to handle credentials expired error. Then, generate new credentials and re-initialize the S3 client.
Instead of using a Role, use a User. (IAM Identities). User credentials can be valid forever. You won't need to update the credentials in this case.
I am facing something similar to How to load file from custom hosted Minio s3 bucket into pandas using s3 URL format?
however, I already have an initialized s3 session (from boto3).
How can I get the credentials returned from it to feed these directly to pandas?
I.e. how can I extract the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from the initialized boto3 s3 client?
You can use session.get_credentials
import boto3
session = boto3.Session()
credentials = session.get_credentials()
AWS_ACCESS_KEY_ID = credentials.access_key
AWS_SECRET_ACCESS_KEY = credentials.secret_key
AWS_SESSION_TOKEN = credentials.token
If you only have access to boto client (like the S3 client), you can find the credentials hidden here:
client = boto3.client("s3")
client._request_signer._credentials.access_key
client._request_signer._credentials.secret_key
client._request_signer._credentials.token
If you don't want to handle credentials (I assume you're using the SSO here), you can load the S3 object directly with pandas: pd.read_csv(s3_client.get_object(Bucket='Bucket', Key ='FileName').get('Body'))
I am trying to move files from a S3 bucket in one account(source account) to S3 bucket in another account(destination account)
I am using sagemaker notebook so I have a sagemaker role.
I also have a role in my team account which has full s3 access and fullsagemaker access and in the trust relationship i have given the destination account role arn and sagemaker role arn.
The destination account also has my team role arn and sagemaker role arn in its trust policy.
I am trying to assume my team role and then I will assume the destination role to copy files.
import boto3
sts_client = boto3.client('sts')
assumed_teamrole_object = sts_client.assume_role(DurationSeconds=1800,
RoleArn='myteamrole',
RoleSessionName='test1')
assumed_destrole_object = sts_client.assume_role(DurationSeconds=1800,
ExternalId='externalid provided by destination account',
RoleArn='destination account role',
RoleSessionName='test2')
temp_credentials = assumed_destrole_object['Credentials']
session=boto3.session.Session(aws_access_key_id=temp_credentials['Access KeyyId'],
aws_secret_access_key=temp_credentials['SecretAccessKey'],
aws_session_token=temp_credentials['SessionToken'],
region_name = 'us-east-1')
client = session.client('s3', aws_access_key_id=temp_credentials['AccessKeyId'],
aws_secret_access_key=temp_credentials['SecretAccessKey'],
aws_session_token=temp_credentials['SessionToken'],
region_name = 'us-east-1')
response = client.list_objects(Bucket='source bucket')
print(response)
When I am running the above script I a getting the error :
An error occurred (AccessDenied) when calling the ListObjects operation: Access Denied
The objects in the source bucket are encrypted. Do I have to add any-permissions to decrypt on my end? Not sure why i am not able to list objects.
When copying files between S3 buckets that belong to different AWS accounts, you will need a single set of credentials that can read from the source bucket and write to the destination bucket.
If, instead, you are using two different credentials, then you will need to download the file with one set of credentials and then upload with another set of credentials, rather than copying the object in one operation.
Therefore, I recommend that you use one set of credentials (eg the myteamrole IAM Role) and then:
Attach a policy to the IAM Role that permits GetObject access on the source bucket, and
Attach a bucket policy to the destination bucket in the other AWS account that permits PutObject access from the above IAM Role
This will permit the CopyObject() operation with the one set of credentials.
I also recommend specifying ACL = bucket-owner-full-control when copying the object. This will grant ownership of the object to the destination AWS Account, which can avoid some permission problems. This will also require PutObjectAcl permissions on the Bucket Policy.
I want Boto3 to get the access and secret key from a config file instead of hard coding them. On my Linux server I set the following environment variable AWS_SHARED_CREDENTIALS_FILE with the value /app/.aws/credentials. In /app/.aws/ I put a file with the name credentials with the following content:
[default]
aws_access_key_id = abcd
aws_secret_access_key = abcd
Of course I used the actual keys instead of abcd.
Python:
import boto3
conn = boto3.client('s3',
region_name="eu-west-1",
endpoint_url="endpoint",
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key,
config=Config(signature_version="s3", s3={'addressing_style': 'path'}))
However it says name'aws_access_key_id' is not defined. How can I fix it? Thanks
Edit:
>>> os.environ['AWS_SHARED_CREDENTIALS_FILE']
'/app/.aws/credentials'
If you have credentials folder with aws credentials already created, this means you don't need to specify them when instantiating your client. The following should work:
import boto3
conn = boto3.client('s3',
region_name="eu-west-1",
endpoint_url="endpoint",
config=Config(signature_version="s3", s3={'addressing_style': 'path'}))
If you are running your application on an EC2 instance, you can also assign a S3 role to your instance and have boto assume that role. Prevents you from having to store keys in your instance.
Look at the section "Assume Role Provider" in the docs:
http://boto3.readthedocs.io/en/latest/guide/configuration.html
Link to IAM roles as well:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
I am running file upload to AWS S3 python script using a cron job.
The result of the cron job is sent via mail to me.
Everything is working fine except the extra message that I'm getting in mail.
The error is sh: aws: command not found.
I tried setting the PATH environment variable in the script;
os.environ["PATH"] = "usr/bin:/usr/local/bin"
and I got rid of the error, but a new one showed up The config profile (247-datapusher-s3) could not be found
Just want to get rid of both of these messages.
*Note: The script is working perfectly in both the cases and the files are successfully uploaded to AWS S3 bucket.
Thanks in advance!! :)
You can check your config file. Edit ~/.aws/config
you will get something like
[default]
aws_access_key_id = KEY_ID
aws_secret_access_key = KEY_PASS
region = us-east-1 (or another region)
[247-datapusher-s3]
aws_access_key_id = KEY_ID
aws_secret_access_key = KEY_PASS
region = us-east-1 (or another region)
you will need to change to
[default]
aws_access_key_id = KEY_ID
aws_secret_access_key = KEY_PASS
region = us-east-1 (or another region)
[profile 247-datapusher-s3]
aws_access_key_id = KEY_ID
aws_secret_access_key = KEY_PASS
region = us-east-1 (or another region)
If you do not even see the 247-datapusher-s3 section and you're using it, you will need to add in your config file.