Can we access bucket with bucket endpoint like .s3.amazonaws.com using python sdk. i don't want access bucket with following bucket = conn.get_bucket(bucket_name).
I don't know why you need to access it this way because the s3 endpoint is a fixed part where only thing changes is the name of your bucket (because it's global).
But, in the end, what you are looking for is not possible unfortunately. You need to provide bucket name for accessing the bucket and running operations on it.
Verified by boto3 documentation and here you can check:
S3 Boto documentation
Related
Im having thousands of objects in all the folders gocc1, gocc2,etc
s3://awss3runner/gocc1/gocc2/goccf/
i just want to delete the objects(50,000+) from goccf and its versions
import boto3
session = boto3.Session()
s3 = session.resource(service_name='s3')
#bucket = s3.Bucket('awss3runner','goccf')if we use this getting error
bucket = s3.Bucket('awss3runner') # (working but if we use this everything in the bucket getting deleted)
bucket.object_versions.delete()
is there anyway to delete goccf objects and its versions
You can use the DeleteObjects API in S3 (https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html)
I would first perform a list operation to enumerate all the objects you wish to delete, then pass that into DeleteObjects. Be very careful as you could accidentally delete other objects in your bucket.
Another option, is to use an S3 lifecycle policy, if this is going to be a one-off operation. With a lifecycle policy you can specify a path in your S3 bucket and set the objects to Expire. They will be asynchronously removed from your S3 bucket https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-expire-general-considerations.html
This is a similar question to this: Is it possible to copy between AWS accounts using AWS CLI? The difference is, I want to do this in python code, and I can't change the s3 bucket policies in the source bucket (it's owned by a 3rd party). I do have the credentials to both buckets.
How do I run a sync command between these two buckets in python code?
To directly copy (eg with CopyObject) objects between Amazon S3 buckets in different accounts, you will need to use a single set of credentials that have:
Read permission on the source bucket
Write permission on the destination bucket
These credentials can come from either account. However, since you cannot change the Bucket policy on the source bucket to reference credentials from your account, you will need to use the credentials that they have provided to you.
Let's say the scenario is:
The source is Bucket-A in Account-A
The destination is Bucket-B in Account-B
You have IAM credentials from Account-A — let's call it User-A
User-A has permission to read from Bucket-A
You will need to :
Add a Bucket Policy to Bucket-B that permits User-A to write to the bucket (PutObject)
When performing the copy, specify "ACL": "bucket-owner-full-control", which will make the objects owned by the destination account. Without this, the objects will continue to be 'owned' by the Account-A even though it is in a bucket owned by Account-B
Finally, please note that boto3 does not natively provide a sync command. You will be responsible for all the sync logic, copying one object at a time.
Do it in Python, like this to call the AWS CLI
import subprocess
cmd='aws s3 sync s3://mybucket s3://mybucket2'
push=subprocess.Popen(cmd, shell=True, stdout = subprocess.PIPE)
print push.returncode
Or there abouts. :-) Wherever you run this from, say an EC2 instance, make sure it has the user or role that has valid permissions to access both buckets.
I have been given a bucket name with ARN Number as below:
arn:aws:iam::<>:user/user-name
I was also given an access key.
I know that this can be done using boto.
Connect to s3 bucket using IAM ARN in boto3
As in the above link do i need to use 'sts'?
if so why am i provided with an access key?
First, I recommend you install the AWS Command-Line Interface (CLI), which provides a command-line for accessing AWS.
You can then store your credentials in a configuration file by running:
aws configure
It will prompt you for the Access Key and Secret Key, which will be stored in a config file.
Then, you will want to refer to S3 — Boto 3 documentation to find out how to access Amazon S3 from Python.
Here's some sample code:
import boto3
client = boto3.client('s3', region_name = 'ap-southeast-2') # Change as appropriate
client.upload_file('/tmp/hello.txt', 'mybucket', 'hello.txt')
I am using AWS recognition on an S3 bucket of data that is currently located in the US-West-1 region. Unfortunately, AWS Rekognition is not supported in that region. I attempted to copy over my bucket into a US-West-2 region, but encountered difficulties in getting metadata. As such, my question is, how do I route my API call to another endpoint, specifically the endpoint 'https://rekognition.us-east-1.amazonaws.com' even though the bucket is based in another region. Any help or advice would be appreciated.
EDIT: I thought it may be relevant to mention, I am running this on Python.
Assuming you are using boto3 in your python script, you should be able to select a region when you create your client resource. Try doing something similar to this:
re_client= boto3.client('rekognition', region_name='us-east-1')
If your question is if you can use AWS Rekognition in one region to access a bucket in another region: As far as I know, you can't. However, you might be able to either migrate yor bucket to the specific region, or use S3 cross-region recplication to access the data from both regions.
I made a publicly listable bucket on google cloud storage. I can see all the keys if I try to list the bucket objects in the browser. I was trying to use the create_anonymous_client() function so that I can list the bucket keys in the python script. It is giving me an exception. I looked up everywhere and still can't find the proper way to use the function.
from google.cloud import storage
client = storage.Client.create_anonymous_client()
a = client.lookup_bucket('publically_listable_bucket')
a.list_blobs()
Exception I am getting:
ValueError: Anonymous credentials cannot be refreshed.
Additional Query: Can I list and download contents of public google cloud storage buckets using boto3, If yes, how to do it anonymously?
I was also struggling with thing and couldn't find an answer anywhere online. Turns out you can access the bucket with just the bucket() method.
I'm not sure why, but this method can take several seconds sometimes.
client = storage.Client.create_anonymous_client()
bucket = client.bucket('publically_listable_bucket')
blobs = list(bucket.list_blobs())
This error means the bucket you are attempting to list does not grant the right permission. You must Give "Storage Object Viewer" or "Storage Legacy Bucket Reader" role to "allUsers".