Develop Reference

Download Blob From Blob Storage Using Python

I am trying to download an excel file on a blob. However, it keeps generating the error "The specified blob does not exist". This error happens at blob_client.download_blob() although I can get the blob_client. Any idea why or other ways I can connect using managed identity?
default_credential = DefaultAzureCredential()
blob_url = BlobServiceClient('url', credential = default_credential)
container_client = blob_url.get_container_client('xx-xx-data')
blob_client = container_client.get_blob_client('TEST.xlsx')
downloaded_blob = blob_client.download_blob()
df=pd.read_excel(downloaded_blob.content_as_bytes(), sheet_name='Test',skiprows=2)

Turns out that I have to also provide 'Reader' access on top of 'Storage Blob Data Contributor' to be able to identify the blob. There was no need for SAS URL.

The reason you're getting this error is because each request to Azure Blob Storage must be an authenticated request. Only exception to this is when you're reading (downloading) a blob from a public blob container. In all likelihood, the blob container holding this blob is having a Private ACL and since you're sending an unauthenticated request, you're getting this error.
I would recommend using a Shared Access Signature (SAS) URL for the blob with Read permission instead of simple blob URL. Since a SAS URL has authorization information embedded in the URL itself (sig portion), you should be able to download the blob provided SAS is valid and has not expired.
Please see this for more information on Shared Access Signature: https://learn.microsoft.com/en-us/rest/api/storageservices/delegate-access-with-shared-access-signature.

Move all files in s3 bucket from s3 account to another using boto3

I'm trying to move the contents of a bucket from account-a to a bucket in account-b which I already have the credentials for both of them.
Here's the code I'm currently using:
import boto3
SRC_AWS_KEY = 'src-key'
SRC_AWS_SECRET = 'src-secret'
DST_AWS_KEY = 'dst-key'
DST_AWS_SECRET = 'dst-secret'
srcSession = boto3.session.Session(
aws_access_key_id=SRC_AWS_KEY,
aws_secret_access_key=SRC_AWS_SECRET
)
dstSession = boto3.session.Session(
aws_access_key_id=DST_AWS_KEY,
aws_secret_access_key=DST_AWS_SECRET
)
copySource = {
'Bucket': 'src-bucket',
'Key': 'test-bulk-src'
}
srcS3 = srcSession.resource('s3')
dstS3 = dstSession.resource('s3')
dstS3.meta.client.copy(CopySource=copySource, Bucket='dst-bucket', Key='test-bulk-dst', SourceClient=srcS3.meta.client)
print('success')
The problem is that when I specify a file's name in the field Key followed by /file.csv it works really fine, but when I set it to copy the whole folder, as showed in the code, it fails and throws this exception:
botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found
What I need to do is to move the contents in one call, not by iterating through the contents of the src-folder, because this is time/money consuming, as I may have thousands of files to be moved.

There is no API call in Amazon S3 to copy folders. (Folders do not actually exist — the Key of each object includes its full path.)
You will need to iterate through each object and copy it.
The AWS CLI (written in Python) provides some higher-level commands that will do this iteration for you:
aws s3 cp --recursive s3://source-bucket/folder/ s3://destination-bucket/folder/
If the buckets are in different accounts, I would recommend:
Use a set of credentials for the destination account (avoids problems with object ownership)
Modify the bucket policy on the source bucket to permit access by the credentials from the destination account (avoids the need to use two sets of credentials)

How to query AWS CloudSearch domain using Python boto3 library?

I'm trying to use boto3 to query my CloudSearch domain using the docs as a guide: http://boto3.readthedocs.io/en/latest/reference/services/cloudsearchdomain.html#client
import boto3
import json
boto3.setup_default_session(profile_name='myprofile')
cloudsearch = boto3.client('cloudsearchdomain')
response = cloudsearch.search(
query="(and name:'foobar')",
queryParser='structured',
returnFields='address',
size=10
)
print( json.dumps(response) )
...but it fails with:
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://cloudsearchdomain.eu-west-1.amazonaws.com/2013-01-01/search"
But how am I supposed to set or configure the endpoint or domain that I want to connect to? I tried adding an endpoint parameter to the request, thinking maybe it was an accidental omission from the docs, but I got this error response:
Unknown parameter in input: "endpoint", must be one of: cursor, expr, facet, filterQuery, highlight, partial, query, queryOptions, queryParser, return, size, sort, start, stats
The docs say:
The endpoint for submitting Search requests is domain-specific. You submit search requests to a domain's search endpoint. To get the search endpoint for your domain, use the Amazon CloudSearch configuration service DescribeDomains action. A domain's endpoints are also displayed on the domain dashboard in the Amazon CloudSearch console.
I know what my search endpoint is, but how do I supply it?

I found a post on a Google forum with the answer. You have to add the endpoint_url parameter into the client constructor e.g.
client = boto3.client('cloudsearchdomain', endpoint_url='http://...')
I hope those docs get updated, because I wasted a lot of time before I figured that out.

import boto3
client = boto3.client('cloudsearchdomain',
aws_access_key_id= 'access-key',
aws_secret_access_key= 'some-secret-key',
region_name = 'us-east-1', # your chosen region
endpoint_url= 'cloudsearch-url'
# endpoint_url is your Search Endpoint as defined in AWS console
)
response = client.search(
query='Foo', # your search string
size = 10
)
Reference response['hits'] for returned results.

How to access image by url on s3 using boto3?

What I want to accomplish is to generate a link to view the file (ex.image or pdf). The item is not accessible by URL (https://[bucket].s3.amazonaws.com/img_name.jpg), I think because its private and not public? (I'm not the owner of the bucket, but he gave me the access_key and secret_key?)
For now, all I can do is to download a file with this code.
s3.Bucket('mybucket').download_file('upload/nocturnes.png', 'dropzone/static/pdf/download_nocturnes.png')
I want to access an image on s3 so I can put it on an HTML, can I view it using the access and secret key?. Thank you for those who can help!

You can accomplish this using a pre-signed URL using the generate_presigned_url function. The caveat is that pre-signed URLs must have an expiration date. I was unable to find documentation on the maximum allowed duration. Here is an example:
url = s3.generate_presigned_url('get_object',
Params={
'Bucket': 'mybucket',
'Key': 'upload/nocturnes.png',
},
ExpiresIn=3600)
print url

For people who want to use generate_presigned_url for a public object and therefore don't want to do the signing part that appends credentials, the best solution I found is to still to use the generate_presigned_url, just that the Client.Config.signature_version needs to be set to botocore.UNSIGNED.
The following returns the public link without the signing stuff.
config.signature_version = botocore.UNSIGNED
boto3.client('s3', config=config).generate_presigned_url('get_object', ExpiresIn=0, Params={'Bucket': bucket, 'Key': key})
The relevant discussions on the boto3 repository are:
https://github.com/boto/boto3/issues/110
https://github.com/boto/boto3/issues/169
https://github.com/boto/boto3/issues/1415

boto3: generate_presigned_url expiring prematurely

I'm using the following code to generate a signed URL:
client.generate_presigned_url('get_object',
{'Bucket': client.bucket, 'Key': s3_filename}, ExpiresIn=expiry_seconds, HttpMethod='GET')
This generates a URL that looks something like this:
https://[bucket_name].s3.amazonaws.com/[path]?AWSAccessKeyId=[access-key-id]&Signature=[signature]&Expires=1478646479&x-amz-security-token=[long_token_string]
The 'Expires' timestamp is 4 days in the future (at the time of writing) and yet when clicking on the link, I get the 'Token has expired' XML response. The link does work for a few hours after generation.
Is there anything else that I should be looking at?
Thanks.

The expiry time of 1478646479 equates to 2016-11-08T23:07:59+00:00, which is in the future, so the expiry time being generated appears correct. (If the clock was wrong on the computer generating the link, this might have been wrong.)
The next thing to check are the permissions associated with the entity (identified by the AWSAccessKeyId) that created the pre-signed URL. In effect, the URL is using the permissions of that entity (eg IAM User, IAM Role, STS credentials) to grant time-limited access to an object. If that entity no longer has permission to access the object, then the pre-signed URL will no longer function.
You can perform tests on pre-signed URLs by using the aws s3 presign command in the AWS Command-Line Interface (CLI).

If you created a presigned URL using a temporary token, then the URL expires when the token expires. To have a URL valid for up to seven days you need to assign IAM user credentials.
Signature Version 4 is a requirement for this to work.
Here is what this looks like in boto3.
import boto3
from botocore.client import Config
s3 = boto3.client('s3',
config=Config(signature_version='s3v4'),
region_name = 'us-east-1',
aws_access_key_id = ‘THE_ID’,
aws_secret_access_key = ‘THE_KEY’
)
response = s3.generate_presigned_url('get_object',
Params={'Bucket': bucket_name,
'Key': ‘key_name’
},
ExpiresIn=expiration
)