Accessing QLDB ledger in another AWS account - python

I'm having trouble accessing the QLDB ledger in another AWS account.
I have granted necessary IAM permission for cross-account access.
I set the credentials in the EC2 where my python script is runing using below code.
sts_client = boto3.client("sts", region_name=region)
response = sts_client.assume_role(
RoleArn="arn:aws:iam::xxx:role/xxx-ec2",
RoleSessionName="RoleSessionname",
)
os.environ["AWS_ACCESS_KEY_ID"] = response["Credentials"]["AccessKeyId"]
os.environ["AWS_SECRET_ACCESS_KEY"] = response["Credentials"]["SecretAccessKey"]
os.environ["AWS_SESSION_TOKEN"] = response["Credentials"]["SessionToken"]
os.environ["AWS_DEFAULT_REGION"] = region
but keep on getting below error
in _get_session
raise ExecuteError(e, True, True)
pyqldb.errors.ExecuteError: Error containing the context of a failure during execute.
botocore.errorfactory.BadRequestException: An error occurred (BadRequestException) when calling the SendCommand operation: The Ledger with name my-ledger is not found
the error is thrown during the execution of the below code.
qldb_driver = QldbDriver(ledger_name='my-ledger', region_name='us-east-1')
result = qldb_driver.execute_lambda(lambda x: read_table(x, table_name))

Found out that the credentials can be passed to QldbDriver function per -> https://github.com/awslabs/amazon-qldb-driver-python/blob/master/pyqldb/driver/qldb_driver.py#L103

Related

Describing AWS SSM parameters using tags failing

I am running a lambda to fetch ssm parameters in AWS and I want to filter using tags. I have tried the method recommended by aws but I keep getting this error:
"errorMessage": "An error occurred (ValidationException) when calling the DescribeParameters operation: An error occurred while calling one AWS dependency service."
Any help as to why?
reference - https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-search.html
https://github.com/spulec/moto/blob/master/tests/test_ssm/test_ssm_boto3.py#L1043-L1045
This is the code that I am running:
import json
import boto3
ssm_client = boto3.client('ssm')
def lambda_handler(event, context):
print("Fetch group A ssm parameter: ")
leg_one_parameter = ssm_client.describe_parameters(
ParameterFilters=[{"Key": "tag:group", "Values":["A"]}]
)['Parameters']
parameter_name = list(map(lambda parameter: parameter['Name'], leg_one_parameter))
return parameter_name
It should work. Validation error may be produced by a bug in the boto3... Once or twice I had a similar issue.
I tried to execute your request and it passed.
leg_one_parameter = ssm.describe_parameters(
ParameterFilters=[{"Key": "tag:group", "Values":["A"]}]
)['Parameters']
leg_one_parameter
[]
I used boto3 in version 1.24.3

Unable to download file from S3 because "A client error (403) occurred when calling the HeadObject operation: Forbidden"

I am trying to download a file in code from an S3 bucket I created through AWS CDK, but got this error "A client error (403) occurred when calling the HeadObject operation: Forbidden". At first I thought it was because I didn't add s3:GetObject action to the IAM policy statement, but I still get that error. Below is how I created the bucket:
# S3
bucket = s3.Bucket(
self, "testS3Bucket", bucket_name=f"test_s3_bucket"
)
service_lambda.add_to_role_policy(
iam.PolicyStatement(
effect=iam.Effect.ALLOW,
actions=[
"s3:ListBucket",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:GetObject",
"s3:HeadObject",
],
resources=[bucket.arn_for_objects("*")],
)
)
Here is the code where I download the file from S3:
def download_file(self, file_name, s3_bucket):
try:
file = self.s3.Object(s3_bucket, file_name).load()
except ClientError as e:
if e.response["Error"]["Code"] == "404":
log.error("File does not exist for partner")
return {}
else:
raise e
except Exception as e:
raise e
return file
Does anybody know how I can get past this issue?
A simpler way to grant your lambda appropriate permissions would be something like this:
bucket = s3.Bucket(
self, "testS3Bucket", bucket_name=f"test_s3_bucket"
)
bucket.grant_read_write(service_lambda.role)
Based on docs
If an encryption key is used, permission to use the key for encrypt/decrypt will also be granted.
Give that a try and see if you still receive a permissions error

S3 unit tests boto client

Having issues writing a unit test for S3 client, it seems the test is trying to use a real s3 client rather than the one i have created for the test here is my example
#pytest.fixture(autouse=True)
def moto_boto(self):
# setup: start moto server and create the bucket
mocks3 = mock_s3()
mocks3.start()
res = boto3.resource('s3')
bucket_name: str = f"{os.environ['BUCKET_NAME']}"
res.create_bucket(Bucket=bucket_name)
yield
# teardown: stop moto server
mocks3.stop()
def test_with_fixture(self):
from functions.s3_upload_worker import (
save_email_in_bucket,
)
client = boto3.client('s3')
bucket_name: str = f"{os.environ['BUCKET_NAME']}"
client.list_objects(Bucket=bucket_name)
save_email_in_bucket(
"123AZT",
os.environ["BUCKET_FOLDER_NAME"],
email_byte_code,
)
This results in the following error
botocore.exceptions.ClientError: An error occurred (ExpiredToken) when calling the PutObject operation: The provided token has expired.
code i am testing looks like this
def save_email_in_bucket(message_id, bucket_folder_name, body):
s3_key = "".join([bucket_folder_name, "/", str(message_id), ".json"])
s3_client.put_object(
Bucket=bucket,
Key=s3_key,
Body=json.dumps(body),
ContentType="application-json",
)
LOGGER.info(
f"Saved email with messsage ID {message_id} in bucket folder {bucket_folder_name}"
)
Not accepting this an an answer but useful for anyone who ends up here, I found a workaround where if I create the s3 client in the function i am trying to test then this approach will work rather than create it globally. I would prefer to find an actual solution though.

Attaching IAM user to IAM Group using Boto3

I am trying to create an IAM policy, to an IAM group using Boto3. So far, I am not able to add IAM user to an IAM group. For "response = iam.add_user_to_group( ", I am getting an error, "Exception has occurred: AttributeError
'iam.ServiceResource' object has no attribute 'add_user_to_group'"
import boto3
iam = boto3.resource('iam') #using resource representing IAM
created_user = iam.create_user(
UserName='some_random_user'
)
print(created_user)
create_group_response = iam.create_group(GroupName = 'Tester')
response = iam.add_user_to_group(
UserName = 'some_random_user', #Name of user
GroupName = 'Tester'
)
response = iam.attach_user_policy(
UserName = 'some_random_user', #Name of user
PolicyArn = 'arn:aws:iam::196687784845:policy/boto-test'
# Policy ARN which you want to asign to user
)
Not sure what seems to be the problem, I am very new to python and boto so might be a very small thing.
According to the doc, add_user_to_group is an action for IAM client (not resource). Use this action on Group resource instead.

Error Getting Managed Identity Access Token from Azure Function

I'm having an issue retrieving an Azure Managed Identity access token from my Function App. The function gets a token then accesses a Mysql database using that token as the password.
I am getting this response from the function:
9103 (HY000): An error occurred while validating the access token. Please acquire a new token and retry.
Code:
import logging
import mysql.connector
import requests
import azure.functions as func
def main(req: func.HttpRequest) -> func.HttpResponse:
def get_access_token():
URL = "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fossrdbms-aad.database.windows.net&client_id=<client_id>"
headers = {"Metadata":"true"}
try:
req = requests.get(URL, headers=headers)
except Exception as e:
print(str(e))
return str(e)
else:
password = req.json()["access_token"]
return password
def get_mysql_connection(password):
"""
Get a Mysql Connection.
"""
try:
con = mysql.connector.connect(
host='<host>.mysql.database.azure.com',
user='<user>#<db>',
password=password,
database = 'materials_db',
auth_plugin='mysql_clear_password'
)
except Exception as e:
print(str(e))
return str(e)
else:
return "Connected to DB!"
password = get_access_token()
return func.HttpResponse(get_mysql_connection(password))
Running a modified version of this code on a VM with my managed identity works. It seems that the Function App is not allowed to get an access token. Any help would be appreciated.
Note: I have previously logged in as AzureAD Manager to the DB and created this user with all privileges to this DB.
Edit: No longer calling endpoint for VMs.
def get_access_token():
identity_endpoint = os.environ["IDENTITY_ENDPOINT"] # Env var provided by Azure. Local to service doing the requesting.
identity_header = os.environ["IDENTITY_HEADER"] # Env var provided by Azure. Local to service doing the requesting.
api_version = "2019-08-01" # "2018-02-01" #"2019-03-01" #"2019-08-01"
CLIENT_ID = "<client_id>"
resource_requested = "https%3A%2F%2Fossrdbms-aad.database.windows.net"
# resource_requested = "https://ossrdbms-aad.database.windows.net"
URL = f"{identity_endpoint}?api-version={api_version}&resource={resource_requested}&client_id={CLIENT_ID}"
headers = {"X-IDENTITY-HEADER":identity_header}
try:
req = requests.get(URL, headers=headers)
except Exception as e:
print(str(e))
return str(e)
else:
try:
password = req.json()["access_token"]
except:
password = str(req.text)
return password
But now I am getting this Error:
{"error":{"code":"UnsupportedApiVersion","message":"The HTTP resource that matches the request URI 'http://localhost:8081/msi/token?api-version=2019-08-01&resource=https%3A%2F%2Fossrdbms-aad.database.windows.net&client_id=<client_idxxxxx>' does not support the API version '2019-08-01'.","innerError":null}}
Upon inspection this seems to be a general error. This error message is propagated even if it's not the underlying issue. Noted several times in Github.
Is my endpoint correct now?
For this problem, it was caused by the wrong endpoint you request for the access token. We can just use the endpoint http://169.254.169.254/metadata/identity..... in azure VM, but if in azure function we can not use it.
In azure function, we need to get the IDENTITY_ENDPOINT from the environment.
identity_endpoint = os.environ["IDENTITY_ENDPOINT"]
The endpoint is like:
http://127.0.0.1:xxxxx/MSI/token/
You can refer to this tutorial about it, you can also find the python code sample in the tutorial.
In my function code, I also add the client id of the managed identity I created in the token_auth_uri but I'm not sure if the client_id is necessary here (In my case, I use user-assigned identity but not system-assigned identity).
token_auth_uri = f"{identity_endpoint}?resource={resource_uri}&api-version=2019-08-01&client_id={client_id}"
Update:
#r "Newtonsoft.Json"
using System.Net;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Primitives;
using Newtonsoft.Json;
public static async Task<IActionResult> Run(HttpRequest req, ILogger log)
{
string resource="https://ossrdbms-aad.database.windows.net";
string clientId="xxxxxxxx";
log.LogInformation("C# HTTP trigger function processed a request.");
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(String.Format("{0}/?resource={1}&api-version=2019-08-01&client_id={2}", Environment.GetEnvironmentVariable("IDENTITY_ENDPOINT"), resource,clientId));
request.Headers["X-IDENTITY-HEADER"] = Environment.GetEnvironmentVariable("IDENTITY_HEADER");
request.Method = "GET";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader streamResponse = new StreamReader(response.GetResponseStream());
string stringResponse = streamResponse.ReadToEnd();
log.LogInformation("test:"+stringResponse);
string name = req.Query["name"];
string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
dynamic data = JsonConvert.DeserializeObject(requestBody);
name = name ?? data?.name;
return name != null
? (ActionResult)new OkObjectResult($"Hello, {name}")
: new BadRequestObjectResult("Please pass a name on the query string or in the request body");
}
For your latest issue, where you are seeing UnsupportedApiVersion, it is probably this issue: https://github.com/MicrosoftDocs/azure-docs/issues/53726
Here are a couple of options that worked for me:
I am assuming you are hosting the Function app on Linux. I noticed that ApiVersion 2017-09-01 works, but you need to make additional changes (instead of "X-IDENTITY-HEADER", use "secret" header). And also use a system-assigned managed identity for your function app, and not a user assigned identity.
When I hosted the function app on Windows, I didn't have the same issues. So if you want to use an user-assigned managed identity, you can try this option instead. (with the api-version=2019-08-01, and X-IDENTITY-HEADER.

Categories

Resources