I am not receiving any messages in my SQS queue when subscribing to an SNS topic via boto3.
Is this an issue with the code or the API credentials I am using? The IAM policy associated with this account has AWS PowerUser privileges, which should mean it has unrestricted access to manage SNS topics and SQS queues.
When I create the equivalent structure through the AWS console (create topic, create queue, subscribe queue to topic) and send a message using either boto3, the AWS CLI, or the AWS console, the message comes through correctly.
I don't think it is an issue with the code because the SubscriptionArn is being returned correctly?
I have tried this with both the US-EAST-1 and AP-SE-1 regions, same result.
Sample code:
#!/usr/bin/env python3
import boto3
import json
def get_sqs_msgs_from_sns():
sqs_client = boto3.client('sqs', region_name='us-east-1')
sqs_obj = boto3.resource('sqs', region_name='us-east-1')
sns_client = boto3.client('sns', region_name='us-east-1')
sqs_queue_name = 'queue1'
topic_name = 'topic1'
# Create/Get Queue
sqs_client.create_queue(QueueName=sqs_queue_name)
sqs_queue = sqs_obj.get_queue_by_name(QueueName=sqs_queue_name)
queue_url = sqs_client.get_queue_url(QueueName=sqs_queue_name)['QueueUrl']
sqs_queue_attrs = sqs_client.get_queue_attributes(QueueUrl=queue_url,
AttributeNames=['All'])['Attributes']
sqs_queue_arn = sqs_queue_attrs['QueueArn']
if ':sqs.' in sqs_queue_arn:
sqs_queue_arn = sqs_queue_arn.replace(':sqs.', ':')
# Create SNS Topic
topic_res = sns_client.create_topic(Name=topic_name)
sns_topic_arn = topic_res['TopicArn']
# Subscribe SQS queue to SNS
sns_client.subscribe(
TopicArn=sns_topic_arn,
Protocol='sqs',
Endpoint=sqs_queue_arn
)
# Publish SNS Messages
test_msg = {'default': {"x":"foo","y":"bar"}}
test_msg_body = json.dumps(test_msg)
sns_client.publish(
TopicArn=sns_topic_arn,
Message=json.dumps({'default': test_msg_body}),
MessageStructure='json')
# Validate Message
sqs_msgs = sqs_queue.receive_messages(
AttributeNames=['All'],
MessageAttributeNames=['All'],
VisibilityTimeout=15,
WaitTimeSeconds=20,
MaxNumberOfMessages=5
)
assert len(sqs_msgs) == 1
assert sqs_msgs[0].body == test_msg_body
print(sqs_msgs[0].body) # This should output dict with keys Message, Type, Timestamp, etc., but only returns the test_msg
if __name__ == "__main__":
get_mock_sqs_msgs_from_sns()
I receive this output:
$ python .\sns-test.py
Traceback (most recent call last):
File ".\sns-test.py", line 55, in <module>
get_sqs_msgs_from_sns()
File ".\sns-test.py", line 50, in get_sqs_msgs_from_sns
assert len(sqs_msgs) == 1
AssertionError
The URL above for the similar question posed for the C# AWS SDK put me in the correct direction for this: I needed to attach a policy to the SQS queue to allow the SNS topic to write to it.
def allow_sns_to_write_to_sqs(topicarn, queuearn):
policy_document = """{{
"Version":"2012-10-17",
"Statement":[
{{
"Sid":"MyPolicy",
"Effect":"Allow",
"Principal" : {{"AWS" : "*"}},
"Action":"SQS:SendMessage",
"Resource": "{}",
"Condition":{{
"ArnEquals":{{
"aws:SourceArn": "{}"
}}
}}
}}
]
}}""".format(queuearn, topicarn)
return policy_document
and
policy_json = allow_sns_to_write_to_sqs(topic_arn, queue_arn)
response = sqs_client.set_queue_attributes(
QueueUrl = queue_url,
Attributes = {
'Policy' : policy_json
}
)
print(response)
Related
I am trying to use the pub sub service on my python application. When I am running the code it get stuck on the last publisher line for some reason and the code never end. The subscriber seems fine.Does someone know what is wrong with my code?
Publisher:
import os
from google.cloud import pubsub_v1
credentials_path = 'PATH/TO/THE/KEY.JSON'
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = credentials_path
publisher = pubsub_v1.PublisherClient()
topic_path = 'projects/PROJECT_NAME/topics/TOPIC_NAME'
# simple garbage text to check if it's working
data = 'A garden sensor is ready!'
data = data.encode('utf-8')
attributes = {
'sensorName': 'garden-001',
'temperature': '75.0',
'humidity': '60'
}
future = publisher.publish(topic_path, data, **attributes)
print(f'published message id {future.result()}') # here it is just waiting forever
Subscriber:
import os
from google.cloud import pubsub_v1
from concurrent.futures import TimeoutError
credentials_path = 'PATH/TO/THE/KEY.JSON'
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = credentials_path
subscriber = pubsub_v1.SubscriberClient()
subscription_path = 'projects/PROJECT_NAME/subscriptions/SUBSCRIPTION_NAME'
def callback(message):
print(f'Received message: {message}')
print(f'data: {message.data}')
if message.attributes:
print("Attributes:")
for key in message.attributes:
value = message.attributes.get(key)
print(f"{key}: {value}")
message.ack()
streaming_pull_future = subscriber.subscribe(
subscription_path, callback=callback)
print(f'Listening for messages on {subscription_path}')
# wrap subscriber in a 'with' block to automatically call close() when done
with subscriber:
try:
streaming_pull_future.result()
except TimeoutError:
streaming_pull_future.cancel()
# block until the shutdown is complete
streaming_pull_future.result()
Google provides decent documentation for using its services including Pub/Sub including a basic Python example that would have helped you avoid your problem.
Aside: your publisher and subscriber snippets set GOOGLE_APPLICATION_CREDENTIALS statically within the code. Don't do this! Set the environment variable before running the code. This way, you can revise the value without changing the code but, more importantly, the value can be set by the runtime e.g. Compute Engine.
Here's a working example based on your code using Application Default Credentials obtained from the environment:
Q="74535931"
BILLING="[YOUR-BILLING-ID]"
PROJECT="$(whoami)-$(date %y%m%d)-${Q}"
gcloud projects create ${PROJECT}
gcloud beta billing projects link ${PROJECT} \
--billing-account=${BILLING}
gcloud services enable pubsub.googleapis.com \
--project=${PROJECT}
ACCOUNT=tester
EMAIL=${ACCOUNT}#${PROJECT}.iam.gserviceaccount.com
gcloud iam service-accounts create ${ACCOUNT} \
--project=${PROJECT}
gcloud iam service-accounts keys create ${PWD}/${ACCOUNT}.json \
--iam-account=${EMAIL}
gcloud projects add-iam-policy-binding ${PROJECT} \
--member=serviceAccount:${EMAIL} \
--role=roles/pubsub.editor
export GOOGLE_APPLICATION_CREDENTIALS=${PWD}/${ACCOUNT}.json
export PROJECT
export PUB="pub"
export SUB="sub"
gcloud pubsub topics create ${PUB} \
--project=${PROJECT}
gcloud pubsub subscriptions create ${SUB} \
--topic=${PUB} \
--project=${PROJECT}
publish.py:
import os
from google.cloud import pubsub_v1
project = os.getenv("PROJECT")
topic = os.getenv("PUB")
topic_path = f"projects/{project}/topics/{topic}"
data = 'A garden sensor is ready!'
data = data.encode('utf-8')
attributes = {
'sensorName': 'garden-001',
'temperature': '75.0',
'humidity': '60'
}
publisher = pubsub_v1.PublisherClient()
future = publisher.publish(topic_path, data, **attributes)
print(f'published message id {future.result()}')
subscribe.py:
import os
from google.cloud import pubsub_v1
from concurrent.futures import TimeoutError
project=os.getenv("PROJECT")
subscription=os.getenv("SUB")
subscription_path = f"projects/{project}/subscriptions/{subscription}"
def callback(message):
print(f'Received message: {message}')
print(f'data: {message.data}')
if message.attributes:
print("Attributes:")
for key in message.attributes:
value = message.attributes.get(key)
print(f"{key}: {value}")
message.ack()
subscriber = pubsub_v1.SubscriberClient()
streaming_pull_future = subscriber.subscribe(
subscription_path, callback=callback)
print(f'Listening for messages on {subscription_path}')
with subscriber:
try:
streaming_pull_future.result()
except TimeoutError:
streaming_pull_future.cancel()
# block until the shutdown is complete
streaming_pull_future.result()
Run python3 subscribe.py:
python3 subscribe.py
Listening for messages on projects/{project}/subscriptions/{sub}
Received message: Message {
data: b'A garden sensor is ready!'
ordering_key: ''
attributes: {
"humidity": "60",
"sensorName": "garden-001",
"temperature": "75.0"
}
}
data: b'A garden sensor is ready!'
Attributes:
humidity: 60
temperature: 75.0
sensorName: garden-001
And in a separate window python3 publish.py:
python3 publish.py
published message id 1234567890123456
I have a json file in S3 bucket and I am pushing that file into kafka topic, Can some one guide me why I am not able to see this data in Kafka topic. My Program has no errors and running correctly.
from kafka import KafkaProducer
import json
import time
import boto3
def json_serializer(data):
return json.dumps(data).encode("utf-8")
producer = KafkaProducer(bootstrap_servers= ['localhost:9092'],
value_serializer=json_serializer)
def read_s3():
s3 = boto3.resource('s3')
bucket = s3.Bucket('s3sparkbucket')
for obj in bucket.objects.all():
key = obj.key
body = obj.get()['Body'].read().decode('utf-8')
return body
if __name__ == "__main__":
body=read_s3()
producer.send("Uber_Eats",body)
print("Done")
KafkaProducer must need 3 params:
bootstrap.servers key.serializer and value.serializer
May be caused by that
I'm attempting to write a GCP Cloud Function in Python that calls the API for creating an IoT device. The initial challenge seems to be getting the appropriate module (specifically iot_v1) loaded within Cloud Functions so that it can make the call.
Example Python code from Google is located at https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/iot/api-client/manager/manager.py. The specific call desired is shown in "create_es_device". Trying to repurpose that into a Cloud Function (code below) errors out with "ImportError: cannot import name 'iot_v1' from 'google.cloud' (unknown location)"
Any thoughts?
import base64
import logging
import json
import datetime
from google.auth import compute_engine
from apiclient import discovery
from google.cloud import iot_v1
def handle_notification(event, context):
#Triggered from a message on a Cloud Pub/Sub topic.
#Args:
# event (dict): Event payload.
# context (google.cloud.functions.Context): Metadata for the event.
#
pubsub_message = base64.b64decode(event['data']).decode('utf-8')
logging.info('New device registration info: {}'.format(pubsub_message))
certData = json.loads(pubsub_message)['certs']
deviceID = certData['device-id']
certKey = certData['certificate']
projectID = certData['project-id']
cloudRegion = certData['cloud-region']
registryID = certData['registry-id']
newDevice = create_device(projectID, cloudRegion, registryID, deviceID, certKey)
logging.info('New device: {}'.format(newDevice))
def create_device(project_id, cloud_region, registry_id, device_id, public_key):
# from https://cloud.google.com/iot/docs/how-tos/devices#api_1
client = iot_v1.DeviceManagerClient()
parent = client.registry_path(project_id, cloud_region, registry_id)
# Note: You can have multiple credentials associated with a device.
device_template = {
#'id': device_id,
'id' : 'testing_device',
'credentials': [{
'public_key': {
'format': 'ES256_PEM',
'key': public_key
}
}]
}
return client.create_device(parent, device_template)
You need to have the google-cloud-iot project listed in your requirements.txt file.
See https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/iot/api-client/manager/requirements.txt
I am currently using python GCP API to create a cloud task queue. My codes are modified from the sample code and the logic is to check if the queue exists or not, if not create a new queue and put new task to that queue. so I use try-except and import from google.api_core import exceptions to handle the error. But the problem right now it keeps saying that my service account don't have permission cloud task. Here is the error.
google.api_core.exceptions.PermissionDenied
google.api_core.exceptions.PermissionDenied: 403 The principal (user or service account) lacks IAM permission "cloudtasks.tasks.create" for the resource "projects/xxxx/locations/us-central1" (or the resource may not exist).
here is my code.
#app.route('/train_model/<dataset_name>/<dataset_id>/', methods=["POST", "GET"])
def train_model(dataset_name,dataset_id):
if request.method == 'POST':
form = request.form
model = form.get('model_name')
date = form.get('date')
datetime_object = datetime.strptime(date, '%Y-%m-%d %H:%M:%S')
timezone = pytz.timezone('Asia/Hong_Kong')
timezone_date_time_obj = timezone.localize(datetime_object)
data=[dataset_id,model]
payload = str(data).encode()
# Create a client.
url = "https://us-central1-xxx.cloudfunctions.net/create_csv"
try:
client = tasks_v2.CloudTasksClient.from_service_account_json(
'./xxxxx.json')
url = "https://us-central1-xxxxxx.cloudfunctions.net/create_csv"
location = 'us-central1'
project = 'xxxxx'
queue = 'testing1'
parent = client.location_path(project, location)
task = {
"http_request": {
'http_method': 'POST',
'url': url,
'body': payload
}}
# set schedule time
timestamp = timestamp_pb2.Timestamp()
timestamp.FromDatetime(timezone_date_time_obj)
task['schedule_time'] = timestamp
response = client.create_task(parent, task)
except exceptions.FailedPrecondition:
location = 'us-central1'
project = 397901391776
# Create a client.
client = tasks_v2.CloudTasksClient.from_service_account_json(
"./xxxx.json")
parent = client.location_path(project, location)
queue = {"name": 'x'}
queue.update(name="projects/xxxxx/locations/us-west2/queues/" + queue #the name of the queue from try.)
response = client.create_queue(parent, queue)
parent = client.queue_path(project, location, queue)
task = {
"http_request": {
'http_method': 'POST',
'url': url,
'body':payload
}}
# set schedule time
timestamp = timestamp_pb2.Timestamp()
timestamp.FromDatetime(timezone_date_time_obj)
task['schedule_time'] = timestamp
response = client.create_task(parent, task)
print(response)
return redirect('/datasetinfo/{}/{}/'.format(dataset_name,dataset_id))
the permission of my service account
I have reproduced your scenario and I managed to get the same issue. The problem is not with the authentication but that the resource doesn't exist.
In order to get the resource path, instead of using the function location_path you should use queue_path. This way, the variable parent will contain the queue's name and the call create_task will be able to find the resource.
Finally, giving the role Editor to a service account may be too much, you should restrict the access to the minimum viable. If this code only needs to create tasks, you should create a custom role with just the required permissions, cloudtasks.tasks.create in this case.
I am creating an SQS queue and adding permissions to that queue for a specific user inline, but the add_permission call is failing in an unexpected way.
import boto3
queue_name = 'test-queue'
sqs_client = boto3.client('sqs')
response = sqs_client.create_queue(QueueName=queue_name)
queue = boto3.resource('sqs').Queue(response.get('QueueUrl')
queue.add_permission(
Label='TestReceivePermissions',
AWSAccountIds=['arn:aws:iam::<account_id>:user/test_user'],
Actions=['ReceiveMessage']
)
When I execute this code, I receive the following error:
botocore.exceptions.ClientError: An error occurred (InvalidParameterValue) when calling the AddPermission operation: Value [arn:aws:iam::<account_id>:user/test_user] for parameter PrincipalId is invalid. Reason: Unable to verify.
However, I am able to add this same permission to the queue via the AWS console. What am I missing here? Is there another approach I should consider?
UPDATE: One viable workaround is to create a managed policy and then attach the user to the policy like so:
import json, boto3
sqs = boto3.resource('sqs')
iam = boto3.resource('iam')
queue_name = 'test-queue'
USER_NAME = 'test_user'
queue = sqs.create_queue(QueueName=queue_name)
policy = iam.create_policy(
PolicyName='{}-ReceiveAccess'.format(queue_name),
PolicyDocument=json.dumps({
'Version': '2012-10-17',
'Statement': [
{
'Effect': 'Allow',
'Action': [
'sqs:ReceiveMessage*'
],
'Resource': [queue.attributes['QueueArn']]
}
]
})
)
policy.attach_user(UserName=USER_NAME)
This works for now, but I still do not understand why the other approach was not working.