With the python google-cloud-pubsub library, acknowledging messages through the subscriber.acknowledge() does not acknowledge my messages. My ack deadline is set at 30 seconds.
Here is my code:
from google.cloud import pubsub_v1
project_id = "$$$$"
subscription_name = "$$$$"
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(project_id, subscription_name)
response = subscriber.pull(subscription_path, max_messages=10, timeout=15)
for msg in response.received_messages:
subscriber.acknowledge(subscription=subscription_path, ack_ids=[msg.ack_id])
Using google-cloud-pubsub==1.0.2
Any idea of what I'm doing wrong?
What I recommend you is referring to Synchronous Pull documentation, then run a sample code in Python to pull and acknowledge messages:
from google.cloud import pubsub_v1
project_id = "Your Google Cloud Project ID"
TODO subscription_name = "Your Pub/Sub subscription name"
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
project_id, subscription_name)
NUM_MESSAGES = 3
response = subscriber.pull(subscription_path, max_messages=NUM_MESSAGES)
ack_ids = []
for received_message in response.received_messages:
print("Received: {}".format(received_message.message.data))
ack_ids.append(received_message.ack_id)
subscriber.acknowledge(subscription_path, ack_ids)
print('Received and acknowledged {} messages. Done.'.format(
len(response.received_messages)))
I can't find definition of ack_ids = [] in your code (you need to define it before starting use it in code). If you will see positive results when running that piece of code, you can assume that there is a bug in your code. Have you provided a full code?
Related
I am trying to use the pub sub service on my python application. When I am running the code it get stuck on the last publisher line for some reason and the code never end. The subscriber seems fine.Does someone know what is wrong with my code?
Publisher:
import os
from google.cloud import pubsub_v1
credentials_path = 'PATH/TO/THE/KEY.JSON'
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = credentials_path
publisher = pubsub_v1.PublisherClient()
topic_path = 'projects/PROJECT_NAME/topics/TOPIC_NAME'
# simple garbage text to check if it's working
data = 'A garden sensor is ready!'
data = data.encode('utf-8')
attributes = {
'sensorName': 'garden-001',
'temperature': '75.0',
'humidity': '60'
}
future = publisher.publish(topic_path, data, **attributes)
print(f'published message id {future.result()}') # here it is just waiting forever
Subscriber:
import os
from google.cloud import pubsub_v1
from concurrent.futures import TimeoutError
credentials_path = 'PATH/TO/THE/KEY.JSON'
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = credentials_path
subscriber = pubsub_v1.SubscriberClient()
subscription_path = 'projects/PROJECT_NAME/subscriptions/SUBSCRIPTION_NAME'
def callback(message):
print(f'Received message: {message}')
print(f'data: {message.data}')
if message.attributes:
print("Attributes:")
for key in message.attributes:
value = message.attributes.get(key)
print(f"{key}: {value}")
message.ack()
streaming_pull_future = subscriber.subscribe(
subscription_path, callback=callback)
print(f'Listening for messages on {subscription_path}')
# wrap subscriber in a 'with' block to automatically call close() when done
with subscriber:
try:
streaming_pull_future.result()
except TimeoutError:
streaming_pull_future.cancel()
# block until the shutdown is complete
streaming_pull_future.result()
Google provides decent documentation for using its services including Pub/Sub including a basic Python example that would have helped you avoid your problem.
Aside: your publisher and subscriber snippets set GOOGLE_APPLICATION_CREDENTIALS statically within the code. Don't do this! Set the environment variable before running the code. This way, you can revise the value without changing the code but, more importantly, the value can be set by the runtime e.g. Compute Engine.
Here's a working example based on your code using Application Default Credentials obtained from the environment:
Q="74535931"
BILLING="[YOUR-BILLING-ID]"
PROJECT="$(whoami)-$(date %y%m%d)-${Q}"
gcloud projects create ${PROJECT}
gcloud beta billing projects link ${PROJECT} \
--billing-account=${BILLING}
gcloud services enable pubsub.googleapis.com \
--project=${PROJECT}
ACCOUNT=tester
EMAIL=${ACCOUNT}#${PROJECT}.iam.gserviceaccount.com
gcloud iam service-accounts create ${ACCOUNT} \
--project=${PROJECT}
gcloud iam service-accounts keys create ${PWD}/${ACCOUNT}.json \
--iam-account=${EMAIL}
gcloud projects add-iam-policy-binding ${PROJECT} \
--member=serviceAccount:${EMAIL} \
--role=roles/pubsub.editor
export GOOGLE_APPLICATION_CREDENTIALS=${PWD}/${ACCOUNT}.json
export PROJECT
export PUB="pub"
export SUB="sub"
gcloud pubsub topics create ${PUB} \
--project=${PROJECT}
gcloud pubsub subscriptions create ${SUB} \
--topic=${PUB} \
--project=${PROJECT}
publish.py:
import os
from google.cloud import pubsub_v1
project = os.getenv("PROJECT")
topic = os.getenv("PUB")
topic_path = f"projects/{project}/topics/{topic}"
data = 'A garden sensor is ready!'
data = data.encode('utf-8')
attributes = {
'sensorName': 'garden-001',
'temperature': '75.0',
'humidity': '60'
}
publisher = pubsub_v1.PublisherClient()
future = publisher.publish(topic_path, data, **attributes)
print(f'published message id {future.result()}')
subscribe.py:
import os
from google.cloud import pubsub_v1
from concurrent.futures import TimeoutError
project=os.getenv("PROJECT")
subscription=os.getenv("SUB")
subscription_path = f"projects/{project}/subscriptions/{subscription}"
def callback(message):
print(f'Received message: {message}')
print(f'data: {message.data}')
if message.attributes:
print("Attributes:")
for key in message.attributes:
value = message.attributes.get(key)
print(f"{key}: {value}")
message.ack()
subscriber = pubsub_v1.SubscriberClient()
streaming_pull_future = subscriber.subscribe(
subscription_path, callback=callback)
print(f'Listening for messages on {subscription_path}')
with subscriber:
try:
streaming_pull_future.result()
except TimeoutError:
streaming_pull_future.cancel()
# block until the shutdown is complete
streaming_pull_future.result()
Run python3 subscribe.py:
python3 subscribe.py
Listening for messages on projects/{project}/subscriptions/{sub}
Received message: Message {
data: b'A garden sensor is ready!'
ordering_key: ''
attributes: {
"humidity": "60",
"sensorName": "garden-001",
"temperature": "75.0"
}
}
data: b'A garden sensor is ready!'
Attributes:
humidity: 60
temperature: 75.0
sensorName: garden-001
And in a separate window python3 publish.py:
python3 publish.py
published message id 1234567890123456
I am new to python and want to write unit testcase for pub/sub asynchronous pull feature . I understand that we need to mock the function for subscriber,event message but I am not sure how that can be done. Also how to test the assertion when we are creating the stream and not returning anything .
from google.cloud import pubsub_v1
def pubsub_listener():
project_id = <<Environment Variable>>
subscription_id = <<Environment Variable>>
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(project_id, subscription_id)
def callback(message: pubsub_v1.subscriber.message.Message) -> None:
logging.info(f"Received {message}.")
message.ack()
streaming_pull_future = subscriber.subscribe(subscription_path, callback=callback)
logging.info(f"Listening for messages on {subscription_path}..\n")
with subscriber:
try:
streaming_pull_future.result(timeout=timeout)
except TimeoutError:
streaming_pull_future.cancel()
streaming_pull_future.result()
if __name__ =='__main__':
pubsub_listener()
It's being described here; for example, where data and mock_context are variable:
def test_pubsub_listener(capsys):
data = {}
mock_context = mock.Mock()
main.pubsub_listener(data, mock_context)
out, err = capsys.readouterr()
assert ...
Just unsure what to assert, while not knowing what it might return,it should respond with "Listening for messages on" and "Received".
Unwrapping pubsub_listener.callback(data) might make it more testable,
because this is the part which processes the actual pub/sub event payload.
Writing unit-tests requires testable code to begin with... which means, you'd have to pass project_id and subscription_id into the function.
This is what makes it not testable - and also the portability is poor:
project_id = "your-project-id"
subscription_id = "your-subscription-id"
For my project using Firebase messaging to send push notification. I have users's firebase tokens stored on the database. Using them I sent push to each user. Total time of sending is about 100 seconds for 100 users. Is there way to send push asynchronously(I mean at one time to send many push notifications)
# Code works synchronously
for user in users:
message = messaging.Message(
notification=messaging.Notification(
title="Push title",
body="Push body"
),
token = user['fcmToken']
)
response = messaging.send(message)
Sure, you could use one of the python concurrency libraries. Here's one option:
from concurrent.futures import ThreadPoolExecutor, wait, ALL_COMPLETED
def send_message(user):
message = messaging.Message(
notification=messaging.Notification(
title="Push title",
body="Push body"),
token = user['fcmToken'])
return messaging.send(message)
with ThreadPoolExecutor(max_workers=10) as executor: # may want to try more workers
future_list = []
for u in users:
future_list.append(executor.submit(send_message, u))
wait(future_list, return_when=ALL_COMPLETED)
# note: we must use the returned self to get the test count
print([future.result() for future in future_list])
If you want to send the same message to all tokens, you can use a single API call with a multicast message. The Github repo has this sample of sending a multicast message in Python:
def send_multicast():
# [START send_multicast]
# Create a list containing up to 500 registration tokens.
# These registration tokens come from the client FCM SDKs.
registration_tokens = [
'YOUR_REGISTRATION_TOKEN_1',
# ...
'YOUR_REGISTRATION_TOKEN_N',
]
message = messaging.MulticastMessage(
data={'score': '850', 'time': '2:45'},
tokens=registration_tokens,
)
response = messaging.send_multicast(message)
# See the BatchResponse reference documentation
# for the contents of response.
print('{0} messages were sent successfully'.format(response.success_count))
# [END send_multicast]
I am trying to get messages from a topic on a message hub on bluemix using Confluent Kafka Python. My code is found below, but something is not working. The topic and the message hub is up and running, so there is probably something with the code.
from confluent_kafka import Producer, KafkaError, Consumer
consumer_settings = {
'bootstrap.servers': 'broker-url-here',
'group.id': 'mygroup',
'default.topic.config': {'auto.offset.reset': 'smallest'},
'sasl.mechanisms': 'PLAIN',
'security.protocol': 'ssl',
'sasl.username': 'username-here',
'sasl.password': 'password-here',
}
c = Consumer(**consumer_settings)
c.subscribe(['topic-here'])
running = True
while running:
msg = c.poll()
if msg.error():
print("Error while retrieving message")
c.close()
sys.exit(10)
elif (msg is not None):
for x in msg:
print(x)
else:
sys.exit(10)
When I run the code, it seems to get stuck at msg = c.poll(). So I guess it is either failing to connect, or failing to retrieve messages. The credentials themselves are correct.
The consume logic look fine but the configuration for the consumer is incorrect.
security.protocol needs to be set to sasl_ssl
ssl.ca.location needs to point to a PEM file containing trusted certificates. The location of that file varies for each OS, but for the most common it's:
Bluemix/Ubuntu: /etc/ssl/certs
Red Hat: /etc/pki/tls/cert.pem
macOS: /etc/ssl/certs.pem
We also have a sample app using this client that can easily be started or deployed to Bluemix: https://github.com/ibm-messaging/message-hub-samples/tree/master/kafka-python-console-sample
I have run into a problem where a Python subscriber using stomp (stomp.py) with a message selector does not receive the messages it should. Interestingly enough, it appears to me at least that the problem is somehow with the sending of the message and not the subscription.
I am using ActiveMQ.
Here's the subscriber code:
class Listener(object):
def __init__(self, count):
if count <= 0:
count = float('inf')
self.count = count
def on_error(self, headers, message):
print("=" * 72)
print('RECEIVED AN ERROR.')
print('Message headers:')
pp = pprint.PrettyPrinter(indent=4)
pp.pprint(headers)
print('Message body:')
print(message)
def on_message(self, headers, message):
print("=" * 72)
print('Message headers:')
pp = pprint.PrettyPrinter(indent=4)
pp.pprint(headers)
print('Message body:')
print(message)
def main():
global conn
args = parse_args()
conn = stomp.Connection([(args.host, args.port)])
conn.set_listener('Listener', Listener(args.count))
conn.start()
conn.connect(login=args.user, passcode=args.password)
if (args.selector):
conn.subscribe(
destination=args.destination,
id=1,
ack='auto',
headers={'selector': args.selector}
)
else:
conn.subscribe(
destination=args.destination,
id=1,
ack='auto'
)
Now I can run this subscriber with a selector such as "type = 'test'".
If I publish a message using Java JMS, the message is received just fine. However, if I publish the identical message from Python it is not.
Here's the relevant Python publishing code:
headers = {}
headers['type'] = 'test'
conn = stomp.Connection12([(args.host, args.port)], auto_content_length=False)
conn.start()
conn.connect(login=args.user, passcode=args.password)
conn.send(body=body, headers=headers, destination=args.destination)
conn.disconnect()
print 'Message sent.'
Some interesting notes from my testing and debugging:
Running the subscriber with a selector receives a matching message sent from Java JMS but not from Python.
Running the subscriber with no selector receives a message sent from Java and also a message sent from Python.
fairly old, but I was currently facing the same issue and so I would like to leave a possible solution here.
At first, according to the documentation, you can provide a field called selectorwith SQL like syntax and should be part of the headers. In your example:
headers = {}
headers['selector'] = "type='test'"
conn = stomp.Connection12([(args.host, args.port)], auto_content_length=False)
conn.start()
conn.connect(login=args.user, passcode=args.password)
conn.send(body=body, headers=headers, destination=args.destination)
conn.disconnect()
print 'Message sent.'
I was also facing the error, that I could not receive any message send from JMS, but after a lot of reading, I found here, that there is a field name JMSType. I changed the code to
headers['selector'] = "type='test' OR JMSType='test'"
With that JMSType in there everything works like expected. Hope that helps somebody