I am using boto library to read messages from SQS queue. My messages have text like this:
{ Command:XXXXXXXXXXX Key:XXXXXXX Input:XXXXXX} . Boto sends with base64 encoded and also reads it, so that if I read the message body then the text is there.
But how can I read the message like
Command = input['Command']
Key = input_message['Key'].split(',')
so that I can use those values for further processing...
I am quite new to Python also
Ok, you seem to have the input in some kind of a format - is it anything standardised? If not, you would need to parse the contents of your message and get the individual keys.
What I have been doing before in my projects was using JSON to facilitate data exchange between platforms.
If you do not have a luxury to edit your incoming data, you would need to do something like this (very naiive example):
input = "{ Command:XXXXXXXXXXX Key:XXXXXXX Input:XXXXXX }"
data = filter(lambda x: ":" in x, input.split())
message_dict = dict()
for item in data:
key, val = item.split(":")
message_dict[key] = val
Consider using good old fashioned JSON to easily send and receive dictionaries acrost the wire.
This test function verifies that the data format is very clear with JSON:
test_sqs.py
import json
import boto3
from moto import mock_sqs
#mock_sqs
def test_sqs():
sqs = boto3.resource('sqs', 'us-east-1')
queue = sqs.create_queue(QueueName='votes')
queue.send_message(MessageBody=json.dumps(
{'Command': 'drink', 'Key': 'beer', 'Input': 'tasty'}))
messages = queue.receive_messages()
assert len(messages) == 1
assert messages[0].body == (
'{"Input": "tasty", "Command": "drink", "Key": "beer"}')
Related
I am trying to send a json formatted message to Slack through a Cloud function using slack_sdk, if I send it like this (not formatted) it works.
client = WebClient(token='xoxb-25.......')
try:
response = client.chat_postMessage(channel='#random', text=DICTIONARY)
I found the documentation on Slack that chat_postMessage supports sending json formats by setting the HTTP headers:
Content-type: application/json
Authorization: Bearer xoxb-25xxxxxxx-xxxx
How would that work applied in my code above? I want to send a big python dictionary and would like to receive it formatted in the Slack channel. I tried adding it in multiple ways and deployment fails.
This is the documentation: https://api.slack.com/web
Bit late, but I hope this can help others who stumble upon this issue in the future.
I think that you've misunderstood the documentation. The JSON support allows for accepting POST message bodies in JSON format, as only application/x-www-form-urlencoded format was supported earlier. Read more here.
To answer your question, you can try to send the dictionary by formatting it or in a code block as Slack API supports markdown.
Reference- Slack Text Formatting.
Sample Code-
from slack_sdk import WebClient
import json
client = WebClient(token="xoxb........-")
json_message = {
"title": "Tom Sawyer",
"author": "Twain, Mark",
"year_written": 1862,
"edition": "Random House",
"price": 7.75
}
# format and send as a text block
formatted_text = f"```{json.dumps(json_message, indent = 2)}```"
client.chat_postMessage(channel = "#general", text = formatted_text)
# format and send as a code block
formatted_code_block = json.dumps(json_message, indent = 2)
client.chat_postMessage(channel = "#general", text = formatted_code_block)
Output-
I'm struggeling to get the properties of events send to azure event hub using python. Currently I don't know how to consume additional event properties I send with a event message.
This is my code of sending messages which works great, BUT on the other hand I would like to consume those properties.
event_data_batch = await producer.create_batch()
i = 0
while i <= 100:
#json_obj = {"TimeStamp": str(datetime.utcnow()),"Name":f"Test{i}", "Metric":"11", "Source": "EventHubMessage"}
json_obj = {"TimeStamp": "timestamp","Name":"Test", "Metric":11, "Source": "EventHubMessage"}
string = json.dumps(json_obj)
Event_data = EventData(body=string)
Event_data.properties = {"Table":"TestTable", "IngestionMappingReference":"TestMapping", "Format":"JSON", "Encoding":"UTF-8"}
event_data_batch.add(Event_data)
i += 1
print(event_data_batch)
return event_data_batch
as you see, I'm sending additional event_data properties {"Table":"TestTable", "IngestionMappingReference":"TestMapping", "Format":"JSON", "Encoding":"UTF-8"} and those I would like to consume with another python application.
Can you please help me by letting me know how to do this? Currently, I'm just able to retrieve the body of the message but not its properties.
Maybe someone of you did already figure this out.
Thanks!
Best,
Chris
In the receiving end of your application you can access the properties on the EventData object like this:
event.properties
which prints:
{b'Table': b'TestTable', b'IngestionMappingReference': b'TestMapping', b'Format': b'JSON', b'Encoding': b'UTF-8'}
If you are following the docs from MSFT: https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-python-get-started-send then simply add this line under the first print statement to print the event properties:
recv.py:
async def on_event(partition_context, event):
# Print the event data.
print("Received the event: \"{}\" from the partition with ID: \"{}\"".format(event.body_as_str(encoding='UTF-8'), partition_context.partition_id))
print(event.properties)
Additionally if you want to consume your properties dict as a dict of string key-value pairs (rather than bytes) you can decode it with a dict comprehension:
{k.decode("utf-8"):v.decode("utf-8") for k,v in event.properties.items()}
which outputs:
{'Table': 'TestTable', 'IngestionMappingReference': 'TestMapping', 'Format': 'JSON', 'Encoding': 'UTF-8'}
I want to make a simple middle-man script which receives data from a JIRA webhook and then parses it and sends it over (to Discord, but that's not important).
From the looks of it, Cloud Functions was a perfect match, so I made a short script which just forwards the request json to a message on Discord to see what data it sends.
import requests
import json
def forward(request):
data = request.json
#tried:
#data = request.get_json()
#data = request.data
#data = request.values
url = 'discord webhook receiver url'
myobj = {'content': str(data)}
requests.post(url, data = myobj)
return '', 200
Set up the webhook on the JIRA side and it kinda works, but not really. First of all it doesn't trigger on some events, like adding a comment or editing a task, only for important(?) events, like adding a new task.
And secondly, it doesn't even send the right looking data for it. This is all i'm getting
{'timestamp': 1609851709546, 'webhookEvent': 'issuelink_created', 'issueLink': {'id': 10016, 'sourceIssueId': 10007, 'destinationIssueId': 10022, 'issueLinkType': {'id': 10004, 'name': 'jira_subtask_link', 'outwardName': 'jira_subtask_outward', 'inwardName': 'jira_subtask_inward', 'style': 'jira_subtask', 'isSubTaskLinkType': True, 'isSystemLinkType': True}, 'sequence': 12, 'systemLink': True}}
And I know those 2 things are wrong because while testing it I've also set up another webhook for Pipedream, where it reacts to all changes, and the data contains names of issues, avatars, links. Everything that's needed for what I'm trying to do. And there aren't any differences between those 2 webhook settings, I have them both with all events selected.
So I've been at it for 2 days now with no breakthrough. Maybe I'm misunderstanding how webhooks work, or maybe cloud functions isn't the service to use for this. So while the question is how to do it in cloud functions, I'm also open to alternatives. (not the ones which do the formatting for you, as that's why I started making this in the first place)
Apparently Jira sends 2 requests on webhook trigger and the one containing all the useful info was just over the limit to be put into a Discord message so it never sent it.
That's what I get for using it for logging.
If anyone is also on the same dumbass path, then what I did to find that out is to save all the request data into a hastebin and send the link to it, like so.
#pack it all into a hastebin
everything = str(request.headers) + "\n" + str(request.data) + "\n" + str(request.args) + "\n" + str(request.form) + "\n" + str(request.method) + "\n" + str(request.remote_addr)
req = requests.post('https://hastebin.com/documents', data=everything)
#send the link id to discord
myobj = {'content': req.text}
x = requests.post(url, data = myobj)
All that's left is to parse and format the json.
When we try to publish the json to Azure Event hub, EventData converting the JSON string. Instead of JSON string I need the JSON only at the consumer end.
response = {"status":"Active", "code":400}
event_batch = await producer.create_batch()
event_batch.add(EventData(json.dumps(response)))
At the consumer end I am getting the event as { body: '{"status":"Active", "code":400}', 'sequence_numbe':1} like this. how can we get only Json at consumer end like this
{ body: {"status":"Active", "code":400}, 'sequence_numbe':1}
Can anyone help?
You could use body_as_json method to achieve your requirement.
Sample Implementation :
jsonbody = event.body_as_json(encoding='UTF-8')
This will directly return us the JSON Object
Alternate Approach :
Get it as string and then convert the same to json object.
jsonbody = json.loads(event.body_as_str(encoding='UTF-8')))
I use python and paho.mqtt for sending messages to cloud
I set up endpoint and route. When I set query string to true, everything works fine
messageDict = {}
systemPropertiesDict = {"contentType": "application/json", "contentEncoding": "utf-8", "iothub-message-source": "deviceMessages", "iothub-enqueuedtime": "2017-05-08T18:55:31.8514657Z"}
messageDict = {"systemProperties": systemPropertiesDict}
messageDict["appProperties"] = {}
body = '{id:1}'
messageDict["body"] = body
root = {"message":messageDict}
msg = json.dumps(root, indent=2).encode('utf-8')
print("Message to send", msg)
self.client.publish(topicName, msg)
But if I set the query string to $body.id = 1, then I don't receive any messages.
Any ideas, guys?
The route not working because the content encoding type is not set. All the "systemProperties" in your code actually as message body not system properties. Content encoding type set by this method doesn't take effect.
Add "$.ct=application%2Fjson&$.ce=utf-8" to the topic. Then it will look like this:
devices/{yourDeviceId}/messages/events/$.ct=application%2Fjson&$.ce=utf-8
But to make the route query works on your message you need use this query string: $body.message.body.id = 1
Two edits to make:
First, change body = '{id:1}' to body = {"id":1} to make the id as a string.
Second, change topicName value to this one:
devices/{yourDeviceId}/messages/events/$.ct=application%2Fjson&$.ce=utf-8
If possible, it is suggesting to use Azure IoT SDK for Python to communicate with Azure IoT Hub.
If you are using a third-party library (like paho-mqtt) you have to specify the content type and the encoding of the message.
IoT Hub route messages with content type "application/json" and content encoding: "utf-8" or "utf-16" or "utf-32".
Using MQTT protocol you can set this info with $.ct and $.ce.
topic example:
devices/{MY_DEVICE_ID}/messages/events/%24.ct=application%2fjson&%24.ce=utf-8
URL encoding of
devices/{MY_DEVICE_ID}/messages/events/$.ct=application/json&$.ce=utf-8
Here you could found more info:
https://azure.microsoft.com/it-it/blog/iot-hub-message-routing-now-with-routing-on-message-body/