As we know , we can send a key with kafka producer which is hashed internally to find which partition in topic data goes to.
I have a producer , where in I am sending a data in JSON format.
kafka-console-producer --broker-list 127.0.0.1:9092 --topic USERPROFILE << EOF
{"user_id" : 100, "firstname":"Punit","lastname":"Gupta", "countrycode":"IN", "rating":4.9 }
{"user_id" : 101, "firstname":"eli","lastname":"eli", "countrycode":"GB", "rating":3.0 }
EOF
Now I want to use "countrycode" as my key , while sending data.
In Normal delimited data we can specify 2 parameters :
--property "parse.key=true"
--property "key.separator=:
But How to do it when sending JSON sata.
I am using confluent's python API for Kafka if there is any thing that I have to write in terms of classed of functions to achieve this, i would be thankful if you can say it in terms of python.
JSON is just a string. The console producer doesn't parse JSON, only the Avro console producer does.
I would avoid key.separator=: since JSON contains :. You could use | character instead, then you just type out
countrycode|{"your":"data"}
In Python, the produce function takes a key, yes. You can parse your data like this in order to extract a value to the key.
key = 'countrycode'
records = [{"user_id" : 100, "firstname":"Punit","lastname":"Gupta", key:"IN", "rating":4.9 },
{"user_id" : 101, "firstname":"eli","lastname":"eli", key:"GB", "rating":3.0 }
]
import json
for r in records:
producer.produce('topic', key=r[key], value=json.dumps(r))
# first record will send a record containing ('IN', { ... 'countrycode':'IN'})
Related
I have a JSON output like this:
{
":output":{
"response":"{ \"ParentId\" : 125, \"ParentKey\" : { \"key\" : \"9aqews-uwdguwdw8-9uw8\", \"identity\" : \"key_ID=674\" } }"
}
}
I'm trying to fetch the content of key, that is: 9aqews-uwdguwdw8-9uw8
Here are somethings that tried:
------------------------------------------------------
${json_data} Parse Json ${output}
Log ${json_data["output"]["response"]}
Log ${json_data["output"]["response"][0][0:10]}
------------------------------------------------------
${json}= Convert To Dictionary ${values}
${j_keys}= Get Dictionary Keys ${json}
Log ${j_keys}
------------------------------------------------------
${values}= Evaluate json.loads($output) json
Log ${values['output']['response'][1]}
-----------------------------------------------------
${KeySP}= Get Substring ${values} "key" ","
Log ${keySP}
------------------------------------------------------
#${parkeydict}= ${values['output']['response']}
#${keyspacedict}= ${parkeydict['ParentKey']}
#Log ${keyspacedict['key']}
------------------------------------------------------
I have tried several other steps, possibilities and keywords,
The best I could parse is till "Log ${json_data["output"]["response"]}" which returns data from 'response'.
It fails even if I convert to Dict and access the 'key', I think that further data after 'response' is completely stored as values.
Can someone help/guide me on how to capture the data in 'key' variable?
Thanks in Advance!
The first problem is that you're trying to use ['output'], but the key is :output.
The second problem is that the value of the "response" key is not a dictionary, it's a string that looks like a dictionary. You must convert it to a dictionary before you try to pull values from it, assuming it is indeed an well-formed json dictionary and not just a string that might look like a dictionary.
This works for me on the exact data provided in the question:
${values}= Evaluate json.loads($output)
${response}= Evaluate json.loads($values[':output']['response'])
${key}= set variable ${response['ParentKey']['key']}
should be equal ${key} 9aqews-uwdguwdw8-9uw8
Note: if you're using a version of robot that is older than 3.2 you'll need to include json as a final argument for the Evaluate command so that robot knows to import the module. Starting with version 3.2 and onward, modules used in the expression are automatically imported.
I'm using Protobuf with the C++ API and I have a standart message I send between 2 different softwares and I want to add a raw nested message as data.
So I added a message like this:
Message main{
string id=1;
string data=2;
}
I tried to serialize some nested messages I made to a string and send it as "data" with "main" message but it doesn't work well on the parser side.
How can I send nested serialized message inside a message using c++ and python api.
Basically, use bytes:
message main {
string id=1;
bytes data=2;
}
In addition to not corrupting the data (string is strictly UTF-8), as long as the payload is a standard message, this is also compatible with changing it later (at either end, or both) to the known type:
``` proto
message main {
string id=1;
TheOtherMessageType data=2;
}
message TheOtherMessageType {...}
(or even using both versions at different times depending on which is most convenient)
I'm trying to build a bridge between a Redis server and MQTT, so that when the Redis database is updated, these updates are dispatched via MQTT to clients.
For this a client (only one, the bridge) connects to the Redis database and starts to monitor it.
My issue is with parsing the commands, more specifically the arguments contained with it, which is a whitespace-separated list of stings.
For example, when I store the following hash in Redis
data = {
"key-3-1-json": "value-1",
"key-3-2-json": 'this "this is \'quoted\' text"',
}
print r18.hmset("test-hash", {
"key-1": "value-1",
"key-2": 'this "this is \'quoted\' text"',
"key-3": json.dumps(data),
})
the client recieves the following
1549578825.1 0 HMSET test-hash "key-3" "{\"key-3-1-json\": \"value-1\", \"key-3-2-json\": \"this \\\"this is 'quoted' text\\\"\"}" "key-2" "this \"this is 'quoted' text\"" "key-1" "value-1"
As you can see I'm already parsing the timestamp, database id, command and key, but the last part, I don't know how to create a list of strings from it.
This message would then be sent over MQTT as
mqtt.publish("redis/mon/0/HMSET/test-hash", json.dumps(args))
where args would be
[
"key-3",
"{\"key-3-1-json\": \"value-1\", \"key-3-2-json\": \"this \\\"this is 'quoted' text\\\"\"}",
"key-2",
"this \"this is 'quoted' text\"",
"key-1",
"value-1"
]
which would probably be the most complex case, since usually the args would be one single string, in the case where r18.set would have been used instead of r18.hmset.
I think there must be some built-in module in Python which could to this as it is like parsing a command line string.
The subprocess module's documentation states that subprocess.Popen() makes use of shlex.split() (shlex: Simple lexical analysis)
Calling shlex.split(args_str) effectively converts the arguments string into the desired list of substrings.
In my Chatfuel block I collect a {{user input}} and POST a JSON in a Zapier webhook. So far so good. After that, my local Pyhon reads this JSON from Zapier storage successfully
url = 'https://store.zapier.com/api/records?secret=password'
response = urllib.request.urlopen(url).read().decode('utf-8')
data = json.loads(response)
and analyze it generating another JSON as output:
json0={
"messages": [
{"text": analysis_output}]
}
Then Python3 posts this JSON in a GET webhook in Zapier:
import requests
r = requests.post('https://hooks.zapier.com/hooks/catch/2843360/8sx1xl/', json=json0)
r.status_code
Zapier Webhook successfully gets the JSON and sends it to Storage.
Key-Value pairs are set and then Chatfuel tries to read from storage:
GET https://store.zapier.com/api/records?secret=password2
But the JSON structure obtained is wrong, what was verified with this code:
url = 'https://store.zapier.com/api/records?secret=password2'
response = urllib.request.urlopen(url).read().decode('utf-8')
data = json.loads(response)
data
that returns:
{'messages': "text: Didn't know I could order several items"}
when the right one for Chatfuel to work should be:
{'messages': [{"text: Didn't know I could order several items"}]}
That is, there are two mais problems:
1) There is a missing " { [ " in the JSON
2) The JSON is appending new information to the existing one, instead of generating a brand new JSON, what cause the JSON to have 5 different parts.
I am looking for possible solutions for this issue.
David here, from the Zapier Platform team.
First off, you don't need quotes around your keys, we take care of that for you. Currently, your json will look like:
{ "'messages'": { "'text'": "<DATA FROM STEP 1>" } }
So the first change is to take out those.
Next, if you want to store an array, use the Push Value Onto List action instead. It takes a top-level key and stores your values in a key in that object called list. Given the following setup:
The resulting structure in JSON is
{ "demo": {"list": [ "5" ]} }
It seems like you want to store an extra level down; an array of json objects:
[ { "text": "this is text" } ]
That's not supported out of the box, as all list items are stored as strings. You can store json strings though, and parse them back into an object when you need to access them like an object!
Does that answer your question?
I'm newbie in python and trying to parse data in my application using these lines of codes
json_str = request.body.decode('utf-8')
py_str = json.loads(json_str)
But I'm getting this error on json.loads
Expecting value: line 1 column 1 (char 0)
this is json formatted data that I send from angular app (Updated)
Object { ClientTypeId: 6, ClientName: "asdasd", ClientId: 0, PhoneNo: "123", FaxNo: "123", NTN: "1238", GSTNumber: "1982", OfficialAddress: "sads", MailingAddress: "asdasd", RegStartDate: "17-Aug-2016", 15 more… }
these are the values that I get in json_str
ClientTypeId=5&ClientName=asdasd&ClientId=0&PhoneNo=123&FaxNo=123&NTN=123&GSTNumber=12&OfficialAddress=adkjh&MailingAddress=adjh&RegStartDate=09-Aug-2016&RegEndDate=16-Aug-2016&Status=1&CanCreateUser=true&UserQuotaFor=11&UserQuotaType=9&MaxUsers=132123&ApplyUserCharges=true&ApplyReportCharges=true&EmailInvoice=true&BillingType=1&UserCharges=132&ReportCharges=123&MonthlyCharges=123&BillingDate=16-Aug-2016&UserSessionId=324
I don't know what's wrong in it.. can anyone mention what's the mistake is??
Your data is not JSON-formatted, not even the one you included in your updated answer. Your data is a JavaScript-object, not an encoded string. Please note the "N" in JSON: Notation -- it is a format inspired from how data is written in JavaScript code, but runtime JavaScript data is not represented in JSON. The "JSON" you pasted is how your browser represents the object to you, it is not proper JSON (that would be {"ClientTypeId": 6, ...} -- note the quotes around the property name).
When sending this data to the server, you have to encode it. You think you are sending it JSON-encoded, but you aren't. You are sending it "web form encoded" (data of type application/x-www-form-urlencoded).
Now either you have to learn how to send the data in JSON format from Angular, or use the correct parsing routine in Python: urllib.parse.parse_qs. Depending on the library you are using, there might be a convenience method to access the data as well, as this is a common use case.