I am processing kafka streams in a python flask server. I read the responses with json and need to extract the udid values from the stream. I read each response with request.json and save it in a dictionary. When i try to parse it fails. the dict contains the following values
dict_items([('data', {'SDKVersion': '7.1.2', 'appVersion': '6.5.5', 'dateTime': '2019-08-05 15:01:28+0200', 'device': 'iPhone', 'id': '3971',....})])
parsing the dict the normal way doesnt work ie event_data['status'] gives error.Perhaps it is because its not a pure dict....?
#app.route('/data/idApp/5710/event/start', methods=['POST'])
def give_greeting():
print("Hola")
event_data = request.json
print(event_data.items())
print(event_data['status'])
#print(event_data['udid'])
#print(event_data['Additional'])
return 'Hello, {0}!'.format(event_data)
The values contained in event data are the following
dict_items([('data', {'SDKVersion': '7.1.2', 'appVersion': '6.5.5', 'dateTime': '2019-08-05 15:01:28+0200', 'device': 'iPhone', 'id': '3971',....})])
The expected result would be this result
print(event_data['status'])->start
print(event_data['udid'])->BAEB347B-9110-4CC8-BF99-FA4039C4599B
print(event_data['SDKVersion'])->7.1.2
etc
the output of
print(event_data.keys()) is dict_keys(['data'])
The data you are expecting is wrapped in an additional data property.
You only need to do one extra step to access this data.
data_dict = request.json
event_data = data_dict['data']
Now you should be able to access the information you want with
event_data['SDKVersion']
...
as you have described above.
As #jonrsharpe stated, this is not an issue with the parsing. The parsing either fails or succeeds, but you will never get a "broken" object (be it dict, list, ...) from parsing JSON.
Related
I would appreciate some help: how could I print just the country from the info obtained via this API call? Thanks!
import requests
import json
url = "https://randomuser.me/api/"
data = requests.get(url).json()
print(data)
You should play a little more with the json in order to learn how to use it, a helpful way to understand them is to go layer by layer printing the keys dict.keys() to see where you should go next if you dont have a documentation
in this particular case it returns a dictionary with the following first layer structure:
{
"results": [ ... ]
"info": { ... }
}
where results contains a single dictionary inside, therefore we can take
data['results'][0] to wok with
there is 'location', and there is a 'country', you can access this in that order to print the country:
print(data['results'][0]['location']['country'])
I have a JSON file hosted locally in my Django directory. It is fetched from that file to a view in views.py, where it is read in like so:
def Stops(request):
json_data = open(finders.find('JSON/myjson.json'))
data1 = json.load(json_data) # deserialises it
data2 = json.dumps(data1) # json formatted string
json_data.close()
return JsonResponse(data2, safe=False)
Using JsonResponse without (safe=False) returns the following error:
TypeError: In order to allow non-dict objects to be serialized set the safe parameter to False.
Similarly, using json.loads(json_data.read()) instead of json.load gives this error:
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
This is confusing to me - I have validated the JSON using an online validator. When the JSON is sent to the frontend with safe=False, the resulting object that arrives is a string, even after calling .json() on it in javascript like so:
fetch("/json").then(response => {
return response.json();
}).then(data => {
console.log("data ", data); <---- This logs a string to console
...
However going another step and calling JSON.parse() on the string converts the object to a JSON object that I can use as intended
data = JSON.parse(data);
console.log("jsonData", data); <---- This logs a JSON object to console
But this solution doesn't strike me as a complete one.
At this point I believe the most likely thing is that there is something wrong with the source JSON - (in the file character encoding?) Either that or json.dumps() is not doing what I think it should, or I am not understanding the Django API's JSONresponse function in a way I'm not aware of...
I've reached the limit of my knowledge on this subject. If you have any wisdom to impart, I would really appreciate it.
EDIT: As in the answer below by Abdul, I was reformatting the JSON into a string with the json.dumps(data1) line
Working code looks like:
def Stops(request):
json_data = open(finders.find('JSON/myjson.json'))
data = json.load(json_data) # deserialises it
json_data.close()
return JsonResponse(data, safe=False) # pass the python object here
Let's see the following lines of your code:
json_data = open(finders.find('JSON/myjson.json'))
data1 = json.load(json_data) # deserialises it
data2 = json.dumps(data1) # json formatted string
You open a file and get a file pointer in json_data, parse it's content and get a python object in data1 and then turn it back into a JSON string and store it into data2. Somewhat redundant right? Next you pass this JSON string to JsonResponse which will further try to serialize it into JSON!! Meaning you then get a string inside a string in JSON.
Try the following code instead:
def Stops(request):
json_data = open(finders.find('JSON/myjson.json'))
data = json.load(json_data) # deserialises it
json_data.close()
return JsonResponse(data, safe=False) # pass the python object here
Note: function names in python should ideally be in snake_case not PascalCase, hence instead of Stops you should use stops. See
PEP 8 -- Style Guide for Python
Code
I have just started with Python programming & working on https://pypi.org/project/PARANOID/ to mask the PII details such as first_name, last_name & email_address.
{
"id": 324324,
"first_name": "John",
"last_name": "Smith",
"email": "john.smith#abc.com"
}
When I am executing paranoid -i my.json -o output, all the fields of my json (id, first_name, last_name & email_address) are getting masked. But I don't want to mask id. For that -l with Xpath to the json has to be provided.
I have tried various combinations for Xpath to json, but still it masks all the fields in the file.
Please guide me.
This library doesn't seems very attractive for me as there's no documentation at all ,the code is very messy and they seem to be overcomplicating some stuff! the only reference to what it can do was here and they just refer to using xpath to process xml not to process json.
I then installed the library locally and verified that xpath doesn't apply to json. But it turns out the library is just a single module( a unique file) with a bunch of functions. So I investigated which functions are being used when you're processing a json file. Only two functions are being used jsonParse2 and maskGenerator. So it was doable to reuse it.
jsonParse2 Doesn't make sense at all for me as they are parsing a json file manually when there are so many easy tools to use such as the json library. I will discard the jsonParse2 function as it was the main problem to filter which keys should be processed.
I will simply reuse the maskGenerator function into my solution then we just pass the keys,values we're interested in to the maskGenerator.
CODE Solution
create an input.json file with your json in the same folder as the solution:
import paranoid
import json
list_not_to_mask = ["id"]
with open("input.json") as input_file:
input_dict =json.loads(input_file.read())
print(input_dict)
output_dict = input_dict
for key in input_dict.keys():
if key in list_not_to_mask:
pass
else:
output_dict[key] = paranoid.maskGenerator(str(input_dict[key]),is_json=True)
print(output_dict)
with open('output.json', 'w') as output_file:
json.dump(output_dict, output_file, ensure_ascii=False, indent=4)
OUTPUT
it will also create a output.json
the input we have is: {'id': 324324, 'first_name': 'John', 'last_name': 'Smith', 'email': 'john.smith#abc.com'}
the output we have is: {'id': 324324, 'first_name': 'Vxqt', 'last_name': 'Yiphq', 'email': 'vuxr.kcicc#muj.jbj'}
I want to make a HTTP wrapper for twitch.tv in python. How would I do this? I know how to use HTTP GET and that's about it. I would like it to work like this:
import twichtvwrapper
twich = twichtvwrapper(useragent, username, password).
channel = twich.channel(channelname)
then all the json properties would go in here like:
print(channel.game) #this would say the last played game
print(channel.displayname) #this would print the display name of the channel
print(cahnnel.link.chat) #this will return the chat link
etc.
How would I go about making this wrapper?
You can convert between Python dicts and JSON strings with the json module of the Standard Library.
import json
# package a python dict as json
dict0 = {'spam': 'data', 'eggs': 'more data'}
pack = json.dumps(dict0)
# turn it back to a dict
dict1 = json.loads(pack)
>>> print dict1['spam']
data
So, if your program gets a JSON response from a HTTP request, just convert it to a dict and use regular Python dict methods to do the rest.
I have a dict like this:
data = {"data":"http://abc/def"}
when I call json.dumps(data) I get this:
'{"data":"http://abc/def"}'
but I want this:
'{"data":"http:\/\/abc\/def"}'
because I use jquery to parse json but seems like it don't understand unescaped solidus, or is there any way to make jquery understand?
UPDATE
For example, here is my json data
{"data": ["http://abc.com/aaaaaaaa/bbbbbbbbb/cccccccccc/xyz.mp3"]}
Here is my success function
function showResult(result) {
$.each(result.data, function(i, item){
link = $('<a>').attr('href', item).text(item)
$("#result").append('<br>')
$("#result").append(link);
});
}
The result should be a hyperlink to
http://abc.com/aaaaaaaa/bbbbbbbbb/cccccccccc/xyz.mp3
But I got a hyperlink to
http://abc.com/aaaaaaaa/bbbbbbbbb/cccccccccc/xyz.mp3
If replace all '/' by '\/', everything is fine
Normally you don't escape forward slashes in JSON, but if you are certain this is your problem you can simply do this:
s = json.dumps(data)
s = s.replace("/", "\\/")