Python fails to json.loads() nested JSON from Kafka - python

I can't loads() a stringified JSON into a valid JSON, using json.dumps(). Here is the code and the result. You can see the erroneous input at error logs. Is there any way I can convert that nested string into valid json? Thanks in advance.
async for value in RedisHandler.get_all_keys():
examined_counter += 1
task = None
try:
task = json.loads(value)
except Exception as convertJSONException:
logger.info("Failed converting '%s' into JSON. Skipping." % value)
continue
prints:
[-- TimeoutHandler] Failed converting '{'description': 'dqwdqdq', 'author': 'spp42admin', 'target_project_uuid': '91115b46-54fa-11e8-9a7c-d4c9ef52f9c6', 'tasklist': [{'opuid': '253e3d0d-5514-11e8-9a7c-d4c9ef52f9c6', 'method': 'split-count', 'parameter': '7', 'optional_parameters': '{"codec":"libx264"}', 's3_credentials': {'service_name': 's3', 'aws_access_key_id': '********', 'aws_secret_access_key': '*********', 'endpoint_url': 'http://localhost:9000', 'bucket_uuid': '91115b46-54fa-11e8-9a7c-d4c9ef52f9c6'}, 'srcpath': 'videos/397939d2-550c-11e8-9a7c-d4c9ef52f9c6/test.mp4', 'destpath': 'operations/253e3d0d-5514-11e8-9a7c-d4c9ef52f9c6', 'videoname': 'test.mp4', 'codec': 'mpeg4', 'progress': 1, 'parent_operation_uuid': '253f87b6-5514-11e8-9a7c-d4c9ef52f9c6', 'estimated_timeout': 1526042364}], 's3_credentials': {'service_name': 's3', 'aws_access_key_id': '**********', 'aws_secret_access_key': '***********', 'endpoint_url': 'http://localhost:9000', 'bucket_uuid': '91115b46-54fa-11e8-9a7c-d4c9ef52f9c6'}, 'video_uuid': '397939d2-550c-11e8-9a7c-d4c9ef52f9c6', 'parent_videodocument_uuid': '397939d3-550c-11e8-9a7c-d4c9ef52f9c6', 'parent_project_uuid': '91115b46-54fa-11e8-9a7c-d4c9ef52f9c6', 'progress': 0, 'operation_uuid': '253f87b6-5514-11e8-9a7c-d4c9ef52f9c6'}' into JSON. Skipping.
optional_parameters is the one causing error.
TLDR: How can I convert string below into JSON, using json.dumps?
'{'description': 'dqwdqdq', 'author': 'spp42admin', 'target_project_uuid': '91115b46-54fa-11e8-9a7c-d4c9ef52f9c6', 'tasklist': [{'opuid': '253e3d0d-5514-11e8-9a7c-d4c9ef52f9c6', 'method': 'split-count', 'parameter': '7', 'optional_parameters': '{"codec":"libx264"}', 's3_credentials': {'service_name': 's3', 'aws_access_key_id': '********', 'aws_secret_access_key': '*********', 'endpoint_url': 'http://localhost:9000', 'bucket_uuid': '91115b46-54fa-11e8-9a7c-d4c9ef52f9c6'}, 'srcpath': 'videos/397939d2-550c-11e8-9a7c-d4c9ef52f9c6/test.mp4', 'destpath': 'operations/253e3d0d-5514-11e8-9a7c-d4c9ef52f9c6', 'videoname': 'test.mp4', 'codec': 'mpeg4', 'progress': 1, 'parent_operation_uuid': '253f87b6-5514-11e8-9a7c-d4c9ef52f9c6', 'estimated_timeout': 1526042364}]}'

Related

How to get specific value from json api with python

So i have code that gets data from one api but i can only use frist 3 keys like message idk why is that code:
import requests
import json
result=requests.get('https://api.etherscan.io/api?module=account&action=tokentx&contractaddress=0x9f8f72aa9304c8b593d555f12ef6589cc3a579a2&page=1&offset=1&sort=asc&apikey=mytoken')
result.status_code
result.text
result.json()
print (result.json()['message]) # work
print (result.json()['gas]) # or any other key dont work
Output from api:
{"status":"1","message":"OK","result":[{"blockNumber":"4620855","timeStamp":"1511634257","hash":"0x5c9b0f9c6c32d2690771169ec62dd648fef7bce3d45fe8a6505d99fdcbade27a","nonce":"5417","blockHash":"0xee385ac028bb7d8863d70afa02d63181894e0b2d51b99c0c525ef24538c44c24","from":"0x731c6f8c754fa404cfcc2ed8035ef79262f65702","contractAddress":"0x9f8f72aa9304c8b593d555f12ef6589cc3a579a2","to":"0x642ae78fafbb8032da552d619ad43f1d81e4dd7c","value":"1000000000000000000000000","tokenName":"Maker","tokenSymbol":"MKR","tokenDecimal":"18","transactionIndex":"55","gas":"3000000","gasPrice":"1000000000","gasUsed":"1594668","cumulativeGasUsed":"4047394","input":"deprecated","confirmations":"7045304"}]}
i can only get status message ect.
when i try gas this is error
Traceback (most recent call last):
File "main.py", line 11, in
print (result.json()[gas])
NameError: name 'gas' is not defined

You should add a few print statements to understand your response and debug
your_reponse = {
'message': 'OK',
'result': [{'blockHash': '0xee385ac028bb7d8863d70afa02d63181894e0b2d51b99c0c525ef24538c44c24',
'blockNumber': '4620855',
'confirmations': '7045304',
'contractAddress': '0x9f8f72aa9304c8b593d555f12ef6589cc3a579a2',
'cumulativeGasUsed': '4047394',
'from': '0x731c6f8c754fa404cfcc2ed8035ef79262f65702',
'gas': '3000000',
'gasPrice': '1000000000',
'gasUsed': '1594668',
'hash': '0x5c9b0f9c6c32d2690771169ec62dd648fef7bce3d45fe8a6505d99fdcbade27a',
'input': 'deprecated',
'nonce': '5417',
'timeStamp': '1511634257',
'to': '0x642ae78fafbb8032da552d619ad43f1d81e4dd7c',
'tokenDecimal': '18',
'tokenName': 'Maker',
'tokenSymbol': 'MKR',
'transactionIndex': '55',
'value': '1000000000000000000000000'}],
'status': '1'}
>>> your_reponse['result']
[{'blockHash': '0xee385ac028bb7d8863d70afa02d63181894e0b2d51b99c0c525ef24538c44c24',
'blockNumber': '4620855',
'confirmations': '7045304',
'contractAddress': '0x9f8f72aa9304c8b593d555f12ef6589cc3a579a2',
'cumulativeGasUsed': '4047394',
'from': '0x731c6f8c754fa404cfcc2ed8035ef79262f65702',
'gas': '3000000',
'gasPrice': '1000000000',
'gasUsed': '1594668',
'hash': '0x5c9b0f9c6c32d2690771169ec62dd648fef7bce3d45fe8a6505d99fdcbade27a',
'input': 'deprecated',
'nonce': '5417',
'timeStamp': '1511634257',
'to': '0x642ae78fafbb8032da552d619ad43f1d81e4dd7c',
'tokenDecimal': '18',
'tokenName': 'Maker',
'tokenSymbol': 'MKR',
'transactionIndex': '55',
'value': '1000000000000000000000000'}]
>>> print(your_reponse['result'][0]['gas'])
3000000
Use this recursive function to work around changes in API response:
def price_of_gas(inp):
def recursive_function(inp):
if type(inp) is list:
for i in inp:
ans = recursive_function(i)
if ans!=None: return ans
elif type(inp) is dict:
if 'gas' in inp: return inp['gas']
for i in inp:
ans = recursive_function(inp[i])
if ans!=None: return ans
else: return None
ans = recursive_function(inp)
return ans if ans else "Could NOT find the gas"
price_of_gas(your_reponse)

See the JSON response carefully. Its a dictionary with three keys (status, message and result). The key result is a list having another dictionary inside.
JSON response:
{
"status":"1",
"message":"OK",
"result":[
{
"blockNumber":"4620855",
"timeStamp":"1511634257",
"hash":"0x5c9b0f9c6c32d2690771169ec62dd648fef7bce3d45fe8a6505d99fdcbade27a",
"nonce":"5417",
"blockHash":"0xee385ac028bb7d8863d70afa02d63181894e0b2d51b99c0c525ef24538c44c24",
"from":"0x731c6f8c754fa404cfcc2ed8035ef79262f65702",
"contractAddress":"0x9f8f72aa9304c8b593d555f12ef6589cc3a579a2",
"to":"0x642ae78fafbb8032da552d619ad43f1d81e4dd7c",
"value":"1000000000000000000000000",
"tokenName":"Maker",
"tokenSymbol":"MKR",
"tokenDecimal":"18",
"transactionIndex":"55",
"gas":"3000000",
"gasPrice":"1000000000",
"gasUsed":"1594668",
"cumulativeGasUsed":"4047394",
"input":"deprecated",
"confirmations":"7045304"
}
]
}
So if you need to access the keys present inside the result dictionary, you have to nest your code accordingly.
result.json['result'][0]['gas']

What data format is this and how can I convert it to XML?

after writing a python script to request some data from a server, I get the response in following structure:
{
'E_AXIS_DATA': {
'item': [
{
'AXIS': '000',
'SET': {
'item': [
{
'TUPLE_ORDINAL': '000000',
'CHANM': '0002',
'CAPTION': 'ECF',
'CHAVL': '0002',
'CHAVL_EXT': None,
'TLEVEL': '00',
'DRILLSTATE': None,
'ATTRIBUTES': None
},
{...
Apparently its not JSON.
After running following command:
results = client.service.RRW3_GET_QUERY_VIEW_DATA("/server")
df = pd.read_json(results)
i get the output meaning that the format is not being accepted as JSON;
ValueError: Invalid file path or buffer object type: <class 'zeep.objects.RRW3_GET_QUERY_VIEW_DATAResponse'>
Any help is welcome.
Thanks

Pandas has DataFrame.read_json() method that can do the trick
import pandas as pd
json_string = '{"content": "a string containing some JSON...." ... etc... }'
df = pd.load_json(json_string)
# Now you can do whatever you like with your dataframe

TypeError: string indices must be integers - Python JSON

Getting TypeError: string indices must be integers on the my_object instantiation here below (3rd line of method)
def get_note_retrieval_body(event):
sns_body = event["Records"][0]["Sns"]
message = json.loads(sns_body["Message"])
my_object = message["data"]["getNotes"]
return my_object
I get the following (account numbers etc are marked with XXX) when i use str(event) and then json format it, so this will tell you what the event looks like
{
'Records': [{
'EventSource': 'aws:sns',
'EventVersion': '1.0',
'EventSubscriptionArn': 'arn:aws:sns:us-east-1:XXXXXX:topic-name-sandbox:7XXXX',
'Sns': {
'Type': 'Notification',
'MessageId': 'd1074c88-ae21-52b6-8a75-1b07d766cfdd',
'TopicArn': 'arn:aws:sns:us-east-1:XXXXXXX:topic-name',
'Subject': None,
'Message': '"{\\"data\\": {\\"getNotes\\": {\\"claimNumber\\": \\"AAAB09000010\\", \\"dateEntered\\": \\"2010-04-22T08:03:53\\",\\"categoryCode\\": \\"fdf49\\",\\"subCategoryCode\\": \\"ATT\\", \\"fileNoteTextDetails\\": [{\\"fileNoteText\\": {\\"fileNoteID\\": \\"112B40FE42934055\\", \\"noteText\\": \\"Send Acknowledgement Letter to Claimant\\", \\"authorID\\": \\"0\\"}, \\"fileNoteAttachments\\": [{\\"attachment\\": {\\"fileName\\": \\"F70F880879D35FC4.doc\\", \\"fileExtension\\": \\".URL\\", \\"dateCreated\\": \\"2010-04-22T08:59:57\\", \\"createdBy\\": \\"CLONER\\", \\"dateUpdated\\": \\"2020-07-30T08:36:19.1903051\\", \\"updatedBy\\": \\"EVERYONE\\"}}]}], \\"fileNoteExtendedEntityData\\": {\\"dateOnDocument\\": \\"2010-04-22T08:59:57\\", \\"serviceDateFrom\\": \\"2010-04-22T08:59:57\\", \\"serviceDateThrough\\": \\"2010-04-22T08:59:57\\", \\"author\\": \\"n0000000\\"}}}}"',
'Timestamp': '2020-07-20T10:50:47.850Z',
'SignatureVersion': '1',
'Signature': 'XXXXXX',
'SigningCertUrl': 'https://sns.us-east-1.amazonaws.com/XXXcert.pem',
'UnsubscribeUrl': 'https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:XXXXXX:topic-name-sandbox:7XX'.
'MessageAttributes': {}
}
}]
}

When you run json.loads on sns_body["Message"], you still get back a string. You can run json.loads twice, and that should solve the issue.
sns_body["Message"] is quoted twice (i.e., it is wrapped in single quotes and double quotes). So when you try to decode it once, you still get back a string, but this time it is only quoted once. Then a second json.loads will decode the string into a dictionary.
def get_note_retrieval_body(event):
sns_body = event["Records"][0]["Sns"]
message = json.loads(json.loads(sns_body["Message"]))
my_object = message["data"]["getNotes"]
return my_object

Twitch API json dictionary error list indices must be integers not str

try:
response = requests.get('https://api.twitch.tv/kraken/streams/followed', headers=headers)
data = response.json()
except (KeyError, ValueError):
print("Error - make sure your OAuth is formatted correctly in oauth.txt")
sys.exit(1)
channelName = data["streams"]["channel"]["name"]
channelGame = data["streams"]["channel"]["game"]
channelViewers = str(data["streams"]["viewers"])
streamType = data["streams"]["stream_type"]
print(channelName, channelGame, channelViewers, streamType)
The error i get is list indices must be integers or slices, not str
From twitch i get a json dictionary:
{'streams': [{'_id': 2011610081, 'game': 'Sports & Fitness', 'broadcast_platform': 'live', 'community_id': '', 'community_ids': [], 'viewers': 12399, 'video_height': 900, 'average_fps':59, 'delay': 0, 'created_at': '2020-06-20T12:06:14Z', 'is_playlist': False, 'stream_type': 'live' } the dictionary is a lot longer
How can i access on it without getting a error

That is a list You Have itrate over it Something like This
Itrate Over it using for loop
for i in data['streams']:
print(i['_id'] ,i['game'])
or select index
data["streams"][0]['_id']

How to correctly serialize API response in Django view

I am calling the Giphy API using another wrapper API which returns a list of dictionaries. I am having hard times to serialize the data to return it to AJAX.
The data is returned as InlineResponse200 with three properties.
(docu)
The problem is that my view is not able to return the JSON properly:
# Traceback
[2020-06-23 14:58:54,086] log: ERROR - Internal Server Error: /get_gifs/
Traceback (most recent call last):
File "C:\Users\Jonas\AppData\Local\Programs\Python\Python38-32\lib\site-packages\django\core\handlers\exception.py", line 34, in inner
response = get_response(request)
File "C:\Users\Jonas\AppData\Local\Programs\Python\Python38-32\lib\site-packages\django\core\handlers\base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "C:\Users\Jonas\AppData\Local\Programs\Python\Python38-32\lib\site-packages\django\core\handlers\base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "C:\Users\Jonas\Desktop\finsphere\finsphere\blog\views.py", line 234, in get_gifs
return JsonResponse(api_response.data[0])
File "C:\Users\Jonas\AppData\Local\Programs\Python\Python38-32\lib\site-packages\django\http\response.py", line 554, in __init__
raise TypeError(
TypeError: In order to allow non-dict objects to be serialized set the safe parameter to False.
[23/Jun/2020 14:58:54] "POST /get_gifs/ HTTP/1.1" 500 17874
If I add safe=False it returns TypeError: Object of type Gif is not JSON serializable
I don't get this since api_response.data[0] is a cristal clear dictionary.
Desired outcome: Get the Giphy object logged in the success function of Ajax.
AJAX
(function($) {
$('#btnSearch').on('click', function(e) {
var query = $('#search').val();
console.log(query);
e.preventDefault();
$.ajax({
type: 'post',
async: true,
url: '/get_gifs/',
data: {
'query': query,
'csrfmiddlewaretoken': window.CSRF_TOKEN // from blog.html
},
success: function(response) {
},
error: function(xhr, status, error) {
// shit happens friends!
}
});
});
}(jQuery));
(Inserted my original -free- API key for reproduction)
Views.py
def get_gifs(request):
# create an instance of the API class
api_instance = giphy_client.DefaultApi()
# API Key
api_key = 'NGSKWrBqtIq1rFU1Ka11D879Y1u4Igia'
# Search term
q = request.POST.get('query')
print(q)
# Query parameters
limit = 2
offset = 0
rating = 'g'
lang = 'en'
fmt = 'json'
try:
# Search Endpoint
api_response = api_instance.gifs_search_get(api_key, q, limit=limit, offset=offset, rating=rating, lang=lang, fmt=fmt)
pprint(api_response)
except ApiException as e:
print("Exception when calling DefaultApi->gifs_search_get: %s\n" % e)
return JsonResponse(api_response.data[0])
API fetched object (pprint api_response)
{'data': [{'bitly_gif_url': 'https://gph.is/g/EJWjdvN',
'bitly_url': 'https://gph.is/g/EJWjdvN',
'content_url': '',
'create_datetime': None,
'embed_url': 'https://giphy.com/embed/J0JGg6doLfmV0yZmIB',
'featured_tags': None,
'id': 'J0JGg6doLfmV0yZmIB',
'images': {'downsized': {'height': '250',
'size': '350582',
'url': 'https://media3.giphy.com/media/J0JGg6doLfmV0yZmIB/giphy.gif?cid=ecefd82565bc1664c2b17e3e4b60d88c736d0c6b5a39d682&rid=giphy.gif',
'width': '478'},
'downsized_large': {'height': '250',
'size': '350582',
'url': 'https://media3.giphy.com/media/J0JGg6doLfmV0yZmIB/giphy.gif?cid=ecefd82565bc1664c2b17e3e4b60d88c736d0c6b5a39d682&rid=giphy.gif',
'width': '478'},
'preview_gif': {'height': '134',
'size': '49623',
'url': 'https://media3.giphy.com/media/J0JGg6doLfmV0yZmIB/giphy-preview.gif?cid=ecefd82565bc1664c2b17e3e4b60d88c736d0c6b5a39d682&rid=giphy-preview.gif',
'width': '256'}},
'import_datetime': '2020-06-15 10:01:39',
'is_anonymous': None,
'is_community': None,
'is_featured': None,
'is_hidden': None,
'is_indexable': None,
'is_realtime': None,
'is_removed': None,
'is_sticker': False,
'rating': 'g',
'slug': 'MITEF-mitefarab-asc2020-J0JGg6doLfmV0yZmIB',
'source': 'www.mitefarab.org',
'source_post_url': 'www.mitefarab.org',
'source_tld': '',
'tags': None,
'trending_datetime': '0000-00-00 00:00:00',
'type': 'gif',
'update_datetime': None,
'url': 'https://giphy.com/gifs/MITEF-mitefarab-asc2020-J0JGg6doLfmV0yZmIB',
'user': {'avatar_url': 'https://media2.giphy.com/avatars/MITEF/8FTlysEjtXzx.jpg',
'banner_url': '',
'display_name': 'MITEF Pan Arab',
'profile_url': 'https://giphy.com/MITEF/',
'twitter': None,
'username': 'MITEF'},
'username': 'MITEF'},
{'bitly_gif_url': 'https://gph.is/g/ZdxQQpP',
'bitly_url': 'https://gph.is/g/ZdxQQpP',
'content_url': '',
'create_datetime': None,
'embed_url': 'https://giphy.com/embed/hTJF0O4vDkJsUi1h8Q',
'featured_tags': None,
'id': 'hTJF0O4vDkJsUi1h8Q',
'images': {'downsized': {'height': '480',
'size': '310971',
'url': 'https://media3.giphy.com/media/hTJF0O4vDkJsUi1h8Q/giphy.gif?cid=ecefd82565bc1664c2b17e3e4b60d88c736d0c6b5a39d682&rid=giphy.gif',
'width': '480'},
'preview': {'height': '480',
'mp4': 'https://media3.giphy.com/media/hTJF0O4vDkJsUi1h8Q/giphy-preview.mp4?cid=ecefd82565bc1664c2b17e3e4b60d88c736d0c6b5a39d682&rid=giphy-preview.mp4',
'mp4_size': '15536',
'width': '480'},
'preview_gif': {'height': '480',
'size': '22387',
'url': 'https://media3.giphy.com/media/hTJF0O4vDkJsUi1h8Q/giphy-preview.gif?cid=ecefd82565bc1664c2b17e3e4b60d88c736d0c6b5a39d682&rid=giphy-preview.gif',
'width': '480'}},
'import_datetime': '2019-07-19 22:27:40',
'is_anonymous': None,
'is_community': None,
'is_featured': None,
'is_hidden': None,
'is_indexable': None,
'is_realtime': None,
'is_removed': None,
'is_sticker': False,
'rating': 'g',
'slug': 'RecargaPay-cashback-recargapay-paguetudopelocelular-hTJF0O4vDkJsUi1h8Q',
'source': 'www.recargapay.com.br',
'source_post_url': 'www.recargapay.com.br',
'source_tld': '',
'tags': None,
'trending_datetime': '0000-00-00 00:00:00',
'type': 'gif',
'update_datetime': None,
'url': 'https://giphy.com/gifs/RecargaPay-cashback-recargapay-paguetudopelocelular-hTJF0O4vDkJsUi1h8Q',
'user': {'avatar_url': 'https://media0.giphy.com/avatars/RecargaPay/msKTiPaVkvqd.png',
'banner_url': 'https://media0.giphy.com/headers/RecargaPay/kg023vdaAaWA.gif',
'display_name': 'RecargaPay',
'profile_url': 'https://giphy.com/RecargaPay/',
'twitter': None,
'username': 'RecargaPay'},
'username': 'RecargaPay'}],
'meta': {'msg': 'OK',
'response_id': '65bc1664c2b17e3e4b60d88c736d0c6b5a39d682',
'status': 200},
'pagination': {'count': 2, 'offset': 0, 'total_count': 10}}

I go though your code everything is correct except return JsonResponse(api_response.data[0]) in your views
JsonResponse:
The first parameter, data, should be a dict instance. If the safe parameter is set to False, it can be any JSON-serializable object. official documentation link
When you say
1. safe=True
return JsonResponse(api_response.data[0])
TypeError: In order to allow non-dict objects to be serialized set the safe parameter to False.
The error is obvious api_response.data[0] is not dict, see point 2 error
2
safe=False
return JsonResponse(api_response.data[0], safe=False)
TypeError: Object of type Gif is not JSON serializable
The data api_response.data[0] you provide to JsonResponse is not a dict type object actually, that's why you got error for first point.
when you say safe=False JsonResponse is trying to serialize object but that object is not json serializable, you can trace-back error
File "....\Python\Python38\lib\json\encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Gif is not JSON serializable
Can be followed the link to see which object can be Json serializable
Coming back to your error.
I don't get this since api_response.data[0] is a cristal clear dictionary.
type of api_response and api_response.data[0]
type(api_response)
<class 'giphy_client.models.inline_response_200.InlineResponse200'>
type(api_response.data[0])
<class 'giphy_client.models.gif.Gif'>
You can follow for giphy_client documentation link for more-details
Solution:
result = api_response.data[0].to_dict() ## NOTE to_dict function of giphy_client.models.gif.Gif
return JsonResponse(result)

render(request, template_name, context=None, content_type=None, status=None, using=None)
render() Combines a given template with a given context dictionary
and returns an HttpResponse object with that rendered text.
You can either use Django defaults JsonResponse class or Django REST framework Response class to return JSON responses.
from django.http import JsonResponse
return JsonResponse(data=api_response.data)
from rest_framework.response import Response
return Response(data=api_response.data)
tried it on the ipython shell and it works just fine.
In [15]: response = Response(api_response.data[0])
In [16]: response
Out[16]: <Response status_code=200, "text/html; charset=utf-8">
response.data gives me the serialized response.

Python has a built in function for converting dicts to json.
import json
data = api_response.data
return render(request, json.dumps(data))
If you use that in your return statement it should return json.

Categories

Resources