I am trying to return a particular string value after getting response from request URL.
Ex.
response =
{
'assets': [
{
'VEG': True,
'CONTACT': '12345',
'CLASS': 'SIX',
'ROLLNO': 'A101',
'CITY': 'CHANDI',
}
],
"body": "**Trip**: 2017\r\n** Date**: 15th Jan 2015\r\n**Count**: 501\r\n\r\n"
}
This is the response which i am getting, from this I need only Date: 15th Jan 2015. I am not sure how to do it.
Any help would be appreciated.
assuming it is a dictionary
a={'body': '**Trip**: 2017\r\n** Date**: 15th Jan 2015\r\n**Count**: 501\r\n\r\n', 'assets': [{'VEG': True, 'CONTACT': '12345', 'CLASS': 'SIX', 'ROLLNO': 'A101', 'CITY': 'CHANDI'}]}
then
required=a['body'].split('\r\n')[1].replace('**','')
print required
result:
' Date: 15th Jan 2015'
Access the key body of the dictionary a
split through \r\n to get a list ['**Trip**: 2017', '** Date**:
15th Jan 2015', '**Count**: 501', '', '']
access it's first index and replace ** with empty('')
a = str(yourdictionary)
print([e for e in a.split("\r\n") if "Date" in e][0].remove("**").strip())
Try this
>>> body = response['body']
>>> bodylist = body.split('\r\n')
>>> for value in bodylist:
... value = value.split(':')
...
>>> for i,value in enumerate(bodylist):
... bodylist[i] = value.split(':')
...
>>> for i, value in enumerate(bodylist):
... if bodylist[i][0] == '** Date**':
... print(bodylist[i][1])
...
15th Jan 2015
I have trown it in the interpreter and this works. I don't know if it is the best code around, but it works ;-)
Related
Can I check how do we convert the below to a dictionary?
code.py
message = event['Records'][0]['Sns']['Message']
print(message)
# this gives the below and the type is <class 'str'>
{
"created_at":"Sat Jun 26 12:25:21 +0000 2021",
"id":1408763311479345152,
"text":"#test I\'m planning to buy the car today \ud83d\udd25\n\n",
"language":"en",
"author_details":{
"author_id":1384883875822907397,
"author_name":"\u1d04\u0280\u028f\u1d18\u1d1b\u1d0f\u1d04\u1d1c\u0299 x NFTs \ud83d\udc8e",
"author_username":"cryptocurrency_x009",
"author_profile_url":"https://xxxx.com",
"author_created_at":"Wed Apr 21 14:57:11 +0000 2021"
},
"id_displayed":"1",
"counter_emoji":{
}
}
I would need to add in additional field called "status" : 1 such that it looks like this:
{
"created_at":"Sat Jun 26 12:25:21 +0000 2021",
"id":1408763311479345152,
"text":"#test I\'m planning to buy the car today \ud83d\udd25\n\n",
"language":"en",
"author_details":{
"author_id":1384883875822907397,
"author_name":"\u1d04\u0280\u028f\u1d18\u1d1b\u1d0f\u1d04\u1d1c\u0299 x NFTs \ud83d\udc8e",
"author_username":"cryptocurrency_x009",
"author_profile_url":"https://xxxx.com",
"author_created_at":"Wed Apr 21 14:57:11 +0000 2021"
},
"id_displayed":"1",
"counter_emoji":{
},
"status": 1
}
Wanted to know what is the best way of doing this?
Update: I managed to do it for some reason.
I used ast.literal_eval(data) like below.
D2= ast.literal_eval(message)
D2["status"] =1
print(D2)
#This gives the below
{
"created_at":"Sat Jun 26 12:25:21 +0000 2021",
"id":1408763311479345152,
"text":"#test I\'m planning to buy the car today \ud83d\udd25\n\n",
"language":"en",
"author_details":{
"author_id":1384883875822907397,
"author_name":"\u1d04\u0280\u028f\u1d18\u1d1b\u1d0f\u1d04\u1d1c\u0299 x NFTs \ud83d\udc8e",
"author_username":"cryptocurrency_x009",
"author_profile_url":"https://xxxx.com",
"author_created_at":"Wed Apr 21 14:57:11 +0000 2021"
},
"id_displayed":"1",
"counter_emoji":{
},
"status": 1
}
Is there any better way to do this? Im not sure so wanted to check...
Can I check how do we convert the below to a dictionary?
As far as I can tell, the data = { } asigns a dictionary with content to the variable data.
I would need to add an additional field called "status" : 1 such that it looks like this
A simple update should do the trick.
data.update({"status": 1})
I found two issues when trying to deserialise the string as JSON
invalid escape I\\'m
unescaped newlines
These can worked around with
data = data.replace("\\'", "'")
data = re.sub('\n\n"', '\\\\n\\\\n"', data, re.MULTILINE)
d = json.loads(data)
There are also surrogate pairs in the data which may cause problems down the line. These can be fixed by doing
data = data.encode('utf-16', 'surrogatepass').decode('utf-16')
before calling json.loads.
Once the data has been deserialised to a dict you can insert the new key/value pair.
d['status'] = 1
I am carrying out a API search on scaleserp and for each search I do i want to put the output in the column in my dataframe.
#Matches the GET request
api_result = requests.get('https://api.scaleserp.com/search', params, verify=False)
print(type(api_result))
#stores the result in JSON
result = api_result.json()
print(type(result))
#Creates a new DataFrame with 'Organic_Results' From the JSON output.
Results_df = (result['organic_results'])
#FOR loop to look at each result and select which output from the JSON is wanted.
for res in Results_df:
StartingDataFrame['JSONDump'] = res
api_result is a requests.models.Response
result is a dict.
RES is a dict.
I want the RES to be put into the column Dump. is this possible?
Updated Code
#Matches the GET request
api_result = requests.get('https://api.scaleserp.com/search', params, verify=False)
#stores the result in JSON
result = api_result.json()
#Creates a new DataFrame with 'Organic_Results' From the JSON output.
Results_df = (result['organic_results'])
#FOR loop to look at each result and select which output from the JSON is wanted.
for res in Results_df:
Extracted_data = {key: res[key] for key in res.keys()
& {'title', 'link', 'snippet_matched', 'date', 'snippet'}}
Extracted_data is a dict and contains the info i need.
{'title': '25 Jun 1914 - Advertising - Trove', 'link': 'https://trove.nla.gov.au/newspaper/article/7280119', 'snippet_matched': ['()', 'charge', 'Dan Whit'], 'snippet': 'I Iron roof, riltibcd II (),. Line 0.139.5. wai at r ar ... Propertb-« entired free of charge. Line 2.130.0 ... AT Dan Whit",\'»\', 6il 02 sturt »L, Prlnce\'»~Brti\'»e,. Line 3.12.0.'}
{'snippet': "Mary Bardwell is in charge of ... I() •. Al'companit'd by: Choppf'd Chitkf'n Li\\f>r Palt·. 1h!iiSC'o Gret'n Salad g iii ... of the overtime as Dan Whit-.", 'title': 'October 16,1980 - Bethlehem Public Library',
'link': 'http://www.bethlehempubliclibrary.org/webapps/spotlight/years/1980/1980-10-16.pdf', 'snippet_matched': ['charge', '()', 'Dan Whit'], 'date': '16 Oct 1980'}
{'snippet': 'CONGRATULATIONS TO DAN WHIT-. TLE ON THE ... jailed and beaten dozens of times. In one of ... ern p()rts ceased. The MIF is not only\xa0...', 'title': 'extensions of remarks - US Government Publishing Office', 'link': 'https://www.gpo.gov/fdsys/pkg/GPO-CRECB-1996-pt5/pdf/GPO-CRECB-1996-pt5-7-3.pdf', 'snippet_matched': ['DAN WHIT', 'jailed', '()'], 'date': '26 Apr 1986'}
{'snippet': 'ILLUSTRATION BY DAN WHIT! By Matt Manning ... ()n the one hand, there are doctors on both ... self-serving will go to jail at the beginning of\xa0...', 'title': 'The BG News May 23, 2007 - ScholarWorks#BGSU - Bowling ...', 'link': 'https://scholarworks.bgsu.edu/cgi/viewcontent.cgi?article=8766&context=bg-news', 'snippet_matched': ['DAN WHIT', '()', 'jail'], 'date': '23 May 2007'}
{'snippet': '$19.95 Charge card number SERVICE HOURS: ... Explorer Advisor Dan Whit- ... lhrr %(OnrwflC or ()utuflrueonlinelfmarketing (arnpaigfl%? 0I - .',
'title': '<%BANNER%> TABLE OF CONTENTS HIDE Section A: Main ...', 'link': 'https://ufdc.ufl.edu/UF00028295/00194', 'snippet_matched': ['Charge', 'Dan Whit', '()'], 'date': 'Listings 1 - 800'}
{'title': 'Lledo Promotional,Bull Nose Morris,Dandy,Desperate Dan ...', 'link': 'https://www.ebay.co.uk/itm/Lledo-Promotional-Bull-Nose-Morris-Dandy-Desperate-Dan-White-Van-/233817683840', 'snippet_matched': ['charges'], 'snippet': 'No additional import charges on delivery. This item will be sent through the Global Shipping Programme and includes international tracking. Learn more- opens\xa0...'}
The problem looks like the length of your organic_results is not the same as the length of your dataframe.
StartingDataFrame['Dump'] = (result['organic_results'])
Here your setting the whole column dump to equal the organic_results which is smaller or larger than your already defined dataframe. I'm not sure what your dataframe already has but you could do this by iterating through the rows and stashing them in that row if you have values you want them to add up with like this:
StartingDataFrame['Dump'] = []*len(StartingDataFrame)
for i,row in StartingDataFrame.iterrows():
StartingDataFrame.at[i,'Dump'] = result['organic_results']
Depending on what your data looks like you could maybe just append it to the dataframe
StartingDataFrame = StartingDataFrame.append(result['organic_results'],ignore_index=True)
Could you show us a sample of what both data source looks like?
I have ['key','value'] format list, which also contain sub-list. How can I convert nested list to JSON format in python
[[' key ', ' 1542633482511430199'
],
['value=>>>BasicData',
[['isConfirmAndOrder', '0'],['brmRequestId', 'BR-2018-0000124'],
['requestType','batch'],['projectId', 'PRJ-2018-0000477'],
['createdOn', 'Mon Nov 19 18:48:02 IST 2018']]
],
['createdBy=>>>BasicData',
[['userId', '999996279'], ['email', 'ITEST275#ITS.JNJ.com'],
['firstName', 'Iris'], ['lastName', 'TEST275'],
['ntId', 'itest275'], ['region', 'NA'],
[' LastAccessTime ', ' 1542639905785 ']]
]
]
Excepted format is
{
"key": "1542633482511430199",
"value=>>>BasicData": {
"isConfirmAndOrder": "0",
"brmRequestId": "BR-2018-0000124"
.
},
"createdBy=>>>BasicData": {
"userId": "999996279",
"email": "ITEST275#ITS.JNJ.com"
.
}
.
}
Actually format of large data is:
[
[
['key11','value11']
['key12',['key13','value13']]
['key14',['key15','value15']]
]
[
['key21','value21']
['key22',['key23','value23']]
['key24',['key25','value25']]
]
]
You can write a simple recursive function for this:
def to_dict_recursive(x):
d = {}
for key, value in x:
if isinstance(value, list):
value = to_dict_recursive(value)
else:
value = value.strip() # get rid of unnecessary whitespace
d[key.strip()] = value
return d
to_dict_recursive(x)
# {'createdBy=>>>BasicData': {'displayName': 'Iris TEST275',
# 'email': 'ITEST275#ITS.JNJ.com',
# 'firstName': 'Iris',
# 'lastName': 'TEST275',
# 'ntId': 'itest275',
# 'region': 'NA',
# 'roles': '[0]CG510_DHF_AP_Role',
# 'userId': '999996279'},
# 'formulaDetails=>>>BasicData': {'CreationTime': '1542633482512',
# 'LastAccessTime': '1542639905785',
# 'batchSizeUnits': 'kg<<<<<<',
# 'hitCount': '1',
# 'version': '1'},
# 'key': '1542633482511430199',
# 'value=>>>BasicData': {'brmRequestId': 'BR-2018-0000124',
# 'createdMonth': 'Nov',
# 'createdOn': 'Mon Nov 19 18:48:02 IST 2018',
# 'department': 'Global Packaging',
# 'gxp': '1',
# 'id': '1542633482511430199',
# 'isConfirmAndOrder': '0',
# 'isFilling': 'false',
# 'projectId': 'PRJ-2018-0000477',
# 'projectName': 'Automation_Product_By_Admin',
# 'requestType': 'batch',
# 'status': 'New',
# 'statusDescription': 'Batch request created',
# 'updatedOn': 'Mon Nov 19 18:48:02 IST 2018'}}
(I ran this in Python 3.6 so the order of the keys in the dictionary representation is different than insertion order. In Python 3.7+ this would be different.)
You can even make this into a dict comprehension:
def to_dict_recursive(x):
return {key.strip(): to_dict_recursive(value) if isinstance(value, list)
else value.strip
for key, value in x}
Since apparently some elements in your object are not a two-element list of key and value, you can add a simple guard against that:
def to_dict_recursive(x):
d = {}
try:
for key, value in x:
if isinstance(value, list):
value = to_dict_recursive(value)
else:
value = value.strip()
d[key.strip()] = value
except ValueError:
return x
return d
x = [[' key ', ' 1542633482511430199'],
["test", ["a", "b", "c"]]
]
to_dict_recursive(x)
# {'key': '1542633482511430199', 'test': ['a', 'b', 'c']}
Note that if mylist is a key-value pair list, then dict(mylist) simply returns a dictionary version of it. The tricky part is traversing deep into those nested lists to replace them with dictionaries. Here's a recursive function that does that:
# Where <kv> is your giant list-of-lists.
def kv_to_dict(kv):
if isinstance(kv, list):
kv = dict(kv)
for k in kv:
if isinstance(kv[k], list):
kv[k] = kv_to_dict(kv[k])
return kv
newdict = kv_to_dict(kvpairs)
Once you have things converted to a dictionary, you can just use json.dumps() to format it as JSON:
import json
as_json = json.dumps(newdict, indent=4)
print(as_json)
I see though that you've tried something similar and got an error. Are you sure that all of the lists in your data are really key-value pairs, and not for example a list of 3 strings?
I have the following code in index.html which grabs a dictionary list and prints out keys and values into a table
$(function() {
$('a#search').bind('click', function() {
$.getJSON('/_search', {
a: $('input[name="a"]').val()
}, function(data) {
var tableData = '<table>'
$.each(data.result, function(key, value){
tableData += '<tr><td>' + ' ' + key + ' ' + '</td>';
alert(key)
$.each(value, function(val){
alert(value[val])
tableData += '<td>' + value[val] + '</td>';
});
tableData += '</tr>';
});
tableData += '</table>';
$('#table').html(tableData);
});
What it is grabbing is a dictionary list from search.py
result = defaultdict(list)
return jsonify(result=result)
result contains the following
defaultdict(<class 'list'>, {'Developer': ['Office Koukan', 'Jorudan', 'Beam Software'], 'Publisher': ['Shouei', 'VAP', 'Hi Tech Expressions'], 'ReleaseDate': ['March 18, 1994', 'November 18, 1994', 'October 1, 1993'], 'Title': ['Idea no Hi', 'Pachinko Hi Hisshouhou', 'hunThe Hunt for Red October']})
However my output is as follows
Developer Publisher ReleaseDate Title
Office Koukan Jorudan Beam Software
Shouei VAP Hi Tech Expressions
March 18, 1994 November 18, 1994 October 1, 1993
Idea no Hi Pachinko Hi Hisshouhou hunThe Hunt for Red October
When the output should be
Developer Publisher ReleaseDate Title
Office Shouei ... ...
Koukan VAP ... ...
Jorudan ... ... ...
Beam Software ... ... ...
Any idea what I might be doing wrong?
This is a sample of what I was referencing (note I took out the part that gets data and hard coded that, pushed the result to a div.
Note that a simple array processes differently than the array of objects using $.each() regarding the index vs they key object name (like the header values you have)
// sample data to use (instead of json call)
var mything = {
'Developer': ['Office Koukan', 'Jorudan', 'Beam Software'],
'Publisher': ['Shouei', 'VAP', 'Hi Tech Expressions'],
'ReleaseDate': ['March 18, 1994', 'November 18, 1994', 'October 1, 1993'],
'Title': ['Idea no Hi', 'Pachinko Hi Hisshouhou', 'hunThe Hunt for Red October']
};
$(function() {
var data = mything;// sample data
// create a table to append rows to
var tableData = $('<table>');
// append a header row
tableData.append('<tr/>');
// reference that header row to append header td to
var header = tableData.find('tr').eq(0);
$.each(data, function(headthing, value) {
header.append('<td>' + headthing + '</td>');
$.each(value, function(idx, myval) {
if (!tableData.find('tr').eq(idx + 1).length) {
tableData.append('<tr/>'); // new row if not one
}
// put the columns in the row for each data
tableData.find('tr').eq(idx + 1).append('<td>' + myval + '</td>');
});
});
// put the table somewhere
$('#results').html(tableData);
});
Here is a fiddle showing it https://jsfiddle.net/MarkSchultheiss/ro5egfu3/1/
Yesterday, I have started with learning python. I want to parse some JSON values now. I have read many of tutorials and spent a lot of time on getting values by multilevel key (if I can call it like that) in my script but nothing works to me. Can you help me please?
This is my JSON output:
{
"future.arte.tv": [
{
"mediaUrl": "http://future.arte.tv/sites/default/files/styles/desktop-span12-940x529/public/berlin.jpg?itok=CvYlNekR",
"micropost": {
"html": "Berlin ",
"plainText": "Berlin"
},
"micropostUrl": "http://future.arte.tv/de/der-erste-weltkrieg-die-rolle-von-wissenschaft-und-technik",
"publicationDate": "Tue Jun 17 20:31:33 CEST 2014",
"relevance": 5.9615083,
"timestamp": 1403029893606,
"type": "image"
}
],
"www.zdf.de": [
{
"mediaUrl": "http://www.zdf.de/ZDFmediathek/contentblob/368/timg94x65blob/9800025",
"micropost": {
"plainText": "Berlin direkt"
},
"micropostUrl": "http://www.zdf.de/ZDFmediathek/hauptnavigation/sendung-a-bis-z",
"publicationDate": "Tue Jun 10 16:25:42 CEST 2014",
"relevance": 3.7259426,
"timestamp": 1402410342400,
"type": "image"
}
]
}
I need to get values stored in "mediaUrl" key so I tried to do
j = json.loads(jsonOutput)
keys = j.keys();
for key in keys:
print key # keys are future.arte.tv and www.zdf.de
print j[key]["mediaUrl"]
but print j[key]["mediaUrl"] causes this error:
TypeError: list indices must be integers, not str
so I tried to do print j[key][0] but the result is not as I wanted to have (I want to have just mediaUrl value... btw j[key][1] causes list index out of range error):
{u'micropostUrl': u'http://www.berlin.de/special/gesundheit-und-beauty/ernaehrung/1692726-215-spargelhoefe-in-brandenburg.html', u'mediaUrl': u'http://berlin.de/binaries/asset/image_assets/42859/ratio_4_3/1371638570/170x130/', u'timestamp': 1403862143675, u'micropost': {u'plainText': u'Spargel', u'html': u'Spargel '}, u'publicationDate': u'Fri Jun 27 11:42:23 CEST 2014', u'relevance': 1.6377668, u'type': u'image'}
Can you give me some advice please?
Here is a list comprehension that should do it
>>> [d[i][0].get('mediaUrl') for i in d.keys()]
['http://www.zdf.de/ZDFmediathek/contentblob/368/timg94x65blob/9800025',
'http://future.arte.tv/sites/default/files/styles/desktop-span12-940x529/public/berlin.jpg?itok=CvYlNekR']
How it works
First you can get a list of the top-level keys
>>> d.keys()
['www.zdf.de', 'future.arte.tv']
Get the corresponding values
>>> [d[i] for i in d.keys()]
[[{'micropostUrl': 'http://www.zdf.de/ZDFmediathek/hauptnavigation/sendung-a-bis-z', 'mediaUrl': 'http://www.zdf.de/ZDFmediathek/contentblob/368/timg94x65blob/9800025', 'timestamp': 1402410342400L, 'micropost': {'plainText': 'Berlin direkt'}, 'publicationDate': 'Tue Jun 10 16:25:42 CEST 2014', 'relevance': 3.7259426, 'type': 'image'}], [{'micropostUrl': 'http://future.arte.tv/de/der-erste-weltkrieg-die-rolle-von-wissenschaft-und-technik', 'mediaUrl': 'http://future.arte.tv/sites/default/files/styles/desktop-span12-940x529/public/berlin.jpg?itok=CvYlNekR', 'timestamp': 1403029893606L, 'micropost': {'plainText': 'Berlin', 'html': 'Berlin '}, 'publicationDate': 'Tue Jun 17 20:31:33 CEST 2014', 'relevance': 5.9615083, 'type': 'image'}]]
For each dictionary, grab the value for the 'mediaUrl' key
>>> [d[i][0].get('mediaUrl') for i in d.keys()]
['http://www.zdf.de/ZDFmediathek/contentblob/368/timg94x65blob/9800025',
'http://future.arte.tv/sites/default/files/styles/desktop-span12-940x529/public/berlin.jpg?itok=CvYlNekR']