Develop Reference

Writing resilient recursive code that will return results from a big json file

I have written a recursive code. I want more experienced people to tell me how resillient and fail-safe is my code:
I have a json file (Json file can be as big as 300MB):
[
{
"modules": {
"webpages": []
},
"webpages": {
"ip_addr": {
"value": "127.0.0.1",
"tags": []
},
"http": {
"status": {
"value": "Unavailable",
"tags": []
},
"title": {
"value": "403 Forbidden",
"tags": [
{
"category": "Server Code",
"match": "403"
},
{
"category": "Interesting Words",
"match": "Forbidden"
}
]
},
"server": {
"value": "Apache",
"tags": [
{
"category": "Apache Server",
"match": "Apache"
}
]
}
},
"redirects": [],
"robottxt": null
}
},
{
"modules": {
"webpages": []
}
}
]
I want to return value keys where tags are populated.
So I want to ignore:
"status": {
"value": "Unavailable",
"tags": []
},
But I want to return the title and server values. I also want to return ip_addr.value
I have written this code:
def getAllValues(nestedDictionary, firstArray, firstObj, firstUseful):
returnedArray = firstArray
tempValue = firstObj
useful = firstUseful
for key, value in nestedDictionary.items():
ipString = nestedDictionary.get("ip_addr")
if ipString is not None:
ipValue = ipString.get("value")
useful = {"ip_add": ipValue}
if isinstance(value, dict):
temp = {
"Key": key,
"useful": useful,
}
getAllValues(value, returnedArray, temp, useful)
else:
if key == "value":
tempValue["value"] = value
if key == "tags" and isinstance(value, list) and len(value) > 0:
tempValue["tags"] = value
returnedArray.append(tempValue)
return returnedArray
The above code should return:
[
{
"Key": "title",
"value": "403 Forbidden",
"useful": { "ip_addr": "127.0.0.1" },
"tags": [
{
"category": "Server Code",
"match": "403"
},
{
"category": "Interesting Words",
"match": "Forbidden"
}
]
},
{
"Key": "server",
"value": "Apache",
"useful": { "ip_addr": "127.0.0.1" },
"tags": [
{
"category": "Apache Server",
"match": "Apache"
}
]
}
]
Its a long post, but hopefully, someone can give me some assurance :)

How to get a value from JSON

This is the first time I'm working with JSON, and I'm trying to pull url out of the JSON below.
{
"name": "The_New11d112a_Company_Name",
"sections": [
{
"name": "Products",
"payload": [
{
"id": 1,
"name": "TERi Geriatric Patient Skills Trainer,
"type": "string"
}
]
},
{
"name": "Contact Info",
"payload": [
{
"id": 1,
"name": "contacts",
"url": "https://www.a3bs.com/catheterization-kits-8000892-3011958-3b-scientific,p_1057_31043.html",
"contacts": [
{
"name": "User",
"email": "Company Email",
"phone": "Company PhoneNumber"
}
],
"type": "contact"
}
]
}
],
"tags": [
"Male",
"Airway"
],
"_id": "0e4cd5c6-4d2f-48b9-acf2-5aa75ade36e1"
}
I have been able to access description and _id via
data = json.loads(line)
if 'xpath' in data:
xpath = data["_id"]
description = data["sections"][0]["payload"][0]["description"]
However, I can't seem to figure out a way to access url. One other issue I have is there could be other items in sections, which makes indexing into Contact Info a non starter.

Hope this helps:
import json
with open("test.json", "r") as f:
json_out = json.load(f)
for i in json_out["sections"]:
for j in i["payload"]:
for key in j:
if "url" in key:
print(key, '->', j[key])

I think your JSON is damaged, it should be like that.
{
"name": "The_New11d112a_Company_Name",
"sections": [
{
"name": "Products",
"payload": [
{
"id": 1,
"name": "TERi Geriatric Patient Skills Trainer",
"type": "string"
}
]
},
{
"name": "Contact Info",
"payload": [
{
"id": 1,
"name": "contacts",
"url": "https://www.a3bs.com/catheterization-kits-8000892-3011958-3b-scientific,p_1057_31043.html",
"contacts": [
{
"name": "User",
"email": "Company Email",
"phone": "Company PhoneNumber"
}
],
"type": "contact"
}
]
}
],
"tags": [
"Male",
"Airway"
],
"_id": "0e4cd5c6-4d2f-48b9-acf2-5aa75ade36e1"
}
You can check it on http://json.parser.online.fr/.
And if you want to get the value of the url.
import json
j = json.load(open('yourJSONfile.json'))
print(j['sections'][1]['payload'][0]['url'])

I think it's worth to write a short function to get the url(s) and make a decision whether or not to use the first found url in the returned list, or skip processing if there's no url available in your data.
The method shall looks like this:
def extract_urls(data):
payloads = []
for section in data['sections']:
payloads += section.get('payload') or []
urls = [x['url'] for x in payloads if 'url' in x]
return urls

This should print out the URL
import json
# open json file to read
with open('test.json','r') as f:
# load json, parameter as json text (file contents)
data = json.loads(f.read())
# after observing format of JSON data, the location of the URL key
# is determined and the data variable is manipulated to extract the value
print(data['sections'][1]['payload'][0]['url'])
The exact location of the 'url' key:
1st (position) of the array which is the value of the key 'sections'
Inside the array value, there is a dict, and the key 'payload' contains an array
In the 0th (position) of the array is a dict with a key 'url'
While testing my solution, I noticed that the json provided is flawed, after fixing the json flaws(3), I ended up with this.
{
"name": "The_New11d112a_Company_Name",
"sections": [
{
"name": "Products",
"payload": [
{
"id": 1,
"name": "TERi Geriatric Patient Skills Trainer",
"type": "string"
}
]
},
{
"name": "Contact Info",
"payload": [
{
"id": 1,
"name": "contacts",
"url": "https://www.a3bs.com/catheterization-kits-8000892-3011958-3b-scientific,p_1057_31043.html",
"contacts": [
{
"name": "User",
"email": "Company Email",
"phone": "Company PhoneNumber"
}
],
"type": "contact"
}
]
}
],
"tags": [
"Male",
"Airway"
],
"_id": "0e4cd5c6-4d2f-48b9-acf2-5aa75ade36e1"}

After utilizing the JSON that was provided by Vincent55.
I made a working code with exception handling and with certain assumptions.
Working Code:
## Assuming that the target data is always under sections[i].payload
from json import loads
line = open("data.json").read()
data = loads(line)["sections"]
for x in data:
try:
# With assumption that there is only one payload
if x["payload"][0]["url"]:
print(x["payload"][0]["url"])
except KeyError:
pass

django - iterate between json response objects

I have a response object that I am receiving from an api call. The response has several objects that are returned in a single call. What I want to do is grab information from each of the objects returned and store them in varialbes to use them within the application. I know to grab info from a json response when it returns a single objects but I am getting confused with multiples objects... I know how to automate the iteration process through something like a forloop... it wont iterate.
here is a sample response that I am getting:
I want to grab the _id from both items.
{
'user':"<class 'synapse_pay_rest.models.users.user.User'>(id=..622d)",
'json':{
'_id':'..6e80',
'_links':{
'self':{
'href':'https://uat-api.synapsefi.com/v3.1/users/..22d/nodes/..56e80'
}
},
'allowed':'CREDIT-AND-DEBIT',
'client':{
'id':'..26a34',
'name':'Charlie Brown LLC'
},
'extra':{
'note':None,
'other':{
},
'supp_id':''
},
'info':{
'account_num':'8902',
'address':'PO BOX 85139, RICHMOND, VA, US',
'balance':{
'amount':'750.00',
'currency':'USD'
},
'bank_long_name':'CAPITAL ONE N.A.',
'bank_name':'CAPITAL ONE N.A.',
'class':'SAVINGS',
'match_info':{
'email_match':'not_found',
'name_match':'not_found',
'phonenumber_match':'not_found'
},
'name_on_account':' ',
'nickname':'SynapsePay Test Savings Account - 8902',
'routing_num':'6110',
'type':'BUSINESS'
},
<class 'synapse_pay_rest.models.nodes.ach_us_node.AchUsNode'>({
'user':"<class 'synapse_pay_rest.models.users.user.User'>(id=..622d)",
'json':{
'_id':'..56e83',
'_links':{
'self':{
'href':'https://uat-api.synapsefi.com/v3.1/users/..d622d/nodes/..6e83'
}
},
'allowed':'CREDIT-AND-DEBIT',
'client':{
'id':'599378ec6aef1b0021026a34',
'name':'Charlie Brown LLC'
},
'extra':{
'note':None,
'other':{
},
'supp_id':''
},
'info':{
'account_num':'8901',
'address':'PO BOX 85139, RICHMOND, VA, US',
'balance':{
'amount':'800.00',
'currency':'USD'
},
'bank_long_name':'CAPITAL ONE N.A.',
'bank_name':'CAPITAL ONE N.A.',
'class':'CHECKING',
'match_info':{
'email_match':'not_found',
'name_match':'not_found',
'phonenumber_match':'not_found'
},
'name_on_account':' ',
'nickname':'SynapsePay Test Checking Account - 8901',
'routing_num':'6110',
'type':'BUSINESS'
},
})
Here is the code that I have:
It wont grab any values...
the iteration needs to be done to the nodes variable which is hte json response object.
def listedLinkAccounts(request):
currentUser = loggedInUser(request)
currentProfile = Profile.objects.get(user = currentUser)
user_id = currentProfile.synapse_id
synapseUser = SynapseUser.by_id(client, str(user_id))
options = {
'page':1,
'per_page':20,
'type': 'ACH-US',
}
nodes = Node.all(synapseUser, **options)
print(nodes)
response = nodes
_id = response["_id"]
print(_id)
return nodes
here is a sample api response from the api documenation:
{
"error_code": "0",
"http_code": "200",
"limit": 20,
"node_count": 5,
"nodes": [
{
"_id": "594e5c694d1d62002f17e3dc",
"_links": {
"self": {
"href": "https://uat-api.synapsefi.com/v3.1/users/594e0fa2838454002ea317a0/nodes/594e5c694d1d62002f17e3dc"
}
},
"allowed": "CREDIT-AND-DEBIT",
"client": {
"id": "589acd9ecb3cd400fa75ac06",
"name": "SynapseFI"
},
"extra": {
"other": {},
"supp_id": "ABC124"
},
"info": {
"account_num": "7443",
"address": "PLACE DE LA REPUBLIQUE 4 CROIX 59170 FR",
"balance": {
"amount": "0.00",
"currency": "USD"
},
"bank_long_name": "3 SUISSES INTERNATIONAL",
"bank_name": "3 SUISSES INTERNATIONAL",
"name_on_account": " ",
"nickname": "Some Account"
},
"is_active": true,
"timeline": [
{
"date": 1498307689471,
"note": "Node created."
},
{
"date": 1498307690130,
"note": "Unable to send micro deposits as node type is not ACH-US."
}
],
"type": "WIRE-INT",
"user_id": "594e0fa2838454002ea317a0"
},
{
...
},
{
...
},
...
],
"page": 1,
"page_count": 1,
"success": true
}

In Python parse a generic JSON structure looking for specific values

I'm trying to parse a JSON structure and looking for specific values.
Here an example of a possibile data to parse:
{
"objects": [
{
"from": 44,
"my_ref": "http://192.168.10.1/index.php",
"allow": "no",
"to": 10
},
{
"from": 20,
"my_ref": "http://192.168.10.2/index.php",
"allow": "mandatory",
"to": 0
}
],
"comment": "My PHP",
"identifiable_with_user": true,
"key": 10,
"link": [
{
"href": "http://192.168.10.1/index.php",
"method": "GET",
"rel": "self",
"type": "website"
},
{
"href": "http://192.168.10.5/identifiable.php",
"method": "GET",
"rel": "account_info"
}
],
"name": "Accounts",
"read_only": true,
"system": true,
"system_key": 20,
"tls_match_ref": "http://192.168.10.5/accounts.php"
}
I need to extract all the values identified by the key *ref (like my_ref, href, etc...) in a generic JSON structure. I just need to extract the URL and then perform some actions:
for entities in JSON_structure
URL= "take the URL corresponding to the ref key"
so something with the URL
I tried using .items() filtering "http://" string but didn't work
URL = {k:v for (k,v) in element_detail.items() if "http://" in k or v}

You should parse the data then look for the keys with ref in it.
That does the trick :
import json
def parse(root):
for k,v in root.items():
if isinstance(v, list):
for e in v:
parse(e)
else:
if 'ref' in k:
print(k + ':' + v)
json_data = '{\
"objects": [\
{ "from": 44, "my_ref": "http://192.168.10.1/index.php", "allow": "no", "to": 10 },\
{ "from": 20, "my_ref": "http://192.168.10.2/index.php", "allow": "mandatory", "to": 0 }\
],\
"comment": "My PHP",\
"identifiable_with_user": true,\
"key": 10,\
"link": [\
{ "href": "http://192.168.10.1/index.php", "method": "GET", "rel": "self", "type": "website" },\
{ "href": "http://192.168.10.5/identifiable.php", "method": "GET", "rel": "account_info" }\
],\
"name": "Accounts",\
"read_only": true,\
"system": true,\
"system_key": 20,\
"tls_match_ref": "http://192.168.10.5/accounts.php"\
}'
data = json.loads(json_data)
parse(data)
Output :
href:http://192.168.10.1/index.php
href:http://192.168.10.5/identifiable.php
my_ref:http://192.168.10.1/index.php
my_ref:http://192.168.10.2/index.php
tls_match_ref:http://192.168.10.5/accounts.php

How to iterate through "nameless" JSON entries in Python?

I have some JSON data from the HubSpot CRM API which, after doing some pagination using Python Code essentially looks like this:
[
{
"dealId": 18039629,
"portalId": 62515,
"isDeleted": false
},
{
"dealId": 18040854,
"portalId": 62515,
"isDeleted": false
}
]
... and now what I'd like to do is:
1) Read one "set" of JSON at a time (meaning dealId, portalId, isDeleted)
2) If isDeleted==false then grab the dealId and portalId and store in variables
3) Use the variables from #2 above to build a URL string that can be used to go back to the HubSpot API and get information on each individual deal (this API endpoint is https://api.hubapi.com/deals/v1/deal/23487870?hapikey=demo (where 23487870 is the dealId from the above JSON)
4) Combine that individual deal-level info into another set of JSON. Specifically I want to grab /properties/dealname/value and properties/dealstage/value from JSON that looks like this:
{
"portalId": 62515,
"dealId": 23487870,
"isDeleted": false,
"properties": {
"dealname": {
"value": "AutomationTestUser-national",
"timestamp": 1457692022120
},
"dealstage": {
"value": "appointmentscheduled",
"timestamp": 1457692022120
}
},
"imports": []
}
5) And then output the final result in JSON something like this:
{
"deals":
[
{
"portalId": 62515,
"dealId": 23487870,
"isDeleted": false,
"properties": {
"dealname": {
"value": "AutomationTestUser-national",
"timestamp": 1457692022120
},
"dealstage": {
"value": "appointmentscheduled",
"timestamp": 1457692022120
}
},
"imports": []
},
{
"portalId": 62515,
"dealId": 23487871,
"isDeleted": false,
"properties": {
"dealname": {
"value": "AutomationTestUser-regional",
"timestamp": 1457692022120
},
"dealstage": {
"value": "appointmentscheduled",
"timestamp": 1457692022120
}
},
"imports": []
}
]
}
... all in Python.
Any help?

import json
output = {"deals": []}
data = """
[
{
"dealId": 18039629,
"portalId": 62515,
"isDeleted": false
},
{
"dealId": 18040854,
"portalId": 62515,
"isDeleted": false
}
]
"""
j = json.loads(data)
for deal in j:
if deal['isDeleted']:
continue # Ignore if deleted
dealId = deal['dealId']
url = 'https://api.hubapi.com/deals/v1/deal/' + str(dealId) + '?hapikey=demo' # Construct new url here
data = getSource(url) # implement getSource to perform web request and get data
j2 = json.loads(data)
output['deals'].append(j2)
print(output)