Develop Reference

Type Error for Path Data. Must be list or null

I am getting the following error when attempting to convert json data to a dataframe. I have successfully used this same method to convert json to a dataframe with similar data in the same script.
The full error:
TypeError: {'success': True, 'data': {'data1': 1, 'data2': 1, 'data3': 1, 'data4': True, 'data5': 0, 'data6': 0, 'data7': False, 'data8': 'ABC', 'start_date': '2000-04-14', 'end_date': '2000-09-23', 'data9': None, 'add_time': '2000-07-12 23:00:11', 'update_time': '2000-06-1420:18:55', 'data10': 1, 'data11': 'custom', 'data12': None}}
has non list value
{'data1': 1, 'data2': 1, 'data3': 1, 'data4': True, 'data5': 0, 'data6': 0, 'data7': False, 'data8': 'ABC', 'start_date': '2000-04-14', 'end_date': '2000-09-23', 'data9': None, 'add_time': '2000-07-12 23:00:11', 'update_time': '2000-06-1420:18:55', 'data10': 1, 'data11': 'custom', 'data12': None}
for path data. Must be list or null.
the function:
def get_subscriptions(id, df):
subscriptions_params = {'api_token': 'abc'}
subscriptions_headers = {'Content-Type': 'application/json'}
subscriptions_response = requests.get('https://url/{}'.format(id), params=subscriptions_params,
headers=subscriptions_headers)
subscriptions_data = subscriptions_response.json()
subscriptions_temp_df = pd.json_normalize(subscriptions_data, record_path=['data'])
I do the exact same thing with a similar (but actually more complex) piece of data with no problems. An example of the response that works:
{'success': True, 'data': [{'data1': 1, 'data2': {'data3': 1, 'name': 'name', 'email': 'email#email.com', 'data4': 0, 'data5': None, 'data6': False, 'data7': 1}, 'data8': {'data9': 1, 'name': 'name', 'email': 'email#email.com', 'data10': 0, 'data11': None, 'data12': True, 'data13': 1}, 'data14': {'data15': True, 'name': 'name' .... etc.
this one is actually massive, where as for the one with issues the error includes the full length of the data.
removed the actual data, but did not change the type of data. strings inside single quotes are just other strings. 1s are just other numbers, etc.
any ideas why one succeeds and another fails?

I do not know what the issue/difference is, but this small change works:
def get_subscriptions(id, df):
subscriptions_params = {'api_token': 'abc'}
subscriptions_headers = {'Content-Type': 'application/json'}
subscriptions_response = requests.get('https://url/{}'.format(id), params=subscriptions_params, headers=subscriptions_headers)
subscriptions_data = subscriptions_response.json()
subscriptions_data = subscriptions_data['data']
subscriptions_temp_df = pd.json_normalize(subscriptions_data)

Uisng Json to format and indent in website requests Python

I am trying to indent and assort the format of balance so that it is easier to read. I want to print the RequestResponse like the Expected output. The balance variable is of type tuple. How could I do such a thing?
import bybit
import json
balance = client.Wallet.Wallet_getBalance(coin="BTC").result()
print(balance)
Output:
({'ret_code': 0, 'ret_msg': 'OK', 'ext_code': '', 'ext_info': '', 'result': {'BTC': {'equity': 0.00208347, 'available_balance': 0.00208347, 'used_margin': 0, 'order_margin': 0, 'position_margin': 0, 'occ_closing_fee': 0, 'occ_funding_fee': 0, 'wallet_balance': 0.00208347, 'realised_pnl': 0, 'unrealised_pnl': 0, 'cum_realised_pnl': 8.347e-05, 'given_cash': 0, 'service_cash': 0}}, 'time_now': '1616685310.655072', 'rate_limit_status': 118, 'rate_limit_reset_ms': 1616685310652, 'rate_limit': 120}, <bravado.requests_client.RequestsResponseAdapter object at 0x000001F5E92EB048>)
Expected Output:
{
"cross_seq": 11518,
"data": [
{
"price": "2999.00",
"side": "Buy",
"size": 9,
"symbol": "BTCUSD"
},
{
"price": "3001.00",
"side": "Sell",
"size": 10,
"symbol": "BTCUSD"
}
],
"timestamp_e6": 1555647164875373,
"topic": "orderBookL2_25.BTCUSD",
"type": "snapshot"
}

I think you provided the wrong expected output since the fields between your output and expected output don't match but in general if you want a better display of a dictionary you can use the json package:
response = {'ret_code': 0, 'ret_msg': 'OK', 'ext_code': '', 'ext_info': '', 'result': {'BTC': {'equity': 0.00208347, 'available_balance': 0.00208347, 'used_margin': 0, 'order_margin': 0, 'position_margin': 0, 'occ_closing_fee': 0, 'occ_funding_fee': 0, 'wallet_balance': 0.00208347, 'realised_pnl': 0, 'unrealised_pnl': 0, 'cum_realised_pnl': 8.347e-05, 'given_cash': 0, 'service_cash': 0}}, 'time_now': '1616685310.655072', 'rate_limit_status': 118, 'rate_limit_reset_ms': 1616685310652, 'rate_limit': 120}
import json
json.loads(json.dumps(response, indent=4, sort_keys=True))
This will give you the following output:
{'ext_code': '',
'ext_info': '',
'rate_limit': 120,
'rate_limit_reset_ms': 1616685310652,
'rate_limit_status': 118,
'result': {'BTC': {'available_balance': 0.00208347,
'cum_realised_pnl': 8.347e-05,
'equity': 0.00208347,
'given_cash': 0,
'occ_closing_fee': 0,
'occ_funding_fee': 0,
'order_margin': 0,
'position_margin': 0,
'realised_pnl': 0,
'service_cash': 0,
'unrealised_pnl': 0,
'used_margin': 0,
'wallet_balance': 0.00208347}},
'ret_code': 0,
'ret_msg': 'OK',
'time_now': '1616685310.655072'}
Another solution is to use pprint
import pprint
pprint.pprint(response)
This will give you the following output:
{'ext_code': '',
'ext_info': '',
'rate_limit': 120,
'rate_limit_reset_ms': 1616685310652,
'rate_limit_status': 118,
'result': {'BTC': {'available_balance': 0.00208347,
'cum_realised_pnl': 8.347e-05,
'equity': 0.00208347,
'given_cash': 0,
'occ_closing_fee': 0,
'occ_funding_fee': 0,
'order_margin': 0,
'position_margin': 0,
'realised_pnl': 0,
'service_cash': 0,
'unrealised_pnl': 0,
'used_margin': 0,
'wallet_balance': 0.00208347}},
'ret_code': 0,
'ret_msg': 'OK',
'time_now': '1616685310.655072'}

Import JSON, then using json.dumps(balance, indent=4) would get you that format.
You could add keyword argument of sort_keys=True if you want them sorted.

Get value of same subkey of json file

I want to convert one json file format to another format.
input json file looks like,
json1 = {"Roll Number":{"0":"12345675890"},"Exam Code":{"0":"125"},"q1":{"0":"A"},"q2":{"0":"B"},"q3":{"0":"B"},"q4":{"0":"C"},"q5":{"0":"C"}
Here you can consider 0 as the 0th student (answersheet of first student)
I want this output format,
json2 = {
"id": 1,
"Roll Number": 12345675890,
"Exam Code": 125,
"qtn_ans_id": 1, #this is question number
"field1": 0,
"field2": 0,
"field3": 1, #if ans is C then field3=1 and else is 0
"field4": 0,
}
in json1 it contains 5 question answers and details and i want to convert it into json2, and my code is
import json
Stored json file into data
with open('json1.json') as data_file:
data = json.load(data_file)
write default format
format = {"table": "omr",
"rows":
[
{
"id": 1,
"regist_no": 1215152,
"exam_id": 125,
"qtn_ans_id": 1,
"field1": 0,
"field2": 0,
"field3": 0,
"field4": 0,
}
]
}
My dummy Code
json2 = {}
for (k, v) in data.items():
if(k=='Roll Number'):
format['rows'][0]['regist_no']=v['0']
if(k=='Exam Code'):
format['rows'][0]['exam_id'] = v['0']
if(v['0']=='A'):
format['rows'][0]['field1'] = 1
elif(v['0']=='B'):
format['rows'][0]['field2'] = 1
elif(v['0']=='C'):
format['rows'][0]['field3'] = 1
elif(v['0']=='D'):
format['rows'][0]['field4'] = 1
json2.append(format['rows'])
wrong output i get is,
[{
'field1': 1,
'field2': 1,
'field3': 1,
'field4': 1
}]
All the field values are 1.
For answer A Right output can be,
[{
'field1': 1,
'field2': 0,
'field3': 0,
'field4': 0
}]
or is there any other way like using pandas data frame.
I know this is long and dummy question but i will be glade if anyone can help. Thank you!

How about something like this?
# assuming your json data looks like this:
json1 = [
{
"Roll Number" : {"0":"12345675890"},
"Exam Code" : {"0":"125"},
"q1" : {"0":"A"},
"q2" : {"0":"B"},
"q3" : {"0":"B"},
"q4" : {"0":"C"},
"q5" : {"0":"C"},
},
{
"Roll Number" : {"0":"12345675891"},
"Exam Code" : {"0":"125"},
"q1" : {"0":"C"},
"q2" : {"0":"B"},
"q3" : {"0":"A"},
"q4" : {"0":"C"},
"q5" : {"0":"D"},
},
]
Then:
# converts old question field to new question field:
def ConvertAnswer(question):
# get letter answer from question:
for key, value in question.items():
letter = value
# pack fields object:
fields = {
'field1' : 1 if letter == 'A' else 0,
'field2' : 1 if letter == 'B' else 0,
'field3' : 1 if letter == 'C' else 0,
'field4' : 1 if letter == 'D' else 0,
}
return fields
# iterate over each student:
for student in json1:
# iterate over each field in student:
for key, value in student.items():
# check for questions (check if key starts with 'q'):
if key[0] == 'q':
# replace question field in student object:
student[key] = ConvertAnswer(value)
Will result in:
# output of json1:
[{'Roll Number': {'0': '12345675890'},
'Exam Code': {'0': '125'},
'q1': {'field1': 1, 'field2': 0, 'field3': 0, 'field4': 0},
'q2': {'field1': 0, 'field2': 1, 'field3': 0, 'field4': 0},
'q3': {'field1': 0, 'field2': 1, 'field3': 0, 'field4': 0},
'q4': {'field1': 0, 'field2': 0, 'field3': 1, 'field4': 0},
'q5': {'field1': 0, 'field2': 0, 'field3': 1, 'field4': 0}},
{'Roll Number': {'0': '12345675891'},
'Exam Code': {'0': '125'},
'q1': {'field1': 0, 'field2': 0, 'field3': 1, 'field4': 0},
'q2': {'field1': 0, 'field2': 1, 'field3': 0, 'field4': 0},
'q3': {'field1': 1, 'field2': 0, 'field3': 0, 'field4': 0},
'q4': {'field1': 0, 'field2': 0, 'field3': 1, 'field4': 0},
'q5': {'field1': 0, 'field2': 0, 'field3': 0, 'field4': 1}}]

My solution in more "pandasonic" than the other answer.
Start from defining a function, which will be used to change
the answer code to 4 answer columns:
def ans(code):
return pd.Series([ int(code == x) for x in 'ABCD' ],
index=[ 'field' + str(i) for i in range(1, 5)])
Read your input file into a DataFrame as follows:
with open('input.json') as data_file:
df = pd.read_json(data_file, orient='columns')
Then reformat it, saving in another DataFrame:
df2 = pd.wide_to_long(df.rename_axis(index='id').reset_index(),
stubnames='q', i='Roll Number', j='qtn_ans_id').reset_index()\
.reindex(columns=['id', 'Roll Number', 'Exam Code', 'qtn_ans_id', 'q'])
For now it contains one column q with answer codes.
So to change it into 4 columns for each field, run:
df2 = df2.join(df2.q.apply(ans)).drop(columns=['q'])
And the last step is to create the output JSON string:
df2.to_json(orient='records')
which you can e.g. save in an output file.
Note: If your input contained e.g. data for 2 students,
the JSON file should have the following form:
{
"Roll Number":{"0":12345675890,"1":23456758901},
"Exam Code":{"0":125,"1":125},
"q1":{"0":"A","1":"A"},
"q2":{"0":"B","1":"B"},
"q3":{"0":"B","1":"C"},
"q4":{"0":"C","1":"D"},
"q5":{"0":"C","1":"C"}
}
(different than the input format proposed in the other answer).

Write json responce for each request into a file

I wrote a code which is making a request to API and recieving output in JSON. So my question is how to write output for each request in file. Now my code is doing the last one request.
import requests
import json
with open("query4.txt", "rt") as file:
data_file = file.read()
for line in data_file.split("\n"):
drX, drY, fromX, fromY, dist = line.split(",")
url = "https://api.openrouteservice.org/directions?"
params = [
["api_key", "my_api_key"],
["coordinates", "%s,%s|%s,%s" % (drY, drX, fromY, fromX)],
["profile", "driving-car"]
]
headers = {
"Accept": "application/json, application/geo+json,"
"application/gpx+xml, img/png; charset=utf-8"}
responce = requests.get(url=url, params=params, headers=headers)
# print(responce.url)
# print(responce.text)
result = json.loads(responce.text)
# print(result)
with open("result.txt", "w+") as f_o:
for rows in result["routes"]:
f_o.writelines(json.dumps(rows["summary"]["distance"])) # depending on how do you want the result
print(result["routes"])
I have an output like this:
{'routes': [{'warnings': [{'code': 1, 'message': 'There may be restrictions on some roads'}], 'summary': {'distance': 899.6, 'duration': 102.1}, 'geometry_format': 'encodedpolyline', 'geometry': 'u~uiHir{iFb#SXzADTlAk#JOJ]#_#CWW{AKo#k#eDEYKo#y#{EGc#G]GYCOa#gCc#iCoBsLNGlAm#VK^Sh#Un#tD', 'segments': [{'distance': 899.6, 'duration': 102.1, 'steps': [{'distance': 22.1, 'duration': 5.3, 'type': 11, 'instruction': 'Head south', 'name': '', 'way_points': [0, 1]}, {'distance': 45.4, 'duration': 10.9, 'type': 1, 'instruction': 'Turn right', 'name': '', 'way_points': [1, 3]}, {'distance': 645.5, 'duration': 52.3, 'type': 0, 'instruction': 'Turn left onto Партизанська вулиця', 'name': 'Партизанська вулиця', 'way_points': [3, 21]}, {'distance': 114.4, 'duration': 20.6, 'type': 1, 'instruction': 'Turn right', 'name': '', 'way_points': [21, 26]}, {'distance': 72.1, 'duration': 13, 'type': 1, 'instruction': 'Turn right', 'name': '', 'way_points': [26, 27]}, {'distance': 0, 'duration': 0, 'type': 10, 'instruction': 'Arrive at your destination, on the left', 'name': '', 'way_points': [27, 27]}]}], 'way_points': [0, 27], 'extras': {'roadaccessrestrictions': {'values': [[0, 1, 0], [1, 3, 2], [3, 27, 0]], 'summary': [{'value': 0, 'distance': 854.2, 'amount': 94.95}, {'value': 2, 'distance': 45.4, 'amount': 5.05}]}}, 'bbox': [38.484536, 48.941171, 38.492904, 48.943022]}], 'bbox': [38.484536, 48.941171, 38.492904, 48.943022], 'info': {'attribution': 'openrouteservice.org | OpenStreetMap contributors', 'engine': {'version': '5.0.1', 'build_date': '2019-05-29T14:22:56Z'}, 'service': 'routing', 'timestamp': 1568280549854, 'query': {'profile': 'driving-car', 'preference': 'fastest', 'coordinates': [[38.485115, 48.942059], [38.492073, 48.941676]], 'language': 'en', 'units': 'm', 'geometry': True, 'geometry_format': 'encodedpolyline', 'instructions_format': 'text', 'instructions': True, 'elevation': False}}}
{'routes': [{'summary': {'distance': 2298, 'duration': 365.6}, 'geometry_format': 'encodedpolyline', 'geometry': 'u~a{Gee`zDLIvBvDpClCtA|AXHXCp#m#bBsBvBmC`AmAtIoKNVLXHPb#c#`A_AFENGzAc#XKZCJ?PDLBH#F?T?PC~CcATOt#Sd#QLKBCBAb#]ZG|#OY_DQ}IE{DC_DAg#Eg#q#aFgBuH^GjBFj#
I did NeverHopeless answer, but i've got the same:
result = json.loads(responce.text)
i = 0
with open(f"result-{i}.txt", "w+") as f_o:
i += 1
for rows in result["routes"]:
f_o.writelines(json.dumps(rows["summary"]["distance"])) # depending on how do you want the result
print(result["routes"])
Output now looks like this
899.622982138.832633191.8
I'm expecting to get this:
2298
2138.8
3263
3191.8
Every value is a distance from different requests so i need to have each on the new line.

You need to open and keep open the output file before your loop:
import requests
import json
with open("query4.txt", "rt") as file:
data_file = file.read()
with open("result.txt", "w") as f_o:
for line in data_file.split("\n"):
drX, drY, fromX, fromY, dist = line.split(",")
url = "https://api.openrouteservice.org/directions?"
params = [
["api_key", "my_api_key"],
["coordinates", "%s,%s|%s,%s" % (drY, drX, fromY, fromX)],
["profile", "driving-car"]
]
headers = {
"Accept": "application/json, application/geo+json,"
"application/gpx+xml, img/png; charset=utf-8"}
responce = requests.get(url=url, params=params, headers=headers)
# print(responce.url)
# print(responce.text)
result = json.loads(responce.text)
# print(result)
for rows in result["routes"]:
print(rows["summary"]["distance"], file=f_o) # depending on how do you want the result
# print(result["routes"])

I think it's better to write results in different files with timestamp. in this way you don't rewrite on your older file and also you can find them easier.
current_time = time.strftime("%m_%d_%y %H_%M_%S", time.localtime())
with open(current_time + ".txt", "w+") as f_o:
for rows in result["routes"]:
f_o.writelines(json.dumps(rows["summary"]["distance"])) # depending on how do you want the result
print(result["routes"])

You need to make this filename "result.txt" dynamic. Currently it is overwriting content.
Perhaps like this:
i = 0 # <----- Keep it outside your for loop or it will be always set to zero
with open(f"result-{i}.txt", "w+") as f_o:
i += 1
Or instead of integers, you may better use timestamp in filename.

Python 3.5 Pandas and MongoDB -json_normalize: raise TypeError("data argument can't be an iterator")

I am trying to do data transformation using pandas on python3.5.
Data is fetched from MongoDB using MongoClient() and json_normalize.
However when i execute below code it throws error as data argument can't be an iterator. Any pointers will help.
Sample Data :
{'bank_code': 'CID005', 'status': 'Init', 'cpgmid': '7847', 'blaze_transId': 'ZI4YQFFOTGG96ZRUQWZS121111632121509-9173782788741', 'currency': 'INR', 'amount': 7800, 'merchant_trans_id': '121111632121509-9173782788741', 'date_time': datetime.datetime(2016, 11, 11, 14, 1, 14, 44000), 'consumer_mobile': 9999999999.0, 'consumer_email': 'test#test.com', '_id': ObjectId('5825cf2a11eae123023730a9')}
{'bank_code': 'CID001', 'status': 'Init', 'cpgmid': '228', 'blaze_transId': '1rjfeklmg2281610111931334hjlm4j8xwl', 'currency': 'INR', 'amount': 651.4, 'merchant_trans_id': '161111569056', 'date_time': datetime.datetime(2016, 11, 11, 14, 1, 14, 333000), 'consumer_mobile': 9999992399.0, 'consumer_email': 'test#air.com', '_id': ObjectId('5825cf2a11eae123023730af')}
{'bank_code': 'CID001', '_id': ObjectId('5825cf2a097752b55d0f17ac'), 'custom_params': {'suppress_trans': 1}, 'currency': 'INR', 'merchant_trans_id': 'BX819215014788728725757', 'date_time': datetime.datetime(2016, 11, 11, 14, 1, 14, 421000), 'consumer_mobile': 0, 'status': 'Init', 'cpgmid': '1656', 'blaze_transId': '1bygejlxl16561610111931423bkgfe1uxx', 'amount': 577, 'consumer_email': 'p.25#gmail.com'}
Code:
start_datetime1 = (datetime.now() - timedelta(days=1)).replace(hour=18, minute=30, second=00, microsecond=0)
start_datetime2 = (datetime.now() - timedelta(days=0)).replace(hour=18, minute=29, second=59, microsecond=0)
client = MongoClient(host_val, int(port_val))
db = client.cit
transactions_collection = db.transactions
cursor = json_normalize(transactions_collection.find({'date_time': {'$lt': start_datetime2, '$gte': start_datetime1}},
{'_id': 1, 'blaze_transId': 1, 'status': 1, 'merchant_trans_id': 1,
'date_time': 1, 'amount': 1, 'status': 1, 'cpgmid': 1, 'currency': 1,
'status_msg': 1, 'bank_code': 1, 'custom_params.suppress_trans': 1,
'consumer_email': 1,'consumer_mobile': 1}))
df_txn = pd.DataFrame(cursor)
Error:
ERROR:root:Exception in fetch
Traceback (most recent call last):
File "/opt/Analytics-services/ETLservices/transformationService/Blazenet_Txns_Fact.py", line 174, in fetchBlazenetTxnsFromDB
'consumer_email': 1,'consumer_mobile': 1}))
File "/usr/local/lib/python3.5/site-packages/pandas/io/json.py", line 717, in json_normalize
return DataFrame(data)
File "/usr/local/lib/python3.5/site-packages/pandas/core/frame.py", line 283, in __init__
raise TypeError("data argument can't be an iterator")
TypeError: data argument can't be an iterator

You need to convert the cursor to a list before passing it to json_normalize.
cursor = transactions_collection.find({'date_time': {'$lt': start_datetime2, '$gte': start_datetime1}},
{'_id': 1, 'blaze_transId': 1, 'status': 1, 'merchant_trans_id': 1,
'date_time': 1, 'amount': 1, 'status': 1, 'cpgmid': 1, 'currency': 1,
'status_msg': 1, 'bank_code': 1, 'custom_params.suppress_trans': 1,
'consumer_email': 1,'consumer_mobile': 1})
df_txn = pd.DataFrame(json_normalize(list(cursor)))
You may also want to look at monary if you want to avoid having the massive ammounts of data converted to a list.

Along with Steves answer changed mongo query to avoid selecting data points which were not required. This is done to as custom_params was not getting flattened if i try to select it in mongo query.
cursor = transactions_collection.find({"date_time": {'$lt': start_datetime2, '$gte': start_datetime1}},{'bankRes':0,'rawDV':0})
df_txn = pd.DataFrame(json_normalize(list(cursor)))