I have a '.txt' file that contains JSON data like below.
[{'tradable': True, 'mode': 'full', 'instrument_token': 4708097, 'last_price': 178.65, 'last_traded_quantity': 5, 'average_traded_price': 180.1, 'volume_traded': 4581928, 'total_buy_quantity': 1282853, 'total_sell_quantity': 1673842, 'ohlc': {'open': 181.95, 'high': 181.95, 'low': 177.8, 'close': 181.0}, 'change': -1.2983425414364609, 'last_trade_time': datetime.datetime(2023, 1, 12, 13, 4, 58), 'oi': 0, 'oi_day_high': 0, 'oi_day_low': 0, 'exchange_timestamp': datetime.datetime(2023, 1, 12, 13, 5, 1), 'depth': {'buy': [{'quantity': 653, 'price': 178.6, 'orders': 8}, {'quantity': 2408, 'price': 178.55, 'orders': 15}, {'quantity': 6329, 'price': 178.5, 'orders': 22}, {'quantity': 9161, 'price': 178.45, 'orders': 24}, {'quantity': 7775, 'price': 178.4, 'orders': 17}], 'sell': [{'quantity': 5726, 'price': 178.7, 'orders': 8}, {'quantity': 4099, 'price': 178.75, 'orders': 11}, {'quantity': 23951, 'price': 178.8, 'orders': 25}, {'quantity': 7446, 'price': 178.85, 'orders': 21}, {'quantity': 11379, 'price': 178.9, 'orders': 21}]}}, {'tradable': True, 'mode': 'full', 'instrument_token': 871681, 'last_price': 972.55, 'last_traded_quantity': 1, 'average_traded_price': 973.85, 'volume_traded': 411290, 'total_buy_quantity': 152925, 'total_sell_quantity': 214765, 'ohlc': {'open': 971.75, 'high': 978.6, 'low': 969.0, 'close': 967.75}, 'change': 0.4959958667011061, 'last_trade_time': datetime.datetime(2023, 1, 12, 13, 4, 53), 'oi': 0, 'oi_day_high': 0, 'oi_day_low': 0, 'exchange_timestamp': datetime.datetime(2023, 1, 12, 13, 5, 4), 'depth': {'buy': [{'quantity': 6, 'price': 972.15, 'orders': 2}, {'quantity': 3, 'price': 972.1, 'orders': 2}, {'quantity': 15, 'price': 972.05, 'orders': 3}, {'quantity': 455, 'price': 972.0, 'orders': 16}, {'quantity': 14, 'price': 971.95, 'orders': 2}], 'sell': [{'quantity': 6, 'price': 972.5, 'orders': 3}, {'quantity': 49, 'price': 972.55, 'orders': 2}, {'quantity': 10, 'price': 972.6, 'orders': 1}, {'quantity': 27, 'price': 972.65, 'orders': 2}, {'quantity': 10, 'price': 972.7, 'orders': 1}]}}]
This data was written to a .txt file after it was recieved from zerodha websocket. Now, I want to read the data from the .txt file using my python script and want to load it as a json. But the json.loads() method in python throws the below error.
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 3 (char 2)
I have tried eval and ast.literal_eval methods in python as well but it didn't solve my problem. All I want is to be able to read the above data as a JSON to my python script. Any leads would be of great help.
Let me start off with THIS IS A BAD IDEA.
The comments on the question call out that this is not a JSON object in a file, rather the result of some other Python process printing the result and putting that in a file. The correct solution would be to modify the producer to use json.dumps() instead.
That aside, here's a DANGEROUS way to read that source file into a Python object.
import datetime # needed for `eval()`
with open('textfile.txt', 'r') as f:
data = eval(f.read())
from pprint import pprint
pprint(data)
This will produce the following output from that input:
[{'average_traded_price': 180.1,
'change': -1.2983425414364609,
'depth': {'buy': [{'orders': 8, 'price': 178.6, 'quantity': 653},
{'orders': 15, 'price': 178.55, 'quantity': 2408},
{'orders': 22, 'price': 178.5, 'quantity': 6329},
{'orders': 24, 'price': 178.45, 'quantity': 9161},
{'orders': 17, 'price': 178.4, 'quantity': 7775}],
'sell': [{'orders': 8, 'price': 178.7, 'quantity': 5726},
{'orders': 11, 'price': 178.75, 'quantity': 4099},
{'orders': 25, 'price': 178.8, 'quantity': 23951},
{'orders': 21, 'price': 178.85, 'quantity': 7446},
{'orders': 21, 'price': 178.9, 'quantity': 11379}]},
'exchange_timestamp': datetime.datetime(2023, 1, 12, 13, 5, 1),
'instrument_token': 4708097,
'last_price': 178.65,
'last_trade_time': datetime.datetime(2023, 1, 12, 13, 4, 58),
'last_traded_quantity': 5,
'mode': 'full',
'ohlc': {'close': 181.0, 'high': 181.95, 'low': 177.8, 'open': 181.95},
'oi': 0,
'oi_day_high': 0,
'oi_day_low': 0,
'total_buy_quantity': 1282853,
'total_sell_quantity': 1673842,
'tradable': True,
'volume_traded': 4581928},
{'average_traded_price': 973.85,
'change': 0.4959958667011061,
'depth': {'buy': [{'orders': 2, 'price': 972.15, 'quantity': 6},
{'orders': 2, 'price': 972.1, 'quantity': 3},
{'orders': 3, 'price': 972.05, 'quantity': 15},
{'orders': 16, 'price': 972.0, 'quantity': 455},
{'orders': 2, 'price': 971.95, 'quantity': 14}],
'sell': [{'orders': 3, 'price': 972.5, 'quantity': 6},
{'orders': 2, 'price': 972.55, 'quantity': 49},
{'orders': 1, 'price': 972.6, 'quantity': 10},
{'orders': 2, 'price': 972.65, 'quantity': 27},
{'orders': 1, 'price': 972.7, 'quantity': 10}]},
'exchange_timestamp': datetime.datetime(2023, 1, 12, 13, 5, 4),
'instrument_token': 871681,
'last_price': 972.55,
'last_trade_time': datetime.datetime(2023, 1, 12, 13, 4, 53),
'last_traded_quantity': 1,
'mode': 'full',
'ohlc': {'close': 967.75, 'high': 978.6, 'low': 969.0, 'open': 971.75},
'oi': 0,
'oi_day_high': 0,
'oi_day_low': 0,
'total_buy_quantity': 152925,
'total_sell_quantity': 214765,
'tradable': True,
'volume_traded': 411290}]
Again, I will restate, THIS IS A BAD IDEA.
Read more on the specifics of eval here: https://realpython.com/python-eval-function/
Your JSON is not valid. You can check that by using an online JSON validator like this one: https://jsonformatter.org/
After entering the “JSON”, you can see it is not in a valid format.
You could either export it right, which I would highly recommend, or you could replace the wrong chars.
Currently, you have three issues:
You are using single quotes instead of double quotes
You are not parsing your datetime object, it looks like you just insert the object, you have to serialize it
You are writing true values as True, but that is the python way and not the JSON way. You either have to write it as true, or you have to pass it as a string. I would recommend the first one.
It could look like this (but I didnt parse datetime right, I just stringified it):
[{"tradable": true, "mode": "full", "instrument_token": 4708097, "last_price": 178.65, "last_traded_quantity": 5, "average_traded_price": 180.1, "volume_traded": 4581928, "total_buy_quantity": 1282853, "total_sell_quantity": 1673842, "ohlc": {"open": 181.95, "high": 181.95, "low": 177.8, "close": 181.0}, "change": -1.2983425414364609, "last_trade_time": "datetime.datetime(2023, 1, 12, 13, 4, 58)", "oi": 0, "oi_day_high": 0, "oi_day_low": 0, "exchange_timestamp": "datetime.datetime(2023, 1, 12, 13, 5, 1)", "depth": {"buy": [{"quantity": 653, "price": 178.6, "orders": 8}, {"quantity": 2408, "price": 178.55, "orders": 15}, {"quantity": 6329, "price": 178.5, "orders": 22}, {"quantity": 9161, "price": 178.45, "orders": 24}, {"quantity": 7775, "price": 178.4, "orders": 17}], "sell": [{"quantity": 5726, "price": 178.7, "orders": 8}, {"quantity": 4099, "price": 178.75, "orders": 11}, {"quantity": 23951, "price": 178.8, "orders": 25}, {"quantity": 7446, "price": 178.85, "orders": 21}, {"quantity": 11379, "price": 178.9, "orders": 21}]}}, {"tradable": true, "mode": "full", "instrument_token": 871681, "last_price": 972.55, "last_traded_quantity": 1, "average_traded_price": 973.85, "volume_traded": 411290, "total_buy_quantity": 152925, "total_sell_quantity": 214765, "ohlc": {"open": 971.75, "high": 978.6, "low": 969.0, "close": 967.75}, "change": 0.4959958667011061, "last_trade_time": "datetime.datetime(2023, 1, 12, 13, 4, 53)", "oi": 0, "oi_day_high": 0, "oi_day_low": 0, "exchange_timestamp": "datetime.datetime(2023, 1, 12, 13, 5, 4)", "depth": {"buy": [{"quantity": 6, "price": 972.15, "orders": 2}, {"quantity": 3, "price": 972.1, "orders": 2}, {"quantity": 15, "price": 972.05, "orders": 3}, {"quantity": 455, "price": 972.0, "orders": 16}, {"quantity": 14, "price": 971.95, "orders": 2}], "sell": [{"quantity": 6, "price": 972.5, "orders": 3}, {"quantity": 49, "price": 972.55, "orders": 2}, {"quantity": 10, "price": 972.6, "orders": 1}, {"quantity": 27, "price": 972.65, "orders": 2}, {"quantity": 10, "price": 972.7, "orders": 1}]}}]
main_dict = {
'NSE:ACC': {'average_price': 0,
'buy_quantity': 0,
'depth': {'buy': [{'orders': 0, 'price': 0, 'quantity': 0},
{'orders': 0, 'price': 0, 'quantity': 0},
{'orders': 0, 'price': 0, 'quantity': 0},
{'orders': 0, 'price': 0, 'quantity': 0},
{'orders': 0, 'price': 0, 'quantity': 0}],
'sell': [{'orders': 0, 'price': 0, 'quantity': 0},
{'orders': 0, 'price': 0, 'quantity': 0},
{'orders': 0, 'price': 0, 'quantity': 0},
{'orders': 0, 'price': 0, 'quantity': 0},
{'orders': 0, 'price': 0, 'quantity': 0}]},
'instrument_token': 5633,
'last_price': 2488.9,
'last_quantity': 0,
'last_trade_time': '2022-09-23 15:59:10',
'lower_circuit_limit': 2240.05,
'net_change': 0,
'ohlc': {'close': 2555.7,
'high': 2585.5,
'low': 2472.2,
'open': 2575},
'oi': 0,
'oi_day_high': 0,
'oi_day_low': 0,
'sell_quantity': 0,
'timestamp': '2022-09-23 18:55:17',
'upper_circuit_limit': 2737.75,
'volume': 0},
}
convert dict to pandas dataframe
for example:
symbol last_price net_change Open High Low Close
NSE:ACC 2488.9 0 2575 2585.5 2472.2 2555.7
I am trying pd.DataFrame.from_dict(main_dict)
but it does not work.
please give the best suggestion.
I would first select the necessary data from your dict and then pass that as input to pd.DataFrame()
df_input = [{
"symbol": symbol,
"last_price": main_dict.get(symbol).get("last_price"),
"net_change": main_dict.get(symbol).get("net_change"),
"open": main_dict.get(symbol).get("ohlc").get("open"),
"high": main_dict.get(symbol).get("ohlc").get("high"),
"low": main_dict.get(symbol).get("ohlc").get("low"),
"close": main_dict.get(symbol).get("ohlc").get("close")
} for symbol in main_dict]
import pandas as pd
df = pd.DataFrame(df_input)
disks={'1':1,'2':2,'3':3,'4':4,'5':5}
mem={'1':11,'2':12,'3':13,'4':14,'5':15}
cpu={'1':21,'2':22,'3':23,'4':24,'5':25}
Stats={}
Values={}
for i in range(1,len(disks)+1):
Index="value"+str(i)
total = disks['%s'%i]
memory = mem['%s'%i]
cpus= cpu['%s'%i]
Values["cpu"]=cpus
Values["total"]=total
Values["memory"]=memory
print(Index) # shows right value
print(Values) # shows right value
Stats[Index]=Values
print(Stats)
print (Stats)
Individual values are printed fine, however the final disctioanary print keys properly as value1, value2 -- however the values are all for the last value as we can see here:
value1
{'cpu': 21, 'total': 1, 'memory': 11}
{'value1': {'cpu': 21, 'total': 1, 'memory': 11}}
value2
{'cpu': 22, 'total': 2, 'memory': 12}
{'value1': {'cpu': 22, 'total': 2, 'memory': 12}, 'value2': {'cpu': 22, 'total': 2, 'memory': 12}}
value3
{'cpu': 23, 'total': 3, 'memory': 13}
{'value1': {'cpu': 23, 'total': 3, 'memory': 13}, 'value2': {'cpu': 23, 'total': 3, 'memory': 13}, 'value3': {'cpu': 23, 'total': 3, 'memory': 13}}
value4
{'cpu': 24, 'total': 4, 'memory': 14}
{'value1': {'cpu': 24, 'total': 4, 'memory': 14}, 'value2': {'cpu': 24, 'total': 4, 'memory': 14}, 'value3': {'cpu': 24, 'total': 4, 'memory': 14}, 'value4': {'cpu': 24, 'total': 4, 'memory': 14}}
value5
{'cpu': 25, 'total': 5, 'memory': 15}
{'value1': {'cpu': 25, 'total': 5, 'memory': 15}, 'value2': {'cpu': 25, 'total': 5, 'memory': 15}, 'value3': {'cpu': 25, 'total': 5, 'memory': 15}, 'value4': {'cpu': 25, 'total': 5, 'memory': 15}, 'value5': {'cpu': 25, 'total': 5, 'memory': 15}}
FINAL value:
{'value1': {'cpu': 25, 'total': 5, 'memory': 15}, 'value2': {'cpu': 25, 'total': 5, 'memory': 15}, 'value3': {'cpu': 25, 'total': 5, 'memory': 15}, 'value4': {'cpu': 25, 'total': 5, 'memory': 15}, 'value5': {'cpu': 25, 'total': 5, 'memory': 15}}
Something I am missing which somehow I am not able to think/figure out. Quick tips welcome
Your program is almost correct, just move Values={} inside for-loop (to make a new dictionary and not use old one):
disks={'1':1,'2':2,'3':3,'4':4,'5':5}
mem={'1':11,'2':12,'3':13,'4':14,'5':15}
cpu={'1':21,'2':22,'3':23,'4':24,'5':25}
Stats={}
for i in range(1,len(disks)+1):
Values={} # <-- move Values here
Index="value"+str(i)
total = disks['%s'%i]
memory = mem['%s'%i]
cpus= cpu['%s'%i]
Values["cpu"]=cpus
Values["total"]=total
Values["memory"]=memory
# print(Index) # shows right value
# print(Values) # shows right value
Stats[Index]=Values
# print(Stats)
print (Stats)
Prints:
{'value1': {'cpu': 21, 'total': 1, 'memory': 11}, 'value2': {'cpu': 22, 'total': 2, 'memory': 12}, 'value3': {'cpu': 23, 'total': 3, 'memory': 13}, 'value4': {'cpu': 24, 'total': 4, 'memory': 14}, 'value5': {'cpu': 25, 'total': 5, 'memory': 15}}
I have below list with nested lists (sort of key,values)
inp1=[{'id': 0, 'name': 98, 'value': 9}, {'id': 1, 'name': 66, 'value': 8}, {'id': 2, 'name': 29, 'value': 5}, {'id': 3, 'name': 99, 'value': 3}, {'id': 4, 'name': 15, 'value': 9}]
Am trying to replace 'name' with 'wid' and 'value' with 'wrt', how can I do it on same list?
My output should be like
inp1=[{'id': 0, 'wid': 98, 'wrt': 9}, {'id': 1, 'wid': 66, 'wrt': 8}, {'id': 2, 'wid': 29, 'wrt': 5}, {'id': 3, 'wid': 99, 'wrt': 3}, {'id': 4, 'wid': 15, 'wrt': 9}]
I tried below, but it doesn't work as list cannot be indexed with string but integer
inp1['name'] = inp1['wid']
inp1['value'] = inp1['wrt']
I tried if I can find any examples, but mostly I found only this for dictionary and not list.
You need to iterate each item, and remove the old entry (dict.pop is handy for this - it removes an entry and return the value) and assign to new keyes:
>>> inp1 = [
... {'id': 0, 'name': 98, 'value': 9},
... {'id': 1, 'name': 66, 'value': 8},
... {'id': 2, 'name': 29, 'value': 5},
... {'id': 3, 'name': 99, 'value': 3},
... {'id': 4, 'name': 15, 'value': 9}
... ]
>>>
>>> for d in inp1:
... d['wid'] = d.pop('name')
... d['wrt'] = d.pop('value')
...
>>> inp1
[{'wid': 98, 'id': 0, 'wrt': 9},
{'wid': 66, 'id': 1, 'wrt': 8},
{'wid': 29, 'id': 2, 'wrt': 5},
{'wid': 99, 'id': 3, 'wrt': 3},
{'wid': 15, 'id': 4, 'wrt': 9}]
def f(item):
if(item.has_key('name') and not item.has_key('wid')):
item['wid']=item.pop('name')
if(item.has_key('value') and not item.has_key('wrt')):
item['wrt']=item.pop('value')
map(f,inp1)
Output:
[{'wrt': 9, 'wid': 98, 'id': 0}, {'wrt': 8, 'wid': 66, 'id': 1}, {'wrt': 5, 'wid': 29, 'id': 2}, {'wrt': 3, 'wid': 99, 'id': 3}, {'wrt': 9, 'wid': 15, 'id': 4}]
I have a list of id's sorted in a proper oder:
ids = [1, 2, 4, 6, 5, 0, 3]
I also have a list of dictionaries, sorted in some random way:
rez = [{'val': 7, 'id': 1}, {'val': 8, 'id': 2}, {'val': 2, 'id': 3}, {'val': 0, 'id': 4}, {'val': -1, 'id': 5}, {'val': -4, 'id': 6}, {'val': 9, 'id': 0}]
My intention is to sort rez list in a way that corresponds to ids:
rez = [{'val': 7, 'id': 1}, {'val': 8, 'id': 2}, {'val': 0, 'id': 4}, {'val': -4, 'id': 6}, {'val': -1, 'id': 5}, {'val': 9, 'id': 0}, {'val': 2, 'id': 3}]
I tried:
rez.sort(key = lambda x: ids.index(x['id']))
However that way is too slow for me, as len(ids) > 150K, and each dict actually had a lot of keys (some values there are strings). Any suggestion how to do it in the most pythonic, but still fastest way?
You don't need to sort because ids specifies the entire ordering of the result. You just need to pick the correct elements by their ids:
rez_dict = {d['id']:d for d in rez}
rez_ordered = [rez_dict[id] for id in ids]
Which gives:
>>> rez_ordered
[{'id': 1, 'val': 7}, {'id': 2, 'val': 8}, {'id': 4, 'val': 0}, {'id': 6, 'val': -4}, {'id': 5, 'val': -1}, {'id': 0, 'val': 9}, {'id': 3, 'val': 2}]
This should be faster than sorting because it can be done in linear time on average, while sort is O(nlogn).
Note that this assumes that there will be one entry per id, as in your example.
I think you are on the right track. If you need to speed it up, because your list is too long and you are having quadratic complexity, you can turn the list into a dictionary first, mapping the ids to their respective indices.
indices = {id_: pos for pos, id_ in enumerate(ids)}
rez.sort(key = lambda x: indices[x['id']])
This way, indices is {0: 5, 1: 0, 2: 1, 3: 6, 4: 2, 5: 4, 6: 3}, and rez is
[{'id': 1, 'val': 7},
{'id': 2, 'val': 8},
{'id': 4, 'val': 0},
{'id': 6, 'val': -4},
{'id': 5, 'val': -1},
{'id': 0, 'val': 9},
{'id': 3, 'val': 2}]