I know how I can save kwargs in a .txt file.
with open(filename, "w") as file:
for key, value in kwargs.items():
file.write("{}: {}\n".format(key, value))
Let's says kwargs = {'page': 1, 'name': 'xyz', 'title': 'xyz'}
But how can I read the file and save the data back to kwargs how it was before it got saved to the file so that the result is kwargs = {'page': 1, 'name': 'xyz', 'title': 'xyz'} after reading the file?
if you're okay with saving it in a slightly different format, you can use the in-built eval function.
Writing your dictionary to a file:
with open(filename, "w") as file:
file.write(str(kwargs))
Reading your dictionary back from the file:
with open(filename, "r") as file:
kwargs = eval(file.read())
Assuming you aren't concerned about the ambiguities, this would be the inverse of what you presented in your question:
with open(filename, "r") as file:
kwargs = {}
for ln in file:
key, _, value = ln.partition(': ')
if value.isdigit():
value = int(value)
kwargs[key] = value
so here is the simple way to go about that, and before we get to the talk about ambiguity with that its simply to show how you could extract what you save in the file. it will be a key/value pair with key and pair are both strings. to cast the value part into the correct type isn't the scope of the solution
the test.txt content:
foo: bar
bas: buz
the code example:
result = {}
with open("/home/mr/test.txt", "r+") as f:
for line in f.readlines():
k, v = line.replace('\n','').split(':')
result.update({k.strip(): v.strip()})
the content of result dict:
{'foo': 'bar', 'bas': 'buz'}
Im writing a default dict (list) to a file with :
def write_to_file(slack_dict):
if os.path.exists("slack_dict"):
print ("slack_dict file already exist, skipping writing")
else:
print ("writing slack_dict to file")
F = open('slack_dict', "w")
F.write(str(slack_dict).encode('UTF-8'))
F.close()
the slack_dict being written to file is :
defaultdict(<type 'list'>, {u'djin_acontentmgmt_sample': [{'color': u'blue',
'status': 'SUCCESS', 'region': u'virginia', 'env': u'int', 'job_no': '122'}]})
How do I remove the unicode strings ? I already have tried F.write(str(slack_dict).encode('UTF-8'))
If I do the following, there are no non-ASCII characters in the 'slack_dict' file, since there are no non-ASCII characters in any of the 'Unicode' strings in the dictionary. Everything is stored exactly as shown (e.g., a literal 'u', not a non-ASCII character).
from collections import defaultdict
slack_dict = defaultdict(list)
slack_dict[u'djin_acontentmgmt_sample'] = [
{'color': u'blue', 'status': 'SUCCESS', 'region': u'virginia', 'env': u'int', 'job_no': '122'}
]
with open('slack_dict', 'w') as f:
f.write(str(slack_dict).encode('UTF-8'))
with open('slack_dict', 'r') as f:
print f.read()
# defaultdict(<type 'list'>, {u'djin_acontentmgmt_sample': [{'color': u'blue', 'status': 'SUCCESS', 'region': u'virginia', 'env': u'int', 'job_no': '122'}]})
Are you trying to eliminate the 'u' markers, e.g., to write json text? In that case, you could try this:
import json
with open('slack_dict', 'w') as f:
json.dump(slack_dict, f)
# equivalent to f.write(json.dumps(slack_dict))
with open('slack_dict', 'r') as f:
print f.read()
# {"djin_acontentmgmt_sample": [{"color": "blue", "status": "SUCCESS", "region": "virginia", "env": "int", "job_no": "122"}]}
I want to convert a file to json.
The file has the format:
Temp=24.0* Humidity=41.0% Date=02/01/17-20:37
Temp=24.0* Humidity=42.0% Date=02/01/17-20:38
Temp=24.0* Humidity=42.0% Date=02/01/17-20:39
I'm using the following code:
list = {}
with open("record.txt") as f:
for line in f:
if not ("Failed" in line):
lists = line.split(" ")
for l in lists:
ll = dict([l.split("=")])
// print(json.dumps(ll))
list.update(ll)
And when I print the dictionary created i get.
>>> print (list)
{'Temp': '29.0*', 'Humidity': '31.0%', 'Date': '15/07/17-10:56\n', '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00Temp': '21.0*'}
I don't understand why. Does anyone knows why i don't get the entire dictionary?
Before converting to json i will also separate to sub-dictionary's by new line /n. Is this possible?
Your input file contains the same set of unique keys three times so a corresponding representation in Python, which can then be serialised to JSON, is an array of dictionaries.
Try this change to your code:
list = []
with open("record.txt") as f:
for line in f:
if not ("Failed" in line):
lists = line.rstrip().split(" ")
ll = {}
for l in lists:
k,v = l.split("=")
ll[k] = v
list.append(ll)
If you then do:
print list
You should get:
print list
[{'Date': '02/01/17-20:37', 'Temp': '24.0*', 'Humidity': '41.0%'}, {'Date': '02/01/17-20:38', 'Temp': '24.0*', 'Humidity': '42.0%'}, {'Date': '02/01/17-20:39', 'Temp': '24.0*', 'Humidity': '42.0%'}]
You can then dump to JSON with:
import json
json.dumps(list)
'[{"Date": "02/01/17-20:37", "Temp": "24.0*", "Humidity": "41.0%"}, {"Date": "02/01/17-20:38", "Temp": "24.0*", "Humidity": "42.0%"}, {"Date": "02/01/17-20:39", "Temp": "24.0*", "Humidity": "42.0%"}]'
You can create a list of dictionaries like this and then can do whatever modification you want to do.
list = {}
complete_list=[]
with open("record.txt") as f:
for line in f:
if not ("Failed" in line):
lists = line.split(" ")
lists[-1] = lists[-1].strip() #it should remove the \n at the end.
for l in lists:
ll = dict([l.split("=")])
// print(json.dumps(ll))
list.update(ll)
complete_list.append(list)
The code below defines a dictionary used to transform field values. Data is read, some of the values are transformed based on this dictionary, and written to a table. It works as-is. The problem, I now want to move this configuration outside the .py file into a JSON configuration file.
lookups = {
11: {
"ST1": ["ABC"],
"UNK01": ["125", "ACD"],
"A": ["52"],
"B": ["91"],
"C": ["92"],
"D": ["95"]
},
10: {
"XYZ01": ["91"],
"XYZ02": ["83"],
"XYZ03": ["27"]
}
}
According to jsonlint.com, in order for the above value being assigned to lookups to be valid JSON, I must quote the 11 and 10 keys. Doing so breaks my Python code and displays TypeError: list indices must be integers, not str.
How do I create valid JSON and minimize changes to my code?
If you want to dump it to a json file:
import json
with open("config.json","w") as f:
json.dump(lookups, f) # dump dict to file
with open("config.json") as f:
s = json.load(f) # load dict from file
print(s)
{'11': {'ST1': ['ABC'], 'A': ['52'], 'D': ['95'], 'UNK01': ['125', 'ACD'], 'B': ['91'], 'C': ['92']}, '10': {'XYZ01': ['91'], 'XYZ03': ['27'], 'XYZ02': ['83']}}
If you need keys as ints you can loop and cast as ints or use pickle:
import pickle
with open("in.pkl","wb") as f:
pickle.dump(lookups, f)
with open("in.pkl","rb") as f:
s = pickle.load(f)
print(s)
{10: {'XYZ03': ['27'], 'XYZ01': ['91'], 'XYZ02': ['83']}, 11: {'UNK01': ['125', 'ACD'], 'B': ['91'], 'D': ['95'], 'ST1': ['ABC'], 'C': ['92'], 'A': ['52']}}
If not just use as is.
If you know what type of data your keys are, a simple int on the keys would suffice:
dictionary_from_json = json.loads(dumped)
newdict = {}
for key, val in dictionary_from_json:
newdict[int(key)] = val
You can extend json.decoder and convert all keys to int when it's possible.
import json
class Json(json.JSONDecoder):
def decode(self,json_string):
default_obj = super(Json,self).decode(json_string)
new_obj = self._rec_serial(default_obj)
return new_obj
def _rec_serial(self,default):
new_dict = {}
for key,value in default.items():
is_dict = isinstance(value,dict)
value = self._rec_serial(value) if is_dict else value
try:
new_dict[int(key)] = value
except ValueError:
new_dict[key] = value
return new_dict
json2= Json()
d = json2.decode(dumped)
Is there a way in Python to serialize a dictionary that is using a tuple as key?
e.g.
a = {(1, 2): 'a'}
simply using json.dumps(a) raises this error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.6/json/__init__.py", line 230, in dumps
return _default_encoder.encode(obj)
File "/usr/lib/python2.6/json/encoder.py", line 367, in encode
chunks = list(self.iterencode(o))
File "/usr/lib/python2.6/json/encoder.py", line 309, in _iterencode
for chunk in self._iterencode_dict(o, markers):
File "/usr/lib/python2.6/json/encoder.py", line 268, in _iterencode_dict
raise TypeError("key {0!r} is not a string".format(key))
TypeError: key (1, 2) is not a string
You can't serialize that as json, json has a much less flexible idea about what counts as a dict key than python.
You could transform the mapping into a sequence of key, value pairs, something like this:
import json
def remap_keys(mapping):
return [{'key':k, 'value': v} for k, v in mapping.iteritems()]
...
json.dumps(remap_keys({(1, 2): 'foo'}))
>>> '[{"value": "foo", "key": [1, 2]}]'
from json import loads, dumps
from ast import literal_eval
x = {(0, 1): 'la-la la', (0, 2): 'extricate'}
# save: convert each tuple key to a string before saving as json object
s = dumps({str(k): v for k, v in x.items()})
# load in two stages:
# (i) load json object
obj = loads(s)
# (ii) convert loaded keys from string back to tuple
d = {literal_eval(k): v for k, v in obj.items()}
See https://stackoverflow.com/a/12337657/2455413.
JSON only supports strings as keys. You'll need to choose a way to represent those tuples as strings.
You could just use str((1,2)) as key because json only expects the keys as strings but if you use this you'll have to use a[str((1,2))] to get the value.
json can only accept strings as keys for dict,
what you can do, is to replace the tuple keys with string like so
with open("file", "w") as f:
k = dic.keys()
v = dic.values()
k1 = [str(i) for i in k]
json.dump(json.dumps(dict(zip(*[k1,v]))),f)
And than when you want to read it, you can change the keys back to tuples using
with open("file", r) as f:
data = json.load(f)
dic = json.loads(data)
k = dic.keys()
v = dic.values()
k1 = [eval(i) for i in k]
return dict(zip(*[k1,v]))
This solution:
Avoids the security risk of eval().
Is short.
Is copy-pastable as save and load functions.
Keeps the structure of tuple as the key, in case you are editing the JSON by hand.
Adds ugly \" to the tuple representation, which is worse than the other str()/eval() methods here.
Can only handle tuples as keys at the first level for nested dicts (as of this writing no other solution here can do better)
def json_dumps_tuple_keys(mapping):
string_keys = {json.dumps(k): v for k, v in mapping.items()}
return json.dumps(string_keys)
def json_loads_tuple_keys(string):
mapping = json.loads(string)
return {tuple(json.loads(k)): v for k, v in mapping.items()}
m = {(0,"a"): "first", (1, "b"): [9, 8, 7]}
print(m) # {(0, 'a'): 'first', (1, 'b'): [9, 8, 7]}
s = json_dumps_tuple_keys(m)
print(s) # {"[0, \"a\"]": "first", "[1, \"b\"]": [9, 8, 7]}
m2 = json_loads_tuple_keys(s)
print(m2) # {(0, 'a'): 'first', (1, 'b'): [9, 8, 7]}
print(m==m2) # True
Here is one way to do it. It will require the key to be json decoded after the main dictionary is decoded and the whole dictionary re-sequenced, but it is doable:
import json
def jsonEncodeTupleKeyDict(data):
ndict = dict()
# creates new dictionary with the original tuple converted to json string
for key,value in data.iteritems():
nkey = json.dumps(key)
ndict[nkey] = value
# now encode the new dictionary and return that
return json.dumps(ndict)
def main():
tdict = dict()
for i in range(10):
key = (i,"data",5*i)
tdict[key] = i*i
try:
print json.dumps(tdict)
except TypeError,e:
print "JSON Encode Failed!",e
print jsonEncodeTupleKeyDict(tdict)
if __name__ == '__main__':
main()
I make no claim to any efficiency of this method. I needed this for saving some joystick mapping data to a file. I wanted to use something that would create a semi-human readable format so it could be edited if needed.
You can actually not serialize tuples as key to json, but you can convert the tuple to a string and recover it, after you have deserialized the file.
with_tuple = {(0.1, 0.1): 3.14} ## this will work in python but is not serializable in json
{(0.1, 0.1): 3.14}
But you cannot serialize it with json. However, you can use
with_string = {str((0.1, 0.1))[1:-1]: 3.14} ## the expression [1,-1] removes the parenthesis surrounding the tuples in python.
{'0.1, 0.1': 3.14} # This is serializable
With a bit of cheating, you will recover the original tuple (after having deserialized the whole file) by treating each key (as str) separately
tuple(json.loads("["+'0.1, 0.1'+"]")) ## will recover the tuple from string
(0.1, 0.1)
It is a bit of overload to convert a string to a tuple using json.loads, but it will work. Encapsulate it and you are done.
Peace out and happy coding!
Nicolas
Here are two functions you could use to convert a dict_having_tuple_as_key into a json_array_having_key_and_value_as_keys and then de-convert it the way back
import json
def json_dumps_dict_having_tuple_as_key(dict_having_tuple_as_key):
if not isinstance(dict_having_tuple_as_key, dict):
raise Exception('Error using json_dumps_dict_having_tuple_as_key: The input variable is not a dictionary.')
list_of_dicts_having_key_and_value_as_keys = [{'key': k, 'value': v} for k, v in dict_having_tuple_as_key.items()]
json_array_having_key_and_value_as_keys = json.dumps(list_of_dicts_having_key_and_value_as_keys)
return json_array_having_key_and_value_as_keys
def json_loads_dictionary_split_into_key_and_value_as_keys_and_underwent_json_dumps(json_array_having_key_and_value_as_keys):
list_of_dicts_having_key_and_value_as_keys = json.loads(json_array_having_key_and_value_as_keys)
if not all(['key' in diz for diz in list_of_dicts_having_key_and_value_as_keys]) and all(['value' in diz for diz in list_of_dicts_having_key_and_value_as_keys]):
raise Exception('Error using json_loads_dictionary_split_into_key_and_value_as_keys_and_underwent_json_dumps: at least one dictionary in list_of_dicts_having_key_and_value_as_keys ismissing key "key" or key "value".')
dict_having_tuple_as_key = {}
for dict_having_key_and_value_as_keys in list_of_dicts_having_key_and_value_as_keys:
dict_having_tuple_as_key[ tuple(dict_having_key_and_value_as_keys['key']) ] = dict_having_key_and_value_as_keys['value']
return dict_having_tuple_as_key
usage example:
my_dict = {
('1', '1001', '2021-12-21', '1', '484'): {"name": "Carl", "surname": "Black", "score": 0},
('1', '1001', '2021-12-22', '1', '485'): {"name": "Joe", "id_number": 134, "percentage": 11}
}
my_json = json_dumps_dict_having_tuple_as_key(my_dict)
print(my_json)
[{'key': ['1', '1001', '2021-12-21', '1', '484'], 'value': {'name': 'Carl', 'surname': 'Black', 'score': 0}},
{'key': ['1', '1001', '2021-12-22', '1', '485'], 'value': {'name': 'Joe', 'id_number': 134, 'percentage': 11}}]
my_dict_reconverted = json_loads_dictionary_split_into_key_and_value_as_keys_and_underwent_json_dumps(my_json)
print(my_dict_reconverted)
{('1', '1001', '2021-12-21', '1', '484'): {'name': 'Carl', 'surname': 'Black', 'score': 0},
('1', '1001', '2021-12-22', '1', '485'): {'name': 'Joe', 'id_number': 134, 'percentage': 11}}
# proof of working 1
my_dict == my_dict_reconverted
True
# proof of working 2
my_dict == json_loads_dictionary_split_into_key_and_value_as_keys_and_underwent_json_dumps(
json_dumps_dict_having_tuple_as_key(my_dict)
)
True
(Using concepts expressed by #SingleNegationElimination to answer #Kvothe comment)
Here's a complete example to encode/decode nested dictionaries with tuple keys and values into/from json. tuple key will be a string in JSON.
values of types tuple or set will be converted to list
def JSdecoded(item:dict, dict_key=False):
if isinstance(item, list):
return [ JSdecoded(e) for e in item ]
elif isinstance(item, dict):
return { literal_eval(key) : value for key, value in item.items() }
return item
def JSencoded(item, dict_key=False):
if isinstance(item, tuple):
if dict_key:
return str(item)
else:
return list(item)
elif isinstance(item, list):
return [JSencoded(e) for e in item]
elif isinstance(item, dict):
return { JSencoded(key, True) : JSencoded(value) for key, value in item.items() }
elif isinstance(item, set):
return list(item)
return item
usage
import json
pydata = [
{ ('Apple','Green') : "Tree",
('Orange','Yellow'):"Orchard",
('John Doe', 1945) : "New York" }
]
jsstr= json.dumps(JSencoded(pydata), indent='\t')
print(jsstr)
#[
# {
# "('Apple', 'Green')": "Tree",
# "('Orange', 'Yellow')": "Orchard",
# "('John Doe', 1945)": "New York"
# }
#]
data = json.loads(jsstr) #string keys
newdata = JSdecoded(data) #tuple keys
print(newdata)
#[{('Apple', 'Green'): 'Tree', ('Orange', 'Yellow'): 'Orchard', ('John Doe', 1945): 'New York'}]
def stringify_keys(d):
if isinstance(d, dict):
return {str(k): stringify_keys(v) for k, v in d.items()}
if isinstance(d, (list, tuple)):
return type(d)(stringify_keys(v) for v in d)
return d
json.dumps(stringify_keys(mydict))