Develop Reference

Convert dictionary of lists to dictionary of dictionary

Current output is of format, dictionary of lists
{
"majestic-service-1.324.02070909": [
"/home/robotics/arm-services/FeaturesDir.yaml",
"/home/robotics/arm-service/majestic-service.tar.gz"
],
}
and I'm looking to change that output format to something like below.(dictionary of dictionary)
{
"majestic-service-1.324.02070909": {
"yaml_file": "/home/robotics/arm-services/FeaturesDir.yaml",
"tar_file": "/home/robotics/arm-services/majestic-service-1.324.02070909.tar.gz",
"kind": "FeaturesDir"
}
}
Corresponding code snippet that I've tried,
output_dict = {}
for file in application_files:
match = re.match(regex_pattern, os.path.basename(file))
if match:
if os.path.exists(os.path.join(os.path.dirname(file), "FeaturesDir.yaml")):
output_dict[file_without_extension(match.string)] = {os.path.join(os.path.dirname(file), "FeaturesDir.yaml")}
output_dict[file_without_extension(match.string)].append(file)
output_dict["Kind"] = "FeaturesDir"
elif os.path.exists(os.path.join(os.path.dirname(file), "output_path/Deviations.yaml")):
output_dict[file_without_extension(match.string)] = {os.path.join(os.path.dirname(file), "output_path/Deviations")}
output_dict[file_without_extension(match.string)].append(file)
output_dict["Kind"] = "Deviations"
# where the function file_without_extension, will return - majestic-service-1.324.02070909 from majestic-service-1.324.02070909.tar.gz
It reports the following error
AttributeError: 'set' object has no attribute 'append'
What am I doing wrong ?

The error occurs because in this line output_dict[file_without_extension(match.string)] = {os.path.join(os.path.dirname(file), "output_path/Deviations")} you're using {} brackets which for python are intended as the data type called set. Sets don't have append attribute, they have add attribute instead but it's a completely different data type. That's why it shows you that kind of error!

constructing a message format from the fetchall result in python

*New to Programming
Question: I need to use the below "Data" (two rows as arrays) queried from sql and use it to create the message structure below.
data from sql using fetchall()
Data = [[100,1,4,5],[101,1,4,6]]
##expected message structure
message = {
"name":"Tom",
"Job":"IT",
"info": [
{
"id_1":"100",
"id_2":"1",
"id_3":"4",
"id_4":"5"
},
{
"id_1":"101",
"id_2":"1",
"id_3":"4",
"id_4":"6"
},
]
}
I tried to create below method to iterate over the rows and then input the values, this is was just a starting, but this was also not working
def create_message(data)
for row in data:
{
"id_1":str(data[0][0],
"id_2":str(data[0][1],
"id_3":str(data[0][2],
"id_4":str(data[0][3],
}
Latest Code
def create_info(data):
info = []
for row in data:
temp_dict = {"id_1_tom":"","id_2_hell":"","id_3_trip":"","id_4_clap":""}
for i in range(0,1):
temp_dict["id_1_tom"] = str(row[i])
temp_dict["id_2_hell"] = str(row[i+1])
temp_dict["id_3_trip"] = str(row[i+2])
temp_dict["id_4_clap"] = str(row[i+3])
info.append(temp_dict)
return info

Edit: Updated answer based on updates to the question and comment by original poster.
This function might work for the example you've given to get the desired output, based on the attempt you've provided:
def create_info(data):
info = []
for row in data:
temp_dict = {}
temp_dict['id_1_tom'] = str(row[0])
temp_dict['id_2_hell'] = str(row[1])
temp_dict['id_3_trip'] = str(row[2])
temp_dict['id_4_clap'] = str(row[3])
info.append(temp_dict)
return info
For the input:
[[100, 1, 4, 5],[101,1,4,6]]
This function will return a list of dictionaries:
[{"id_1_tom":"100","id_2_hell":"1","id_3_trip":"4","id_4_clap":"5"},
{"id_1_tom":"101","id_2_hell":"1","id_3_trip":"4","id_4_clap":"6"}]
This can serve as the value for the key info in your dictionary message. Note that you would still have to construct the message dictionary.

Why am I getting this error "TypeError: string indices must be integers" when trying to fetch data from an api?

json file =
{
"success": true,
"terms": "https://curr
"privacy": "https://cu
"timestamp": 162764598
"source": "USD",
"quotes": {
"USDIMP": 0.722761,
"USDINR": 74.398905,
"USDIQD": 1458.90221
}
}
The json file is above. i deleted lot of values from the json as it took too many spaces. My python code is in below.
import urllib.request, urllib.parse, urllib.error
import json
response = "http://api.currencylayer.com/live?access_key="
api_key = "42141e*********************"
parms = dict()
parms['key'] = api_key
url = response + urllib.parse.urlencode(parms)
mh = urllib.request.urlopen(url)
source = mh.read().decode()
data = json.loads(source)
pydata = json.dumps(data, indent=2)
print("which curreny do you want to convert USD to?")
xm = input('>')
print(f"Hoe many USD do you want to convert{xm}to")
value = input('>')
fetch = pydata["quotes"][0]["USD{xm}"]
answer = fetch*value
print(fetch)
--------------------------------
Here is the
output
"fetch = pydata["quotes"][0]["USD{xm}"]
TypeError: string indices must be integers"

First of all the JSON data you posted here is not valid. There are missing quotes and commas. For example here "terms": "https://curr. It has to be "terms": "https://curr",. The same at "privacy" and the "timestamp" is missing a comma. After i fixed the JSON data I found a solution. You have to use data not pydata. This mean you have to change fetch = pydata["quotes"][0]["USD{xm}"] to fetch = data["quotes"][0]["USD{xm}"]. But this would result in the next error, which would be a KeyError, because in the JSON data you provided us there is no array after the "qoutes" key. So you have to get rid of this [0] or the json data has to like this:
"quotes":[{
"USDIMP": 0.722761,
"USDINR": 74.398905,
"USDIQD": 1458.90221
}]
At the end you only have to change data["quotes"]["USD{xm}"] to data["quotes"]["USD"+xm] because python tries to find a key called USD{xm} and not for example USDIMP, when you type "IMP" in the input.I hope this fixed your problem.

Python use .json as config file with class instances

I'm creating an application that stores it's config in a dictionary. I know I can write this to a JSON file and read this every time the app starts. But the problem is that this dictionary also contains objects. Like so(LED is an imported module with the classes APALedstrip and Arduino)
rooms['livingroom'] = {
"room":data.room(name = 'livingroom',dataKeys = dataKeys),
"lights":{
"LedStrip":LED.APALedstrip(name = 'livingroom',
room = 'livingroom')
}
}
rooms['bed'] = {
"room":data.room(name = 'bed', dataKeys = dataKeys),
"lights":{
"LedStrip":LED.Arduino(name ='bed',
serialPort = 'ttyUSB0',
room = 'livingroom',
master = {'room':'livingroom', 'light':'LedStrip'},
roomSensors = 'livingroom')
}
}
I'm curious is it also possible to store this in an JSON file like so? And when it's imported into a dictionary that the objects are still created?

You need to serialize your objects. One way is to use "pickle".
Pickle convert an object to bytes, so the next step is to convert them to string using base64.
I choose base64 because it's safe for non-ASCII characters
In order to automatically save and retrieve the rooms use
save_rooms() and retrieve_rooms()
import codecs
import json
import pickle
def save_rooms(rooms):
for room in rooms:
# find all LedStrip objects
if 'lights' in rooms[room] and 'LedStrip' in rooms[room]['lights']:
lights = rooms[room]['lights']['LedStrip']
# encode object to bytes with pickle and then to string with base64
rooms[room]['lights']['LedStrip'] = codecs.encode(pickle.dumps(lights),
"base64").decode()
with open("rooms.json", "w") as f:
json.dump(rooms, f)
def retrieve_rooms():
with open("rooms.json") as f:
rooms = json.load(f)
for room in rooms:
# find all LedStrip objects
if 'lights' in rooms[room] and 'LedStrip' in rooms[room]['lights']:
lights = rooms[room]['lights']['LedStrip']
# decode from string to bytes with base64 and then from bytes to object with pickle
rooms[room]['lights']['LedStrip'] = pickle.loads(codecs.decode(lights.encode(), "base64"))
return rooms
rooms = {}
rooms['livingroom'] = {
"room": data.room(name='livingroom', dataKeys=dataKeys),
"lights": {
"LedStrip": LED.APALedstrip(name='livingroom',
room='livingroom')
}
}
rooms['bed'] = {
"room": data.room(name='bed', dataKeys=dataKeys),
"lights": {
"LedStrip": LED.Arduino(name='bed',
serialPort='ttyUSB0',
room='livingroom',
master={'room': 'livingroom', 'light': 'LedStrip'},
roomSensors='livingroom')
}
}
save_rooms(rooms)
loaded_rooms = retrieve_rooms()
In addition I implemented the logic so you can save any variation of rooms as long as you keep the structure the same.
ex.
rooms['kitchen'] = {
"room": data.room(name='kitchen', dataKeys=dataKeys),
"lights": {
"LedStrip": LED.APALedstrip(name='kitchen',
room='kitchen')
}
}

Remove specific JSON oobjects from a File and then store this file

This is just a part of my json file which looks like:
"network_lo": "127.0.0.0",
"ec2_block_device_mapping_root": "/dev/sda1",
"selinux": "false",
"uptime_seconds": 127412,
"ec2_reservation_id": "r-cd786568",
"sshdsakey": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"ec2_block_device_mapping_ami": "/dev/sda1",
"memorysize": "3.66 GB",
"swapsize": "0.00 kB",
"netmask": "255.255.255.192",
"uniqueid": "24wq0see",
"kernelmajversion": "3.2",
I have a Python scipt which download this file.. i want to parse this file and remove a number of objects like "swapsize","sshdsakey"
sqs = boto.sqs.connect_to_region("ap-west-1")
q = sqs.get_queue("deathvally")
m = q.read(visibility_timeout=15)
if m == None:
print "No message!"
else:
with open('download.json', 'w') as json_data:
print m.get_body()
json_data.write(m.get_body())
json_data.close()
# I want a logic here which can simply delete the specific json objects
# Something like this is what i tried but didn't work...
# clean_data = json.load(json_data)
# for element in clean_data: ##
# del element['sshdsakey']
# json_data.write(clean_data)
I basically need to parse the fetched json file and then remove the specific objects and then just write this new modified stuff in a file.

json.loads will decode JSON string into Python dictionary (Although format you provided is not a valid JSON format, there have to be curly braces on each side), then you can delete the needed keys with del , encode dictionary back to JSON string with json.dumps and write the resultit
clean_data = json.loads(json_data.read())
del clean_data[your_key]
with open(your_file_to_write, 'w') as f:
f.write(json.dumps(clean_data))

You can parse your json using loads from native json module.
Then delete an element from the dict using del
import json
keys_to_remove = ['sshdsakey', 'selinux']
json_str = '''{
"network_lo": "127.0.0.0",
"ec2_block_device_mapping_root": "/dev/sda1",
"selinux": "false",
"uptime_seconds": 127412,
"ec2_reservation_id": "r-cd786568",
"sshdsakey": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}'''
data = json.loads(json_str)
for key in keys_to_remove:
if key in data:
del data[key]
print data

You need to first convert the JSON object string into a Python dict, delete the keys from it, and then write to to the output file.
import json
sqs = boto.sqs.connect_to_region("ap-west-1")
q = sqs.get_queue("deathvally")
m = q.read(visibility_timeout=15)
if m is None:
print "No message!"
else:
KEYS_TO_REMOVE = "swapsize", "sshdsakey", "etc"
with open('download.json', 'w') as json_data:
json_obj = json.loads(m.get_body())
for key in KEYS_TO_REMOVE:
try:
del json_obj[key]
except KeyError:
pass
json_data.write(json.dumps(json_obj, indent=4))