Develop Reference

constructing a message format from the fetchall result in python

*New to Programming
Question: I need to use the below "Data" (two rows as arrays) queried from sql and use it to create the message structure below.
data from sql using fetchall()
Data = [[100,1,4,5],[101,1,4,6]]
##expected message structure
message = {
"name":"Tom",
"Job":"IT",
"info": [
{
"id_1":"100",
"id_2":"1",
"id_3":"4",
"id_4":"5"
},
{
"id_1":"101",
"id_2":"1",
"id_3":"4",
"id_4":"6"
},
]
}
I tried to create below method to iterate over the rows and then input the values, this is was just a starting, but this was also not working
def create_message(data)
for row in data:
{
"id_1":str(data[0][0],
"id_2":str(data[0][1],
"id_3":str(data[0][2],
"id_4":str(data[0][3],
}
Latest Code
def create_info(data):
info = []
for row in data:
temp_dict = {"id_1_tom":"","id_2_hell":"","id_3_trip":"","id_4_clap":""}
for i in range(0,1):
temp_dict["id_1_tom"] = str(row[i])
temp_dict["id_2_hell"] = str(row[i+1])
temp_dict["id_3_trip"] = str(row[i+2])
temp_dict["id_4_clap"] = str(row[i+3])
info.append(temp_dict)
return info

Edit: Updated answer based on updates to the question and comment by original poster.
This function might work for the example you've given to get the desired output, based on the attempt you've provided:
def create_info(data):
info = []
for row in data:
temp_dict = {}
temp_dict['id_1_tom'] = str(row[0])
temp_dict['id_2_hell'] = str(row[1])
temp_dict['id_3_trip'] = str(row[2])
temp_dict['id_4_clap'] = str(row[3])
info.append(temp_dict)
return info
For the input:
[[100, 1, 4, 5],[101,1,4,6]]
This function will return a list of dictionaries:
[{"id_1_tom":"100","id_2_hell":"1","id_3_trip":"4","id_4_clap":"5"},
{"id_1_tom":"101","id_2_hell":"1","id_3_trip":"4","id_4_clap":"6"}]
This can serve as the value for the key info in your dictionary message. Note that you would still have to construct the message dictionary.

DataFrame to JSON giving absurd output

I have this code
eventSummary = {}
eventSummary['jobClassID'] = data['jobClassID'].split()
eventSummary['modelID'] =data['modelID'].split()
eventSummary['startDate'] = pd.to_datetime(start).date().strftime("%Y-%m-%d").split()
eventSummary['hourlySummary']=merged_df.to_json(orient = 'records')
dataF = pd.DataFrame(eventSummary)
dataF.to_json(path_or_buf = '*\merged2.json', orient="records")
which is giving me json like this :
{
"jobClassID":"NorthernRegion|FarNorthDistrict|GeneralDuties|Constable|FIR|Senior|Lead|5",
"modelID":"4",
"startDate":
"2019-01-01",
"hourlySummary":
"[{\"Day\":1,\"Hour\":0,\"Month\":1,\"priority\":3,\"Count\":3,\"Hourly Season\":1.5589041096,\"Weekly Season\":1.9205974843,\"Monthly Season\":2.0497311828,\"Noise Factor\":0.6517894485,\"groupId\":0,\"K (Service)\":0.6955185685,\"Theta (Service)\":10924.8330481261,\"K (Travel)\":0.6819906983,\"Theta (Travel)\":5694.9711557023}]
}
But I want output to be like this:
{
"jobClassID":["NorthernRegion|FarNorthDistrict|GeneralDuties|Constable|FIR|Senior|Lead|5"],
"modelID":["4"],
"startDate":["2019-01-01"],
"hourlySummary":[
{"Day":1,"Hour":0,"Month":1,"priority":3,"Count":3,"Hourly Season":1.5589041096,"Weekly Season":1.9205974843,"Monthly Season":2.0497311828,"Noise Factor":0.6517894485,"groupId":0,"K (Service)":0.6955185685,"Theta (Service)":10924.8330481261,"K (Travel)":0.6819906983,"Theta (Travel)":5694.9711557023}]
}
How to achieve this?
Thanks in advance.

to_json returns a string, not a dictionary which is what you want. You should use to_dictmethod instead.

eventSummary['jobClassID'] = [data['jobClassID'].split()]
eventSummary['modelID'] = [data['modelID'].split()]
eventSummary['startDate'] = [pd.to_datetime(start).date().strftime("%Y-%m-
%d").split()]
eventSummary['hourlySummary']= [merged_df.to_dict(orient='records')]
dataF = pd.DataFrame(eventSummary)
dataF.to_json(path_or_buf = '*\merged12.json', orient="records", lines = True)
This code is giving results as expected.
Thanks for your efforts, everyone.

Getting certain information from string

I'm new to python as was wondering how I could get the estimatedWait and routeName from this string.
{
"lastUpdated": "07:52",
"filterOut": [],
"arrivals": [
{
"routeId": "B16",
"routeName": "B16",
"destination": "Kidbrooke",
"estimatedWait": "due",
"scheduledTime": "06: 53",
"isRealTime": true,
"isCancelled": false
},
{
"routeId":"B13",
"routeName":"B13",
"destination":"New Eltham",
"estimatedWait":"29 min",
"scheduledTime":"07:38",
"isRealTime":true,
"isCancelled":false
}
],
"serviceDisruptions":{
"infoMessages":[],
"importantMessages":[],
"criticalMessages":[]
}
}
And then save this to another string which would be displayed on the lxterminal of the raspberry pi 2. I would like only the 'routeName' of B16 to be saved to the string. How do I do that?

You just have to deserialise the object and then use the index to access the data you want.
To find only the B16 entries you can filter the arrivals list.
import json
obj = json.loads(json_string)
# filter only the b16 objects
b16_objs = filter(lambda a: a['routeName'] == 'B16', obj['arrivals'])
if b16_objs:
# get the first item
b16 = b16_objs[0]
my_estimatedWait = b16['estimatedWait']
print(my_estimatedWait)

You can use string.find() to get the indices of those value identifiers
and extract them.
Example:
def get_vaules(string):
waitIndice = string.find('"estimatedWait":"')
routeIndice = string.find('"routeName":"')
estimatedWait = string[waitIndice:string.find('"', waitIndice)]
routeName = string[routeIndice:string.find('"', routeIndice)]
return estimatedWait, routeName
Or you could just deserialize the json object (highly recommended)
import json
def get_values(string):
jsonData = json.loads(string)
estimatedWait = jsonData['arrivals'][0]['estimatedWait']
routeName = jsonData['arrivals'][0]['routeName']
return estimatedWait, routeName
Parsing values from a JSON file using Python?

Python utility for parsing blocks?

I have a file that starts something like:
databaseCons = {
main = {
database = "readable_name",
hostname = "hostdb1.serv.com",
instances= {
slaves = {
conns = "8"
}
}
maxconns = "5",
user = "user",
pass = "pass"
}
}
So, what I'd like to do is parse this out into a dict of sub-dicts, something like:
{'main': {'database': 'readable_name', 'hostname': 'hostdb1.serv.com', 'maxconns': '5', 'instances': {'slave': {'maxCount': '8'}}, 'user': 'user', 'pass': 'pass'}}
I think the above makes sense... but please feel free to edit this if it doesn't. Basically I want the equivalent of:
conns = '8'
slave = dict()
slave['maxCount'] = conns
instances = dict()
instances['slave'] = slave
database = 'readable_name'
hostname = 'hostdb1.serv.com'
maxconns = '5'
user = 'user'
pas = 'pass'
main = dict()
main['database'] = database
main['hostname'] = hostname
main['instances'] = instances
main['maxconns'] = maxconns
main['user'] = user
main['pass'] = pas
databaseCons = dict()
databaseCons['main'] = main
Are there any modules out there that can handle this sort of parsing? Even what I've suggested above looks messy.. there's got to be a better way I'd imagine.

Here is a pyparsing parser for your config file:
from pyparsing import *
def to_dict(t):
return {k:v for k,v in t}
series = Forward()
struct = Suppress('{') + series + Suppress('}')
value = quotedString.setParseAction(removeQuotes) | struct
token = Word(alphanums)
assignment = Group(token + Suppress('=') + value + Suppress(Optional(",")))
series << ZeroOrMore(assignment).setParseAction(to_dict)
language = series + stringEnd
def config_file_to_dict(filename):
return language.parseFile(filename)[0]
if __name__=="__main__":
from pprint import pprint
pprint(config_file_to_dict('config.txt'))

Python: Is there a better way to construct a dictionary?

There are multiple dimensions and metrics that I want to pull from Google Analytics via their API, and this requires dimension:metric(s) pairing. Naturally, I thought dictionary might be a good option for this. To minimize typing, and increase readability, this is what I tried.
dim_dict = {
0:'',
1:'ga:searchUsed',
2:'ga:searchKeyword',
3:'ga:pageTitle',
4:'ga:operatingSystem',
5:'ga:goalPreviousStep3'
}
metric_dict = {
1:'ga:sessions',
2:'ga:sessionDuration',
3:'bounceRate',
4:'pageviews',
5:'ga:searchSessions',
6:'ga:goalCompletionsAll'
}
dim_metric_dict = {
0:[1,2,3,4,5,6],
1:[1,2,3,4,5,6],
2:[1,2,3,4,5,6],
3:[1,2,3,4,5,6],
4:[1,2,3,4,5,6],
5:[6]
}
query_dict = {}
for dim_key in dim_dict.keys():
met = []
for metric_key in dim_metric_dict[dim_key]:
met.append(metric_dict[metric_key])
query_dict.update({dim_dict[dim_key]:met})
Then, I'm using the following code to make API requests:
def get_ga_kpi(start_date, end_date, dimensions='', sort='ga:pageviews'):
service = build('analytics', 'v3', http=http)
metrics = query_dict[dimensions]
if sort not in metrics:
sort = metrics[0]
data_query = service.data().ga().get(**{
'ids': 'ga:#######',
'metrics': '%s' % (','.join(metrics)),
'dimensions': '%s' % (dimensions),
'start_date':'%s' % (start_date),
'end_date':'%s' % (end_date),
'sort': '-%s' % (sort)
})
return feed['rows']
The code works as intended, but I'm wondering if there is a better way to approach this problem. Thanks for any input!

I don't know if this is much better, but you could use a dictionary comprehension and enumerate to avoid typing out all of the numbers.
For example, the dim_dict construction could become:
dim_list = ['','ga:searchUsed','ga:searchKeyword','ga:pageTitle','ga:operatingSystem','ga:goalPreviousStep3']
dim_dict = {key:value for (key,value) in enumerate(dim_list)}