Develop Reference

constructing a message format from the fetchall result in python

*New to Programming
Question: I need to use the below "Data" (two rows as arrays) queried from sql and use it to create the message structure below.
data from sql using fetchall()
Data = [[100,1,4,5],[101,1,4,6]]
##expected message structure
message = {
"name":"Tom",
"Job":"IT",
"info": [
{
"id_1":"100",
"id_2":"1",
"id_3":"4",
"id_4":"5"
},
{
"id_1":"101",
"id_2":"1",
"id_3":"4",
"id_4":"6"
},
]
}
I tried to create below method to iterate over the rows and then input the values, this is was just a starting, but this was also not working
def create_message(data)
for row in data:
{
"id_1":str(data[0][0],
"id_2":str(data[0][1],
"id_3":str(data[0][2],
"id_4":str(data[0][3],
}
Latest Code
def create_info(data):
info = []
for row in data:
temp_dict = {"id_1_tom":"","id_2_hell":"","id_3_trip":"","id_4_clap":""}
for i in range(0,1):
temp_dict["id_1_tom"] = str(row[i])
temp_dict["id_2_hell"] = str(row[i+1])
temp_dict["id_3_trip"] = str(row[i+2])
temp_dict["id_4_clap"] = str(row[i+3])
info.append(temp_dict)
return info

Edit: Updated answer based on updates to the question and comment by original poster.
This function might work for the example you've given to get the desired output, based on the attempt you've provided:
def create_info(data):
info = []
for row in data:
temp_dict = {}
temp_dict['id_1_tom'] = str(row[0])
temp_dict['id_2_hell'] = str(row[1])
temp_dict['id_3_trip'] = str(row[2])
temp_dict['id_4_clap'] = str(row[3])
info.append(temp_dict)
return info
For the input:
[[100, 1, 4, 5],[101,1,4,6]]
This function will return a list of dictionaries:
[{"id_1_tom":"100","id_2_hell":"1","id_3_trip":"4","id_4_clap":"5"},
{"id_1_tom":"101","id_2_hell":"1","id_3_trip":"4","id_4_clap":"6"}]
This can serve as the value for the key info in your dictionary message. Note that you would still have to construct the message dictionary.

How to make a nested dictionary based on a list of URLs?

I have this list of hierarchical URLs:
data = ["https://python-rq.org/","https://python-rq.org/a","https://python-rq.org/a/b","https://python-rq.org/c"]
And I want to dynamically make a nested dictionary for every URL for which there exists another URL that is a subdomain/subfolder of it.
I already tried the follwoing but it is not returning what I expect:
result = []
for key,d in enumerate(data):
form_dict = {}
r_pattern = re.search(r"(http(s)?://(.*?)/)(.*)",d)
r = r_pattern.group(4)
if r == "":
parent_url = r_pattern.group(3)
else:
parent_url = r_pattern.group(3) + "/"+r
print(parent_url)
temp_list = data.copy()
temp_list.pop(key)
form_dict["name"] = parent_url
form_dict["children"] = []
for t in temp_list:
child_dict = {}
if parent_url in t:
child_dict["name"] = t
form_dict["children"].append(child_dict.copy())
result.append(form_dict)
This is the expected output.
{
"name":"https://python-rq.org/",
"children":[
{
"name":"https://python-rq.org/a",
"children":[
{
"name":"https://python-rq.org/a/b",
"children":[
]
}
]
},
{
"name":"https://python-rq.org/c",
"children":[
]
}
]
}
Any advice?

This was a nice problem. I tried going on with your regex method but got stuck and found out that split was actually appropriate for this case. The following works:
data = ["https://python-rq.org/","https://python-rq.org/a","https://python-rq.org/a/b","https://python-rq.org/c"]
temp_list = data.copy()
# This removes the last "/" if any URL ends with one. It makes it a lot easier
# to match the URLs and is not necessary to have a correct link.
data = [x[:-1] if x[-1]=="/" else x for x in data]
print(data)
result = []
# To find a matching parent
def find_match(d, res):
for t in res:
if d == t["name"]:
return t
elif ( len(t["children"])>0 ):
temp = find_match(d, t["children"])
if (temp):
return temp
return None
while len(data) > 0:
d = data[0]
form_dict = {}
l = d.split("/")
# I removed regex as matching the last parentheses wasn't working out
# split does just what you need however
parent = "/".join(l[:-1])
data.pop(0)
form_dict["name"] = d
form_dict["children"] = []
option = find_match(parent, result)
if (option):
option["children"].append(form_dict)
else:
result.append(form_dict)
print(result)
[{'name': 'https://python-rq.org', 'children': [{'name': 'https://python-rq.org/a', 'children': [{'name': 'https://python-rq.org/a/b', 'children': []}]}, {'name': 'https://python-rq.org/c', 'children': []}]}]

CSV to elasticsearch with python SerializationError

When i try to send the bulk_data to the local elasticsearch, my data isn't loaded because of the SerializationError.
I already tried to fill the empty cells in the csv file, but that wasn't the solution.
from elasticsearch import Elasticsearch
bulk_data = []
header = []
count = 0
for row in csv_file_object:
if count > 0 :
data_dict = {}
for i in range(len(row)):
row = row.rstrip()
data_dict[header[i]] = row[i]
op_dict = {
"index": {
"_index": INDEX_NAME,
"_type": TYPE_NAME,
}
}
bulk_data.append(op_dict)
bulk_data.append(data_dict)
else:
header = row
count = count+1
# create ES client, create index
es = Elasticsearch(hosts = [ES_HOST])
if es.indices.exists(INDEX_NAME):
print("deleting '%s' index..." % (INDEX_NAME))
res = es.indices.delete(index = INDEX_NAME)
res = es.bulk(index = INDEX_NAME, body = bulk_data, refresh = True)
See image for the SerializationError and bulk_data values:
Please note: the \n is added by the serialization process itself.

I try to repond to you but I can't understand one thing. How you retrieve your field name from data? In your code I see that you retrieve it from a list called header that is empty? I can't understand how you take this value.. Check my answer i don't know if i understand well
from elasticsearch import Elasticsearch
from elasticsearch import helpers
index_name = "your_index_name"
doc_type = "your_doc_type"
esConnector = Elasticsearch(["http://192.168.1.1:9200/"])
# change your ip here
count = 0
def generate_data(csv_file_object)
with open(csv_file_object, "r") as f:
for line in f:
line = line.split(",").rstrip()
data_dict = {header[count]: line}
obj={
'_op_type': 'index',
'_index': index_name,
'_type': doc_type,
'_id': count+1,
'_source': data_dict
}
count +=1
yield obj
for success, info in helpers.parallel_bulk(client=esConnector, actions=generate_data(csv_file_object), thread_count=4):
if not success:
print 'Doc failed', info

Combine two list into a Dict, Tuple

I am creating a rest-api using the Python+Flask_Restful combo, find it amazing.
Currently, i am allowing user to run a sql-query using the browser and it works fine, but the problem is, headers information is not displayed in the response.
Here is the code that i am using:
class RunCustomeSQL(Resource):
def get(self, enter_db_name, query):
if not os.path.isfile(enter_db_name+'.db'):
raise BadRequest("Database Doesn't Exist. Please select a valid database")
conn = sqlite3.connect(enter_db_name+'.db')
search_out = []
cursor = conn.execute(query)
row = None
for row in cursor:
search_out.append(row)
if not row: #This means empty response
raise BadRequest("No Results Found")
conn.commit()
conn.close()
return search_out
While, this code works great, it doesn't print the header values in the json-response. The current response is :
[
[
"dusiri_bibi",
"11",
"----------",
" srt/None ",
"14.30 MB",
"2017-12-13 23:43:54",
"C:/Test_Software/vc_redist.x64.exe"
],
]
Expected Output :
[
[
"Machine Name" : "dusiri_bibi",
"LABEL" : "11",
"PERMISSIONS" : "----------",
"USER" : " srt/None ",
"SIZE" : "14.30 MB",
"CREATED" : "2017-12-13 23:43:54",
"FILENAME" : "C:/Test_Software/vc_redist.x64.exe"
],
]
All the above text such as "machine name, label etc." are my table headers, I am not sure how to print them along with my output.
What if the user runs select user, size from table_name only
or
What if the user runs select * from table_name
In both scenario's, the output should display the table headers
Thanks
UPDATE #1 (25 April) : I managed to answer my first question and able to display a proper json response if the user selects the SELECT * statement in SQL but still facing issue with the second piece of it
Here is the answer to first part if anyone is looking for it : Using Regex
row = None
if re.search(r'(?<=SELECT)(.*)(?=FROM)',query, re.IGNORECASE).group(1) == ' * ':
for row in cursor:
search_out.append({'NAME' : row[0], 'LABEL_NUMBER' : row[1], 'PERM' : row[2], 'USER' : row[3] , 'SIZE' : row[4], 'DATE' : row[5], 'FILENAME' : row[6]})
if not row: #This means empty response
raise BadRequest("No Results Found")
Part II : Unanswered Query:
For the second piece, i now have two list :
list_1 : [[u'LABEL_NUMBER', u'PERM', u'FILENAME']]
list_2 : [(u'11', u'----------', u'C:/Test_Software/26.avi'), (u'11', u'----------', u'C:/Test_Software/6.avi'), (u'11', u'-rwx------', u'C:/Test_Software/Debug/Current_Frame1.avi'), (u'10', u'-rwxrwx---', u'C:/Windows/WinSxS/boxed-split.avi')]
As you can see, i have two list and i want to combine them into a dict to show the response like this:
[
{
LABEL_NUMBER : '11' ,
PERM : '-----------',
FILENAME : 'C:/Test_Software/26.avi'
},
...
....
......
{
LABEL_NUMBER : '10' ,
PERM : '-rwxrwx---',
FILENAME : 'C:/Windows/WinSxS/boxed-split.avi'
},
]
i am using the following code to do the same :
chunks = [list_2[idx:idx+3] for idx in range(0, len(list_2), 3)]
output = []
for each in chunks:
output.append(dict(zip(list_1, each)))
print(output)
But, this is failing with "TypeError: unhashable type: 'list'", i understand that lists are mutable and which is why i am getting this error but then how can i get the desired dict response? what am i doing wrong here?

You can use a list comprehension combined with zip for this:
list_1 = [[u'LABEL_NUMBER', u'PERM', u'FILENAME']]
list_2 = [(u'11', u'----------', u'C:/Test_Software/26.avi'), (u'11', u'----------', u'C:/Test_Software/6.avi'), (u'11', u'-rwx------', u'C:/Test_Software/Debug/Current_Frame1.avi'), (u'10', u'-rwxrwx---', u'C:/Windows/WinSxS/boxed-split.avi')]
d = [dict(zip(list_1[0], i)) for i in list_2]
Result:
[{'FILENAME': 'C:/Test_Software/26.avi',
'LABEL_NUMBER': '11',
'PERM': '----------'},
{'FILENAME': 'C:/Test_Software/6.avi',
'LABEL_NUMBER': '11',
'PERM': '----------'},
{'FILENAME': 'C:/Test_Software/Debug/Current_Frame1.avi',
'LABEL_NUMBER': '11',
'PERM': '-rwx------'},
{'FILENAME': 'C:/Windows/WinSxS/boxed-split.avi',
'LABEL_NUMBER': '10',
'PERM': '-rwxrwx---'}]

How can I put excel data to the dictionary?

I wanna put excel data to the dictionary.
Excel is
views.py is
#coding:utf-8
from django.shortcuts import render
import xlrd
book3 = xlrd.open_workbook('./data/excel.xlsx')
sheet3 = book3.sheet_by_index(0)
large_item = None
data_dict = {}
for row_index in range(1,sheet3.nrows):
rows3 = sheet3.row_values(row_index)
large_item = rows3[1] or large_item
data_dict = rows3
Now when I printed out print(data_dict),['', '4', '10', 'Karen', ''] was shown.Before,I wrote data_dict.extend(rows3) in place of data_dict = rows3,but in that time dict has not extend error happens.My ideal output is
data_dict = {
1: {
user_id: 1,
name_id: 1,
name: Blear,
age: 40,
man: false,
employee: leader,
},
2: {
user_id: 2,
name_id: 5,
・
　　　　　　 ・
　　　　　　 ・
},
・
　　　　　　 ・
　　　　　　 ・
}
How should I write to achieve my goal?

Your problem is :
data_dict = rows3
This doesn't add rows3 to data_dict, this set is value. So data_dict is equal to the last row.
To add element to a dict you need to do this:
data_dict[KEY] = VALUE
Your key will be the row index.
Now, you want another dict like VALUE
{
user_id: 1,
name_id: 1,
name: Blear,
age: 40,
man: false,
employee: leader,
}
So for each row you need to construct this dict, use headers and cell value to do it.
I don't test this code, it's just to give you an idea to how to do it.
#coding:utf-8
from django.shortcuts import render
import xlrd
book3 = xlrd.open_workbook('./data/excel.xlsx')
sheet3 = book3.sheet_by_index(0)
headers = sheet3.row_values(0)
large_item = None
data_dict = {}
for row_index in range(1,sheet3.nrows):
rows3 = sheet3.row_values(row_index)
large_item = rows3[1] or large_item
# Create dict with headers and row values
row_data = {}
for idx_col,value in enumerate(rows3):
header_value = headers[idx_col]
# Avoid to add empty column. A column in your example
if header_value:
row_data[headers[idx_col]] = value
# Add row_data to your data_dict with
data_dict[row_index] = row_data

You can use python's library pandas for an easy solution:
from pandas import *
xls = ExcelFile('your_excel_file.xls')
df = xls.parse(xls.sheet_names[0])
df.to_dict()