Develop Reference

Cannot get ordered result

I'm working with Python 3.5.2 and I'm trying to get a dictionary ordered by key by using OrderedDict.
Here is what I'm trying:
import re
from collections import OrderedDict
BRACKETS_PATTERN = re.compile(r"(?P<info>.*)?\((?P<bracket_info>.*?)\)")
def transform_vertical(vertical, trans=True):
# elearning & Massive Online Open Courses (MOOCs) => ELEARNING_AND_MASSIVE_ONLINE_OPEN_COURSES
# Repair & Maintenance (SMB) => SMB_REPAIR_AND_MAINTENANCE
# Digital Advertising & Marketing/Untagged Agencies => DIGITAL_ADVERTISING_AND_MARKETING_OR_UNTAGGED_AGENCIES
if not trans:
return vertical
else:
v = vertical.replace(" & ", "_AND_").replace(", ", "_AND_").replace("/", "_OR_")
brackets_search_result = BRACKETS_PATTERN.search(v)
result = v
if brackets_search_result:
bracket_info = brackets_search_result.group("bracket_info")
info = brackets_search_result.group("info")
if bracket_info.upper() in ("SMB", "CBV"): # todo more prefix
result = bracket_info.upper() + "_" + info
else:
result = info
result = result.replace(" ", "_").upper().strip("_")
return result
VERTICAL_MAP = OrderedDict({
"GAMING": OrderedDict({
"MOBILE_GAMING": 1,
"AR_OR_VR_GAMING": 1,
"CONSOLE_AND_CROSS_PLATFORM_GAMING": 1,
"ESPORTS": 1,
"PC_GAMING": 1,
"REAL_MONEY_GAMING": 1,
}),
"TRAVEL": OrderedDict({
"AUTO_RENTAL": 1,
"RAILROADS": 1,
"HOTEL_AND_ACCOMODATION": 1,
"RIDE_SHARING_OR_TAXI_SERVICES": 1,
"TOURISM_AND_TRAVEL_SERVICES": 1,
"TOURISM_BOARD": 1,
"AIR": 1,
"TRAVEL_AGENCIES_AND_GUIDES_AND_OTAS": 1,
"CRUISES_AND_MARINE": 1,
})
})
s = list(VERTICAL_MAP[transform_vertical("Gaming")].keys())
print(s)
And I get non-ordered result like:
['REAL_MONEY_GAMING', 'AR_OR_VR_GAMING', 'MOBILE_GAMING', 'CONSOLE_AND_CROSS_PLATFORM_GAMING', 'ESPORTS', 'PC_GAMING']
Expected result:
[ 'MOBILE_GAMING', 'AR_OR_VR_GAMING','CONSOLE_AND_CROSS_PLATFORM_GAMING', 'ESPORTS', 'PC_GAMING', 'REAL_MONEY_GAMING']
What's wrong with my code and how to get an ordered result?

Dictionaries are not insertion ordered in Python 3.5.
You are instantiating the ordered dicts with arbitrarily ordered regular dicts. Construct each of the ordered dicts from a list of (key, value) tuples.

Trying to get a weighted average out of a dictionary of grades data

I am trying to return the weighted average of the student's grades based on the last definition. I have the dictionaries defined, but think my attempt to pull the numbers out is incorrect.
def Average(lst):
return sum(lst) / len(lst)
# Driver Code
lst = [1,2,3,4,5]
average = Average(lst)
print("Average of the list =", average)
def get_weighted_average(student):
return average('homework')*0.10 + average('quizzes')*0.30 + average('tests')*.60
#driver code
students = [steve, alice, tyler]
print(get_weighted_average('steve'))
How to get a weighted average out of a dictionary of grades above?

What is the primary source of your data? Text? Anyway, it looks like you have something like this in mind.
Imperative approach
1 - Your "database"
students_marks = {
'steve':{
'homework':[1,2,3,4,5],
'quizzes' :[5,4,3,2,1],
'tests' :[0,0,0,0,0],
},
'alice':{
'homework':[5,4,3,2,1],
'quizzes' :[0,0,0,0,0],
'tests' :[1,2,3,4,5],
},
}
use case:
>>> students_marks['steve']
{'homework': [1, 2, 3, 4, 5], 'quizzes': [5, 4, 3, 2, 1], 'tests': [0, 0, 0, 0, 0]}
>>> students_marks['steve']['homework']
[1, 2, 3, 4, 5]
2 - The definition of average and get_weighted_average
def average(lst):
return sum(lst)/len(lst) # Python3
#return sum(lst)/float(len(lst)) # Python2
def get_weighted_average(student_name):
student_marks = students_marks[student_name]
return round(
average(student_marks['homework'])*.1
+ average(student_marks['quizzes'])*.3
+ average(student_marks['tests'])*.6
, 2)
use case:
>>> get_weighted_average('steve')
1.2
>>> get_weighted_average('alice')
2.1
or using list
>>> students_names = ['steve', 'alice']
>>> [get_weighted_average(name) for name in students_names]
[1.2, 2.1]
or using dict
>>> {name:get_weighted_average(name) for name in students_names}
{'steve': 1.2, 'alice': 2.1}
Object-Oriented (OO) approach
All this being shown, what you want to do would probably be better done by programming in an OO manner. A quick example
class Student(object):
homeworks_weight = .1
quizzes_weight = .3
tests_weight = .6
def __init__(self, name, homeworks_marks, quizzes_marks, tests_marks):
self.name = name
self.homeworks_marks = homeworks_marks
self.quizzes_marks = quizzes_marks
self.tests_marks = tests_marks
#staticmethod
def average(marks):
return sum(marks)/len(marks)
def get_gpa(self, rd=2):
return round(
self.average(self.homeworks_marks)*self.homeworks_weight
+ average(self.quizzes_marks)*self.quizzes_weight
+ average(self.tests_marks)*self.tests_weight
, rd)
use case:
>>> steve = Student(
name = 'Steve',
homeworks_marks = [1,2,3,4,5],
quizzes_marks = [5,4,3,2,1],
tests_marks = [0,0,0,0,0]
)
>>> steve.get_gpa()
1.2
>>> steve.homeworks_marks
[1, 2, 3, 4, 5]

parse empty string using json

I was wondering if there was a way to use json.loads in order to automatically convert an empty string in something else, such as None.
For example, given:
data = json.loads('{"foo":"5", "bar":""}')
I would like to have:
data = {"foo":"5", "bar":None}
Instead of:
data = {"foo":"5", "bar":""}

You can use a dictionary comprehension:
data = json.loads('{"foo":"5", "bar":""}')
res = {k: v if v != '' else None for k, v in data.items()}
{'foo': '5', 'bar': None}
This will only deal with the first level of a nested dictionary. You can use a recursive function to deal with the more generalised nested dictionary case:
def updater(d, inval, outval):
for k, v in d.items():
if isinstance(v, dict):
updater(d[k], inval, outval)
else:
if v == '':
d[k] = None
return d
data = json.loads('{"foo":"5", "bar":"", "nested": {"test": "", "test2": "5"}}')
res = updater(data, '', None)
{'foo': '5', 'bar': None,
'nested': {'test': None, 'test2': '5'}}

You can also accomplish this with the json.loads object_hook parameter. For example:
import json
import six
def empty_string2none(obj):
for k, v in six.iteritems(obj):
if v == '':
obj[k] = None
return obj
print(json.loads('{"foo":"5", "bar":"", "hello": {"world": ""}}',
object_hook=empty_string2none))
This will print
{'foo': '5', 'bar': None, 'hello': {'world': None}}
This way, you don't need additional recursion.

I did some trial and error and it is impossible to parse None into a String using json.loads() you will have to use json.loads() with json.dumps() like I do in this example:
import json
data = json.loads('{"foo":"5", "bar":"%r"}' %(None))
data2 = json.loads(json.dumps({'foo': 5, 'bar': None}))
if data2['bar'] is None:
print('worked')
print(data['bar'])
else:
print('did not work')

Extract path for each terminal node

I have a python nested dictionary structure that looks like the below.
This is a small example but I have larger examples that can have varying levels of nesting.
From this, I need to extract a list with:
One record for each terminal 'leaf' node
A string, list, or object representing the logical path leading up to that node
(e.g. 'nodeid_3: X < 0.500007 and X < 0.279907')
I've spent the larger part of this weekend trying to get something working and am realizing just how bad I am with recursion.
# Extract json string
json_string = booster.get_dump(with_stats=True, dump_format='json')[0]
# Convert to python dictionary
json.loads(json_string)
{u'children': [{u'children': [
{u'cover': 2291, u'leaf': -0.0611795, u'nodeid': 3},
{u'cover': 1779, u'leaf': -0.00965727, u'nodeid': 4}],
u'cover': 4070,
u'depth': 1,
u'gain': 265.811,
u'missing': 3,
u'no': 4,
u'nodeid': 1,
u'split': u'X',
u'split_condition': 0.279907,
u'yes': 3},
{u'cover': 3930, u'leaf': -0.0611946, u'nodeid': 2}],
u'cover': 8000,
u'depth': 0,
u'gain': 101.245,
u'missing': 1,
u'no': 2,
u'nodeid': 0,
u'split': u'X',
u'split_condition': 0.500007,
u'yes': 1}

You data structure is recursive. If a node has a children key, then we can consider that it is not terminal.
To analyze your data, you need a recursive function which keeps track of the ancestors (the path).
I would implement this like that:
def find_path(obj, path=None):
path = path or []
if 'children' in obj:
child_obj = {k: v for k, v in obj.items()
if k in ['nodeid', 'split_condition']}
child_path = path + [child_obj]
children = obj['children']
for child in children:
find_path(child, child_path)
else:
pprint.pprint((obj, path))
If you call:
find_path(data)
You get 3 results:
({'cover': 2291, 'leaf': -0.0611795, 'nodeid': 3},
[{'nodeid': 0, 'split_condition': 0.500007},
{'nodeid': 1, 'split_condition': 0.279907}])
({'cover': 1779, 'leaf': -0.00965727, 'nodeid': 4},
[{'nodeid': 0, 'split_condition': 0.500007},
{'nodeid': 1, 'split_condition': 0.279907}])
({'cover': 3930, 'leaf': -0.0611946, 'nodeid': 2},
[{'nodeid': 0, 'split_condition': 0.500007}])
Of course, you can replace the call to pprint.pprint() by a yield to turn this function into a generator:
def iter_path(obj, path=None):
path = path or []
if 'children' in obj:
child_obj = {k: v for k, v in obj.items()
if k in ['nodeid', 'split_condition']}
child_path = path + [child_obj]
children = obj['children']
for child in children:
# for o, p in iteration_path(child, child_path):
# yield o, p
yield from iter_path(child, child_path)
else:
yield obj, path
Note the usage of yield from for the recursive call. You use this generator like below:
for obj, path in iter_path(data):
pprint.pprint((obj, path))
You can also change the way child_obj object is build to match your needs.
To keep the order of objects: reverse the if condition: if 'children' not in obj: ….

Python CFFI convert structure to dictionary

There is a way to initialize structure with dictionary:
fooData= {'y': 1, 'x': 2}
fooStruct = ffi.new("foo_t*", fooData)
fooBuffer = ffi.buffer(fooStruct)
Is there some ready function to do the conversion?
fooStruct = ffi.new("foo_t*")
(ffi.buffer(fooStruct))[:] = fooBuffer
fooData= convert_to_python( fooStruct[0] )
Do I have to use ffi.typeof("foo_t").fields by myself?
I come up with this code so far:
def __convert_struct_field( s, fields ):
for field,fieldtype in fields:
if fieldtype.type.kind == 'primitive':
yield (field,getattr( s, field ))
else:
yield (field, convert_to_python( getattr( s, field ) ))
def convert_to_python(s):
type=ffi.typeof(s)
if type.kind == 'struct':
return dict(__convert_struct_field( s, type.fields ) )
elif type.kind == 'array':
if type.item.kind == 'primitive':
return [ s[i] for i in range(type.length) ]
else:
return [ convert_to_python(s[i]) for i in range(type.length) ]
elif type.kind == 'primitive':
return int(s)
Is there a faster way?

Arpegius' solution works fine for me, and is quite elegant. I implemented a solution based on Selso's suggestion to use inspect. dir() can substitute inspect.
from inspect import getmembers
from cffi import FFI
ffi = FFI()
from pprint import pprint
def cdata_dict(cd):
if isinstance(cd, ffi.CData):
try:
return ffi.string(cd)
except TypeError:
try:
return [cdata_dict(x) for x in cd]
except TypeError:
return {k: cdata_dict(v) for k, v in getmembers(cd)}
else:
return cd
foo = ffi.new("""
struct Foo {
char name[6];
struct {
int a, b[3];
} item;
} *""",{
'name': b"Foo",
'item': {'a': 3, 'b': [1, 2, 3]}
})
pprint(cdata_dict(foo))
Output:
{'item': {'a': 3, 'b': [1, 2, 3]}, 'name': b'Foo'}

This code infortunately does not work for me, as some struct members are "pointer" types, it leads to storing "none" in the dict.
I am a Python noob, but maybe the inspect module would be another starting point, and a shorter way to print "simple" data. Then we would iterate over the result in order to unroll data structure.
For example with the following example :
struct foo {
int a;
char b[10];
};
Using inspect.getmembers( obj ) I have the following result :
[('a', 10), ('b', <cdata 'char[10]' 0x7f0be10e2824>)]

Your code is fine.
Even if there was a built-in way in CFFI, it would not be what you need here. Indeed, you can say ffi.new("foo_t*", {'p': p1}) where p1 is another cdata, but you cannot recursively pass a dictionary containing more dictionaries. The same would be true in the opposite direction: you would get a dictionary that maps field names to "values", but the values themselves would be more cdata objects anyway, and not recursively more dictionaries.