Develop Reference

Finding and averaging elements of multiple dataframes that are within 10% of each other

I have objects that store values are dataframes. I have been able to compare if values from two dataframes are within 10% of each other. However, I am having difficulty extending this to multiple dataframes. Moreover, I am wondering how I should apporach this problem if dataframes are not the same size?
def add_well_peak(self, *other):
if len(self.Bell) == len(other.Bell): #if dataframes ARE the same size
for k in range(len(self.Bell)):
for j in range(len(other.Bell)):
if int(self.Size[k]) - int(self.Size[k])*(1/10) <= int(other.Size[j]) <= int(self.Size[k]) + int(self.Size[k])*(1/10):
#average all
For example, in the image below, there are objects that contain dataframes (i.e., self, other1, other2). The colors represent matches (i.e, values that are within 10% of each other). If a match exist, then average the values. If a match does not exist still include the unmatch number. I want to be able to generalize this for any number of objects greater or equal than 2 (other 1, other 2, other 3, other ....). Any help would be appreciated. Please let me know if anything is unclear. This is my first time posting. Thanks again.
matching data

Results:
Using my solution on the dataframes of your image, I get the following:
Threshold outlier = 0.2:
0
0 1.000000
1 1493.500000
2 5191.333333
3 35785.333333
4 43586.500000
5 78486.000000
6 100000.000000
Threshold outlier = 0.5:
0 1
0 1.000000 NaN
1 1493.500000 NaN
2 5191.333333 NaN
3 43586.500000 35785.333333
4 78486.000000 100000.000000
Explanations:
The lines are averaged peaks, the columns representing the different values obtained for these peaks. I assumed the average emanating from the biggest number of elements was the legitimate one, and the rest within the THRESHOLD_OUTLIER were the outliers (should be sorted, the more probable you are as a legitimate peak, the more you are on the left (the 0th column is the most probable)). For instance, on line 3 of the 0.5 outlier threshold results, 43586.500000 is an average coming from 3 dataframes, while 35785.333333 comes from only 2, thus the most probable is the first one.
Issues:
The solution is quite complicated. I assume a big part of it could be removed, but I can't see how for the moment, and as it works, I'll certainly leave the optimization to you.
Still, I tried commenting my best, and if you have any question, do not hesitate!
Files:
CombinationLib.py
from __future__ import annotations
from typing import Dict, List
from Errors import *
class Combination():
"""
Support class, to make things easier.
Contains a string `self.combination` which is a binary number stored as a string.
This allows to test every combination of value (i.e. "101" on the list `[1, 2, 3]`
would signify grouping `1` and `3` together).
There are some methods:
- `__add__` overrides the `+` operator
- `compute_degree` gives how many `1`s are in the combination
- `overlaps` allows to verify if combination overlaps (use the same value twice)
(i.e. `100` and `011` don't overlap, while `101` and `001` do)
"""
def __init__(self, combination:str) -> Combination:
self.combination:str = combination
self.degree:int = self.compute_degree()
def __add__(self, other: Combination) -> Combination:
if self.combination == None:
return other.copy()
if other.combination == None:
return self.copy()
if self.overlaps(other):
raise CombinationsOverlapError()
result = ""
for c1, c2 in zip(self.combination, other.combination):
result += "1" if (c1 == "1" or c2 == "1") else "0"
return Combination(result)
def __str__(self) -> str:
return self.combination
def compute_degree(self) -> int:
if self.combination == None:
return 0
degree = 0
for bit in self.combination:
if bit == "1":
degree += 1
return degree
def copy(self) -> Combination:
return Combination(self.combination)
def overlaps(self, other:Combination) -> bool:
for c1, c2 in zip(self.combination, other.combination):
if c1 == "1" and c1 == c2:
return True
return False
class CombinationNode():
"""
The main class.
The main idea was to build a tree of possible "combinations of combinations":
100-011 => 111
|---010-001 => 111
|---001-010 => 111
At each node, the combination applied to the current list of values was to be acceptable
(all within THREASHOLD_AVERAGING).
Also, the shorter a path, the better the solution as it means it found a way to average
a lot of the values, with the minimum amount of outliers possible, maybe by grouping
the outliers together in a way that makes sense, ...
- `populate` fills the tree automatically, with every solution possible
- `path` is used mainly on leaves, to obtain the path taken to arrive there.
"""
def __init__(self, combination:Combination) -> CombinationNode:
self.combination:Combination = combination
self.children:List[CombinationNode] = []
self.parent:CombinationNode = None
self.total_combination:Combination = combination
def __str__(self) -> str:
list_paths = self.recur_paths()
list_paths = [",".join([combi.combination.combination for combi in path]) for path in list_paths]
return "\n".join(list_paths)
def add_child(self, child:CombinationNode) -> None:
if child.combination.degree > self.combination.degree and not self.total_combination.overlaps(child.combination):
raise ChildDegreeExceedParentDegreeError(f"{child.combination} > {self.combination}")
self.children.append(child)
child.parent = self
child.total_combination += self.total_combination
def path(self) -> List[CombinationNode]:
path = []
current = self
while current.parent != None:
path.append(current)
current = current.parent
path.append(current)
return path[::-1]
def populate(self, combination_dict:Dict[int, List[Combination]]) -> None:
missing_degrees = len(self.combination.combination)-self.total_combination.degree
if missing_degrees == 0:
return
for i in range(min(self.combination.degree, missing_degrees), 0, -1):
for combination in combination_dict[i]:
if not self.total_combination.overlaps(combination):
self.add_child(CombinationNode(combination))
for child in self.children:
child.populate(combination_dict)
def recur_paths(self) -> List[List[CombinationNode]]:
if len(self.children) == 0:
return [self.path()]
paths = []
for child in self.children:
for path in child.recur_paths():
paths.append(path)
return paths
Errors.py
class ChildDegreeExceedParentDegreeError(Exception):
pass
class CombinationsOverlapError(Exception):
pass
class ToImplementError(Exception):
pass
class UncompletePathError(Exception):
pass
main.py
from typing import Dict, List, Set, Tuple, Union
import pandas as pd
from CombinationLib import *
best_depth:int = -1
best_path:List[CombinationNode] = []
THRESHOLD_OUTLIER = 0.2
THRESHOLD_AVERAGING = 0.1
def verif_averaging_pct(combination:Combination, values:List[float]) -> bool:
"""
For a given combination of values, we must have all the values within
THRESHOLD_AVERAGING of the average of the combination
"""
avg = 0
for c,v in zip(combination.combination, values):
if c == "1":
avg += v
avg /= combination.degree
for c,v in zip(combination.combination, values):
if c == "1"and (v > avg*(1+THRESHOLD_AVERAGING) or v < avg*(1-THRESHOLD_AVERAGING)):
return False
return True
def recursive_check(node:CombinationNode, depth:int, values:List[Union[float, int]]) -> None:
"""
Here is where we preferencially ask for a small number of bigger groups
"""
global best_depth
global best_path
# If there are more groups than the current best way to do, stop
if best_depth != -1 and depth > best_depth:
return
# If all the values of the combination are not within THRESHOLD_AVERAGING, stop
if not verif_averaging_pct(node.combination, values):
return
# If we finished the list of combinations, and this way is the best, keep it, stop
if len(node.children) == 0:
if best_depth == -1 or depth < best_depth:
best_depth = depth
best_path = node.path()
return
# If we are still not finished (not every value has been used), continue
for cnode in node.children:
recursive_check(cnode, depth+1, values)
def groups_from_list(values:List[Union[float, int]]) -> List[List[Union[float, int]]]:
"""
From a list of values, get the smallest list of groups of elements
within THRESHOLD_AVERAGING of each other.
It implies that we will try and recursively find the biggest group possible
within the unsused values (i.e. groups with combinations of size [3, 1] are prefered
over [2, 2])
"""
global best_depth
global best_path
groups:List[List[float]] = []
# Generate all the combinations (I used binary for this)
combination_dict:Dict[int, List[Combination]] = {}
for i in range(1, 2**len(values)):
combination = format(i, f"0{len(values)}b") # Here is the binary conversion
counter = 0
for c in combination:
if c == "1":
counter += 1
if counter not in combination_dict:
combination_dict[counter] = []
combination_dict[counter].append(Combination(combination))
# Generate of the combinations of combinations that use all values (without using one twice)
combination_trees:List[List[CombinationNode]] = []
for key in combination_dict:
for combination in combination_dict[key]:
cn = CombinationNode(combination)
cn.populate(combination_dict)
combination_trees.append(cn)
best_depth = -1
best_path = None
for root in combination_trees:
recursive_check(root, 0, values)
# print(",".join([combination.combination.combination for combination in best_path]))
for combination in best_path:
temp = []
for c,v in zip(combination.combination.combination, values):
if c == "1":
temp.append(v)
groups.append(temp)
return groups
def averages_from_groups(gs:List[List[Union[float, int]]]) -> List[float]:
"""Computing the averages of each group"""
avgs:List[float] = []
for group in gs:
avg = 0
for elt in group:
avg += elt
avg /= len(group)
avgs.append(avg)
return avgs
def end_check(ds:List[pd.DataFrame], ids:List[int]) -> bool:
"""Check if we finished consuming all the dataframes"""
for d,i in zip(ds, ids):
if i < len(d[0]):
return False
return True
def search(group:List[Union[float, int]], values_list:List[Union[float, int]]) -> List[int]:
"""Obtain all the indices corresponding to a set of values"""
# We will get all the indices in values_list of the values in group
# If a value is present in group, all the occurences of this value will be too,
# so we can use a set and search every occurence for each value.
indices:List[int] = []
group_set = set(group)
for value in group_set:
for i,v in enumerate(values_list):
if value == v:
indices.append(i)
return indices
def threshold_grouper(total_list:List[Union[float, int]]) -> pd.DataFrame:
"""Building a 2D pd.DataFrame with the averages (x) and the outliers (y)"""
result_list:List[List[Union[float, int]]] = [[total_list[0]]]
result_index = 0
total_index = 1
while total_index < len(total_list):
# Only checking if the bigger one is within THRESHOLD_OUTLIER of the little one.
# If it is the case, the opposite is true too.
# If yes, it is an outlier
if result_list[result_index][0]*(1+THRESHOLD_OUTLIER) >= total_list[total_index]:
result_list[result_index].append(total_list[total_index])
# Else it is a new peak
else:
result_list.append([total_list[total_index]])
result_index += 1
total_index += 1
result:pd.DataFrame = pd.DataFrame(result_list)
return result
def dataframes_merger(dataframes:List[pd.DataFrame]) -> pd.DataFrame:
"""Merging the dataframes, with THRESHOLDS"""
# Store the averages for the within 10% cells, in ascending order
result = []
# Keep tabs on where we are regarding each dataframe (needed for when we skip cells)
curr_indices:List[int] = [0 for _ in range(len(dataframes))]
# Repeat until all the cells in every dataframe has been seen once
while not end_check(dataframes, curr_indices):
# Get the values of the current indices in the dataframes
curr_values = [dataframe[0][i] for dataframe,i in zip(dataframes, curr_indices)]
# Get the largest 10% groups from the current list of values
groups = groups_from_list(curr_values)
# Compute the average of these groups
avgs = averages_from_groups(groups)
# Obtain the minimum average...
avg_min = min(avgs)
# ... and its index
avg_min_index = avgs.index(avg_min)
# Then get the group corresponding to the minimum average
avg_min_group = groups[avg_min_index]
# Get the indices of the values included in this group
indices_to_increment = search(avg_min_group, curr_values)
# Add the average to the result merged list
result.append(avg_min)
# For every element in the average we added, increment the corresponding index
for index in indices_to_increment:
curr_indices[index] += 1
# Re-assemble the dataframe, taking the threshold% around average into account
result = threshold_grouper(result)
print(result)
df1 = pd.DataFrame([1, 1487, 5144, 35293, 78486, 100000])
df2 = pd.DataFrame([1, 1500, 5144, 36278, 45968, 100000])
df3 = pd.DataFrame([1, 5286, 35785, 41205, 100000])
dataframes_merger([df3, df2, df1])

How to get the maximum value of a key in dict?

I would like to get the URL of a video with maximum resolution.
If I had the following dictionary, would it be easiest to get the URL of the video with the maximum size?
Is it best to split the size by partition, format the number into an int type, and then get the value?
videos = {
'size_476_306': 'https://www.....',
'size_560_360': 'https://www.....',
'size_644_414': 'https://www.....',
'size_720_480': 'https://www.....',
}
Solved
I couldn't figure out lambda, so I implemented it in a different way.
for size, v_url in videos.items():
max_size_info = size.split('_')
split_size = int(max_size_info[1])
size_array.append(split_size)
video_array.append(v_url)
max_size = size_array.index(max(size_array))
if str(size_array[max_size]) in v_url:
print(size,v_url)

Is it best to split the size by partition, format the number into an int type, and then get the value?
Yes, that's the way I'd do it:
>>> max(videos.items(), key=lambda i: int.__mul__(*map(int, i[0].split("_")[1:])))
('size_720_480', 'https://www.....')
Here's a slightly more verbose version with a named function:
>>> def get_resolution(key: str) -> int:
... """
... Gets the resolution (as pixel area) from 'size_width_height'.
... e.g. get_resolution("size_720_480") -> 345600
... """
... _, width, height = key.split("_")
... return int(width) * int(height)
...
>>> max(videos, key=get_resolution)
'size_720_480'
Given that expression that gives us the largest key, we can easily get the corresponding value:
>>> videos[max(videos, key=get_resolution)]
'https://www.....'
or we could get the key, value pair by taking the max of items(), here using a much simpler lambda that just translates key, value into get_resolution(key):
>>> max(videos.items(), key=lambda i: get_resolution(i[0]))
('size_720_480', 'https://www.....')

increase value of a dictionary using dictionary comprehension

I would like to write a simple code to store histogram data in a dictionary, I want to use dictionary comprehension build the final histogram dict. in python. I can achieve that in multiple ways as shown in code below but I have a problem with method two, here is how it works:
Enumerate the list of random letters to use that index to access a slice of the list starting at the current iteration to the end
check if letters exists in the slice
if yes, add the letter as index to the dictionary and increase its value by 1
if no, add the letter as index to the dictionary and set its value to 1
my problem is where is this temporary dictionary to add to before the dict. Comprehension is exhausted. I tried the final dict “histogram_dict” but it’s not the same.
Again, The main goal is to rewrite the first code in terms of dictionary comprehension as in the second code.
Thanks
import random
rnd_letters_list=[chr(random.randrange(97,122)) for i in range(21)]
histogram_dict={}
#=============first method ==============
# for letter in rnd_letters_list:
# if letter in histogram_dict:
# histogram_dict[letter]+=1
# else:
# histogram_dict[letter]=1
# print(histogram_dict)
#=============second method ==============
histogram_dict={letter: XXX_dict[letter]+=1
if letter in rnd_letters_list[indx::]
else XXX_dict[letter]=1
for indx,letter in enumerate(rnd_letters_list)}
print(histogram_dict)
#=============third method ==============
# histogram_dict={letter:rnd_letters_list.count(letter)
# for letter in rnd_letters_list}
# print(histogram_dict)
#=============fourth method ==============
# from collections import Counter
# print(dict(
# Counter(rnd_letters_list).most_common()))

I had the same problem a few weeks ago. I extendet the counter class.
Counter itself extends the dictionary.
Counter Docs
from typing import Counter, Dict
class Histogram(Counter):
"""
A that saves discrete values for a histogram in an efficient manor.
"""
def population(self) -> int:
return sum(self.values())
def mean(self) -> float:
population = 0
average_value = 0
for key, value in self.items():
average_value += key * value
population += value
return average_value / population
def y_mean(self) -> float:
entry_counts: int = len(self.items())
sum_of_values: int = sum(self.values())
return sum_of_values / (entry_counts * sum_of_values)
def median(self, alpha: float = 0.5) -> float:
position: int = round(alpha * self.population())
keys = sorted(self.keys())
for key in keys:
position -= self[key]
if position < 0:
return key
return float("NaN")
def variance(self) -> float:
pop: int = self.population()
if pop < 2:
return 0
var: float = 0
mean: float = self.mean()
for key, value in self.items():
var += float(value) * ((float(key) - mean) ** 2)
return var / (pop - 1)
def normed(self) -> Dict[int, float]:
dictionary: Dict[int, float] = {}
population = self.population()
for key, value in self.items():
dictionary[key] = value / population
return dictionary

Knapsack recursive function

I have a list named self.items where the elements are:
items = [dict(id=0, w=4, v=12),
dict(id=1, w=6, v=10),
dict(id=2, w=5, v=8),
dict(id=3, w=7, v=11),
dict(id=4, w=3, v=14),
dict(id=5, w=1, v=7),
dict(id=6, w=6, v=9)]
With this I had to do a list of lists, where every element has all the possible combinations including the empty case, so finally my list of lists has more or less this appearence:
[[],[{id:0,w:4,v:12}],....,[{id:0,w:4,v:12}, {id:1,w:6,v:10}]....]
Now I have to found a recursive function to search what combination of elements has the max weight permitted and the max value.
def recursive(self, n, max_weight):
""" Recursive Knapsack
:param n: Number of elements
:param max_weight: Maximum weight allowed
:return: max_value
"""
self.iterations += 1
result = 0
if max_weight > self.max_weight: #they gave me self.max_weight as a variable of __init__ method which shows me what is the maximum weight permitted
self.recursive(self, self.items+self.iterations, max_weight)
if max_weight < self.max_weight:
self.recursive(self, self.items+self.iterations, max_weight)
else:
result = self.items['v']+result
return result
I think that my error is in this line:
result = self.items['v']+result
But I cannot find it.

I just have found the solution to this recursive problem:
(I'm from Spain so the variable "cantidad" also means "quantity")
def recursive(self, n, max_weight):
""" Recursive Knapsack
:param n: Number of elements
:param max_weight: Maximum weight allowed
:return: max_valu
"""
self.iterations += 1
result = 0
cantidad = 0
quantity = 0
if max_weight == 0 or n == 1:
if max_weight >= self.items[n-1]['w'] :
cantidad= self.items[n-1]['v']
return max(cantidad,quantity)
else:
if max_weight >= self.items[n-1]['w']:
cantidad = self.items[n-1]['v']+self.recursive(n-1,max_weight-self.items[n-1]['w'])
quantity = self.recursive(n-1,max_weight)
result = max(cantidad, quantity)
return result
I put this code into a program proportioned by the university I am studying at and it returns to me the correct result:
Method: recursive
Iterations:107
Max value:44 expected max_value:44

Can't get linking multiple functions and a class to work

I have a program which holds many functions and I have no idea how to link them together :/
When I'm working on the python's shell everything is working fine because everything is in one definition but when I divide them into different definitions it doesn't work.
I have a different Metals each one of them has a weight and value and a name, etc. I want to make a class for those metals and then sort them in a descending order based on there valuePerBar once and valuePerWeight once
those are my metals :
Gold, 1 bar, 5 weight per bar, 750 value per bar
Silver, 1 bar, 1 weight per bar, 400 value per bar
Rhodium, 1 bar, 4 weight per bar, 500 value per bar
Platinum, 1 bar, 6 weight per bar, 1000 value per bar
and this is the my class:
class Metal(rit_object):
"""
Represents a single metal type, composed of:
:slot name (str): The name of the metal
:slot totalBars (int): The total number of bars
:slot weightPerBar (int): The weight of a single bar
:slot valuePerBar (int): The value of a single bar
:slot valuePerWeight (float): The value per weight of the metal
:slot barsTaken (int): The number of bars added to the satchel
"""
__slots__ = ( 'name' , 'totalBars' , 'weightPerBar', 'valuePerBar', 'valuePerWeight', 'barsTaken' )
_types = ( str , int , int, int, float, int )
Note: rit_object is a private class which makes my work easier and all it does is to put types for each parameter ! ex: name --> str and totalBars --> int ..etc
the way I am doing the sorting in the shell is :
class Metal(rit_object):
"""
Represents a single metal type, composed of:
:slot name (str): The name of the metal
:slot totalBars (int): The total number of bars
:slot weightPerBar (int): The weight of a single bar
:slot valuePerBar (int): The value of a single bar
:slot valuePerWeight (float): The value per weight of the metal
:slot barsTaken (int): The number of bars added to the satchel
"""
__slots__ = ( 'name' , 'totalBars' , 'weightPerBar', 'valuePerBar', 'valuePerWeight', 'barsTaken' )
_types = ( str , int , int, int, float, int )
platinum = Metal("platinum", 1, 6, 1000, 166.666667, 0 )
gold = Metal("gold", 1, 5, 750, 150.0, 0 )
rhodium = Metal("rhodium", 1, 4, 500, 125.0, 0 )
silver = Metal("silver", 1, 1, 4, 400.0, 0 )
Metals = [
Metal("platinum", 1, 6, 1000, 166.666667, 0 ),
Metal("gold", 1, 5, 750, 150.0, 0 ) ,
Metal("rhodium", 1, 4, 500, 125.0, 0 ),
Metal("silver", 1, 1, 4, 400.0, 0 )
]
def getKey(Metal):
return name.valuePerBar
sorted(customlist, key=getKey, reverse=True)
and everything works perfect, now I want to make a program with multiple definition to make it well-organiezed but my problems is I DO NOT KNOW HOW TO LINK THE FUNCTIONS !!
this is my program:
"""
Author: Sean Strout (sps#cs.rit.edu)
Author: <<< YOUR NAME HERE >>>
This class represents the types of metal bars that Greedo can
store in his satchel. Each type of bar is a separate Metal
object. This module also has routines that work with metals,
e.g. creation, reading from a file, and sorting based on
various criteria.
Language: Python 3
"""
from rit_object import * # rit_object class
class Metal(rit_object):
"""
Represents a single metal type, composed of:
:slot name (str): The name of the metal
:slot totalBars (int): The total number of bars
:slot weightPerBar (int): The weight of a single bar
:slot valuePerBar (int): The value of a single bar
:slot valuePerWeight (float): The value per weight of the metal
:slot barsTaken (int): The number of bars added to the satchel
"""
__slots__ = ( 'name' , 'totalBars' , 'weightPerBar', 'valuePerBar', 'valuePerWeight', 'barsTaken' )
_types = ( str , int , int, int, float, int )
def createMetal(name, totalBars, weightPerBar, valuePerBar):
"""
Create and return a new Metal object.
:param name (str): The name of the metal
:param totalBars (int): The total number of bars
:param weightPerBar (int): The weight of a single bar
:param valuePerBar (int): The value of a single bar
:return: A newly initialized Metal object
:rtype: Metal
"""
platinum = Metal("platinum", 1, 6, 1000, 166.666667, 0 )
gold = Metal("gold", 1, 5, 750, 150.0, 0 )
rhodium = Metal("rhodium", 1, 4, 500, 125.0, 0 )
silver = Metal("silver", 1, 1, 4, 400.0, 0 )
def readMetals(Metals):
"""
Read the metals from a file whose format is:
metalName totalBars weightPerBar valuePerBar
:param fileName (str): The name of the file
:return: A list of Metal objects
:rtype: list
"""
Metals = [
Metal("platinum", 1, 6, 1000, 166.666667, 0 ),
Metal("gold", 1, 5, 750, 150.0, 0 ) ,
Metal("rhodium", 1, 4, 500, 125.0, 0 ),
Metal("silver", 1, 1, 4, 400.0, 0 )
]
print (name.valuePerBar)
def getKey(Metal):
return name.valuePerBar
def sortMetalsByValuePerBar():
"""
Sort the metals by value per bar using insertion sort. The list of
metals is modified in place to be ordered by value per bar.
:param metals (list of Metal): The list of metals
:return: None
:rtype: NoneType
"""
return sorted(Metals, key=getKey, reverse=True)
def getKey2(Metal):
return name.weightPerBar
def sortMetalsByValuePerWeight(metals):
"""
Sort the metals by value per weight using insertion sort. The list of
metals is modified in place to be ordered by value per weight.
:param metals (list of Metal): The list of metals
:return: None
:rtype: NoneType"""
return sorted(Metals, key=getKey, reverse=True)
def printMetals(metals):
"""
Display the metals to standard output.
:param metals (list of Metal): The list of metals
:return: None
:rtype: NoneType
"""
if Q == a:
getKey(Metal)
sortMetalsByValuePerBar()
else:
getKey2(Metal)
sortMetalsByValuePerWeight(metals)
Q = input ("Enter 'a' for descending order or 'b' for ascending order")
Metal(rit_object)
createMetal(name, totalBars, weightPerBar, valuePerBar)
readMetals(Metals)
printMetals(metals)