Related
I have the following dataframe:
columns = pd.date_range(start="2022-05-21", end="2022-06-30")
data = [
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5],
[5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
[5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5]
]
df = pd.DataFrame(data, columns=columns)
2022-05-21 2022-05-22 2022-05-23 ... 2022-06-28 2022-06-29 2022-06-30
0 0 0 0 ... 5 5 5
1 5 5 5 ... 1 1 1
2 5 5 5 ... 5 5 5
I have to take the first and last column index for every distinct value in the order they are. The correct output for this dataframe will be:
[
[
{'value': 0, 'start': '2022-05-21', 'end': '2022-05-31'},
{'value': 2, 'start': '2022-06-01', 'end': '2022-06-19'},
{'value': 5, 'start': '2022-06-20', 'end': '2022-06-30'}
],
[
{'value': 5, 'start': '2022-05-21', 'end': '2022-05-31'},
{'value': 2, 'start': '2022-06-01', 'end': '2022-06-19'},
{'value': 1, 'start': '2022-06-20', 'end': '2022-06-30'}
],
[
{'value': 5, 'start': '2022-05-21', 'end': '2022-05-31'},
{'value': 2, 'start': '2022-06-01', 'end': '2022-06-19'},
{'value': 5, 'start': '2022-06-20', 'end': '2022-06-30'}
]
]
My best approach for the moment is:
series_set = df.apply(frozenset, axis=1)
container = []
for index in range(len(df.index)):
row = df.iloc[[index]]
values = series_set.iloc[[index]]
inner_container = []
for value in values[index]:
single_value_series = row[row.columns[row.isin([value]).all()]]
dates = single_value_series.columns
result = dict(value=value, start=dates[0].strftime("%Y-%m-%d"), end=dates[-1].strftime("%Y-%m-%d"))
inner_container.append(result)
container.append(inner_container)
The result is:
[
[
{'value': 0, 'start': '2022-05-21', 'end': '2022-05-31'},
{'value': 2, 'start': '2022-06-01', 'end': '2022-06-19'},
{'value': 5, 'start': '2022-06-20', 'end': '2022-06-30'}
],
[
{'value': 1, 'start': '2022-06-20', 'end': '2022-06-30'},
{'value': 2, 'start': '2022-06-01', 'end': '2022-06-19'},
{'value': 5, 'start': '2022-05-21', 'end': '2022-05-31'}
],
[
{'value': 2, 'start': '2022-06-01', 'end': '2022-06-19'},
{'value': 5, 'start': '2022-05-21', 'end': '2022-06-30'}
]
]
It has several problems, only the first array is correct :)
When I convert dataframe to frozenset it is sorted and order is changed and also if some value appears more than once it is removed.
I will appreciate any idea and guidance. What I want to avoid is iterating the dataframe.
Thank you!
You can first transpose DataFrame by DataFrame.T and then aggregate minimal and maximal index with convert values to strings by Series.dt.strftime, last convert to dictionaries by DataFrame.to_dict.
For get consecutive groups is compared shifted values with Series.cumsum.
df1 = df.T.reset_index()
L = [df1.groupby(df1[x].ne(df1[x].shift()).cumsum())
.agg(value=(x, 'first'),
start=('index', 'min'),
end=('index', 'max'))
.assign(start=lambda x: x['start'].dt.strftime('%Y-%m-%d'),
end=lambda x: x['end'].dt.strftime('%Y-%m-%d'))
.to_dict(orient='records') for x in df1.columns.drop('index')]
print (L)
[[{'value': 0, 'start': '2022-05-21', 'end': '2022-05-31'},
{'value': 2, 'start': '2022-06-01', 'end': '2022-06-19'},
{'value': 5, 'start': '2022-06-20', 'end': '2022-06-30'}],
[{'value': 5, 'start': '2022-05-21', 'end': '2022-05-31'},
{'value': 2, 'start': '2022-06-01', 'end': '2022-06-19'},
{'value': 1, 'start': '2022-06-20', 'end': '2022-06-30'}],
[{'value': 5, 'start': '2022-05-21', 'end': '2022-05-31'},
{'value': 2, 'start': '2022-06-01', 'end': '2022-06-19'},
{'value': 5, 'start': '2022-06-20', 'end': '2022-06-30'}]]
I have a nested list, and for each list inside I want to create a dictionary that will contain another dictionary with the words related to a certain word as a key and the times they appear as the value. For example:
from
sentences = [["i", "am", "a", "sick", "man"],
["i", "am", "a", "spiteful", "man"],
["i", "am", "an", "unattractive", "man"],
["i", "believe", "my", "liver", "is", "diseased"],
["however", "i", "know", "nothing", "at", "all", "about", "my",
"disease", "and", "do", "not", "know", "for", "certain", "what", "ails", "me"]]
part of the dictionary returned would be:
{ "man": {"i": 3, "am": 3, "a": 2, "sick": 1, "spiteful": 1, "an": 1, "unattractive": 1}, "liver": {"i": 1, "believe": 1, "my": 1, "is": 1, "diseased": 1}...}
with as many keys as there are distinct words in the passage.
I've tried this:
d = {}
for row in sentences:
for words in rows:
if words not in d:
d[words] = 1
else:
d[words] += 1
But is only the way to count them, how could I use d as a value for another dictionary?
from collections import defaultdict
data = {}
for sentence in sentences:
for word in sentence:
data[word] = defaultdict(lambda: 0)
for sentence in sentences:
length = len(sentence)
for index1, word1 in enumerate(sentence):
for num in range(0, length - 1):
index2 = (index1 + 1 + num) % length
word2 = sentence[index2]
data[word1][word2] += 1
print(data)
sentences = [["i", "am", "a", "sick", "man"],
["i", "am", "a", "spiteful", "man"],
["i", "am", "an", "unattractive", "man"],
["i", "believe", "my", "liver", "is", "diseased"],
["however", "i", "know", "nothing", "at", "all", "about", "my",
"disease", "and", "do", "not", "know", "for", "certain", "what", "ails", "me"]]
# "as many keys as there are distinct words in the passage"
# Well then we need to start by finding the distinct words.
# sets always help for this.
# first we flatten the list. If you don't know what this is doing,
# search "flatten nested list Python". This is a common pattern:
flat_list = [term for group in sentences for term in group]
# now use set to find distinct words
distinct_words = set(flat_list)
# variable for final dictionary
result = {}
# define this function first. See invocation below
def find_related_counts(word):
# a nice way to do counts us with
# setdefault. If the term has already
# been counted, then it just increments.
# otherwise, it will create the key and
# initialise it to the default
related_counts = {}
for group in sentences:
# is "word" related to the terms in this group?
if word in group:
# yes it is! add the other terms:
for other in group:
# except, presumably, the word itself
if other != word:
related_counts.setdefault(other, 0)
related_counts[other] += 1
return related_counts
# for each word we have a key, and must find the value
for word in distinct_words:
# when dealing with nested anythings, it helps to
# make a function, so you don't have so much
# nesting in one place and separate things out
# nicely instead
value = find_related_counts(word)
result[word] = value
print(result)
print(result["man"])
OUTPUT:
{'spiteful': {'i': 1, 'am': 1, 'a': 1, 'man': 1}, 'and': {'however': 1, 'i': 1, 'know': 2, 'nothing': 1, 'at': 1, 'all': 1, 'about': 1, 'my': 1, 'disease': 1, 'do': 1, 'not': 1, 'for': 1, 'certain': 1, 'what': 1, 'ails': 1, 'me': 1}, 'unattractive': {'i': 1, 'am': 1, 'an': 1, 'man': 1}, 'nothing': {'however': 1, 'i': 1, 'know': 2, 'at': 1, 'all': 1, 'about': 1, 'my': 1, 'disease': 1, 'and': 1, 'do': 1, 'not': 1, 'for': 1, 'certain': 1, 'what': 1, 'ails': 1, 'me': 1}, 'diseased': {'i': 1, 'believe': 1, 'my': 1, 'liver': 1, 'is': 1}, 'sick': {'i': 1, 'am': 1, 'a': 1, 'man': 1}, 'man': {'i': 3, 'am': 3, 'a': 2, 'sick': 1, 'spiteful': 1, 'an': 1, 'unattractive': 1}, 'do': {'however': 1, 'i': 1, 'know': 2, 'nothing': 1, 'at': 1, 'all': 1, 'about': 1, 'my': 1, 'disease': 1, 'and': 1, 'not': 1, 'for': 1, 'certain': 1, 'what': 1, 'ails': 1, 'me': 1}, 'believe': {'i': 1, 'my': 1, 'liver': 1, 'is': 1, 'diseased': 1}, 'i': {'am': 3, 'a': 2, 'sick': 1, 'man': 3, 'spiteful': 1, 'an': 1, 'unattractive': 1, 'believe': 1, 'my': 2, 'liver': 1, 'is': 1, 'diseased': 1, 'however': 1, 'know': 2, 'nothing': 1, 'at': 1, 'all': 1, 'about': 1, 'disease': 1, 'and': 1, 'do': 1, 'not': 1, 'for': 1, 'certain': 1, 'what': 1, 'ails': 1, 'me': 1}, 'certain': {'however': 1, 'i': 1, 'know': 2, 'nothing': 1, 'at': 1, 'all': 1, 'about': 1, 'my': 1, 'disease': 1, 'and': 1, 'do': 1, 'not': 1, 'for': 1, 'what': 1, 'ails': 1, 'me': 1}, 'an': {'i': 1, 'am': 1, 'unattractive': 1, 'man': 1}, 'my': {'i': 2, 'believe': 1, 'liver': 1, 'is': 1, 'diseased': 1, 'however': 1, 'know': 2, 'nothing': 1, 'at': 1, 'all': 1, 'about': 1, 'disease': 1, 'and': 1, 'do': 1, 'not': 1, 'for': 1, 'certain': 1, 'what': 1, 'ails': 1, 'me': 1}, 'a': {'i': 2, 'am': 2, 'sick': 1, 'man': 2, 'spiteful': 1}, 'am': {'i': 3, 'a': 2, 'sick': 1, 'man': 3, 'spiteful': 1, 'an': 1, 'unattractive': 1}, 'however': {'i': 1, 'know': 2, 'nothing': 1, 'at': 1, 'all': 1, 'about': 1, 'my': 1, 'disease': 1, 'and': 1, 'do': 1, 'not': 1, 'for': 1, 'certain': 1, 'what': 1, 'ails': 1, 'me': 1}, 'about': {'however': 1, 'i': 1, 'know': 2, 'nothing': 1, 'at': 1, 'all': 1, 'my': 1, 'disease': 1, 'and': 1, 'do': 1, 'not': 1, 'for': 1, 'certain': 1, 'what': 1, 'ails': 1, 'me': 1}, 'not': {'however': 1, 'i': 1, 'know': 2, 'nothing': 1, 'at': 1, 'all': 1, 'about': 1, 'my': 1, 'disease': 1, 'and': 1, 'do': 1, 'for': 1, 'certain': 1, 'what': 1, 'ails': 1, 'me': 1}, 'for': {'however': 1, 'i': 1, 'know': 2, 'nothing': 1, 'at': 1, 'all': 1, 'about': 1, 'my': 1, 'disease': 1, 'and': 1, 'do': 1, 'not': 1, 'certain': 1, 'what': 1, 'ails': 1, 'me': 1}, 'liver': {'i': 1, 'believe': 1, 'my': 1, 'is': 1, 'diseased': 1}, 'know': {'however': 1, 'i': 1, 'nothing': 1, 'at': 1, 'all': 1, 'about': 1, 'my': 1, 'disease': 1, 'and': 1, 'do': 1, 'not': 1, 'for': 1, 'certain': 1, 'what': 1, 'ails': 1, 'me': 1}, 'at': {'however': 1, 'i': 1, 'know': 2, 'nothing': 1, 'all': 1, 'about': 1, 'my': 1, 'disease': 1, 'and': 1, 'do': 1, 'not': 1, 'for': 1, 'certain': 1, 'what': 1, 'ails': 1, 'me': 1}, 'all': {'however': 1, 'i': 1, 'know': 2, 'nothing': 1, 'at': 1, 'about': 1, 'my': 1, 'disease': 1, 'and': 1, 'do': 1, 'not': 1, 'for': 1, 'certain': 1, 'what': 1, 'ails': 1, 'me': 1}, 'disease': {'however': 1, 'i': 1, 'know': 2, 'nothing': 1, 'at': 1, 'all': 1, 'about': 1, 'my': 1, 'and': 1, 'do': 1, 'not': 1, 'for': 1, 'certain': 1, 'what': 1, 'ails': 1, 'me': 1}, 'ails': {'however': 1, 'i': 1, 'know': 2, 'nothing': 1, 'at': 1, 'all': 1, 'about': 1, 'my': 1, 'disease': 1, 'and': 1, 'do': 1, 'not': 1, 'for': 1, 'certain': 1, 'what': 1, 'me': 1}, 'me': {'however': 1, 'i': 1, 'know': 2, 'nothing': 1, 'at': 1, 'all': 1, 'about': 1, 'my': 1, 'disease': 1, 'and': 1, 'do': 1, 'not': 1, 'for': 1, 'certain': 1, 'what': 1, 'ails': 1}, 'what': {'however': 1, 'i': 1, 'know': 2, 'nothing': 1, 'at': 1, 'all': 1, 'about': 1, 'my': 1, 'disease': 1, 'and': 1, 'do': 1, 'not': 1, 'for': 1, 'certain': 1, 'ails': 1, 'me': 1}, 'is': {'i': 1, 'believe': 1, 'my': 1, 'liver': 1, 'diseased': 1}}
{'i': 3, 'am': 3, 'a': 2, 'sick': 1, 'spiteful': 1, 'an': 1, 'unattractive': 1}
I am trying to write a python function where for each key (the dates), the value would be the sum of that day's result and the previous day(s) (sort of following the same logic as the fibonacci sequence).
For example, I have:
{20200516: {'Level1': 0, 'Level2': 1, 'Level3': 0, 'Level4': 0}, 20200517: {'Level1': 0, 'Level2': 0, 'Level3': 0, 'Level4': 1}, 20200518: {'Level1': 1, 'Level2': 0, 'Level3': 0, 'Level4': 0}, 20200519: {'Level1': 0, 'Level2': 1, 'Level3': 0, 'Level4': 1}}
but I want to have:
{20200516: {'Level1': 0, 'Level2': 1, 'Level3': 0, 'Level4': 0}, 20200517: {'Level1': 0, 'Level2': 1, 'Level3': 0, 'Level4': 1}, 20200518: {'Level1': 1, 'Level2': 1, 'Level3': 0, 'Level4': 1}, 20200519: {'Level1': 1, 'Level2': 2, 'Level3': 0, 'Level4': 2}
What I have done until now:
def summing(d):
'''
each key after the first one is the sum of the one before and its own result
>>> {20200516: {'Level1': 0, 'Level2': 1, 'Level3': 0, 'Level4': 0}, 20200517: {'Level1': 0,
'Level2': 0, 'Level3': 0, 'Level4': 1}, 20200518: {'Level1': 1, 'Level2': 0, 'Level3':
0, 'Level4': 0}, 20200519: {'Level1': 0, 'Level2': 1, 'Level3': 0, 'Level4': 1}}
{20200516: {'Level1': 0, 'Level2': 1, 'Level3': 0, 'Level4': 0}, 20200517: {'Level1': 0,
'Level2': 1, 'Level3': 0, 'Level4': 1}, 20200518: {'Level1': 1, 'Level2': 1, 'Level3': 0, '
Level4': 1}, 20200519: {'Level1': 1, 'Level2': 2, 'Level3': 0, 'Level4': 2}
'''
#STILL IN PROGRESS
c={}
for key in d:
if key == 20200516:
c[20200516]=d[20200516]
else:
c[key]=d[key-1]+d[key]
return c
You made a good effort, but you can't just add dicts like that. Here's a minimal change to get from your input to desired output, by using dict comprehension to add the value for each entry in the daily record:
from pprint import pprint
def summing_oneday(d1, d2):
return {key: d1[key] + d2[key] for key in d2}
def summing(data):
result = {}
for day in sorted(data.keys()):
if not result:
result[day] = data[day]
else:
result[day] = summing_oneday(previous, data[day])
previous = result[day]
return result
data = {20200516: {'Level1': 0, 'Level2': 1, 'Level3': 0, 'Level4': 0}, 20200517: {'Level1': 0, 'Level2': 0, 'Level3': 0, 'Level4': 1}, 20200518: {'Level1': 1, 'Level2': 0, 'Level3': 0, 'Level4': 0}, 20200519: {'Level1': 0, 'Level2': 1, 'Level3': 0, 'Level4': 1}}
pprint(summing(data))
I'm assuming all the keys are present on all the daily records. Otherwise we'll have to deal with that.
The transaction csv looks like this and I add them to list as shown below.
Bread Milk
Bread Diapers Beer Eggs Beer
[{'Bread': 1, 'Milk': 1, '': 7}, {'Bread': 1, 'Diapers': 1, 'Beer': 6, 'Eggs': 1}, {'Milk': 1, 'Diapers': 1, 'Beer': 6, 'Cola': 1}, {'Bread': 1, 'Milk': 1, 'Diapers': 1, 'Beer': 6}, {'Bread': 1, 'Milk': 1, 'Diapers': 2, 'Cola': 1, 'Chips': 2, 'Beer': 1, '': 1}, {'Bread': 1, 'Milk': 1, '': 7}, {'Bread': 1, 'Cola': 1, 'Beer': 3, 'Milk': 1, 'Chips': 1, 'Diapers': 3, '': 1}, {'Milk': 1, 'Bread': 1, 'Beer': 4, 'Cola': 1, 'Diapers': 1, 'Chips': 1}, {'Bread': 1, 'Milk': 2, 'Diapers': 2, 'Beer': 2, 'Chips': 2}, {'Bread': 2, 'Beer': 3, 'Diapers': 3, 'Milk': 1}]
I would like to consider only the list which contains the count 3 Diapers.
I would expect the transactions to return only as shown below:
{'Bread': 2, 'Beer': 3, 'Diapers': 3, 'Milk': 1}
{'Bread': 1, 'Cola': 1, 'Beer': 3, 'Milk': 1, 'Chips': 1, 'Diapers': 3, '': 1}
{'Bread', 'Beer', 'Diapers', 'Milk'}
{'Bread', 'Cola', 'Beer', 'Milk', 'Chips', 'Diapers', ''}
The code i have is:
def M():
li = []
# Open the csv file
with open('transaction.csv') as fp:
DataCaptured = csv.reader(fp, delimiter=',')
# Iterate through each word in csv and add it's counter to the row
for row in DataCaptured:
li.append(dict(Counter(row)))
if li['Diaper']==3: ---> I am missing this logic not sure how to get it.
# Return the list of counters
return li
print(M())
li=[{'Bread': 1, 'Milk': 1, '': 7}, {'Bread': 1, 'Diapers': 1, 'Beer': 6, 'Eggs': 1}, {'Milk': 1, 'Diapers': 1, 'Beer': 6, 'Cola': 1}, {'Bread': 1, 'Milk': 1, 'Diapers': 1, 'Beer': 6}, {'Bread': 1, 'Milk': 1, 'Diapers': 2, 'Cola': 1, 'Chips': 2, 'Beer': 1, '': 1}, {'Bread': 1, 'Milk': 1, '': 7}, {'Bread': 1, 'Cola': 1, 'Beer': 3, 'Milk': 1, 'Chips': 1, 'Diapers': 3, '': 1}, {'Milk': 1, 'Bread': 1, 'Beer': 4, 'Cola': 1, 'Diapers': 1, 'Chips': 1}, {'Bread': 1, 'Milk': 2, 'Diapers': 2, 'Beer': 2, 'Chips': 2}, {'Bread': 2, 'Beer': 3, 'Diapers': 3, 'Milk': 1}]
for d in li:
if 'Diapers' in d and d['Diapers']==3:
print(d)
OUTPUT:
{'Bread': 1, 'Cola': 1, 'Beer': 3, 'Milk': 1, 'Chips': 1, 'Diapers': 3, '': 1}
{'Bread': 2, 'Beer': 3, 'Diapers': 3, 'Milk': 1}
Is there a cleaner/more pythonic way of summing the contents of a list of nested dicts? Here's what I'm doing, but I suspect that there may be a better way:
list_of_nested_dicts = [{'class1': {'TP': 1, 'FP': 0, 'FN': 2}, 'class2': {'TP': 0, 'FP': 0, 'FN': 0}, 'class3': {'TP': 0, 'FP': 0, 'FN': 0}, 'class4': {'TP': 1, 'FP': 0, 'FN': 2}},
{'class1': {'TP': 1, 'FP': 0, 'FN': 2}, 'class2': {'TP': 0, 'FP': 0, 'FN': 0}, 'class3': {'TP': 0, 'FP': 0, 'FN': 0}, 'class4': {'TP': 1, 'FP': 0, 'FN': 2}},
{'class1': {'TP': 1, 'FP': 0, 'FN': 2}, 'class2': {'TP': 0, 'FP': 0, 'FN': 0}, 'class3': {'TP': 0, 'FP': 0, 'FN': 0}, 'class4': {'TP': 1, 'FP': 0, 'FN': 2}},
{'class1': {'TP': 1, 'FP': 0, 'FN': 2}, 'class2': {'TP': 0, 'FP': 0, 'FN': 0}, 'class3': {'TP': 0, 'FP': 0, 'FN': 0}, 'class4': {'TP': 1, 'FP': 0, 'FN': 2}}]
total_counts = {k:{'TP': 0, 'FP': 0, 'FN': 0} for k in list_of_nested_dicts[0].keys()}
for d in list_of_nested_dicts:
for label,counts_dict in d.items():
for k,v in counts_dict.items():
total_counts[label][k] += v
print(total_counts)
(Assuming all keys are exactly the same, but values could be any integer)
You can have a slightly tighter code using collections (similar result to #blhsing)
import collections
counts = collections.defaultdict(collections.Counter)
for d in list_of_nested_dicts:
for k, v in d.items():
counts[k].update(v)
This will give you a defaultdict of counters instead of only dicts, but they behave similarly. You can also explicitly cast them to dicts at the end if you want.
{'class1': {'FN': 8, 'FP': 0, 'TP': 4},
'class2': {'FN': 0, 'FP': 0, 'TP': 0},
'class3': {'FN': 0, 'FP': 0, 'TP': 0},
'class4': {'FN': 8, 'FP': 0, 'TP': 4}}
vs
defaultdict(<class 'collections.Counter'>,
{'class1': Counter({'FN': 8, 'TP': 4, 'FP': 0}),
'class2': Counter({'TP': 0, 'FP': 0, 'FN': 0}),
'class3': Counter({'TP': 0, 'FP': 0, 'FN': 0}),
'class4': Counter({'FN': 8, 'TP': 4, 'FP': 0})})
One thing in your code that stands out as "unclean" is the fact that you are hard-coding the keys of the sub-dicts in the initialization of total_counts. You can avoid such hard-coding by using the dict.setdefault and dict.get methods as you iterate over the items of the sub-dicts instead:
total_counts = {}
for d in list_of_nested_dicts:
for label, counts_dict in d.items():
for k, v in counts_dict.items():
total_counts[label][k] = total_counts.setdefault(label, {}).get(k, 0) + v