I am trying to sum Django query with a condition. Suppose I have got some data like this:
| Name | Type |
---------------
| a | x |
| b | z |
| c | x |
| d | x |
| e | y |
| f | x |
| g | x |
| h | y |
| i | x |
| j | x |
| k | x |
| l | x |
And these types are string and they have values, like x = 1, y = 25, z = -3
How can I sum up all the values without a loop? Currently using a loop.
data = A.objects.all()
sum = 0
mapp = {'x': 1, 'y': 25, 'z': -3}
for datum in list(data):
sum = sum + mapp[datum.type]
print(sum)
To perform the calculation inside the database use Queryset.aggregate() with an aggregate expression that uses Case/When:
from django.db.models import Sum, Case, When
A.objects.all().aggregate(
amount=Sum(Case(
When(type="x", then=1),
When(type="y", then=25),
When(type="z", then=-3),
default=0,
output_field=FloatField()
))
)
You could shorthand it like:
sum(mapp.get(o['Type'],0) for o in data)
or simpler if you trust the data to have all valid types:
sum(mapp[o['Type']] for o in data)
(Don't trust the data though)
If the mapping x = 1, y = 25 etc... is coming from another table, you can use some SQL voodoo to let the database handle the summation for you. Other wise you have to (one way or another) loop through all results and sum them up.
You could also theoretically just count the distinct amount of x, y and z in the table and have sum = x*count_of_x + y*count_of_y + z*count_of_z based on how the example is structured.
Related
I have a multiindexed dataframe where the index levels have multiple categories, something like this:
|Var1|Var2|Var3|
|Level1|Level2|Level3|----|----|----|
| A | A | A | | | |
| A | A | B | | | |
| A | B | A | | | |
| A | B | B | | | |
| B | A | A | | | |
| B | A | B | | | |
| B | B | A | | | |
| B | B | B | | | |
In summary, and specifically in my case, Level 1 has 2 levels, Level 2 has 24, Level 3 has 6, and there are also Levels 4 (674) and Level 5 (9) (with some minor variation depending on specific higher-level values - Level1 == 1 actually has 24 Level2s, but Level1 == 2 has 23).
I need to generate all possible combinations of 3 at Level 5, then calculate their means for Vars 1-3.
I am trying something like this:
# Resulting df to be populated
df_result = pd.DataFrame([])
# Retrieving values at Level1
lev1s = df.index.get_level_values("Level1").unique()
# Looping through each Level1 value
for lev1 in lev1s:
# Filtering df based on Level1 value
df_lev1 = df.query('Level1 == ' + str(lev1))
# Repeating...
lev2s = df_lev1.index.get_level_values("Level2").unique()
for lev2 in lev2s:
df_lev2 = df_lev1.query('Level2 == ' + str(lev2))
# ... until Level3
lev3s = df_lev2.index.get_level_values("Level3").unique()
# Creating all combinations
combs = itertools.combinations(lev3s, 3)
# Looping through each combination
for comb in combs:
# Filtering values in combination
df_comb = df_wl.query('Level3 in ' + str(comb))
# Calculating means using groupby (groupby might not be necessary,
# but I don't believe it has much of an impact
df_means = df_comb.reset_index().groupby(['Level1', 'Level2']).mean()
# Extending resulting dataframe
df_result = df_result.append(df_means)
The thing is, after a little while, this process gets really slow. Since I have around 2 * 24 * 6 * 674 levels and 84 combinations (of 9 elements, 3 by 3), I am expecting more than 16 million df_meanss to be calculated.
Is there any more efficient way to do this?
Thank you.
I have some data as follows:
+--------+------+
| Reason | Keys |
+--------+------+
| x | a |
| y | a |
| z | a |
| y | b |
| z | b |
| x | c |
| w | d |
| x | d |
| w | d |
+--------+------+
I want to get the Reason corresponding to the first occurrence of each Key. Like here, I should get Reasons x,y,x,w for Keys a,b,c,d respectively. After that, I want to compute the percentage of each Reason, as in a metric for how many times each Reason occurs. Thus x = 2/4 = 50%. And w,y = 25% each.
For the percentage, I think I can use something like value_counts(normalize=True) * 100, based on the previous step. What is a good way to proceed?
You are right about the second step and the first step could be achieved by
summary = df.groupby("Keys").first()
You can using drop_duplicates
df.drop_duplicates(['Reason'])
Out[207]:
Reason Keys
0 x a
1 y a
2 z a
6 w d
I have data from a platform that records a users events - whether answers to polls, or clickstream data. I am trying to bring together a number of related datasets, each of which has a session_id column.
Each dataset began as a csv that was read in as a series of nested lists. Not every session will have a user answering a question, or completing certain actions, so each dataset will not contain an entry for every session -- however, every session exists in at least one of the datasets.
assume there are 5 sessions recorded:
e.g. dataset 1:
SessionID |a | b | c | d
1 | x | x | x | x
2 | x | x | x | x
5 | x | x | x | x
e.g. dataset 2:
SessionID |e | f | g | h
1 | x | x | x | x
3 | x | x | x | x
5 | x | x | x | x
e.g. dataset 3:
SessionID |i | j | k | l
2 | x | x | x | x
3 | x | x | x | x
4 | x | x | x | x
How would I construct this:
SessionID |a | b | c | d | e | f | h | i |j | k | l
1 | x | x | x | x | x | x | x | x | - | - | - | -
2 | x | x | x | x | - | - | - | - | x | x | x | x
3 | - | - | - | - | x | x | x | x | x | x | x | x
4 | - | - | - | - | - | - | - | - | x | x | x | x
5 | x | x | x | x | x | x | x | x | - | - | - | -
By far the easiest way to do this is to import each csv into pandas:
merged_df = pd.merge(dataset1, dataset2, how = 'outer', on="sessionID")
pd.merge(merged_df, dataset3, how = 'outer', on="sessionID")
however the requirements are that I not use any external libraries.
I'm struggling to find a workable logic to detect gaps in the sessionID, and then pad the lists with null data so the three lists would be simply added together.
Any ideas?
How do you define "external libraries"? Does sqlite3 qualify as external or internal?
If it doesn't and you want to think about the problem in terms of relational operations, you could slam your tables into a sqlite3 file and take it from there.
If the number of datasets is finite, you could create a class Session, containing a dictionary where each column (a to j) would be a key. If you are proficient, you could use the __getattr__ function to use a "dot" notation when you need it. For the "table", I would simply use a dictionary, with the key as the id, then fill up your dictionary in three steps (dataset1, dataset2, dataset3). In this way you wouldn't have to worry about gaps.
So I have a dataframe with some values. This is my dataframe:
|in|x|y|z|
+--+-+-+-+
| 1|a|a|b|
| 2|a|b|b|
| 3|a|b|c|
| 4|b|b|c|
I would like to get number of unique values of each row, and number of values that are not equal to value in column x. The result should look like this:
|in | x | y | z | count of not x |unique|
+---+---+---+---+---+---+
| 1 | a | a | b | 1 | 2 |
| 2 | a | b | b | 2 | 2 |
| 3 | a | b | c | 2 | 3 |
| 4 | b | b |nan| 0 | 1 |
I could come up with some dirty decisions here. But there must be some elegant way of doing that. My mind is turning around dropduplicates(that does not work on series); turning into array and .unique(); df.iterrows() that I want to evade; and .apply on each row.
Here are solutions using apply.
df['count of not x'] = df.apply(lambda x: (x[['y','z']] != x['x']).sum(), axis=1)
df['unique'] = df.apply(lambda x: x[['x','y','z']].nunique(), axis=1)
A non-apply solution for getting count of not x:
df['count of not x'] = (~df[['y','z']].isin(df['x'])).sum(1)
Can't think of anything great for unique. This uses apply, but may be faster, depending on the shape of the data.
df['unique'] = df[['x','y','z']].T.apply(lambda x: x.nunique())
I want to read in T1 and write it out as T2 (note both are .csv).
T1 contains duplicate rows; I don't want to write duplicates in T2.
T1
+------+------+---------+---------+---------+
| Type | Year | Value 1 | Value 2 | Value 3 |
+------+------+---------+---------+---------+
| a | 8 | x | y | z |
| b | 10 | q | r | s |
+------+------+---------+---------+---------+
T2
+------+------+---------+-------+
| Type | Year | Value # | Value |
+------+------+---------+-------+
| a | 8 | 1 | x |
| a | 8 | 2 | y |
| a | 8 | 3 | z |
| b | 10 | 1 | q |
| ... | ... | ... | ... |
+------+------+---------+-------+
Currently, I have this excruciatingly slow code to filter out duplicates:
no_dupes = []
for row in reader:
type = row[0]
year = row[1]
index = type,age
values_list = row[2:]
if index not in no_dupes:
for i,j in enumerate(values_list):
line = [type, year, str(i+1), str(j)]
writer.writerow(line) #using csv module
no_dupes.append(index)
I cannot exagerate how slow this code is when T1 gets large.
Is there a faster way to filter out duplicates from T1 as I write to T2?
I think you want something like this:
no_dupes = set()
for row in reader:
type, year = row[0], row[1]
values_list = row[2:]
for index, value in enumerate(values_list, start=1):
line = (type, year, index, value)
no_dupes.add(line)
for t in no_dupes:
writer.writerow(t)
If possible convert reader to a set and iterate over the set instead, then there is no possibility of dups