I have a list of dictionaries
example_list = [{'email':'myemail#email.com'},{'email':'another#email.com'}]
and a dataframe with an 'Email' column
I need to compare the list against the dataframe and return the values that are not in the dataframe.
I can certainly iterate over the list, check in the dataframe, but I was looking for a more pythonic way, perhaps using list comprehension or perhaps a map function in dataframes?
To return those values that are not in DataFrame.email, here's a couple of options involving set difference operations—
np.setdiff1d
emails = [d['email'] for d in example_list)]
diff = np.setdiff1d(emails, df['Email']) # returns a list
set.difference
# returns a set
diff = set(d['email'] for d in example_list)).difference(df['Email'])
One way is to take one set from another. For a functional solution you can use operator.itemgetter:
from operator import itemgetter
res = set(map(itemgetter('email'), example_list)) - set(df['email'])
Note - is syntactic sugar for set.difference.
I ended up converting the list into a dataframe, comparing the two dataframes by merging them on a column, and then creating a dataframe out of the missing values
so, for example
example_list = [{'email':'myemail#email.com'},{'email':'another#email.com'}]
df_two = pd.DataFrame(item for item in example_list)
common = df_one.merge(df_two, on=['Email'])
df_diff = df_one[(~df_one.Email.isin(common.Email))]
Related
Description
I have 2 lists
List1=['curentColumnName1','curentColumnName2','currentColumnName3']
List2=['newColumnName1','newColumnName2','newColumnName3']
Their is a dataframe df which contains all columns
I want to check like if column 'curentColumnName1 is present in dataframe,if yes then rename it to newColumnName1
Need to do this for all columns if those are present in dataframe
How to achieve this scenario using pyspark
Just iterate over the first list, check if it in the column list, and rename:
for i in range(len(List1)):
if List1[i] in df.columns:
df = df.withColumnRenamed(List1[i], List2[i])
P.S. Instead of two lists, it's better to use dictionary - it's easier to maintain, and you can avoid errors when you add/remove elements only in one list
Here is anotherway of doing it in one-liner :
from functools import reduce
df = reduce(
lambda a, b: a.withColumnRenamed(b[0], b[1]),
zip(List1, List2),
df,
)
You can achieve in one line:
df.selectExpr(*[f"{old_col} AS {new_col}" for old_col, new_col in zip(List1, List2)]).show()
I have 4 different dataframes containing time series data that all have the same structure.
My goal is to take each individual dataframe and pass it through a function I have defined that will group them by datestamp, sum the columns and return a new dataframe with the columns I want. So in total I want 4 new dataframes that have only the data I want.
I just looked through this post:
Loop through different dataframes and perform actions using a function
but applying this did not change my results.
Here is my code:
I am putting the dataframes in a list so I can iterate through them
dfs = [vds, vds2, vds3, vds4]
This is my function I want to pass each dataframe through:
def VDS_pre(df):
df = df.groupby(['datestamp','timestamp']).sum().reset_index()
df = df.rename(columns={'datestamp': 'Date','timestamp':'Time','det_vol': 'VolumeVDS'})
df = df[['Date','Time','VolumeVDS']]
return df
This is the loop I made to iterate through my dataframe list and pass each one through my function:
for df in dfs:
df = VDS_pre(df)
However once I go through my loop and go to print out the dataframes, they have not been modified and look like they initially did. Thanks for the help!
However once I go through my loop and go to print out the dataframes, they have not been modified and look like they initially did.
Yes, this is actually the case. The reason why they have not been modified is:
Assignment to an item in a for item in lst: loop does not have any effect on both the lst and the identifier/variables from which the lst items got their values as it is demonstrated with following code:
v1=1; v2=2; v3=3
lst = [v1,v2,v3]
for item in lst:
item = 0
print(lst, v1, v2, v3) # gives: [1, 2, 3] 1 2 3
To achieve the result you expect to obtain you can use a list comprehension and the list unpacking feature of Python:
vds,vds2,vds3,vds4=[VDS_pre(df) for df in [vds,vds2,vds3,vds4]]
or following code which is using a list of strings with the identifier/variable names of the dataframes:
sdfs = ['vds', 'vds2', 'vds3', 'vds4']
for sdf in sdfs:
exec(str(f'{sdf} = VDS_pre(eval(sdf))'))
Now printing vds, vds2, vds3 and vds4 will output the modified dataframes.
Pandas frame operations return new copy of data. Your snippet store the result in df variable which is not stored or updated to your initial list. This is why you don't have any stored result after execution.
If you don't need to keep original frames, you may simply overwrite them:
for i, df in enumerate(dfs):
dfs[i] = VDS_pre(df)
If not just use a second list and append result to it.
l = []
for df in dfs:
df2 = VDS_pre(df)
l.append(df2)
Or even better use list comprehension to rewrite this snippet into a single line of code.
Now you are able to store the result of your processing.
Additionally if your frames have the same structure and can be merged as a single frame, you may consider to first concat them and then apply your function on it. That would be totally pandas.
I want to create n DataFrames using the value s as the name of each DataFrame, but I only could create a list full of DataFrames. It's possible to change this list in each of the DataFrames inside it?
#estacao has something like [ABc,dfg,hil,...,xyz], and this should be the name of each DataFrame
estacao = dados.Station.unique()
for s,i in zip(estacao,range(126)):
estacao[i] = dados.groupby('Station').get_group(s)
I'd use a dictionary here. Then you can name the keys with s and the values can each be the dataframe corresponding to that group:
groups = dados.Station.unique()
groupby_ = datos.groupby('Station')
dataframes = {s: groupby_.get_group(s) for s in groups}
Then calling each one by name is as simple as:
group_df = dataframes['group_name']
If you REALLY NEED to create DataFrames named after s (which I named group in the following example), using exec is the solution.
groups = dados.Station.unique()
groupby_ = dados.groupby('Station')
for group in groups:
exec(f"{group} = groupby_.get_group('{group:s}')")
CAVEAT
See this answer to understand why using exec and eval commands is not always desirable.
Why should exec() and eval() be avoided?
I've a DataFrame df of let us say one thousand rows, and I'd like to split it to 10 lists where each list contains a DataFrame of 100 rows. So list
zero = df[0:99]
one = df[100:199]
two = df[200:299]
...
nine = df[900:900]
What could be a good (preferably) oneliner for this?
Assuming index is a running integer (can use .reset_index() if not)
[d for g,d in df.groupby(df.index//100)]
Returns a list of dataframes.
Like this maybe:
list_of_dfs = [df.loc[i:i+size-1,:] for i in range(0, len(df),1000)]
I have about 30 columns for which I am looking to change a value if that value is in that row for a column of lists. This is a little hard to describe verbally so here is some code for what I'm talking about:
test = test.groupby('RealId')['Player'].apply(list).reset_index(name='Invalids')
test.index = te["RealId"]
test.drop("RealId", axis='columns', inplace=True)
test = test.join(te, on="RealId", how="left")
test['PA_14'].isin(test['Invalids'])
The PA_14 column is a normal Series of strings, while Invalids is a Series of lists of strings. What I would like is for that last line to output a boolean vector, but isin() doesn't appear to work with a series of lists. How I do this relatively quickly considering that it needs to be done 30 more times?
Your last line actually tests for each element (str) of test['PA_14'] if it is one of the elements (list) of test['Invalids'].
You can try:
boolean_index = [s in list_s for (s, list_s) in zip(test['PA_14'], test['Invalids'])]