Develop Reference

How to assign a dataframe variable based on a dictionary key in a loop

My searching was unable to find a solution for this one. I hope it is simple and just missed it.
I am trying to assign a dataframe variable based on a dictionary key. I want to loop through a dictionary of keys 0, 1, 2 3... and save the dataframe as df_0, df_1, df_2 ... I am able to get the key and values working and can assign one dataframe, but cannot find a way to assign new dataframes based on the keys.
I tried How to create a new dataframe with every iteration of for loop in Python but it didn't seem to work.
Here is what I tried:
docs_dict = {0: '2635_base', 1: '2635_tri'}
for keys, docs in docs_dict.items():
print(keys, docs)
df = pd.read_excel(Path(folder_loc[docs]) / file_name[docs], sheet_name=sheet_name[docs], skiprows=3)}
Output: 0 2635_base 1 2635_tri from the print statement, and %whos DataFrame > df as excepted.
What I would like to get is: df_0 and df_1 based on the excel files in other dictionaries which work fine.
df[keys] = pd.read_excel(Path(folder_loc[docs]) / file_name[docs], sheet_name=sheet_name[docs], skiprows=3)
produces a ValueError: Wrong number of items passed 26, placement implies 1
SOLVED thanks to RubenB for pointing me to How do I create a variable number of variables? and answer by #rocky-li using globals()
for keys, docs in docs_dict.items():
print(keys, docs)
globals()['df_{}'.format(keys)] = pd.read_excel(...)}
>> Output: dataframes df_0, df_1, ...

You might want to try a dict comprehension as such (substitute pd.read_excel(...docs...) with whatever you need to read the dataframe from disc):
docs_dict = {0: '2635_base', 1: '2635_tri'}
dfs_dict = {k: pd.read_excel(...docs...) for k, docs in docs_dict.items()}

Python---loop over a dictionary with dataframes

I have a dictionary of dataframes
list_of_dfs={'df1:Dataframe','df2:Dataframe','df3:Dataframe','df4:Dataframe'}
Each data frame contains the same variables (price, volume, price,"Sell/Purchase") that I want to manipulate to end up with a new subset of DataFrames. My new dataframes have to filter the variable called "Sell/Purchase" by the observations that have "Sell" in the variable.
sell=df[df["Sale/Purchase"]=="Sell"]
My question is how do I loop over the dictionary in order to get a new dictionary with this new subset?
I dont know how to write this command to do the loop. I know it has to start like this:
# Create an empty dictionary called new_dfs to hold the results
new_dfs = {}
# Loop over key-value pair
for key, df in list_of_dfs.items():
But then due to my small knowledge of looping over a dictionary of dataframes I dont know how to write the filter command. I would be really thankful if someone can help me.
Thanks in advance.

Try this,
dict_of_dfs={'df1':'Dataframe','df2':'Dataframe','df3':'Dataframe','df4':'Dataframe'}
# Create an empty dictionary called new_dfs to hold the results
new_dfs = {}
# Loop over key-value pair
for key, df in dict_of_dfs.items():
new_dfs[key] = df[df["Sale/Purchase"]=="Sell"]
Explanation:
new_dfs = {} # Here we have created a empty dictionary.
# dictionary contains keys and values.
# to add keys and values to our dictionary,
# we need to do it as shown below,
new_dfs[our_key_1] = our_value_2
new_dfs[our_key_2] = our_value_2
.
.
.

You can map a function:
lambda df: df[df["Sale/Purchase"] == "Sell"]
HOW:
Syntax = map(fun, iter)
map(lambda df: df[df["Sale/Purchase"] == "Sell"], list_of_dfs)
You can map it on the a list, or set
For dict:
df_dict = {k: df[df["Sale/Purchase"]=="Sell"] for k, df in list_of_dfs.items()}

Something like:
sells = {k: v for (k, v) in list_of_df.items() if v["Sale/Purchase"] == "Sell"}
This pattern is called dictionary comprehension. According to this question this is the fastest and most Pythonic approach.
You should provide an example of the data you are dealing with for more precise answer.

Assign dict key as dataframe name and value as dataframe itself

I have one dictionary of several pandas dataframes. It looks like this:
key Value
A pandas dataframe here
B pandas dataframe here
C pandas dataframe here
I need to extract dataframes from dict as a separate part and assign dict key as a name.
Desired output should be as many separate dataframes as many values of my dict have.
A = dict.values() - this is first dataframe
B = dict.values() - this is second dataframe
Note that dataframes names are dict keys.
I tried this code but without any success.
for key, value in my_dict_name.items():
key = pd.DataFrame.from_dict(value)
Any help would be appreciated.

It is not recommended, but possible:
Thanks # Willem Van Onsem for better explanation:
It is a quite severe anti-pattern, especially since it can override existing variables, and one can never exclude that scenario
a = pd.DataFrame({'a':['a']})
b = pd.DataFrame({'b':['b']})
c = pd.DataFrame({'c':['c']})
d = {'A':a, 'B':b, 'C':c}
print (d)
{'A': a
0 a, 'B': b
0 b, 'C': c
0 c}
for k, v in d.items():
globals()[k] = v
print (A)
a
0 a
I think here the best is MultiIndex if same columns or index values in each DataFrame, also dictionary of DataFrame is perfectly OK.

python dictionary with multiple keys as tuple and df values

Below id my DataFrame named df2
ID,'AI','DB','ML','Python','IR'
0,1,1,0,0,0
1,1,1,1,0,1
2,1,1,0,1,1
3,1,0,1,0,1
I want to create a dictionary such that the first row and index become tuple and the values become values of the dictionary, something like
{
(0,"AI"):1,(0,"DB"):1,(0,"ML"):0,(0,"Python"):0,(0,"IR"):0,
(1,"AI"):1,(1,"DB"):1,(1,"ML"):1,(1,"Python"):0,(1,"IR"):0,
(2,"AI"):1,(2,"DB"):1,(2,"ML"):1,(2,"Python"):0,(2,"IR"):1,
(3,"AI"):1,(3,"DB"):0,(3,"ML"):1,(3,"Python"):0,(3,"IR"):1,
}
My trial so far your_dict = dict(zip(es, df2))
print(your_dict) but,does not produce my desired output

Create every combination of indices/columns, then create a dictionary with the key as a tuple of these values and the value is the value in the dataframe at that postion.
import itertools
{(x,y):df[y][x] for x, y in itertools.product(df.index, df.columns)}

Building a dataframe in an efficient way from dictionary

I have large set of data that I have process and generated a dictionary. Now I want to create a dataframe from this dictionary. Vales of the dictionary are list of tuples. From those values I need to find out the unique values to build the columns of the dataframe:
d = {'0001': [('skiing',0.789),('snow',0.65),('winter',0.56)],'0002': [('drama', 0.89),('comedy', 0.678),('action',-0.42) ('winter',-0.12),('kids',0.12)],'0003': [('action', 0.89),('funny', 0.58),('sports',0.12)],'0004': [('dark', 0.89),('Mystery', 0.678),('crime',0.12), ('adult',-0.423)],'0005': [('cartoon', -0.89),('comedy', 0.678),('action',0.12)],'0006': [('drama', -0.49),('funny', 0.378),('Suspense',0.12), ('Thriller',0.78)],'0007': [('dark', 0.79),('Mystery', 0.88),('crime',0.32), ('adult',-0.423)]}
(size of the dictionary close to 800,000 records)
I iterate over the dictionary to find out the unique headers:
col_headers = []
entities = []
for key, scores in d.iteritems():
entities.append(key)
d[key] = dict(scores)
col_headers.extend(d[key].keys())
col_headers = list(set(col_headers))
I believe this take long time to process. Using dict might also be an issue since its much slower. Further more when I construct the data frame raw by raw it further slows down the process:
df = pd.DataFrame(columns=col_headers, index=entities)
for k in d:
df.loc[k] = pd.Series(d[k])
df.fillna(0.0, axis=1)
How can I speed up this process to reduce to the process time?

#ajcr almost gets it.
But you probably also need to unwrap the internal key-value pairs into a dictionary along the way.
df = pd.DataFrame.from_dict({ k: dict(v) for k,v in d.items() },
orient="index").fillna(0)
Then optionally, if you want to homogenize the style of column titles:
df.columns = [c.lower() for c in df.columns]
If you wanted to go entirely crazy, you could then sort the columns:
df = df.sort(axis=1)