I am parsing through two separate csv files with the goal of finding matching customerID's and dates to manipulate balance.
In my for loop, at some point there should be a match as I intentionally put duplicate ID's and dates in my csv. However, when parsing and attempting to match data, the matches aren't working properly even though the values are the same.
main.py:
transactions = pd.read_csv(INPUT_PATH, delimiter=',')
accounts = pd.DataFrame(
columns=['customerID', 'MM/YYYY', 'minBalance', 'maxBalance', 'endingBalance'])
for index, row in transactions.iterrows():
customer_id = row['customerID']
date = formatter.convert_date(row['date'])
minBalance = 0
maxBalance = 0
endingBalance = 0
dict = {
"customerID": customer_id,
"MM/YYYY": date,
"minBalance": minBalance,
"maxBalance": maxBalance,
"endingBalance": endingBalance
}
print(customer_id in accounts['customerID'] and date in accounts['MM/YYYY'])
# Returns False
if (accounts['customerID'].equals(customer_id)) and (accounts['MM/YYYY'].equals(date)):
# This section never runs
print("hello")
else:
print("world")
accounts.loc[index] = dict
accounts.to_csv(OUTPUT_PATH, index=False)
Transactions CSV:
customerID,date,amount
1,12/21/2022,500
1,12/21/2022,-300
1,12/22/2022,100
1,01/01/2023,250
1,01/01/2022,300
1,01/01/2022,-500
2,12/21/2022,-200
2,12/21/2022,700
2,12/22/2022,200
2,01/01/2023,300
2,01/01/2023,400
2,01/01/2023,-700
Accounts CSV
customerID,MM/YYYY,minBalance,maxBalance,endingBalance
1,12/2022,0,0,0
1,12/2022,0,0,0
1,12/2022,0,0,0
1,01/2023,0,0,0
1,01/2022,0,0,0
1,01/2022,0,0,0
2,12/2022,0,0,0
2,12/2022,0,0,0
2,12/2022,0,0,0
2,01/2023,0,0,0
2,01/2023,0,0,0
2,01/2023,0,0,0
Expected Accounts CSV
customerID,MM/YYYY,minBalance,maxBalance,endingBalance
1,12/2022,0,0,0
1,01/2023,0,0,0
1,01/2022,0,0,0
2,12/2022,0,0,0
2,01/2023,0,0,0
Where does the problem come from
Your Problem comes from the comparison you're doing with pandas Series, to make it simple, when you do :
customer_id in accounts['customerID']
You're checking if customer_id is an index of the Series accounts['customerID'], however, you want to check the value of the Series.
And in your if statement, you're using the pd.Series.equals method. Here is an explanation of what does the method do from the documentation
This function allows two Series or DataFrames to be compared against each other to see if they have the same shape and elements. NaNs in the same location are considered equal.
So equals is used to compare between DataFrames and Series, which is different from what you're trying to do.
One of many solutions
There are multiple ways to achieve what you're trying to do, the easiest is simply to get the values from the series before doing the comparison :
customer_id in accounts['customerID'].values
Note that accounts['customerID'].values returns a NumPy array of the values of your Series.
So your comparison should be something like this :
print(customer_id in accounts['customerID'].values and date in accounts['MM/YYYY'].values)
And use the same thing in your if statement :
if (customer_id in accounts['customerID'].values and date in accounts['MM/YYYY'].values):
Alternative solutions
You can also use the pandas.Series.isin function that given an element as input return a boolean Series showing whether each element in the Series matches the given input, then you will just need to check if the boolean Series contain one True value.
Documentation of isin : https://pandas.pydata.org/docs/reference/api/pandas.Series.isin.html
It is not clear from the information what does formatter.convert_date function does. but from the example CSVs you added it seems like it should do something like:
def convert_date(mmddyy):
(mm,dd,yy) = mmddyy.split('/')
return mm + '/' + yy
in addition, make sure that data types are also equal
(both date fields are strings and also for customer id)
Newbie here. Just as the title says, I have a list of dataframes (each dataframe is a class of students). All dataframes have the same columns. I have made certain columns global.
BINARY_CATEGORIES = ['Gender', 'SPED', '504', 'LAP']
for example. These are yes/no or male/female categories, and I have already changed all of the data to be 1's and 0's for these columns. There are several other columns which I want to ignore as I iterate.
I am trying to accept the list of classes (dataframes) into my function and perform calculations on each dataframe using only my BINARY_CATEGORIES list of columns. This is what I've got, but it isn't making it through all of the classes and/or all of the columns.
def bal_bin_cols(classes):
i = 0
c = 0
for x in classes:
total_binary = classes[c][BINARY_CATEGORIES[i]].sum()
print(total_binary)
i+=1
c+=1
Eventually I need a new dataframe from this all of the sums corresponding to the categories and the respective classes. print(total binary) is just a place holder/debugger. I don't have that code yet that will populate the dataframe from the results of the above code, but I'd like it to be the classes as the index and the total calculation as the columns.
I know there's probably a vectorized way to do this, or enum, or groupby, but I will take a fix to my loop. I've been stuck forever. Please help.
Try something like:
Firstly create a dictionary:
d={
'male':1,
'female':0,
'yes':1,
'no':0
}
Finally use replace():
df[BINARY_CATEGORIES]=df[BINARY_CATEGORIES].replace(d.keys(),d.values(),regex=True)
This is my first post to the coding community, so I hope I get the right level of detail in my request for help!
Background info:
I want to repeat (loop) command in a df using a variable that contains a list of options. While the series 'amenity_options' contains a simple list of specific items (let's say only four amenities as the example below) the df is a large data frame with many other items. My goal is the run the operation below for each item in the 'amenity_option' until the end of the list.
amenity_options = ['cafe','bar','cinema','casino'] # this is a series type with multiple options
df = df[df['amenity'] == amenity_options] # this is my attempt to select the the first value in the series (e.g. cafe) out of dataframe that contains such a column name.
df.to_excel('{}_amenity.xlsx, format('amenity') # wish to save the result (e.g. cafe_amenity) as a separate file.
Desired result:I wish to loop step one and two for each and every item available in the list (e.g. cafe, bar, cinema...). So that I will have separate excel files in the end. Any thoughts?
What #Rakesh suggested is correct, you probably just need one more step.
df = df[df['amenity'].isin(amenity_options)]
for key, g in df.groupby('amenity'):
g.to_excel('{}_amenity.xlsx'.format(key))
After you call groupby() of your df, you will get 4 groups so that you can directly loop on them.
The key is the group key, which are cafe, bar and etc. and the g is the sub-dataframe that specifically filtered by that key.
Seems like you just need a simple for loop:
for amenity in amenity_options:
df[df['amenity'] == amenity].to_excel(f"{amenity}_amenity.xlsx")
I'm trying to run a function (row_extract) over a column in my dataframe, that returns three values that I then want to add to three new columns.
I've tried running it like this
all_data["substance", "extracted name", "name confidence"] = all_data["name"].apply(row_extract)
but I get one column with all three values. I'm going to iterate over the rows, but that doesn't seem like a very efficient system - any thoughts?
This is my current solution, but it takes an age.
for index, row in all_data.iterrows():
all_data.at[index, "substance"], all_data.at[index, "extracted name"], all_data.at[index, "name confidence"] = row_extract(row["name"])
Check what the type of your function output is or what the datatypes are. It seems like that's a string.
You can use the "split" method on a string to separate them.
https://docs.python.org/2/library/string.html
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.str.split.html
Alternatively, adjust your function to return more than one value.
E.g.
def myfunc():
...
...
return x, y, z
I am currently working to process some data that are imported to Python as a dataframe that has 10000 rows and 20 columns. The columns store sample names and chemical element. The daaaframe is currently indexed by both sample name and time, appearing as so:
[1]: https://i.stack.imgur.com/7knqD.png .
From this dataframe, I want to create individual arrays for each individual sample, of which there are around 25, with a loop. I have generated an index and array of the sample names, which yields an array that appears as so
samplename = fuegodataframe.index.levels[0]
samplearray = samplename.to_numpy()
array(['AC4-EUH41', 'AC4-EUH79N', 'AC4-EUH79S', 'AC4-EUH80', 'AC4-EUH81',
'AC4-EUH81b', 'AC4-EUH82N', 'AC4-EUH82W', 'AC4-EUH84',
'AC4-EUH85N', 'AC4_EUH48', 'AC4_EUH48b', 'AC4_EUH54N',
'AC4_EUH54S', 'AC4_EUH60', 'AC4_EUH72', 'AC4_EUH73', 'AC4_EUH73W',
'AC4_EUH78', 'AC4_EUH79E', 'AC4_EUH79W', 'AC4_EUH88', 'AC4_EUH89',
'bhvo-1', 'bhvo-2', 'bir-1', 'bir-2', 'gor132-1', 'gor132-2',
'gor132-3', 'sc ol-1', 'sc ol-2'], dtype=object)
I have also created a dictionary with keys of each of these variable names. I am now wondering how I would use this dictionary to generate individual variables for each of these samples that capture all the rows in which a sample is found.
I have tried something along these lines:
for ii in sampledictionary.keys():
if ii == sampledictionary[ii]:
sampledictionary[ii] = fuegodataframe.loc[sampledictionary[ii]]
but this fails. How would I actually go about doing something like this? Is this possible?
I think you're asking how to generate variables dynamically rather than assign your output to a key in your dictionary.
In Python there is a globals function globals() that will output all the variable names defined in the document.
You can assign new variables dynamically to this dictionary
globals()[f'variablename_{ii}'] = fuegodataframe.loc[sampledictionary[ii]]
etc.
if ii was 0 then variablename_0 would be available with the assigned value.
In general this is not considered good practice but it is required sometimes.