Develop Reference

Is there a way to compare two dataframes and report which column is different in Pyspark?

I'm using df1.subtract(df2).rdd.isEmpty() to compare two dataframes (assuming the schema of these two df are the same, or at least we expect them to be the same), but if one of the column doesn't match, I can't tell from the output logs, and it takes long time for me to find out the issue in the data (and it's exhausting when the datasets are quite big)
Is there a way that we can compare two df and return which column doesn't match with Pyspark? Thanks a lot.

You could use the chispa library, it is a great tool for comparing data frames.

Rearranging CSV rows to match another CSV based on data in multiple columns

I am wondering if there is a fast way to rearrange the rows of a csv using pandas so that it could match the order of the rows in another csv that have the same data, but arranged differently. To be clear, these two csvs have the same data in the form of several numeric features spread across several columns. I tried doing loops that matches each row of data with its counterpart by comparing the values in multiple columns, but this prove too slow for my purposes.

You should use pandas DataFrame:
"read_csv"__both files.
Convert both to "DataFrame".
Use "merge".
"to_csv"__use to save.
share your data..

Pandas cross merge while combining a number of columns (Python)

I am working in python's pandas with 4 dataframes, each of which have a column of unique identifiers followed by multiple columns of shared attributes. I'd like to generate the cartesian product of the first column, but sum the remaining columns of shared attributes for each unique combination.
I have a working version of that that separates the first column out into a separate frame, performs the cross merge, and then looks up the shared attributes for each of its unique elements but it is horribly slow. There has got to be a smarter way to do this. Anyone have any tips?

Iterating through big data with pandas, large and small dataframes

This is my first post here and it’s based upon an issue I’ve created and tried to solve at work. I’ll try to precisely summarize my issue as I’m having trouble wrapping my head around a preferred solution. #3 is a real stumper for me.
Grab a large data file based on a parquet - no problem
Select 5 columns from the parquet and create a dataframe - no problem
import pandas
df = pd.read_parquet(’/Users/marmicha/Downloads/sample.parquet’,
columns=["ts", "session_id", "event", "duration", "sample_data"])
But here is where it gets a bit tricky for me. One column(a key column) is called "session_id" . Many values are unique. Many duplicate values(of session_id) exist and have multiple associated entry rows of data. I wish to iterate through the master dataframe, create a unique dataframe per session_id. Each of these unique (sub) dataframes would have a calculation done that simply gets the SUM of the "duration" column per session_id. Again that SUM would be unique per unique session_id, so each sub dataframe would have it's own SUM with a row added with that total listed along with the session_id I'm thinking there is a nested loop formula that will work for me but every effort has been a mess to date.
Ultimately, I'd like to have a final dataframe that is a collection of these unique sub dataframes. I guess I'd need to define this final dataframe, and append it with each new sub dataframe as I iterate through the data. I should be able to do that simply
Finally, write this final df to a new parquet file. Should be simple enough so I won't need help with that.
But that is my challenge in a nutshell. The main design I’d need help with is #3. I’ve played with interuples and iterows

I think the groupby function will work:
df.groupby('session_id')['duration'].sum()
More info here:
https://pandas.pydata.org/pandas-docs/stable/user_guide/groupby.html

Pandas DataFrames in Dict

I have an excel file with several tabs, each reporting quarterly account values.
I want to create a dataframe to group_by accounts and report by Period.
I manage to generate a period_index and read the file with pd.ExcelFileParse (https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.ExcelFile.parse.html) and get a dict with tab names as keyes and the period-data as dataframes. So far, good.
Now, when I loop through the dict to either append the different data frames or concatenate them, Pandas generates errors that only series and DataFrames can do that, not the dict object.
How can I generate one DF from the different dataframes in the dict?
thanks in advance,
kind regards,
Marc