I have two dataframe df and buyers.
I need to apply two different type of join inner and leftanti on them and take 1% sample from leftanit and then do the union of these two resultant. I tried the below
buyr = df.join(buyers,on=['key'],how='inner')
non_buyr = df.join(buyers,on=['key'],how='leftanti')
onepct = non_buyr.sample(False,0.01,seed=100)
df_final = buyr.unionAll(onepct)
But due to this we have two stages for df and each stage of 731 partition take around 4hr to complete
Is there any way to perform the inner and left anti join in single step instead of two or any other efficient method to perform the same ?
The Inner + Left anti is equal to a simple Left join.
Then, you can filter the line where F.col("buyers.key").isNotNull() # The inner and F.col("buyers.key").isNull() # The left anti.
Related
If I have two tables, I can easily combine them in SQL using something like
SELECT a.*, b.* FROM table_1 a, table_2 b
WHERE (a.id < 1000 OR b.id > 700)
AND a.date < b.date
AND (a.name = b.first_name OR a.name = b.last_name)
AND (a.location = b.origin OR b.destination = 'home')
and there could be many more conditions. Note that this is just an example and the set of conditions may be anything.
The two easiest solutions in pandas that support any set of conditions are:
Compute a cross product of the tables and then filter one condition at a time.
Loop over one DataFrame (apply, itertuples, ...) and filter the second DataFrame in each iteration. Append the filtered DataFrames from each iteration.
In case of huge datasets (at least a few million rows per DataFrame), the first solution is impossible because of the required memory and the second one is considered an anti-pattern (https://stackoverflow.com/a/55557758/2959697). Either solution will be rather slow.
What is the pandaic way to proceed in this general case?
Note that I am not only interested in a solution to this particular problem but in the general concept of how to translate these types of statements. Can I use pandas.eval? Is it possible to perform a "conditional merge"? Etc.
I learned join methods in sql, and I know that inner join means returning only the intersections of the two different tables that we want to set.
I thought for python the concept is same. But I have problem understanding the certain code.
crsp1=pd.merge(crsp, crsp_maxme, how='inner', on=['jdate','permco','me'])
crsp1=crsp1.drop(['me'], axis=1)
crsp2=pd.merge(crsp1, crsp_summe, how='inner', on=['jdate','permco'])
If I understood correctly, the first line merges table crsp and crsp_maxme with intersection on column 'jdate', 'permco', 'me'. So the table crsp1 would have 3 columns.
The second line drops the 'me' column of table crsp1.
The last lien would merge newly adjusted table crsp1 and crsp_summe with inner join, with intersection on 'jdate' and 'permco'. Which makes newly merged table crsp2 only having 2 columns.
However, the code explanation from line 2 says that the second and third lines drop 'me' column from crsp1 and then replace it with 'me' from crsp_summe table, which I had problem understanding.
Could anyone clarify these lines for me?
PS: I thought it isn't necessary to explain what the table crsp, crsp_summe, and crsp_maxme since they are all framed by inner join function. So please excuse the lack of background info.
The merge() functions on parameter specifies on what columns you want to make joins. how specifies what type of join you want to apply (similar to sql joins as outer, inner, left, right etc.).
Ex:
suppose there are two tables A and B containing columns as A['x1','x2','x3'] and B['x2','y1'] so joining them based on 'x1' (as it is common column in both table) would produce A_join_B_on_x1['A_B_x1','A_x2','A_x3','B_y1'] and the join will based on how you want to join.
in your current code consider,
A = crsp1
B = crsp_maxme
C = crsp_summe
Now in your program your
first line merges your A,B on ['jdate','permco','me'] columns and creates a new dataframe A_B containing ['jdate','permco','me',...'+columns_from_both_tables(A)(B)'] as inner join (i.e rows which are common in both A,B based on ['jdate','permco','me'] columns)
second line drops 'me' column from A_B dataframe. so it will be something like
['jdate','permco',...'+columns_from_both_tables(A)(B)']
third line merges your A_B,C on ['jdate','permco'] and creates ['jdate','permco',...'+columns_from_both_tables(A_B)(C)'] as inner join (i.e rows which are common in both A_B,C based on ['jdate','permco','me'] columns)
I've multiple dataframes (with exacly the same structure, same variables...) and they all starts with "df_".
What I would like to do is to join all these dataframes into one.
I can do it manually, but I have many data frames and their names can change.
frames = [df_24_10000, df_48_10000, df_64_20000, df_82_30000]
result = pd.concat(frames)
Is it possible to join all data frames that starts with "df_"?
This is normally the sign of a design problem. If you find yourself trying to group a number of objects by their names, it means that should have been elements of the same container (list, dict, set, map or whatever) from the very beginning.
Said differently, if instead of df_24_10000, df_48_10000, df_64_20000, you had dfs['24_10000'], dfs['48_10000'], dfs['64_20000'], the join would simply be:
result = pd.concat(dfs.values)
In pandas, is there an equivalent merge or merge_asof operation that can accomplish the SQL equivalent of an:
INNER JOIN number_table as n on n.N <= t.some_integer_field
where n is a number/tally table dataframe with a single column of integers(1 to 1000)
and t is a table with some integer field you would like to "deaggregate"
Any tips would be most appreciated!
In SQL, the INNER JOIN without equality is equivalent to CROSS JOIN and ON can be replaced with WHERE. Technically, even with equality! So your need of:
INNER JOIN number_table as n ON n.N <= t.some_integer_field
Can be replaced as:
CROSS JOIN number_table as n WHERE n.N <= t.some_integer_field
And because cross joins are cartesian products, run the same process in pandas where you assign a column in both dataframes of same value and merge on it which returns all possible combinations of rows from both sets since key will align.
df_number['key'] = 1 # OR df_number.assign(key=1)
df_table['key'] = 1
# CROSS JOIN WITH CONDITIONAL FILTER
pd.merge(df_table, df_number, on='key').query('N < some_integer_field')
Now performance of CJs is another question!
I have this large query I am trying to perform. I perform a series of joins, and then from that resulting relation I want to perform another join and filter out certain tuples.
SELECT *
FROM
(
SELECT *
FROM
market_instrument
inner join exchange_instrument
on market_instrument.id = exchange_instrument.instrument_id
inner join Table1 on market_instrument.id = Table1.instrument_id
left join Table2 on market_instrument.id = Table2.instrument_id
left join `options`on market_instrument.id = `options`.instrument_id
left join Table3 on market_instrument.id = Table3.instrument_id
) as R
inner join Table4 on R.instrument_id = Table4.instrument_id
where Table4.fill_timestamp between CURDATE() - INTERVAL 30 DAY AND NOW();
R is the "series of joins" I'm referring to. I want to inner join R with Table4 and then filter out the resulting relation for the last 30 days (where the date attribute is Table4.fill_timestamp). I'm using SQLAlchemy so I thought about somehow saving R to some result relation variable and performing a separate query on that, but I don't know how SQLAlchemy handles that, so I wanted to try doing the entire query in SQL first.
I keep getting the Duplicate Column Name "instrument_id" error. instrument_id is the primary key for all tables except market_instrument, where it's the same but it's called id instead. What can I do to get around this issue?
The problem is that R has all the columns from several tables, and more than one of those tables has a column named "instrument_id". You have not assigned aliases to any of those column names, so SQL does not know which instrument_id column you mean when you say "R.instrument_id".
If market_instrument is the only table with an id column then you could join on R.id instead of R.instrument_id.
Alternatively, another group of solutions involves assigning different names to some or all of the columns in R. For example,
SELECT
market_instrument.*,
exchange_instrument.*,
Table1.instrument_id AS the_one_true_id,
Table1.another_column,
Table1.yet_another_column,
...
Table2.*,
options.*,
Table3.*
FROM
market_instrument
inner join exchange_instrument
on market_instrument.id = exchange_instrument.instrument_id
inner join Table1 on market_instrument.id = Table1.instrument_id
left join Table2 on market_instrument.id = Table2.instrument_id
left join `options`on market_instrument.id = `options`.instrument_id
left join Table3 on market_instrument.id = Table3.instrument_id
With the above, you could then join on R.the_one_true_id. Alternatively, you could leave your current join as it is, and rename all the instrument_id columns but one. It might (or might not) be convenient to do that in the context of replacing R with a full-fledged VIEW in your schema.
Alternatively, your select list could enumerate all the columns of all the tables in the join. That might be tedious, but if you really do need all of them, then you will need to do that to disambiguate the other duplicate names, which include, at least, the various other instrument_id columns. Presented with such a task, however, perhaps you would discover that you don't really need every one of them.
As yet another alternative, you could add more columns instead of renaming existing ones. For example,
SELECT
*
exchange_instrument.instrumentId AS ei_instrument_id,
Table1.instrument_id AS t1_instrument_id,
Table2.instrument_id AS t2_instrument_id,
options.instrument_id AS op_instrument_id,
Table3.instrument_id AS t3_instrument_id
FROM
...
Then you can access, say, R.t1_instrument_id, whose name is presumably unique.