Develop Reference

Create new Data frame from an existing one in pyspark

I created this dataframe with pySpark from txt file that includes searches queries and user ID.
`spark = SparkSession.builder.getOrCreate()
df = spark.read.option("header", "true") \
.option("delimiter", "\t") \
.option("inferSchema", "true") \
.csv("/content/drive/MyDrive/my_data.txt")
df.select("AnonID","Query").show()`
And it look like that:
+------+--------------------+
|AnonID| Query|
+------+--------------------+
| 142| rentdirect.com|
| 142|www.prescriptionf...|
| 142| staple.com|
| 142| staple.com|
| 142|www.newyorklawyer...|
| 142|www.newyorklawyer...|
| 142| westchester.gov|
| 142| space.comhttp|
| 142| dfdf|
| 142| dfdf|
| 142| vaniqa.comh|
| 142| www.collegeucla.edu|
| 142| www.elaorg|
| 142| 207 ad2d 530|
| 142| 207 ad2d 530|
| 142| broadway.vera.org|
| 142| broadway.vera.org|
| 142| vera.org|
| 142| broadway.vera.org|
| 142| frankmellace.com|
| 142| ucs.ljx.com|
| 142| attornyleslie.com|
| 142|merit release app...|
| 142| www.bonsai.wbff.org|
| 142| loislaw.com|
| 142| rapny.com|
| 142| whitepages.com|
| 217| lottery|
| 217| lottery|
| 217| ameriprise.com|
| 217| susheme|
| 217| united.com|
| 217| mizuno.com|
| 217|p; .; p;' p; ' ;'...|
| 217|p; .; p;' p; ' ;'...|
| 217|asiansexygoddess.com|
| 217| buddylis|
| 217|bestasiancompany.com|
| 217| lottery|
| 217| lottery|
| 217| ask.com|
| 217| weather.com|
| 217| wellsfargo.com|
| 217|www.tabiecummings...|
| 217| wanttickets.com|
| 217| yahoo.com|
| 217| -|
| 217| www.ngo-quen.org|
| 217| -|
| 217| vietnam|
+------+--------------------+
What I want to do is that each user ID will be a row and each query will be in a column.
+------+------------+---------
|ID | 1 | 2 | 3 .......
+------+------------+---------
|142| query1|query2| query3
|217| query1|query2| query3
|993| query1|query2| query3
|1268| query1|query2| query3
|1326| query1|query2| query3
.
.
.
I tried to switch between rows and columns with the help of a search I did on Google, but I didn't succeed.

You can group the dataframe by AnonID, and then pivot the Query column to create new columns for each unique query:
import pyspark.sql.functions as F
df = df.groupBy("AnonID").pivot("Query").agg(F.first("Query"))
If you have a lot of distinct values try
df = df.groupBy("AnonID").agg(F.collect_list("Query").alias("Queries"))
You can then rename the columns to 1, 2, 3, etc.
df = df.selectExpr("AnonID", *[f"`{i+1}` as `{i+1}`" for i in range(len(df.columns)-1)])

Date difference in years in PySpark dataframe

I come from Pandas background and new to Spark. I have a dataframe which has id, dob, age as columns. I want to get the age of the user from his dob (in some cases age column is NULL).
+----+------+----------+
| id | age | dob |
+----+------+----------+
| 1 | 24 | NULL |
| 2 | 25 | NULL |
| 3 | NULL | 1/1/1973 |
| 4 | NULL | 6/6/1980 |
| 5 | 46 | |
| 6 | NULL | 1/1/1971 |
+----+------+----------+
I want a new column which will calculate age from dob and current date.
I tried this, but not getting any results from it:
df.withColumn("diff",
datediff(to_date(lit("01-06-2020")),
to_date(unix_timestamp('dob', "dd-MM-yyyy").cast("timestamp")))).show()

You need to compute the date difference and convert the result to years, something like this:
df.withColumn('diff',
when(col('age').isNull(),
floor(datediff(current_date(), to_date(col('dob'), 'M/d/yyyy'))/365.25))\
.otherwise(col('age'))).show()
Which produces:
+---+----+--------+----+
| id| age| dob|diff|
+---+----+--------+----+
| 1| 24| null| 24|
| 2| 25| null| 25|
| 3|null|1/1/1973| 47|
| 4|null|6/6/1980| 39|
| 5| 46| null| 46|
| 6|null|1/1/1971| 49|
+---+----+--------+----+
It preserves the age column where not null and computes the difference (in days) between dob and today where age is null. The result is then converted to years (by dividing by 365.25; you may want to confirm this) then floored.

I believe it is more appropriate to use months_between when it comes to year difference. we should use datediff only when if you need difference in days
Approach-
val data =
"""
| id | age | dob
| 1 | 24 |
| 2 | 25 |
| 3 | | 1/1/1973
| 4 | | 6/6/1980
| 5 | 46 |
| 6 | | 1/1/1971
""".stripMargin
val stringDS = data.split(System.lineSeparator())
.map(_.split("\\|").map(_.replaceAll("""^[ \t]+|[ \t]+$""", "")).mkString(","))
.toSeq.toDS()
val df = spark.read
.option("sep", ",")
.option("inferSchema", "true")
.option("header", "true")
.csv(stringDS)
df.show(false)
df.printSchema()
/**
* +---+----+--------+
* |id |age |dob |
* +---+----+--------+
* |1 |24 |null |
* |2 |25 |null |
* |3 |null|1/1/1973|
* |4 |null|6/6/1980|
* |5 |46 |null |
* |6 |null|1/1/1971|
* +---+----+--------+
*
* root
* |-- id: integer (nullable = true)
* |-- age: integer (nullable = true)
* |-- dob: string (nullable = true)
*/
Find age
df.withColumn("diff",
coalesce(col("age"),
round(months_between(current_date(),to_date(col("dob"), "d/M/yyyy"),true).divide(12),2)
)
).show()
/**
* +---+----+--------+-----+
* | id| age| dob| diff|
* +---+----+--------+-----+
* | 1| 24| null| 24.0|
* | 2| 25| null| 25.0|
* | 3|null|1/1/1973|47.42|
* | 4|null|6/6/1980|39.99|
* | 5| 46| null| 46.0|
* | 6|null|1/1/1971|49.42|
* +---+----+--------+-----+
*/
round it to 0 if you want age in whole number

Using months_between like in this answer, but in a different approach:
in my table, I don't have 'age' column yet;
for rounding to full years I use .cast('int').
from pyspark.sql import functions as F
df = df.withColumn('age', (F.months_between(current_date(), F.col('dob')) / 12).cast('int'))
If system date is UTC and your locale is different, a separate date function may be needed:
from pyspark.sql import functions as F
def current_local_date():
return F.from_utc_timestamp(F.current_timestamp(), 'Europe/Riga').cast('date')
df = df.withColumn('age', (F.months_between(current_local_date(), F.col('dob')) / 12).cast('int'))

Pyspark merge 2 dataframes without losing data

I'm looking for joining 2 pyspark dataframes without losing any data inside. Easiest way would be showing you with example. Might even count them together and sort. If null in the desktop or phone column, then it should equal 0 in the output.
I tried:
desktop_df.join(phone_df, on='query')\
.fillna(0).orderBy("desktop", ascending=False)\
.show(20)
(doesn't have the total column yet, so I'm ordering it by count1)
But this approach doesn't seems to be working - doesn't show zeros at all.
desktop_df:
query |desktop|
----------------
query1 | 12 |
----------------
query2 | 23 |
----------------
query3 | 8 |
----------------
query4 | 11 |
----------------
query6 | 45 |
----------------
query9 | 89 |
phone_df:
query | phone |
----------------
query1 | 21 |
----------------
query2 | 33 |
----------------
query4 | 11 |
----------------
query5 | 55 |
----------------
query6 | 45 |
----------------
query7 | 1234 |
----------------
query8 | 4321 |
----------------
query10| 10 |
----------------
query11| 1 |
Output I'm looking for:
query | desktop| phone | total |
--------------------------------
query8 | 0 | 4321 | 4321 |
--------------------------------
query7 | 0 | 1234 | 1234 |
--------------------------------
query6 | 45 | 45 | 90 |
--------------------------------
query9 | 89 | 0 | 89 |
--------------------------------
query2 | 23 | 33 | 56 |
--------------------------------
query5 | 0 | 55 | 55 |
--------------------------------
query1 | 12 | 21 | 33 |
--------------------------------
query4 | 11 | 11 | 22 |
--------------------------------
query10| 0 | 10 | 10 |
--------------------------------
query3 | 8 | 0 | 8 |
--------------------------------
query11| 0 | 1 | 1 |
Wanted solutions
df = desktop_df.join(phone_df, on=["query"], how='fullouter').fillna(0).withColumn("total",col("desktop")+col("phone")).show(200)
or
from pyspark.sql.functions import lit
from pyspark.sql.functions import col
from pyspark.sql.functions import max
desktop_df = df.filter("hwType == 'DESKTOP'").groupby("query").count().orderBy("count", ascending=False).withColumnRenamed('count','desktop')
phone_df = df.filter("hwType == 'PHONE'").groupby("query").count().orderBy("count", ascending=False).withColumnRenamed('count','phone')
# add missing column to each dataframe
desktop_df = desktop_df.withColumn('phone', lit(0)).select('query', 'desktop', 'phone')
phone_df = phone_df.withColumn('desktop', lit(0)).select('query', 'desktop', 'phone')
# union all and agg to select max value
phone_df.unionAll(desktop_df).groupBy('query').agg(max(col('desktop')).alias('desktop'), max(col('phone')).alias('phone'))
# withColumn('total', col('desktop') + col('phone')) \
# .orderBy(col('total').desc()) \
# .show()

May be try inner join on query column. And find "Total" by adding column values.
df = desktop_df.join(phone_df, desktop_df.query==phone_df.query,"full").select(desktop_df.query,"count1","count2").fillna(0).withColumn("total",col("count1")+col("count2"))

You can do that using unionAll and then groupBy.
Example:
desktop_data = [("query1", 12), ("query2", 23), ("query3", 8),
("query4", 11), ("query6", 45), ("query9", 89)]
phone_data = [("query1", 21), ("query2", 33), ("query4", 11), ("query5", 55), ("query6", 45),
("query7", 1234), ("query8", 4321), ("query10", 10), ("query11", 1)]
desktop_df = spark.createDataFrame(desktop_data, ['query', 'count1'])
phone_df = spark.createDataFrame(phone_data, ['query', 'count2'])
# add missing column to each dataframe
desktop_df = desktop_df.withColumn('count2', lit(0)).select('query', 'count1', 'count2')
phone_df = phone_df.withColumn('count1', lit(0)).select('query', 'count1', 'count2')
# union all and agg to select max value
phone_df.unionAll(desktop_df) \
.groupBy('query').agg(max(col('count1')).alias('count1'), max(col('count2')).alias('count2')) \
.withColumn('total', col('count1') + col('count2')) \
.orderBy(col('total').desc()) \
.show()
+-------+------+------+-----+
| query|count1|count2|total|
+-------+------+------+-----+
| query8| 0| 4321| 4321|
| query7| 0| 1234| 1234|
| query6| 45| 45| 90|
| query9| 89| 0| 89|
| query2| 23| 33| 56|
| query5| 0| 55| 55|
| query1| 12| 21| 33|
| query4| 11| 11| 22|
|query10| 0| 10| 10|
| query3| 8| 0| 8|
|query11| 0| 1| 1|
+-------+------+------+-----+

Find column names of interconnected row values - Spark

I have a Spark dataframe that adheres to the following structure:
+------+-----------+-----------+-----------+------+
|ID | Name1 | Name2 | Name3 | Y |
+------+-----------+-----------+-----------+------+
| 1 | A,1 | B,1 | C,4 | B |
| 2 | D,2 | E,2 | F,8 | D |
| 3 | G,5 | H,2 | I,3 | H |
+------+-----------+-----------+-----------+------+
For every row I want to find in which column the value of Y is denoted as the first element. So, ideally I want to retrieve a list like: [Name2,Name1,Name2].
I am not sure how and whether it works to convert first to a RDD, then use a map function and convert the result back to DataFrame.
Any ideas are welcome.

You can probably try this piece of code :
df.show()
+---+-----+-----+-----+---+
| ID|Name1|Name2|Name3| Y|
+---+-----+-----+-----+---+
| 1| A,1| B,1| C,4| B|
| 2| D,2| E,2| F,8| D|
| 3| G,5| H,2| I,3| H|
+---+-----+-----+-----+---+
from pyspark.sql import functions as F
name_cols = ["Name1", "Name2", "Name3"]
cond = F
for col in name_cols:
cond = cond.when(F.split(F.col(col),',').getItem(0) == F.col("Y"), col)
df.withColumn("whichName", cond).show()
+---+-----+-----+-----+---+---------+
| ID|Name1|Name2|Name3| Y|whichName|
+---+-----+-----+-----+---+---------+
| 1| A,1| B,1| C,4| B| Name2|
| 2| D,2| E,2| F,8| D| Name1|
| 3| G,5| H,2| I,3| H| Name2|
+---+-----+-----+-----+---+---------+

PySpark Dataframe identify distinct value on one column based on duplicate values in other columns

I have a pyspark dataframe like: where c1,c2,c3,c4,c5,c6 are the columns
+----------------------------+
|c1 | c2 | c3 | c4 | c5 | c6 |
|----------------------------|
| a | x | y | z | g | h |
| b | m | f | l | n | o |
| c | x | y | z | g | h |
| d | m | f | l | n | o |
| e | x | y | z | g | i |
+----------------------------+
I want to extract c1 values for the rows which have same c2,c3,c4,c5 values but different c1 values.
Like, 1st, 3rd & 5th rows have same values for c2,c3,c4 & c5 but different c1 value. So the output should be a, c & e.
(update)
similarly, 2nd & 4th rows have same values for c2,c3,c4 & c5 but different c1 value. So the output should also contain b & d
How can I obtain such result ? I have tried applying groupby but I don't understand how to obtain distinct values for c1.
UPDATE:
Output should be a Dataframe of c1 values
# +-------+
# |c1_dups|
# +-------+
# | a,c,e|
# | b,e|
# +-------+
My Approach:
m = data.groupBy('c2','c3','c4','c5)
but I'm not understanding how to retrieve the values in m. I'm new to pyspark dataframes hence very much confused

This is actually very simple, let's create some data first :
schema = ['c1','c2','c3','c4','c5','c6']
rdd = sc.parallelize(["a,x,y,z,g,h","b,x,y,z,l,h","c,x,y,z,g,h","d,x,f,y,g,i","e,x,y,z,g,i"]) \
.map(lambda x : x.split(","))
df = sqlContext.createDataFrame(rdd,schema)
# +---+---+---+---+---+---+
# | c1| c2| c3| c4| c5| c6|
# +---+---+---+---+---+---+
# | a| x| y| z| g| h|
# | b| x| y| z| l| h|
# | c| x| y| z| g| h|
# | d| x| f| y| g| i|
# | e| x| y| z| g| i|
# +---+---+---+---+---+---+
Now the fun part, you'll just need to import some functions, group by and explode as following :
from pyspark.sql.functions import *
dupes = df.groupBy('c2','c3','c4','c5') \
.agg(collect_list('c1').alias("c1s"),count('c1').alias("count")) \ # we collect as list and count at the same time
.filter(col('count') > 1) # we filter dupes
df2 = dupes.select(explode("c1s").alias("c1_dups"))
df2.show()
# +-------+
# |c1_dups|
# +-------+
# | a|
# | c|
# | e|
# +-------+
I hope this answers your question.