Develop Reference

PySpark DataFrame showing different results when using .select()

Why is .select() showing/parsing values differently to I don't use it?
I have this CSV:
CompanyName, CompanyNumber,RegAddress.CareOf,RegAddress.POBox,RegAddress.AddressLine1, RegAddress.AddressLine2,RegAddress.PostTown,RegAddress.County,RegAddress.Country,RegAddress.PostCode,CompanyCategory,CompanyStatus,CountryOfOrigin,DissolutionDate,IncorporationDate,Accounts.AccountRefDay,Accounts.AccountRefMonth,Accounts.NextDueDate,Accounts.LastMadeUpDate,Accounts.AccountCategory,Returns.NextDueDate,Returns.LastMadeUpDate,Mortgages.NumMortCharges,Mortgages.NumMortOutstanding,Mortgages.NumMortPartSatisfied,Mortgages.NumMortSatisfied,SICCode.SicText_1,SICCode.SicText_2,SICCode.SicText_3,SICCode.SicText_4,LimitedPartnerships.NumGenPartners,LimitedPartnerships.NumLimPartners,URI,PreviousName_1.CONDATE, PreviousName_1.CompanyName, PreviousName_2.CONDATE, PreviousName_2.CompanyName,PreviousName_3.CONDATE, PreviousName_3.CompanyName,PreviousName_4.CONDATE, PreviousName_4.CompanyName,PreviousName_5.CONDATE, PreviousName_5.CompanyName,PreviousName_6.CONDATE, PreviousName_6.CompanyName,PreviousName_7.CONDATE, PreviousName_7.CompanyName,PreviousName_8.CONDATE, PreviousName_8.CompanyName,PreviousName_9.CONDATE, PreviousName_9.CompanyName,PreviousName_10.CONDATE, PreviousName_10.CompanyName,ConfStmtNextDueDate, ConfStmtLastMadeUpDate
"ATS CAR RENTALS LIMITED","10795807","","",", 1ST FLOOR ,WESTHILL HOUSE 2B DEVONSHIRE ROAD","ACCOUNTING FREEDOM","BEXLEYHEATH","","ENGLAND","DA6 8DS","Private Limited Company","Active","United Kingdom","","31/05/2017","31","5","28/02/2023","31/05/2021","TOTAL EXEMPTION FULL","28/06/2018","","0","0","0","0","49390 - Other passenger land transport","","","","0","0","http://business.data.gov.uk/id/company/10795807","","","","","","","","","","","","","","","","","","","","","12/06/2023","29/05/2022"
"ATS CARE LIMITED","10393661","","","UNIT 5 CO-OP BUILDINGS HIGH STREET","ABERSYCHAN","PONTYPOOL","TORFAEN","WALES","NP4 7AE","Private Limited Company","Active","United Kingdom","","26/09/2016","30","9","30/06/2023","30/09/2021","UNAUDITED ABRIDGED","24/10/2017","","0","0","0","0","87900 - Other residential care activities n.e.c.","","","","0","0","http://business.data.gov.uk/id/company/10393661","17/05/2018","ATS SUPPORT LIMITED","22/12/2017","ATS CARE LIMITED","","","","","","","","","","","","","","","","","09/10/2022","25/09/2021"
I'm reading the csv like so:
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
_file = "/path/dir/BasicCompanyDataAsOneFile-2022-08-01.csv"
df = spark.read.csv(_file, header=True, quote='"', escape="\"")
Focusing on the CompanyCategory column, we should see Private Limited Company for both lines. But this is what I get instead when using select():
df.select("CompanyCategory").show(truncate=False)
+-----------------------+
|CompanyCategory |
+-----------------------+
|DA6 8DS |
|Private Limited Company|
+-----------------------+
df.select("CompanyCategory").collect()
[Row(CompanyCategory='DA6 8DS'),
Row(CompanyCategory='Private Limited Company')]
vs when not using select():
from pprint import pprint
for row in df.collect():
pprint(row.asDict())
{' CompanyNumber': '10795807',
...
'CompanyCategory': 'Private Limited Company',
'CompanyName': 'ATS CAR RENTALS LIMITED',
...}
{' CompanyNumber': '10393661',
...
'CompanyCategory': 'Private Limited Company',
'CompanyName': 'ATS CARE LIMITED',
...}
Using asDict() for readability.
SQL doing the same thing:
df.createOrReplaceTempView("companies")
spark.sql('select CompanyCategory from companies').show()
+--------------------+
| CompanyCategory|
+--------------------+
|Private Limited C...|
| DA6 8DS|
+--------------------+
As you can see when not using select() the CompanyCategory values are showing correctly. Why is this happening? What can I do to avoid this?
Context: I'm trying to creating dimension tables which is why I'm selecting a single column. The next phase is to drop duplicates, filter, sort, etc.
Edit:
Here are two example values in the actual CSV that are throwing things off:
CompanyName of """ BORA "" 2 LTD"
1st line address of ", 1ST FLOOR ,WESTHILL HOUSE 2B DEVONSHIRE ROAD"
Note:
These values from two separate distinct lines in the CSV.
These values are copy and pasted from the CSV opened in text editor like Notepad or VSCode).
Tried and failed:
df = spark.read.csv(_file, header=True) - completely picks up incorrect column.
df = spark.read.csv(_file, header=True, escape='\"') - exact same thing described in original question above. So same results.
df = spark.read.csv(_file, header=True, escape='""') - since the CSV escapes quotes using two double quotes, then I guess using two double quotes as escape param would do the trick? But getting following error:
Py4JJavaError: An error occurred while calling o276.csv.
: java.lang.RuntimeException: escape cannot be more than one character

When reading the csv, the parameters quote and escape are set to the same value ('"'=="\"" returns True in Python).
I would guess that configuring both parameters in this way will somehow disturb the parser that Spark uses to separate the single fields. After removing the escape parameter you can process the remaining " with regexp_replace:
from pyspark.sql import functions as F
df = spark.read.csv(<filename>, header=True, quote='"')
cols = [F.regexp_replace(F.regexp_replace(
F.regexp_replace("`" + col + "`", '^"', ''),
'"$', ''), '""', '"').alias(col) for col in df.columns]
df.select(cols).show(truncate=False)
Probably there is a smart regexp that can combine all three replace operations into one...

This is an issue when reading a single column from CSV file vs. when reading all the columns:
df = spark.read.csv('companies-house.csv', header=True, quote='"', escape="\"")
df.show() # correct output (all columns loaded)
df.toPandas() # same as pd.read_csv()
df.select('CompanyCategory').show() # wrong output (trying to load a single column)
df.cache() # all columns will be loaded and used in any subsequent call
df.select('CompanyCategory').show() # correct output
The first select() performs a different (optimized) read than the second, so one possible workaround would be to cache() the data immediately. This will however load all the columns, not just one (although pandas and COPY do the same).
The problematic part of the CSV is the RegAddress.POBox column where empty value is saved as ", instead of "",. You can check this by incrementally loading more columns:
df.unpersist() # undo cache() operation (for testing purposes only)
df.select(*[f"`{c}`" for c in df.columns[:3]], 'CompanyCategory').show() # wrong
df.select(*[f"`{c}`" for c in df.columns[:4]], 'CompanyCategory').show() # OK

itertools and combinations all coming back the same

I'm writing some code to bring back a unique ID for each event that comes in in a given version. The value can repeat in a future version as the prefix for the version will change. I have the version information but I'm struggling to bring back the uid. I found some code that seems to produce what I need, found here and have to implement it for what I want but I am facing an issue.
I have the information I need as a dataframe and when I run the code it returns all values as the same unique value. I suspect that the issue stems from how I am using the used set from the example and it isn't being properly stored hence why it returns the same info each time.
Is anyone able to provide some hint on where to look as I can't seem to work out how to persist the information to change it for each row. Side note, I can't use Pandas so I can't use the udf function in there and the uuid module is no good as the requirement is to keep it short to allow easy human typing for searching. I've posted the code below.
import itertools
import string
from pyspark.sql.functions import udf
from pyspark.sql.types import StringType
#udf(returnType=StringType())
def uid_generator(id_column):
valid_chars = set(string.ascii_lowercase + string.digits) - set('lio01')
used = set()
unique_id_generator = itertools.combinations(valid_chars, 6)
uid = "".join(next(unique_id_generator)).upper()
while uid in used:
uid = "".join(next(unique_id_generator))
return uid
used.add(uid)
#uuid_udf = udf(uuid_generator,)
df2 = df_uid_register_input.withColumn('uid', uid_generator(df_uid_register_input.record))
The output is:

In the function definition, you have the argument id_column, but you never use that argument in the function body. And it seems that you haven't tried to use the version column either.
What may be easier for you, is not to aim for true uniqueness, but use one of hash functions. Even though in theory they don't give unique results, but practically, it's just ridiculously unlikely that one would get the same hash for different inputs.
from pyspark.sql import functions as F
df = spark.createDataFrame(
[(1, 1, 2),
(2, 1, 2),
(3, 1, 2)],
['record', 'job_id', 'version'])
df = df.select(
'*',
F.sha1(F.concat_ws('_', 'record', 'version')).alias('uid1'),
F.sha2(F.concat_ws('_', 'record', 'version'), 0).alias('uid2'),
F.md5(F.concat_ws('_', 'record', 'version')).alias('uid3'),
)
df.show()
# +------+------+-------+--------------------+--------------------+--------------------+
# |record|job_id|version| uid1| uid2| uid3|
# +------+------+-------+--------------------+--------------------+--------------------+
# | 1| 1| 2|486cbd63f94d703d2...|0c79023f435b2e9e6...|ab35e84a215f0f711...|
# | 2| 1| 2|f5d7b663eea5f2e69...|48fccc7ee00b72959...|5229803558d4b7895...|
# | 3| 1| 2|982bde375462792cb...|ad9a5c5fb1bc135d8...|dfe3a334fc99f298a...|
# +------+------+-------+--------------------+--------------------+--------------------+

How do filter with multiple contains in pyspark

I'm going to do a query with pyspark to filter row who contains at least one word in array. For example, the dataframe is:
"content" "other"
My father is big. ...
My mother is beautiful. ...
I'm going to travel. ...
I have an array:
array=["mother","father"]
And the output must be this:
"content" "other"
My father is big. ...
My mother is beautiful. ...
A simple filter for word in array.

I think this solution works. Let me know what you think.
import pyspark.sql.functions as f
phrases = ['bc', 'ij']
df = spark.createDataFrame([
('abcd',),
('efgh',),
('ijkl',)
], ['col1'])
(df
.withColumn('phrases', f.array([f.lit(element) for element in phrases]))
.where(f.expr('exists(phrases, element -> col1 like concat("%", element, "%"))'))
.drop('phrases')
.show()
)
output
+----+
|col1|
+----+
|abcd|
|ijkl|
+----+

Had the same thoughts as #ARCrow but using instr.
lst=["mother","father"]
DataFrame
data= [
(1,"My father is big."),
(2, "My mother is beautiful"),
(3,"I'm going to travel.")
]
df=spark.createDataFrame(data, ("id",'content'))
Solution
df=(df
.withColumn('phrases', f.array([f.lit(element) for element in lst]))
.where(f.expr('exists(phrases, element -> instr (content, element)>=1)'))
.drop('phrases')
)
df.show()
Outcome
+---+--------------------+
| id| content|
+---+--------------------+
| 1| My father is big.|
| 2|My mother is beau...|
+---+--------------------+

Taking some the same configuration as #wwnde,
data= [
(1,"My father is big."),
(2, "My mother is beautiful"),
(3,"I'm going to travel.")
]
df=spark.createDataFrame(data, ("id",'content'))
Solution
words = ["father", "mother"]
conditions = " or ".join([f"content like '%{word}%'" for word in words])
(
df
.filter(F.expr(conditions))
.show(truncate=False)
)
+---+----------------------+
|id |content |
+---+----------------------+
|1 |My father is big. |
|2 |My mother is beautiful|
+---+----------------------+

We made the Fugue project to port native Python or Pandas code to Spark or Dask. This lets you can keep the logic very readable by expressing it in native Python. Fugue can then port it to Spark for you with one function call.
First, we setup,
import pandas as pd
array=["mother","father"]
df = pd.DataFrame({"sentence": ["My father is big.", "My mother is beautiful.", "I'm going to travel. "]})
and then we can create a native Python function to express the logic:
from typing import List, Dict, Any, Iterable
def myfilter(df: List[Dict[str,Any]]) -> Iterable[Dict[str, Any]]:
for row in df:
for value in array:
if value in row["sentence"]:
yield row
and then test it on Pandas:
from fugue import transform
transform(df, myfilter, schema="*")
Because of works on Pandas, we can execute it on Spark by specifying the engine:
import fugue_spark
transform(df, myfilter, schema="*", engine="spark").show()
+---+--------------------+
| id| sentence|
+---+--------------------+
| 0| My father is big.|
| 1|My mother is beau...|
+---+--------------------+
Note we need .show() because Spark evaluates lazily. Schema is also a Spark requirement so Fugue interprets the "*" as all columns in = all columns out.
The fugue transform function can take both Pandas DataFrame inputs and Spark DataFrame inputs.
Edit:
You can replace the myfilter function above with a Pandas implementation like this:
def myfilter(df: pd.DataFrame) -> pd.DataFrame:
res = df.loc[df["sentence"].str.contains("|".join(array))]
return res
and Fugue will be able to port it to Spark the same way. Fugue knows how to adjust to the type hints and this will be faster than the native Python implementation because it takes advantage of Pandas being vectorized.

PySpark groupby elements with key of their occurence

I have this data in a DATAFRAME:
id,col
65475383,acacia
63975914,acacia
65475383,excelsa
63975914,better
I want to have a dictionary that will contain the column "word" and every id that is associated with it, something like this:
word:key
acacia: 65475383,63975914
excelsa: 65475383
better: 63975914
I tried groupBy, but that is a way to aggregate data, how to approach this problem?

I'm not sure if you intend to have the result as a Python dictionary or as a Dataframe (it is not clear from your question).
However, if you do want a Dataframe then one way to calculate that is:
from pyspark.sql.functions import collect_list
idsByWords = df \
.groupBy("col") \
.agg(collect_list("id").alias("ids")) \
.withColumnRenamed("col", "word")
This will result in:
idsByWords.show(truncate=False)
+-------+--------------------+
|word |ids |
+-------+--------------------+
|excelsa|[65475383] |
|better |[63975914] |
|acacia |[65475383, 63975914]|
+-------+--------------------+
Then you can turn that dataframe into a Python dictionary :
d = {r.asDict()["word"]: r.asDict()["ids"] for r in idsByWords.collect()}
To finally get:
{
'excelsa': [65475383],
'better': [63975914],
'acacia': [65475383, 63975914]
}
Note that collect may crash your driver program if it exceeds your driver memory.

How to convert RDD into Dataframe with Pyspark?

I have an RDD below, which I have received from a client. How can I convert this RDD into a Dataframe?
["Row(Moid=2, Tripid='11', Tstart='2007-05-28 08:53:14.040', Tend='2007-05-28 08:53:16.040', Xstart='9738.73', Ystart='103.246', Xend='9743.73', Yend='114.553')"]

Note: This is not really an answer, but I don't understand as to what OP is asking about. Writing this in comment section would not have been possible, but may be we can take it forward from here.
OP says that he/she receives an RDD (purportedly a single element) from Client -
["Row(Moid=2, Tripid='11', Tstart='2007-05-28 08:53:14.040', Tend='2007-05-28 08:53:16.040', Xstart='9738.73', Ystart='103.246', Xend='9743.73', Yend='114.553')"]
Now, OP wants to translate that to a DataFrame. To translate that, one has to de-string the Row object, but OP has to clarify what he needs.
from pyspark.sql import Row
rdd_from_client = [Row(Moid=2, Tripid='11', Tstart='2007-05-28 08:53:14.040', Tend='2007-05-28 08:53:16.040', Xstart='9738.73', Ystart='103.246', Xend='9743.73', Yend='114.553')]
df = sqlContext.createDataFrame(rdd_from_client)
df.show(truncate=False)
+----+-----------------------+------+-----------------------+-------+-------+-------+-------+
|Moid|Tend |Tripid|Tstart |Xend |Xstart |Yend |Ystart |
+----+-----------------------+------+-----------------------+-------+-------+-------+-------+
|2 |2007-05-28 08:53:16.040|11 |2007-05-28 08:53:14.040|9743.73|9738.73|114.553|103.246|
+----+-----------------------+------+-----------------------+-------+-------+-------+-------+