apache spark - replace group by with reducebykey

Question

I have the following code in dataframe. I am using the following statement to perform group by and it is not performing well on big datasets.

npa = npa.groupby(f.col('phonecode').alias('npa')).agg(f.mean('latitude').alias('latitude'), f.mean('longitude').alias('longitude'))

NPA dataset:-
+---------+---------+-----------+------------+
|phonecode| latitude|  longitude|country     |
+---------+---------+-----------+------------+
|   425   |47.496384|-121.784424|          US|
|   360   |47.540962|  -122.6358|          US|

Please advise how I can use reducebykey to solve the performance issue.

question from:https://stackoverflow.com/questions/65648960/replace-group-by-with-reducebykey

Answer 1

Waitting for answers