hive - Percentile vs percentile_approx in SparkSQL

Question

I am using percentile in one of my sql's but due to large data skewness, the query is failing with out of memory error. Even after increasing memory, the query is failing. I found this percentile_approx function which works on my dataset but there are differences between percentile and percentile_approx.

Can some kind soul please explain the differences?

My dataset contains dollar amounts up to 2 decimal points. The number of rows will vary from month to month but it's usually over 300 million rows.

Thanks

question from:https://stackoverflow.com/questions/66067352/percentile-vs-percentile-approx-in-sparksql

Answer 1

Waitting for answers