java - Spark : get specific value of the column map if the key exists

Question

I have a column "price_map" in my dataset and it's a HashMap that is like this :

(usd_price,100),
(eur_price,200),
(jpy_price,500)...

Now I want to retrieve the price value of usd_price (i.e. 100), when the key usd_price exists in the map, and put it as a new row in my dataset.

I tried something like this:

    dataset = dataset.withColumn("usd_price", when(
            size(map_keys(dataset.col(price_map))),
            map_values(dataset.col(price_map))
                    .getField("usd_price"))
            .otherwise(lit("no data"))
    );

But it tells me that

org.apache.spark.sql.AnalysisException: cannot resolve 'map_values(`price_map`)['usd_price']' due to data type mismatch: argument 2 requires integral type, however, ''usd_price'' is of string type.;;

I want to know what is the correct way to do this ?

question from:https://stackoverflow.com/questions/65920888/spark-get-specific-value-of-the-column-map-if-the-key-exists

Answer 1

You can just get the map value directly using getItem, and use coalesce to replace null in case the key is not found.

dataset2 = dataset.withColumn(
    "usd_price",
    coalesce(
        dataset.col("price_map").getItem("usd_price"),
        lit("no data")
    )
);