Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
1.1k views
in Technique[技术] by (71.8m points)

java - MapReduce to Spark

I have a MapReduce job written in Java. It depends on multiple classes. I want to run the MapReduce job on Spark.

What steps should I follow to do the same?

I need to make changes only to the MapReduce class?

Thanks!

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Answer

0 votes
by (71.8m points)

This is a very broad question, but the short of it is:

  1. Create an RDD of the input data.
  2. Call map with your mapper code. Output key-value pairs.
  3. Call reduceByKey with your reducer code.
  4. Write the resulting RDD to disk.

Spark is more flexible than MapReduce: there is a great variety of methods that you could use between steps 1 and 4 to transform the data.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome to OStack Knowledge Sharing Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...