reduce

The reduce method passes the elements of the RDD to a function to be operated upon and then returns the computed output having the same data type as that of the input function. The simplest implementation of the reduce() function is of adding, multiplying, or counting all the elements of the RDD:

Java 7:

//reduce()
JavaRDD<Integer> intRDD = sparkContext.parallelize(Arrays.asList(1,4,3));
Integer sumInt=intRDD.reduce(new Function2<Integer, Integer, Integer>() {
private static finallongserialVersionUID = 1L;
@Override
public Integer call(Integer v1, Integer v2) throws Exception {
return v1+v2;
}
});
System.out.println("The sum of all the elements of RDD using reduce is "+sumInt);

Java 8:

JavaRDD<Integer> intRDD = sparkContext.parallelize(Arrays.asList(1,4,3));Integer 
sumInt=intRDD.reduce((a,b)->a+b);
System.out.println("The sum of all the elements of RDD using reduce is "+sumInt);
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.116.49.247