edu.iu.dsc.tws.comms.shuffle

Class FSKeyedSortedMerger2

java.lang.Object

edu.iu.dsc.tws.comms.shuffle.FSKeyedSortedMerger2

All Implemented Interfaces:

Shuffle

public class FSKeyedSortedMerger2
extends java.lang.Object
implements Shuffle

Sorted merger implementation todo add support to handling large values. When tuples have large values, since we are opening multiple files at the same time, when reading, this implementation overloads heap

Constructor Summary

Constructors

Constructor and Description
FSKeyedSortedMerger2(long maxBytesInMemory, long maxBytesToAFile, java.lang.String dir, java.lang.String opName, MessageType kType, MessageType dType, java.util.Comparator kComparator, int tar, boolean groupByKey, int parallelIOAllowance) Create a key based sorted merger

Method Summary

Modifier and Type	Method and Description
void	add(java.lang.Object key, byte[] data, int length) Add the data to the file
void	add(Tuple tuple) Add the data to the file
void	clean() Cleanup the directories
RestorableIterator<java.lang.Object>	readIterator() This method gives the values
void	run() This method saves the data to file system
void	switchToReading() Switch to reading, we cannot add after this

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.iu.dsc.tws.comms.shuffle.Shuffle

add

Constructor Detail

FSKeyedSortedMerger2

public FSKeyedSortedMerger2(long maxBytesInMemory,
                            long maxBytesToAFile,
                            java.lang.String dir,
                            java.lang.String opName,
                            MessageType kType,
                            MessageType dType,
                            java.util.Comparator kComparator,
                            int tar,
                            boolean groupByKey,
                            int parallelIOAllowance)

Create a key based sorted merger

Method Detail

add

public void add(Tuple tuple)

Add the data to the file

Specified by:

add in interface Shuffle

Parameters:

tuple - tuple

add

public void add(java.lang.Object key,
                byte[] data,
                int length)

Add the data to the file

Specified by:

add in interface Shuffle

Parameters:

key - the object key

data - the data

length - length of the data

switchToReading

public void switchToReading()

Description copied from interface: Shuffle

Switch to reading, we cannot add after this

Specified by:

switchToReading in interface Shuffle

run

public void run()

This method saves the data to file system

Specified by:

run in interface Shuffle

readIterator

public RestorableIterator<java.lang.Object> readIterator()

This method gives the values

Specified by:

readIterator in interface Shuffle

Returns:

an iterator

clean

public void clean()

Cleanup the directories

Specified by:

clean in interface Shuffle