edu.iu.dsc.tws.comms.shuffle

Class FSKeyedSortedMerger

java.lang.Object

edu.iu.dsc.tws.comms.shuffle.FSKeyedSortedMerger

All Implemented Interfaces:

Shuffle

public class FSKeyedSortedMerger
extends java.lang.Object
implements Shuffle

Sorted merger implementation todo add support to handling large values. When tuples have large values, since we are opening multiple files at the same time, when reading, this implementation overloads heap

Constructor Summary

Constructors

Constructor and Description
FSKeyedSortedMerger(int maxBytesInMemory, int maxRecsInMemory, java.lang.String dir, java.lang.String opName, MessageType kType, MessageType dType, java.util.Comparator<java.lang.Object> kComparator, int tar) Create a key based sorted merger

Method Summary

Modifier and Type	Method and Description
void	add(java.lang.Object key, byte[] data, int length) Add the data to the file
void	clean() Cleanup the directories
java.util.Iterator<java.lang.Object>	readIterator() This method gives the values
void	run() This method saves the data to file system
void	switchToReading() Switch to reading, we cannot add after this

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.iu.dsc.tws.comms.shuffle.Shuffle

add, add

Constructor Detail

FSKeyedSortedMerger

public FSKeyedSortedMerger(int maxBytesInMemory,
                           int maxRecsInMemory,
                           java.lang.String dir,
                           java.lang.String opName,
                           MessageType kType,
                           MessageType dType,
                           java.util.Comparator<java.lang.Object> kComparator,
                           int tar)

Create a key based sorted merger

Method Detail

add

public void add(java.lang.Object key,
                byte[] data,
                int length)

Add the data to the file

Specified by:

add in interface Shuffle

Parameters:

key - the object key

data - the data

length - length of the data

switchToReading

public void switchToReading()

Description copied from interface: Shuffle

Switch to reading, we cannot add after this

Specified by:

switchToReading in interface Shuffle

run

public void run()

This method saves the data to file system

Specified by:

run in interface Shuffle

readIterator

public java.util.Iterator<java.lang.Object> readIterator()

This method gives the values

Specified by:

readIterator in interface Shuffle

Returns:

an iterator

clean

public void clean()

Cleanup the directories

Specified by:

clean in interface Shuffle