edu.iu.dsc.tws.comms.shuffle

Class FileLoader

java.lang.Object

edu.iu.dsc.tws.comms.shuffle.FileLoader

public final class FileLoader
extends java.lang.Object

Method Summary

Modifier and Type	Method and Description
static <any>	openFilePart(java.lang.String fileName, long startOffSet, int maxSize, MessageType keyType, MessageType dataType, KryoSerializer deserializer)
static OpenFilePart	openPart(java.lang.String fileName, long startOffSet, long maxSize, MessageType keyType, MessageType dataType, KryoSerializer deserializer) Reads a file part upto max size from start offset
static OpenFile	openSavedPart(java.lang.String fileName) Open a saved part and load it as a memory mapped file
static OpenFile	openSavedPart(java.lang.String fileName, long position, long length) Open a saved part and load it as a memory mapped file
static java.util.List<java.lang.Object>	readFile(java.lang.String fileName, MessageType dataType, KryoSerializer deserializer)
static java.util.List<Tuple>	readFile(java.lang.String fileName, MessageType keyType, MessageType dataType, KryoSerializer deserializer)
static long	saveKeyValues(java.util.List<Tuple> records, java.util.List<java.lang.Integer> sizes, long size, java.lang.String outFileName, MessageType keyType) Save the list of records to the file system
static long	saveKeyValues(java.util.List<Tuple> records, long size, java.lang.String outFileName, MessageType keyType) This method accepts a Array of lists instead of a list of tuples
static void	saveObjects(java.util.List<byte[]> records, java.util.List<java.lang.Integer> sizes, long size, java.lang.String outFileName) Save the list of records to the file system
static void	saveSizes(java.util.List<java.lang.Integer> sizes, java.lang.String outFileName) Save the list of records to the file system

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

saveObjects

public static void saveObjects(java.util.List<byte[]> records,
                               java.util.List<java.lang.Integer> sizes,
                               long size,
                               java.lang.String outFileName)

Save the list of records to the file system

Parameters:

records - records to be written

size - total size of the records

outFileName - out file name

saveKeyValues

public static long saveKeyValues(java.util.List<Tuple> records,
                                 java.util.List<java.lang.Integer> sizes,
                                 long size,
                                 java.lang.String outFileName,
                                 MessageType keyType)

Save the list of records to the file system

Parameters:

records - records to be written

size - total size of the records

outFileName - out file name

Returns:

maximum size of the tuple written to this file

saveKeyValues

public static long saveKeyValues(java.util.List<Tuple> records,
                                 long size,
                                 java.lang.String outFileName,
                                 MessageType keyType)

This method accepts a Array of lists instead of a list of tuples

readFile

public static java.util.List<Tuple> readFile(java.lang.String fileName,
                                             MessageType keyType,
                                             MessageType dataType,
                                             KryoSerializer deserializer)

readFile

public static java.util.List<java.lang.Object> readFile(java.lang.String fileName,
                                                        MessageType dataType,
                                                        KryoSerializer deserializer)

openFilePart

public static <any> openFilePart(java.lang.String fileName,
                                 long startOffSet,
                                 int maxSize,
                                 MessageType keyType,
                                 MessageType dataType,
                                 KryoSerializer deserializer)

openPart

public static OpenFilePart openPart(java.lang.String fileName,
                                    long startOffSet,
                                    long maxSize,
                                    MessageType keyType,
                                    MessageType dataType,
                                    KryoSerializer deserializer)

Reads a file part upto max size from start offset

Parameters:

fileName - name

startOffSet - start offset in bytes

maxSize - max size to read

keyType - key type

dataType - data type

deserializer - the deserializer

Returns:

OpenFilePart with read information and null if fails

saveSizes

public static void saveSizes(java.util.List<java.lang.Integer> sizes,
                             java.lang.String outFileName)

Save the list of records to the file system

Parameters:

outFileName - out file name

openSavedPart

public static OpenFile openSavedPart(java.lang.String fileName)

Open a saved part and load it as a memory mapped file

Parameters:

fileName - name of the file

Returns:

the bytebuffer and the file channel

openSavedPart

public static OpenFile openSavedPart(java.lang.String fileName,
                                     long position,
                                     long length)

Open a saved part and load it as a memory mapped file

Parameters:

fileName - name of the file

position - the position to start reading

length - length of the bytes to read

Returns:

the bytebuffer and the file channel