edu.iu.dsc.tws.data.api.splits

Class GenericCSVInputSplit

java.lang.Object

edu.iu.dsc.tws.data.api.splits.LocatableInputSplit<OT>

edu.iu.dsc.tws.data.api.splits.FileInputSplit<OT>

edu.iu.dsc.tws.data.api.splits.DelimitedInputSplit<java.lang.String>

edu.iu.dsc.tws.data.api.splits.GenericCSVInputSplit

All Implemented Interfaces:

InputSplit<java.lang.String>, java.io.Serializable

public class GenericCSVInputSplit
extends DelimitedInputSplit<java.lang.String>

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class edu.iu.dsc.tws.data.api.splits.FileInputSplit

FileInputSplit.InputSplitOpenThread

Field Summary

Fields inherited from class edu.iu.dsc.tws.data.api.splits.DelimitedInputSplit

RECORD_DELIMITER

Fields inherited from class edu.iu.dsc.tws.data.api.splits.FileInputSplit

enumerateNestedFiles, MAX_SPLIT_SIZE_DISCREPANCY, minSplitSize, numSplits, openTimeout, splitLength, splitStart, stream

Constructor Summary

Constructors

Constructor and Description
GenericCSVInputSplit(int num, Path file, long start, long length, java.lang.String[] hosts) Constructs a split with host information.

Method Summary

Modifier and Type	Method and Description
void	configure(Config parameters) Configures this input format by reading the path to the file from the configuration and the string that defines the record delimiter.
java.lang.String	getCharsetName()
java.lang.String	readRecord(java.lang.String reusable, byte[] bytes, int readOffset, int numBytes) This function parses the given byte array which represents a serialized record.
void	setCharsetName(java.lang.String charsetName)

Methods inherited from class edu.iu.dsc.tws.data.api.splits.DelimitedInputSplit

close, getBufferSize, getCharset, getDelimiter, getDelimiterString, getLineLengthLimit, nextRecord, open, reachedEnd, readLine, setBufferSize, setDelimiter, setDelimiterString, setLineLengthLimit

Methods inherited from class edu.iu.dsc.tws.data.api.splits.FileInputSplit

equals, getLength, getMinSplitSize, getNumSplits, getPath, getStart, hashCode, isEnumerateNestedFiles, open, setEnumerateNestedFiles, setMinSplitSize, setNumSplits, toString

Methods inherited from class edu.iu.dsc.tws.data.api.splits.LocatableInputSplit

getHostnames, getSplitNumber

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

GenericCSVInputSplit

public GenericCSVInputSplit(int num,
                            Path file,
                            long start,
                            long length,
                            java.lang.String[] hosts)

Constructs a split with host information.

Parameters:

num - the number of this input split

file - the file name

start - the position of the first byte in the file to process

length - the number of bytes in the file to process (-1 is flag for "read whole file")

hosts - the list of hosts containing the block, possibly null

Method Detail

getCharsetName

public java.lang.String getCharsetName()

Overrides:

getCharsetName in class DelimitedInputSplit<java.lang.String>

setCharsetName

public void setCharsetName(java.lang.String charsetName)

Overrides:

setCharsetName in class DelimitedInputSplit<java.lang.String>

configure

public void configure(Config parameters)

Description copied from class: DelimitedInputSplit

Configures this input format by reading the path to the file from the configuration and the string that defines the record delimiter.

Specified by:

configure in interface InputSplit<java.lang.String>

Overrides:

configure in class DelimitedInputSplit<java.lang.String>

Parameters:

parameters - The configuration object to read the parameters from.

readRecord

public java.lang.String readRecord(java.lang.String reusable,
                                   byte[] bytes,
                                   int readOffset,
                                   int numBytes)
                            throws java.io.IOException

Description copied from class: DelimitedInputSplit

This function parses the given byte array which represents a serialized record. The function returns a valid record or throws an IOException.

Specified by:

readRecord in class DelimitedInputSplit<java.lang.String>

Parameters:

reusable - An optionally reusable object.

bytes - Binary data of serialized records.

readOffset - The offset where to start to read the record data.

numBytes - The number of bytes that can be read starting at the offset position.

Returns:

Returns the read record if it was successfully deserialized.

Throws:

java.io.IOException - if the record could not be read.